Itheum Docs
  • 👋Getting Started
  • Infrastructure
    • 🚆AI Data Workforce
      • Token Utility for the AI Data Workforce
      • Join the Workforce
      • Liveliness Staking Rewards
    • 🖼️Data NFT
      • Data NFT Types
        • Data NFT-FT
        • Data NFT-LEASE
        • Data NFT-PH (Plug-In Hybrid)
      • Data NFT Generative Image Tool
    • 🤖NFMe ID Vaults
    • 🔋Liveliness - On-Chain Reputation
      • Data Creator Liveliness Bonding
        • Liveliness Score States
      • Liveliness Staking
      • FAQ - Liveliness staking
    • 🔓Data Marshal Network
  • Apps
    • 🔥<BiTz> XP System
      • Bonus BiTz for protocol usage
    • 💹Data NFT Marketplace
      • Listing a Data NFT
      • Procuring a Data NFT
      • FAQ - Data NFT Marketplace
    • 📡Data DEX
      • Minting a Data NFT
        • Store Data and Mint a Data NFT - Step-by-Step Tutorial
        • Creator Donations For Community Airdrops
      • Using the claims Portal
      • CanaryNet Guardrails
        • Guardrail : Trading Data NFTs on 3rd Party NFT Marketplaces
    • 🔍Itheum Explorer
  • Integrators
    • 🔋Liveliness Staking Guides
      • Liveliness Staking Guide : Solana
      • Liveliness Staking Guide : MultiversX
    • 📙Data Streams Guides
      • Data Asset Storage Options
      • Data Stream URL Rules
      • Zedge Storage
        • Static File on IPFS
        • Dymamic File on IPFS + DNS Link
        • Dymamic File on IPFS + IPNS
        • Music Data NFT Compatible Dynamic Data Stream on IPFS + IPNS
        • Trailblazer Data NFT Compatible Dynamic Data Stream on IPFS + IPNS
      • Amazon Web Services (AWS)
        • Storage : AWS S3
          • Data NFT Streaming Automation - Multiple files
          • Manual upload of file to AWS for Data NFT Streaming
          • Data NFT Streaming Automation - Trailblazer
        • Hosting : AWS S3 + Cloudflare
          • Task 1: Use a domain name to "sit in front" of your AWS S3 Bucket Public URL
          • Task 2: Convert your AWS S3 Bucket into a "website"
          • Task 3: Use Cloudflare to connect your Domain Name to your S3 Bucket securely
          • Troubleshooting
      • Akord - Arweave blockchain
      • MultiversX Native Auth Protected API
    • 📗Data DEX Guides
      • MultiversX Blockchain
        • Guide 1 : Get Started with the Data DEX on MultiversX
          • Section 1: Setting up wallets on the MultiversX Blockchain - Devnet
          • Section 2: Getting xEGLD Gas tokens to pay for transactions - MultiversX Devnet
          • Section 3: Getting ITHEUM devnet tokens via the Data DEX - MultiversX Devnet
        • Guide 2: Get Started with Itheum Enterprise
        • Itheum Ecosystem Actions Catalogue
      • Astar Network
        • Guide 1 : Get Started with the Data DEX on Astar Network
          • Section 1: Setting up wallets on Astar Network - Shibuya Testnet
          • Section 2: Getting ITHEUM devnet tokens via the Data DEX - Shibuya Testnet
        • Guide 2 : Procure Data NFTs from the peer-to-peer Data NFT Marketplace on Astar Network
        • Guide 3: Use the “Web3 Gamer Passport” App on the Astar Network to trade your PlayStation Data
    • 💳Supported Wallets
      • MultiversX DeFi Wallet
      • Ledger Wallet
      • xPortal Wallet
      • xAlias (Login with Google)
    • 📕Trailblazer Guides
      • How to Acquire a Trailblazer
      • How to view a Trailblazer
      • How to List a Trailblazer
    • 📘Data Coalition DAOs (DC DAOs) Guides
      • Appointer > Delegator Pattern for Data NFT "Deputizing"
  • Developers
    • 👨‍💻Software Development Kits (SDKs)
      • Data NFT SDK
        • Guide 1 : Minting a Custom Data NFT Collection with Authenticated Data Streams (via SDK)
        • Guide 2 : Unlocking Data NFTs via MultiversX Native Auth
        • Guide 3 : Using Nested Streams to Access Nested Data Assets from a Primary Data Stream
        • Guide 4: Use the Data NFT "Deputy" Feature to delegate access of your Data NFTs to a Smart Contract
        • Guide 5: Preparing a Data Stream containing a password to protect a URL
      • Enterprise SDK
        • Guide 1 : Using Itheum Enterprise to Mint a Data NFT Collection (e.g. NFT Loyalty Card Solution)
      • Data Marshal Network SDK
        • Guide 1 : Make your Regular NFT Collection to be Data NFT-PH Compatible
    • 🥋Data Marshal Network
      • Data Marshal Node Gateway Endpoints
      • Data Marshal Transit Flags and Headers
    • 🛂Tech Support - Discord
      • Portal Bridge Support
    • 🛒Release Notes
      • Data DEX
      • Itheum Explorer
      • Data NFT SDK
      • Enterprise SDK
      • Data Marshal Network
    • 🔐Security
      • 🐞Bug Bounty
      • ℹ️Security Audit
  • Protocol
    • $ITHEUM Token
    • 🌉Token Bridge
      • FAQ - Omni-Chain Portal Bridge
      • $ITHEUM Token Multi-Chain Max Supply Rebalancing Transactions Audit
    • 🏆Token Rewards
      • Badges
    • 🧨Token Burning
      • Phase 1 : Token Burn Program
    • 🏛️Governance
      • Itheum Ecosystem DAO
        • Version 1: How it Works
      • Itheum xPand DAO
        • Itheum xPand Grants Program
          • Code Of Conduct
          • Announcement Guidelines
          • Cohorts vs Alpha Builders
        • Program 1: MultiversX Post-Hackathon Accelerator
        • Program 2: xPand DAO Music Data NFT Growth
    • 💪Hackathons and Dev Challenges
      • MultiversX xDay Hackathon
        • Project Ideas > MultiversX Dev Tooling and Infra
        • Project Ideas > Itheum
        • Test Data NFT Catalog
      • Community Test Events
        • Portal Traveler 🌀 : Test the Itheum Omni-Chain Portal (Bridge)
        • APR for Liveliness 🎖️: Test the Bonding + Staking Rewards Module
        • Minting and Bonding on Solana
  • R&D
    • 🏢Itheum Enterprise
    • 🗳️Data Coalition DAOs (DC DAOs)
    • 🎏Trailblazer
      • FAQ - Trailblazer
  • Legal
    • ⚖️Ecosystem Tools Terms
      • Datadex
        • Terms Of Use
        • Privacy Policy
      • Liveliness Bonding: Penalties and Slashing Terms
      • BiTz XP
        • Give BiTz
      • Omni-Chain Portal Bridge
      • Gamer Passport
        • Data Collection and Storage
    • 👮Content Guidelines
    • Itheum Data License
    • Terminology Disclaimer
    • Protocol Docs, Token Disclaimer
Powered by GitBook
On this page
  • Building a "Music Playlist" Data NFT
  • Target User Story:
  • Step 1: Configure Data Stream Origin "Backend" for Nested Streams
  • Step 2: Installation of Itheum Data NFT SDK
  • Step 3: Enable Native Auth Login via MultiversX SDK Core
  • Step 4: Obtain an Auth "Session Token"
  • Step 5: "View Data" via a Native Auth session and handle the response
  • Step 6: Working with the Nested Stream
  • Other Use Cases for Nested Streams
  1. Developers
  2. Software Development Kits (SDKs)
  3. Data NFT SDK

Guide 3 : Using Nested Streams to Access Nested Data Assets from a Primary Data Stream

PreviousGuide 2 : Unlocking Data NFTs via MultiversX Native AuthNextGuide 4: Use the Data NFT "Deputy" Feature to delegate access of your Data NFTs to a Smart Contract

Last updated 1 year ago

This guide walks you through how to use Data Streams configured to be “nested streams” and how client-side apps can use the Itheum SDK to interact with these nested streams.

Firstly, what are nested streams and who are they needed? As detailed in the Data Streams Guides, we see that Data NFTs provide "web3 licensed" access to Data Streams, i.e., Data NFTs are a license to use your data. In most cases, your Data Stream will be "dynamic," where it can evolve to have its data change over time. An example of a dynamic Data Stream is could be some on-chain analytics data you have sourced, maintained, and kept updated, similar to the . The Data Marshal can "stream out" the complete Data Stream once ownership of the Data NFT has been proved. The Data Marshal is also responsible for protecting the primary Data Stream link and aims never to expose the original data location (it only streams out the content).

The "hide original URL location and stream" mechanism of the Data Marshal works well when those above-mentioned "dynamic" Data Streams are mostly also "self-contained" and "complete" and don't contain links within them that need to be "hidden" as they are part of the original outcome of the Data Stream. For example, image files, PDFs, SVG documents, etc are most likely "complete" documents. So, in this situation, the Data Marshal does not need to hide and protect any secondary links located within the original data.

But what if your use-case requires that you have nested links within the original Data Stream that you also want to "hide" and protect, and use the same "hide original URL location and stream" features of the Data Marshal?

This is where "nested streams" come into play, but to get your head around it - let's look at a real-world example of nested streams being built on Itheum.

Building a "Music Playlist" Data NFT

You could build a music playlist or music player experience using Data NFTs, nested streams, and the Itheum Data NFT SDK. In this use case, your primary Data Stream has a list of songs hosted in some location and would want the Data Marshal to mask/hide that data to protect the source/origin of the individual files. You can think of your primary "base" version of the Data Stream as a sort of manifest or instruction file that can be used by the client app to unpack a complete user experience. Nested streams allow for this functionality; in the example of the music player, you can unpack the original Data Stream as usual, but if the origin Data Stream server has flagged it as the “nested stream,” the Data Marshal is smart enough to discover and hide the links within it that need to be protected.

You can then use the Itheum Data NFT SDK to iterate and stream the nested links while preserving the origin location of all the nested data assets. In the context of the music player, you get the primary "base" version of the nested stream, which provides all the data needed to render the music interaction (song titles, album art, etc.), but then you use the Data NFT SDK to play the actual music assets when the user clicks on play, next, back etc. We will explore the code needed to do this below.


🏎️ Ready to start? Here are a few assumptions and pre-conditions

  • You need @itheum/sdk-mx-data-nft v2.1.0 or above to follow this guide.

  • This guide assumes that you are integrating with Itheum via an application built on the MultiversX blockchain, you use TypeScript/JavaScript as your programming language, and that you use that use the @multiversx/sdk-core and @multiversx/sdk-dapp libraries to authenticate and sign and track on-chain transactions. However, these libraries are optional and you can use any MultiversX tooling to interact with the blockchain.

  • To open a Data NFT and view its data using the Data NFT SDK, you have implemented MultiversX Native Auth integration with Itheum as per this guide Guide 2 : Unlocking Data NFTs via MultiversX Native Auth

  • You have control of the origin Data Stream publishing system (see Step 1 setup section below), as you will need to set HTTP Response Headers on the Data Stream.


Target User Story:

The target user story builds on the above Building a "Music Playlist" Data NFT section.

As an integrator of Itheum web3 Data Brokerage infrastructure:

  • I'd like to build a Music Player app that supports all Data NFTs that follow a standard "music player nested stream" format.

    • This Music Player app empowers the musicians who minted their songs as a Data NFT with the means to truly own their original music data.

    • This Music Player app can also evolve into a "decentralized Spotify," where I build and maintain the "front-end" music player features (and possibly charge a subscription fee for "premium" features), and musicians around the world can own their music data and earn royalties directly from the trade of their songs on my app.

Let's begin...


Step 1: Configure Data Stream Origin "Backend" for Nested Streams

We will make the "Response Header" name configurable, but for now, as we are still in "beta" mode, the target header needs to be :

x-amz-meta-marshal-deep-fetch = 1

Cannot add custom HTTP Response Headers? No worries, we got you!

If you are using a static, decentralized data stream that's hosted on IPFS, IPNS, or even on a centralized location and you CANNOT add HTTP Response headers, you can still trigger the Nested Stream workflow for your data by adding admf-nestedstream=1 parameter to your Data Stream URL when minting.

Here are some example Data Stream URLs with this parameter:

  1. https://gateway.lighthouse.storage/ipfs/QmSpzKpcayVeAoMP89HAP5EijbAdUsg85PdGA4k4jxYqdh?dmf-nestedstream=1

  2. ipns://my-hash?dmf-nestedstream=1

  3. ipfs://my-CID?dmf-nestedstream=1

When the Data Marshal sees this HTTP header (or query string parameter) during the transit traffic, it will respond as per the rules of a nested stream. If the header is not found, it defaults to the usual behavior of a general Data Stream.

Your nested stream should ideally follow a data format that is similar to the below sample JSON document that details the nested stream payload for the Music Player app use-case mentioned above.

{
  "data_stream": {
    "name": "tokentunes:musiverse:musicx",
    "creator": "Manu",
    "created_on": "2023-05-22T05:37:17Z",
    "last_modified_on": "2023-06-10T14:00:19Z",
    "marshalManifest": {
      "totalItems": 3,
      "nestedStream": true
    }
  },
  "data": [
    {
      "idx": 1,
      "date": "2023-01-02T00:00:00Z",
      "category": "Beats",
      "artist": "Emancipator",
      "album": "Soon It Will Be Cold Enough",
      "title": "Beats are best",
      "file": "https://itheum-static.s3.ap-southeast-2.amazonaws.com/hosteddataassets/musicblazer/manu-song-1.mp3",
      "cover_art_url": "https://521dimensions.com/img/open-source/amplitudejs/album-art/soon-it-will-be-cold-enough.jpg"
    },
    {
      "idx": 2,
      "date": "2023-01-09T00:00:00Z",
      "category": "Funk",
      "artist": "Emancipator",
      "album": "Soon It Will Be Cold Enough",
      "title": "Funk is best",
      "file": "https://itheum-static.s3.ap-southeast-2.amazonaws.com/hosteddataassets/musicblazer/manu-song-2.mp3",
      "cover_art_url": "https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcT0hBNmQVoxqDWLdEKSZPnmbanHXKFMjzDrYA&usqp=CAU"
    },
    {
      "idx": 3,
      "date": "2023-01-16T00:00:00Z",
      "category": "Rock",
      "artist": "Emancipator",
      "album": "Soon It Will Be Cold Enough",
      "title": "Rock is best",
      "file": "https://itheum-static.s3.ap-southeast-2.amazonaws.com/hosteddataassets/musicblazer/manu-song-3.mp3",
      "cover_art_url": "https://coverartworks.com/wp-content/uploads/2021/05/yeter-750px.jpg"
    }
  ]
}

Here is a full sample file for you to inspect JSON content and HTTP response header:

https://itheum-static.s3.ap-southeast-2.amazonaws.com/hosteddataassets/musicblazer/stream_1.json

In the data format above, the most important items that need to be included are :

  • The general JSON file skeleton structure seen above needs to be maintained.

  • The data key must exist as a first-level key and should have a list of data item objects.

  • Each data item in the data list MUST have a idx , date and file field.

  • idx can start with 0 or 1 depending on your choice. We prefer 1.

  • date should be in the format "2023-01-16T00:00:00Z"

  • The file should be a public URL that follows the standard Data Stream URL Rules

When the Data Marshal sees the header as above, it will serve the contents of the file but it will filter out the file links to individual assets, but we will be able to programmatically interact with the Data Marshal and fetch and stream these file links by using the idx index value.

Let’s now see how we can interact with the data marshal and get it to fetch and serve individual assets.

Step 2: Installation of Itheum Data NFT SDK

npm install --save @itheum/sdk-mx-data-nft

Step 3: Enable Native Auth Login via MultiversX SDK Core

Step 4: Obtain an Auth "Session Token"

Step 5: "View Data" via a Native Auth session and handle the response

Please note that in addition to the code described above where you call dataNftToOpen.viewDataViaMVXNativeAuth, you may also want to use the stream: true setting to ensure the JSON version of the "base" data streams out and does not "auto-download" in your browser. See the next step for a code example of using the stream param.

Step 6: Working with the Nested Stream

If we followed all the steps above, we now have access to the base version of the nested stream. If we print out the response from the previous, you will notice that you get the raw Data Stream contents, but as the Data Marshal has detected it to be a “nested stream”, it has extracted out and "hidden" all the file attributes from the data list. (Hint: compare the payload below with the original JSON document above to see the changes)

{
  "data_stream": {
    "name": "tokentunes:musiverse:musicx",
    "creator": "Manu",
    "created_on": "2023-05-22T05:37:17Z",
    "last_modified_on": "2023-06-10T14:00:19Z",
    "marshalManifest": {
      "totalItems": 3,
      "nestedStream": true
    }
  },
  "data": [
    {
      "idx": 1,
      "date": "2023-01-02T00:00:00Z",
      "category": "Beats",
      "artist": "Emancipator",
      "album": "Soon It Will Be Cold Enough",
      "title": "Beats are best",      
      "cover_art_url": "https://521dimensions.com/img/open-source/amplitudejs/album-art/soon-it-will-be-cold-enough.jpg"
    },
    {
      "idx": 2,
      "date": "2023-01-09T00:00:00Z",
      "category": "Funk",
      "artist": "Emancipator",
      "album": "Soon It Will Be Cold Enough",
      "title": "Funk is best",      
      "cover_art_url": "https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcT0hBNmQVoxqDWLdEKSZPnmbanHXKFMjzDrYA&usqp=CAU"
    },
    {
      "idx": 3,
      "date": "2023-01-16T00:00:00Z",
      "category": "Rock",
      "artist": "Emancipator",
      "album": "Soon It Will Be Cold Enough",
      "title": "Rock is best",      
      "cover_art_url": "https://coverartworks.com/wp-content/uploads/2021/05/yeter-750px.jpg"
    }
  ]
}

You can set up the UI with this “base” version as needed. In our case study, we can set a Music Player UI and load images, titles, etc.

And when a user selects a song and clicks on “play,” we can now interactively call the Data Marshal and “fetch” the nested file links.

For example, let’s assume the user wants to play Beats are best with idx “1”. Similar to the previous step where we "View Data", we now need to now call viewDataWithNativeAuth again, but we add the target idx of the link we want to stream as a new nestedIdxToStream param. Also, do not forget to use the stream: true setting to ensure the data streams out.

const res: ViewDataReturnType = await dataNftToOpen.viewDataViaMVXNativeAuth({
  mvxNativeAuthOrigins: ["http://localhost:3000"],
  mvxNativeAuthMaxExpirySeconds: 3000,
  fwdHeaderKeys: "authorization", // forward the authorization header to the data stream server (so it can validate the sesison and get caller address etc)
  fwdHeaderMapLookup: {
    "authorization": `Bearer ${nativeAuthToken}`,
  },
  stream: true, // this ensures the data streams and "does not download"
  nestedIdxToStream: 1  // We want to stream the song "Beats are best" so we send it's "idx"
});

And that's it... using the above interactive coding method, we can iterate and interact with the "base" version of the Data Stream. By doing so, we can build a fully functional Audio Player app.

Other Use Cases for Nested Streams

As you would have seen by now, nested streams enable unique flexibility to Data NFTs as they allow for some programmatic, interactive iteration of links placed within a "base" Data Stream file. There are many use cases for this that we can explore, for example:

  1. Any use case that requires a "manifest" or "instruction" type "base" file that contains "URLs" to other assets:

    1. Audio Player

    2. Video Player

    3. Slideshows

    4. Live Streams etc

  2. "Paginating" through large datasets

We'd love to see what you can build with nested streams...

Your Data Stream could be hosted by any of the data hosting service options provided inData Streams Guides. If you are not requiring the behaviour of a "nested stream", you do not need to do anything extra above hosting your data and making it available as a public URL. But in the event you do need to configure your Data Stream as a nested stream, then your backend Data Stream server will need to send down a custom .

Example of how the response header is sent by the Origin Data Stream server

In your NodeJS/JavaScript/TypeScript app, install the SDK @itheum/sdk-mx-data-nft by following the . If you are using NPM then it would be.

This step was described in the guide section , once you complete this you can come back to continue with the next steps.

This step was also described in the guide section , once you complete this you can come back to continue with the next steps.

This step was also described in the guide section and once you complete these you can come back to continue with the next steps.

👨‍💻
🏁
MultiversX Bubbles Data NFT
HTTP Response Header
instructions here
Step 2: Using SDK Core to enable MultiversX Wallet Login with Native Auth Enabled
Step 3: Grab a Native Auth "Session Token"
Step 4: Open a Data NFT and "View Data" via a Native Auth session
Bonus Step 5: How can I parse non-JSON Data Streams?