How to upload data using web3.storage

In this how-to guide, you'll learn how to store data programmatically for your development projects using the web3.storage client library in JavaScript using your (developer-owned) Space. This includes various architecture options for the data pipeline for your users to upload to web3.storage, which then makes your data available on the decentralized IPFS network with persistent long-term storage provided by Filecoin.

Later in this section, we also cover uploading data using the CLI or web console. If you just want to quickly store a few files using web3.storage rather than include upload functionality in an app or service you're building, you may want to hop down there.

Using the CLI

If you followed the Create Account and Create Space sections, you will already have the CLI set up with a Space. However, you might be using the CLI on a new machine, in which case you can follow these instructions:

1. Install the CLI from npm using your command line: npm install -g @web3-storage/w3cli.
2. Run w3 login [email protected] in the command line using your email address. Click on the validation link sent to your email.
3. After successfully running login, your CLI Agent has been delegated access to all Spaces associated with your email address. You can see a list of these Spaces using w3 space ls and select the one you'd like to upload to using w3 space use <space_did>.

When the right Space is selected, you are ready to upload! You can do so by running w3 up <path>.

There are a few useful flags (check out the reference docs or w3 up --help to see a full list):

--no-wrap    # Don't wrap input files with a directory.
-H, --hidden # Include paths that start with ".".
-c, --car    # File is a CAR file.

Using the JS client

Installing the client

In your JavaScript project, add the web3.storage package to your dependencies:

npm install @web3-storage/w3up-client

Creating a client instance

The package provides a static create function (opens in a new tab) that returns a Client object (opens in a new tab). How you initialize it depends on the environment the client is used in: persistent or ephemeral.

Examples of persistent environments:

• A browser application
• A terminal application
• An installed application (e.g. Electron)

Examples of ephemeral environments:

• AWS Lambda or server side workers
• Running inside Docker instances
• CI

Claim delegations via email validation

A new client can claim access to their existing Spaces by validating their email address.

You can use web3.storage's email authorization flow to give permissions to your client. This can be good if your environment will be persistent (otherwise it would be prohibitive to click an email validation link every time the client is re-instantiated).

When a Space is created, access permissions are delegated to your email address. We use a special kind of DID for this, a did:mailto:. These UCANs are stashed with the web3.storage service. When you validate your email address with a new Agent DID, web3.storage issues a UCAN attestation, that says your Agent DID is owned by your email address. It also returns the UCAN permissions you previously stashed. You can then use the returned UCANs, along with the attestation to prove you are authorized to perform actions.

import { create } from '@web3-storage/w3up-client'
const client = await create()

By default, constructing a client like this will re-use state persisted by other clients because create constructs the client with a store that persists data between processes and client instances.

Once you have created a client, you can login with your email address. Calling login will cause an email to be sent to the given address.

await client.login('[email protected]')

Once a user clicks the confirmation link in the email, the login method will resolve. Make sure to check for errors, as login will fail if the email is not confirmed within the expiration timeout. Authorization needs to happen only once per agent. This also claims all delegations available with your email address, so from there, you can select the Space you'd like to use.

await client.setCurrentSpace('did:key:...') // select the relevant Space DID that is associated with your account

Bring your own delegations

An option that works for any backend environment is for a developer to create and provision a Space, and then delegate access to a different Agent DID that will be used by the client. This is especially useful if you're using the client in a serverless environment (e.g., AWS Lambda).

In your command line where w3cli is configured with the Space you want to use (e.g., where you created the Space):

# The following command returns what will be your Agent private key and DID
w3 key create
 
# ❗️ Store the private key (starting "Mg...") in environment variable KEY
 
# The following command creates a UCAN delegation from the w3cli agent to the
# agent you generated above.
#
# Use `w3 space use` prior to this to set the Space you intend on delegating
# access to.
#
# If you want to limit permissions being passed to the Agent, you can specify
# permissions to give, e.g., `--can 'store/add' --can 'upload/add'` limits to
# just being able to upload.
w3 delegation create <did_from_ucan-key_command_above> | base64
 
# ❗️ Store the output in environment variable PROOF

Then, when you initialize and configure the client, you can pass in this Agent and UCAN.

import * as Client from '@web3-storage/w3up-client'
import { StoreMemory } from '@web3-storage/w3up-client/stores/memory'
import { importDAG } from '@ucanto/core/delegation'
import { CarReader } from '@ipld/car'
import * as Signer from '@ucanto/principal/ed25519'
 
async function main () {
  // Load client with specific private key
  const principal = Signer.parse(process.env.KEY)
  const store = new StoreMemory()
  const client = await Client.create({ principal, store })
  // Add proof that this agent has been delegated capabilities on the space
  const proof = await parseProof(process.env.PROOF)
  const space = await client.addSpace(proof)
  await client.setCurrentSpace(space.did())
  // READY to go!
}
 
/** @param {string} data Base64 encoded CAR file */
async function parseProof (data) {
  const blocks = []
  const reader = await CarReader.fromBytes(Buffer.from(data, 'base64'))
  for await (const block of reader.blocks()) {
    blocks.push(block)
  }
  return importDAG(blocks)
}

If you're doing this in a non-persistent or serverless backend, you might consider using an in-memory Store (opens in a new tab) for your Agent information rather than the default on-disk:

import { StoreMemory } from '@web3-storage/access/stores/store-memory'
const client = await Client.create({ principal, store: new StoreMemory() })

Uploading to web3.storage

Now that your client instance is setup to interact with your Space, you're ready to upload! Call uploadFile to upload a single file, or uploadDirectory to upload multiple files.

There are two main options to getting content into your Space:

• Upload data to web3.storage from the backend client itself (e.g., you're storing data that your users are uploading to your backend)
• Upload data to web3.storage directly from your user's environment (like your application's user's browser) by delegating a UCAN that has permission to upload to your Space

Upload from backend client directly

You are already set up to upload using your client instance as data becomes available to your backend - you can call uploadFile or uploadDirectory with it.

import { create } from '@web3-storage/w3up-client'
import { filesFromPaths } from 'files-from-path'
 
// e.g "./best-gifs"
const path = process.env.PATH_TO_FILES
const files = await filesFromPaths([path])
 
const client = await create()
const directoryCid = await client.uploadDirectory(files)

In the example above, directoryCid resolves to an IPFS directory.

Delegate UCAN for your user to upload directly

Your backend instance can also be used to delegate upload permissions directly to your user to upload. The code snippet below shows an example of how you might set up a client instance in your application frontend and how it might interact with your backend client. You can see how the frontend client Agent DID is used for the backend client to delegate permissions to; from there, it will be the frontend client that will call the upload method.

Backend

import { CarReader } from '@ipld/car'
import * as DID from '@ipld/dag-ucan/did'
import * as Delegation from '@ucanto/core/delegation'
import * as Signer from '@ucanto/principal/ed25519'
import * as Client from '@web3-storage/w3up-client'
import { StoreMemory } from '@web3-storage/w3up-client/stores/memory'
 
async function backend(did) {
  // Load client with specific private key
  const principal = Signer.parse(process.env.KEY)
  const store = new StoreMemory()
  const client = await Client.create({ principal, store })
 
  // Add proof that this agent has been delegated capabilities on the space
  const proof = await parseProof(process.env.PROOF)
  const space = await client.addSpace(proof)
  await client.setCurrentSpace(space.did())
 
  // Create a delegation for a specific DID
  const audience = DID.parse(did)
  const abilities = ['store/add', 'upload/add']
  const expiration = Math.floor(Date.now() / 1000) + (60 * 60 * 24) // 24 hours from now
  const delegation = await client.createDelegation(audience, abilities, { expiration })
 
  // Serialize the delegation and send it to the client
  const archive = await delegation.archive()
  return archive.ok
}
 
/** @param {string} data Base64 encoded CAR file */
async function parseProof(data) {
  const blocks = []
  const reader = await CarReader.fromBytes(Buffer.from(data, 'base64'))
  for await (const block of reader.blocks()) {
    blocks.push(block)
  }
  return Delegation.importDAG(blocks)
}

Frontend

import * as Delegation from '@ucanto/core/delegation'
import * as Client from '@web3-storage/w3up-client'
 
async function frontend() {
  // Create a new client
  const client = await Client.create()
 
  // Fetch the delegation from the backend
  const apiUrl = `/api/w3up-delegation/${client.agent().did()}`
  const response = await fetch(apiUrl)
  const data = await response.arrayBuffer()
 
  // Deserialize the delegation
  const delegation = await Delegation.extract(new Uint8Array(data))
  if (!delegation.ok) {
    throw new Error('Failed to extract delegation', { cause: delegation.error })
  }
 
  // Add proof that this agent has been delegated capabilities on the space
  const space = await client.addSpace(delegation.ok)
  client.setCurrentSpace(space.did())
 
  // READY to go!
}

Preparing files and uploading

You are now ready to upload using the client! In general, the easiest way to upload data is using the uploadFile or uploadDirectory method.

uploadFile expects a "Blob like" input, which can be a Blob (opens in a new tab) or File (opens in a new tab) when running in a browser. On Node.js, see the files-from-path library (opens in a new tab), which can load compatible objects from the local filesystem. By default, files uploaded to web3.storage will be wrapped in an IPFS directory listing. This preserves the original filename and makes links more human-friendly than CID strings, which look like random gibberish.

uploadDirectory requires File-like objects instead of Blobs, as the file's name property is used to build the directory hierarchy.

You can control the directory layout and create nested directory structures by using / delimited paths in your filenames:

const files = [
  new File(['some-file-content'], 'readme.md'),
  new File(['import foo'], 'src/main.py'),
  new File([someBinaryData], 'images/example.png'),
]
 
const directoryCid = await client.uploadDirectory(files)

In the example above, directoryCid resolves to an IPFS directory with the following layout:

.
├──images
|  └──example.png
├──readme.md
└──src
   └──main.py

There are a few different ways of creating File objects available, depending on your platform.

In the browser, you can use a file input element (opens in a new tab) to allow the user to select files for upload:

function getFiles () {
  const fileInput = document.querySelector('input[type="file"]')
  return fileInput.files
}

You can also manually create File objects using the native File constructor provided by the browser runtime. This is useful when you want to store data created by your application, instead of files from the user's computer.

function makeFileObjects () {
  // You can create File objects from a Blob of binary data
  // see: https://developer.mozilla.org/en-US/docs/Web/API/Blob
  // Here we're just storing a JSON object, but you can store images,
  // audio, or whatever you want!
  const obj = { hello: 'world' }
  const blob = new Blob([JSON.stringify(obj)], { type: 'application/json' })
 
  const files = [
    new File(['contents-of-file-1'], 'plain-utf8.txt'),
    new File([blob], 'hello.json')
  ]
  return files
}

In Node.js, the files-from-path library (opens in a new tab) reads File objects from the local file system. The filesFromPaths helper asynchronously returns an array of Files that you can use directly with the uploadDirectory client method:

import { filesFromPaths } from 'files-from-path'

You can also manually create File objects by importing a Node.js implementation of File from the web3.storage package. This is useful when you want to store data created by your application, instead of files from the user's computer.

async function getFiles (path) {
  const files = await filesFromPaths([path])
  console.log(`read ${files.length} file(s) from ${path}`)
  return files
}
 
function makeFileObjects () {
  // You can create File objects from a Buffer of binary data
  // see: https://nodejs.org/api/buffer.html
  // Here we're just storing a JSON object, but you can store images,
  // audio, or whatever you want!
  const obj = { hello: 'world' }
  const buffer = Buffer.from(JSON.stringify(obj))
  const files = [
    new File(['contents-of-file-1'], 'plain-utf8.txt'),
    new File([buffer], 'hello.json')
  ]
  return files
}

Next steps

Learn more about how to fetch your data using the CID in the next section, retrieve.