How to store data using Web3.Storage

In this how-to guide, you'll learn how to store data programmatically for your development projects using the Web3.Storage client libraries in JavaScript and Go. This includes making your data available on the decentralized IPFS network with persistent long-term storage provided by Filecoin — all for free.

If you just want to quickly store a few files using Web3.Storage rather than include upload functionality in an app or service you're building, you may want to skip this guide for now and simply use the Files page on the Web3.Storage site.

For developers, Web3.Storage provides a simple interface for storing data using syntax inspired by familiar web APIs such as fetch and File. This guide focuses on JavaScript client library and Go client library, which are the simplest way to use Web3.Storage programmatically.

If you're using another language, see the HTTP API reference for details on working with the underlying HTTP API.

Uploading data to Web3.Storage using a client library requires a free API token, which in turn requires a Web3.Storage account. If you already have an account and a token, read on. If not, have a look at the quickstart guide to get up and running in just a few minutes.

CAUTION

All data uploaded to Web3.Storage is available to anyone who requests it using the correct CID. Do not store any private or sensitive information in an unencrypted form using Web3.Storage.

Installing the client

In your JavaScript project, add the web3.storage package to your dependencies:

npm install web3.storage

Creating a client instance

First, create a Web3.Storage client object, passing in an API token to its constructor:

import { Web3Storage } from 'web3.storage'

function getAccessToken () {
  // If you're just testing, you can paste in a token
  // and uncomment the following line:
  // return 'paste-your-token-here'

  // In a real app, it's better to read an access token from an
  // environement variable or other configuration that's kept outside of
  // your code base. For this to work, you need to set the
  // WEB3STORAGE_TOKEN environment variable before you run your code.
  return process.env.WEB3STORAGE_TOKEN
}

function makeStorageClient () {
  return new Web3Storage({ token: getAccessToken() })
}

Preparing files for upload

The Web3.Storage client's put method accepts an array of File objects.

There are a few different ways of creating File objects available, depending on your platform.

In the browser, you can use a file input element to allow the user to select files for upload:

function getFiles () {
  const fileInput = document.querySelector('input[type="file"]')
  return fileInput.files
}

You can also manually create File objects using the native File constructor provided by the browser runtime. This is useful when you want to store data created by your application, instead of files from the user's computer.

function makeFileObjects () {
  // You can create File objects from a Blob of binary data
  // see: https://developer.mozilla.org/en-US/docs/Web/API/Blob
  // Here we're just storing a JSON object, but you can store images,
  // audio, or whatever you want!
  const obj = { hello: 'world' }
  const blob = new Blob([JSON.stringify(obj)], { type: 'application/json' })

  const files = [
    new File(['contents-of-file-1'], 'plain-utf8.txt'),
    new File([blob], 'hello.json')
  ]
  return files
}
Tip

When uploading multiple files, try to give each file a unique name. All the files in a put request will be bundled into one content archive, and linking to the files inside is much easier if each file has a unique, human-readable name.

Uploading to Web3.Storage

Once your files are ready, uploading is a simple method call on the client object.

IMPORTANT

Deleting files from the Web3.Storage site's Files page will remove them from the file listing for your account, but that doesn't prevent nodes on the decentralized storage network from retaining copies of the data indefinitely. Do not use Web3.Storage for data that may need to be permanently deleted in the future.

Once you have an array of Files, you can upload them with the client's put method:

async function storeFiles (files) {
  const client = makeStorageClient()
  const cid = await client.put(files)
  console.log('stored files with cid:', cid)
  return cid
}

Showing progress to the user

The put method has some options that can be passed in to get progress on the upload as it happens in the background. There are two callback parameters you can use: onRootCidReady, and onStoredChunk.

The onRootCidReady callback is invoked as soon as the client has calculated the content identifier (CID) that identifies the data being uploaded. Because this calculation happens locally on the client, the callback is invoked before the upload begins.

As each chunk of data is uploaded, the onStoredChunk callback gets invoked with the size of the chunk in bytes passed in as a parameter.

Here's a simple example of using the callbacks to print the progress of an upload to the console:

async function storeWithProgress (files) {
  // show the root cid as soon as it's ready
  const onRootCidReady = cid => {
    console.log('uploading files with cid:', cid)
  }

  // when each chunk is stored, update the percentage complete and display
  const totalSize = files.map(f => f.size).reduce((a, b) => a + b, 0)
  let uploaded = 0

  const onStoredChunk = size => {
    uploaded += size
    const pct = totalSize / uploaded
    console.log(`Uploading... ${pct.toFixed(2)}% complete`)
  }

  // makeStorageClient returns an authorized Web3.Storage client instance
  const client = makeStorageClient()

  // client.put will invoke our callbacks during the upload
  // and return the root cid when the upload completes
  return client.put(files, { onRootCidReady, onStoredChunk })
}

Directory wrapping

By default, files uploaded to Web3.Storage will be wrapped in an IPFS directory listing. This preserves the original filename and makes links more human-friendly than CID strings, which look like random gibberish.

The CID you get back from the client when uploading is the CID of the directory, not the file itself! To link to the file itself using an IPFS URI, just add the filename to the CID, separated by a / like this: ipfs://<cid>/<filename>.

To make a gateway link, use https://<cid>.ipfs.<gateway-host>/<filename> or https://<gateway-host>/ipfs/<cid>/<filename>, where <gateway-host> is the address of an HTTP gateway like dweb.link.

Once uploaded, you can retrieve the directory or list the contents without downloading it.

To avoid having your files wrapped in a directory listing, set the wrapWithDirectory: option to false when uploading using the JavaScript client.

Storing IPFS Content Archives

So far we've focused on using the put method, which accepts regular files and packs them into an IPFS Content Archive (CAR) file before uploading to Web3.Storage. If you're already using IPFS in your application, or if you want more control over the IPLD graph used to structure your data, you can construct your own CAR files and upload them directly.

See Working with CAR files for more information about Content Archives, including how to create and manipulate them with code or command-line tools.

Once you have a Content Archive, you can use the putCar client method to upload it to Web3.Storage.

The putCar method accepts a CarReader, which is a type defined by the @ipld/car package.

You can create a CarReader from a Uint8Array using the fromBytes static method:

import { CarReader } from '@ipld/car';

// assume loadCarData returns the contents of a CAR file as a Uint8Array
const carBytes = await loadCarData();
const reader = await CarReader.fromBytes(carBytes);

const client = makeStorageClient();
const cid = await client.putCar(reader);
console.log('Uploaded CAR file to Web3.Storage! CID:', cid);

See the putCar reference documentation for more information about putCar, including optional parameters.

The Working with CAR files guide has more information about the @ipld/car package, including how to implement the loadCarData function and other ways to construct a CarReader.

Next steps

The client returns an IPFS content identifier (CID) that can be used to fetch your files over IPFS. Once uploaded, your data is immediately available for retrieval via IPFS and will be stored with Filecoin storage providers within 48 hours. To learn how to fetch your data using the Web3.Storage client, or directly from IPFS using a gateway or the IPFS command line, see the how-to guide on retrieval.

You can also get more information about the status of your data. See the query how-to guide to learn how to get more details about your data, including the status of any Filecoin storage deals.

Was this information helpful?

Help us improve this site!

Edit this page on GitHub or open an issue for it
Last updated on 06/24/2022