How to upload from CI

Publish files to IPFS and Filecoin via web3.storage from a Continuous Integration server.

Using w3cli locally, the steps are

Create a space that you want CI to upload to (or use an existing one)
Create a signing key for CI to use with w3 key create
Create a proof that delegates capabilities to upload to your space to that key

Then in the CI environment

Install w3cli from npm
Import the signing key by setting it as W3_PRINCIPAL in the env
Import the proof by passing it to w3 space add
Upload your files with w3 up

Using Github? Use the add-to-web3 Action https://github.com/marketplace/actions/add-to-web3 (opens in a new tab)

Example

Using w3cli locally create a signing key and a proof.

# Create key and did. Use the output `key` value as W3_PRINCIPAL in CI.
$ w3 key create --json > ci-key.json
 
# Extract the did as $AUDIENCE
$ AUDIENCE=$(jq -r .did ci-key.json)
 
# Create a signed proof that you delegate capabilties to that key. 
$ w3 delegation create $AUDIENCE -c store/add -c upload/add --base64
mAYIEAP8OEaJlcm9vdHOAZ3ZlcnNpb24BwwUBcRIg+oHTbzShh1WzBo9ISkonCW+KAcy/+zW8Zb...
 
# Pass the output to `w3 space add` in ci

Then on the CI side (in github flavour (opens in a new tab))

steps:
  - run: npm install -g @web3-storage/w3cli
 
  - run: w3 space add ${{ inputs.proof }}
    env:
      W3_PRINCIPAL: ${{ inputs.secret_key }}
 
  - run: w3 up ${{ inputs.path_to_add }}
    env:
      W3_PRINCIPAL: ${{ inputs.secret_key }}

The rest of this document explains the process in more detail.

Create a space

On your local machine with w3cli (opens in a new tab) installed and logged in (see: https://web3.storage/docs/quickstart/ (opens in a new tab)) run

$ w3 space create

and follow the instructions. (See: https://web3.storage/docs/how-to/create-space/#using-the-cli (opens in a new tab) if you get stuck.)

If you want to use an existing space, make sure it is set as your current space using w3 space ls and w3 space use

Create a signing key

On your local machine, use w3cli (opens in a new tab) to generate a new a new Ed25519 private signing key for CI to use.

# Use the `did` in the input to the next command. 
# Use `key` as your `secret_key` for add_to_web3.
$ w3 key create --json
{
  "did": "did:key:z6Mk...",
  "key": "MgCaT7Se2QX9..."
}

Keep the key safe. It will be used by CI to sign requests to web3.storage.

The did from the command above is the public decentalised identifier for that private key.

Create a proof

On your local machine, use w3cli (opens in a new tab) to delegate capabilties to upload to our space to the public DID for the signing key we created.

Our CI environment doesn't need to list our uploads or change our billing plan so we only delegate the store/add and upload/add capabilities to it.

Pass the did for the signing key as the audience parameter. We are delegating capabilities to that key.

$ AUDIENCE=did:key:z6Mk...
 
# Delegate capabilities to the `did` we created above.
$ w3 delegation create $AUDIENCE -c 'store/add' -c 'upload/add' --base64
mAYIEAP8OEaJlcm9vdHOAZ3ZlcnNpb24BwwUBcRIg+oHTbzShh1WzBo9ISkonCW+KAcy/+zW8Zb...

The output is a base64 encoded UCAN proof, signed by your local key. It can only be used as proof by the signing key we specified by the DID we passed in.

Now we have a signing key and a proof we can use in the CI environment.

Install `w3cli` in CI

Install it from npm with the --global flag to make the w3 command available.

$ npm i --global @web3-storage/w3cli

Import the signing key

Set W3_PRINCIPAL=<the signing key> in the CI environment. The w3 commmand will use the value as the signing key to use. see: https://github.com/web3-storage/w3cli?tab=readme-ov-file#w3\_principal (opens in a new tab)

The value is the key we created above with w3 key create. The key must be the one for the did that was used to create the proof.

Import the proof

Set W3_PROOF=<the proof> in the CI environment.

In your CI job definition, run the w3 space add command to import the proof that it can upload to the space we created.

$ w3 space add $W3_PROOF

Upload your files

In your CI job definition, run the w3 up command to upload the files you want to publish on IPFS and store in Filecoin.

$ w3 up <path to files>

that path might be the dist or output directory of a previous step that built your static website or collected some stats from a job.

Once that command returns succesfully, you are done, your files are content addressed and available over IPFS.

If you want to capture the CID for your uploads pass the --json flag and use jq to extract it

# write the output as json to a file
$ w3 up <path to files> --json > ./w3_up_output.json
 
# extract the root cid from the output and set it as an env var.
$ CID=$(jq --raw-output '.root."/"' ./w3_up_output.json)

Github Action: add-to-web3

The add-to-web3 (opens in a new tab) action is a lightweight wrapper around w3cli (opens in a new tab). You create the key and proof as before, and the action configures and runs w3 in CI for you.

Use it in your Github Workflow like this

uses: web3-storage/add-to-web3@v3
id: w3up
with:
  path_to_add: 'dist'
  secret_key: ${{ secrets.W3_PRINCIPAL }}
  proof: ${{ secrets.W3_PROOF }}
 
# use the outputs in subsequent steps
# "bafkreicysg23kiwv34eg2d7qweipxwosdo2py4ldv42nbauguluen5v6am"
- run: echo ${{ steps.w3up.outputs.cid }}
 
# "https://bafkreicysg23kiwv34eg2d7qweipxwosdo2py4ldv42nbauguluen5v6am.ipfs.w3s.link"
- run: echo ${{ steps.w3up.outputs.url }}

It uploads path_to_add to web3.storage.

It outputs the root CID as cid and IPFS Gateway URL as url for subsequent steps in your workflow to use.

We use the action to publish this website (👁️⁂👁️) to IPFS when a PR is merged to main. We use the cid output from the action as input to update a dns record (opens in a new tab) called a DNSlink for the domain.

Setting the DNSLink announces the new root CID for the website, so IPFS-aware browsers (opens in a new tab) and IPFS Companion (opens in a new tab) can load the site over IPFS.

Publish IPNS names HTTP Bridge