Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Publishing

REPS publishes verified terminal runs to an S3 bucket at a target supplied by you. There are no built-in hosts, provider presets, saved remotes, or target environment variables. Under the selected prefix, the bucket contains only runs/ and, when the prefix declares a license, its license.json.

Set up an AWS profile

Create the bucket and configure a named AWS profile using your storage provider’s instructions. For Hugging Face Storage Buckets, follow its S3 client and AWS profile setup. For Amazon S3, follow the AWS credentials guide. Keep endpoint, region, addressing and checksum settings in your AWS configuration, and credentials in the provider’s supported credential source.

--profile research selects that AWS profile. Omit it to use boto3’s normal credential resolution. REPS creates a boto3 session and its S3 client without overriding endpoint, region or client configuration. Ambient AWS settings are not passed into the experiment process; credential sets explicitly selected for the experiment still are.

Publish when a run finishes

Add --publish and optionally an AWS --profile to the experiment command. This complete mock-model example executes one run and publishes it:

nix run .#inspect-hello -- --set model=mockllm/model --set limit=0 --set epochs=1 --set 'generate_args={}' --publish s3://my-bucket/reps-v1 --profile research

The run’s data directory still follows --data-dir, REPS_DATA_DIR, then the XDG default.

--profile NAME selects the AWS profile for publication. Model credentials use --credential SET=NAME; both flags can appear in the same command.

--license SPDX declares the license. The license is resolved before the run starts; if it cannot be resolved, the command exits 2 without running.

Publication requires a terminal state, a pinned clean provenance.fetch_ref, and a passing verify audit. An unpinned run is refused, including with --dry-run; a later release label cannot supply its missing revision. Failed and interrupted runs can publish when their evidence passes that audit. A publishing error is logged without changing the run state or exit code.

Publish saved runs

Run the batch command against a local data directory:

nix run .#reps-runner -- publish --to s3://my-bucket/reps-v1 --profile research --data-dir "$HOME/reps-data"

With no stems, it considers all runs. Add --experiment NAME to filter by the experiment recorded on each card, or supply one or more positional stems: <condition>-<experiment> for a whole condition, or <condition>-<experiment>/<run> for one run. Stems are relative to runs/. Each run is verified independently; a rejected run does not stop the batch. The command returns a nonzero exit code if any run fails to publish.

--dry-run verifies and prints the exact object keys and byte sizes, including compressed streams. It reads the license declaration and checks the bucket for existing run keys but writes nothing. Verification resolves the manifest in the same way as reps-runner verify: from REPS_MANIFEST, then the experiment’s file in REPS_MANIFESTS, or a manifest catalog built from the current checkout.

Publication checks both destination run keys with HEAD. If either already exists, it refuses to overwrite that run, including a partial upload. Other runs continue. Choose a new prefix when you need different immutable objects. The publisher uses only HeadObject, GetObject and PutObject: it reads the license declaration, then uploads each stream first and its card last.

Each run uploads exactly these two objects:

<prefix>/runs/<condition>-<experiment>/<run>/events.jsonl.zst
<prefix>/runs/<condition>-<experiment>/<run>/run.json

The stream is compressed at zstd level 19 in a single frame with Content-Type: application/zstd, preserving its original bytes when decompressed. The card is copied byte-for-byte with Content-Type: application/json. Workspaces are never uploaded. See the published layout.

Declare the license

A prefix declares at most one license for all runs under it, in <prefix>/license.json:

{"v":0,"spdx":"CC-BY-4.0"}

spdx is an SPDX license identifier. The experiment command (with --publish) and reps-runner publish both accept --license SPDX. Without it, an existing declaration applies. If the prefix declares none, a terminal shows this menu:

1) CC-BY-4.0 (default)
2) CC0-1.0
3) CC-BY-SA-4.0
4) CC-BY-NC-SA-4.0
5) CC-BY-NC-ND-4.0
6) No license
7) Other: an identifier from https://spdx.org/licenses/<id>.html

Enter a number or type an identifier; an empty answer selects CC-BY-4.0. Without a terminal, or with the experiment command’s --non-interactive, the command fails when the prefix declares none and --license is absent. Each publish logs the license in effect as LICENSE s3://BUCKET/PREFIX SPDX.

--license NONE, or option 6, publishes without a license. It writes no license.json and logs LICENSE s3://BUCKET/PREFIX undeclared, and the index records null for the store. If the prefix already declares a license, NONE warns (publish: WARNING: … declares SPDX; publishing under it) and the runs publish under that declaration. NOASSERTION is refused; use NONE. A hand-written license.json that names NONE or NOASSERTION declares nothing: the index records null with a warning, and the publisher refuses the prefix.

The declaration is written once, before the first run uploads, and never replaced. Repeating the declared identifier is accepted. A different identifier other than NONE fails before any run uploads: the batch publishes nothing, and the experiment command does not start the run. To change the license, choose a new prefix.

To declare the license of an existing prefix, publish from an empty data directory:

nix run .#reps-runner -- publish --to s3://my-bucket/reps-v1 --profile research --license CC-BY-4.0 --data-dir "$(mktemp -d)"

The index copies each store’s declaration into index.json.

Build and serve an index

Experiment buckets contain only runs/ and license.json. A separate, replaceable index lets a static site discover those runs. Write stores.yaml yourself; REPS does not infer a public URL from an S3 address. Store lists accept YAML (.yaml or .yml) or JSON. data.example.org below is an example store hostname.

v: 0
stores:
  - s3: s3://my-experiment-bucket/reps-v1
    profile: research
    url: https://data.example.org/experiments/reps-v1
    experiments:
      - govsim
    exclude:
      - runs:
          - <run-id>
        reason: Provider outage invalidated these results.

url must expose the same prefix over public HTTPS. Optional experiments, conditions and runs lists filter exact recorded values and intersect. Leave a filter out to include everything; an empty list includes nothing.

An optional exclude list removes runs even when the inclusion filters select them. Each entry requires a non-empty list of run IDs and a non-empty reason, because the store list is the site operator’s record of what it chose not to show. A run ID can appear in only one exclusion entry per store. Exclusions are logged with their reasons on every rebuild, including --dry-run; unknown IDs warn without failing the build. Exclusions change only the site’s index and never touch bucket objects.

Store profiles select source credentials. The index command writes to a local directory that the site deploys. The index reads only ListObjectsV2 and GetObject, using each store’s profile without endpoint, region or client-configuration overrides. Every rebuild reads each store’s license.json and logs LICENSE s3://BUCKET/PREFIX SPDX, or undeclared with a warning when the declaration is absent or malformed.

Run cards are cached under $XDG_CACHE_HOME/reps/index (or ~/.cache/reps/index), keyed by profile (default when omitted), bucket, full object key and the listed ETag. Every rebuild still lists the store, so the listing determines membership and changed ETags trigger a fetch. The cache does not affect the index output; built_at still records each rebuild’s time. Use --cache-dir DIR to choose another cache directory or --no-cache to bypass cache reads and writes. --dry-run can read existing entries but never writes them. Store summaries report cached and fetched cards before filters and exclusions.

Build the index with this repository’s public outputs:

catalog="$(nix build .#manifests --no-link --print-out-paths)"
nix run .#reps-runner -- index --stores stores.yaml --catalog "$catalog" --to site/index

Deploy the generated site/index directory alongside your site. The index command reads the stores with AWS profiles; clients read public run objects without those credentials.

--catalog DIR is required and points to the built manifest directory: <name>.json files and optional assets/<name>/ trees, as with verify --catalog. The site’s catalog is derived from the index, so an experiment with no published runs never appears.

--dry-run prints each store’s license and filtered run count and the files and sizes it would write, without creating or deleting anything. Every invocation rebuilds the whole index from complete run-object pairs. It deletes the output directory and rewrites it in full, writing index.json last. Shards for experiments no longer present are removed, together with their catalog entries and assets. Unreadable, malformed or disappeared cards produce warnings and are skipped, so healthy runs can still be indexed. Failure to list a store aborts the rebuild before replacing the destination.

The root index.json lists experiment names, their run counts, the total count, stores and built_at. stores has one {"url":"<public HTTPS base>","license":"<SPDX identifier>"} entry per store, in store-list order; license is null for an undeclared or malformed declaration. Each experiments/<experiment>/index.jsonl contains one row {"store":"<public HTTPS base>","card":<run.json as a JSON value>} per run. The card is compactly re-serialized with Unicode retained and key order preserved. This derived cache is never authoritative. Read the original card and compressed stream from the row’s store for the recorded bytes and decompressed lines.

The index includes catalog.json with current manifests, shared render hints and versioned experiment hints for indexed experiments only. README assets are copied to catalog/assets/<name>/, dereferencing the manifest directory’s symlinks. A missing manifest warns and leaves the experiment in the index through its runs, without a manifest or experiment assets.