What is REPS?

REPS packages replications of multi-agent safety experiments, runs them on your machine, and saves their inputs, results and execution records together. This repository provides the experiments, runner and Nix library.

What would you like to do?

See the Roadmap for planned work on execution and recorded data.

How are experiments and results organized?

For example, choosing a model and dataset limit sets inputs for an evaluation. Running that configuration twice saves two separate records. Changing the model lets you compare another configuration. A run’s record can include messages, model requests and responses, scores, logs and files; the experiment determines what it emits.

An experiment defines a program and its inputs. A condition groups runs with the same experiment name, declared source content and complete set of input values. A run is one execution of a condition. Repeating a condition creates another run with its own ID and record.

For precise field and option definitions, use the references for commands, manifests, stored files and events.

Run your first local experiment

Use the terminal to run inspect-hello, a small evaluation with fixed mock replies. You need Nix installed on Linux or macOS and an internet connection for the initial build. This example needs no model credentials.

Inspect the inputs

Ask the experiment for its manifest:

nix run .#inspect-hello -- --describe

The commands on this page follow your Nix command settings, including downloaded source or a local checkout. The default uses classic Nix and requires no checkout.

The manifest describes every parameter and result. Each parameter must be supplied explicitly; an initial value is a suggestion, not a command-line default. For this example, choose mockllm/model, both samples (limit=0), one pass (epochs=1) and no generation overrides (generate_args={}).

Check the command

nix run .#inspect-hello -- \
  --set model=mockllm/model --set limit=0 --set epochs=1 \
  --set 'generate_args={}' --dry-run

This validates the parameter names and types and prints the resolved inputs, condition ID and seed without executing the experiment. For a real model, configure credentials before running.

Execute one run

Remove --dry-run:

nix run .#inspect-hello -- \
  --set model=mockllm/model --set limit=0 --set epochs=1 \
  --set 'generate_args={}'

The generated launcher builds the pinned program and starts the runner with its manifest and source identity. Expect two completed samples, zero errors and score 1.0. Each invocation creates one run. Ctrl-C interrupts execution while retaining the partial record.

The terminal prints the run ID and store path. The default data directory is ~/.local/share/reps, or $XDG_DATA_HOME/reps when set. REPS_DATA_DIR or --data-dir DIR selects another directory; see run storage.

Inspect the result

Open run.json in the printed run directory for inputs, provenance, state and derived results. events.jsonl contains the event envelopes, including each result and the final run.end. Follow the terminal reading example, or add --json to the experiment command to stream the events to standard output while they are saved.

A completed process does not establish that every evaluation item succeeded; check the recorded results and errors too. Before collecting real-model results, audit the first run with reps-runner verify.

Roadmap

What works today?

• Local CLI execution through Nix.
• Saved credential profiles.
• Experiment contributions through ordinary in-repository pull requests.

What comes next?

• Record source, environment and external inputs, and test with disposable runs so we can derive additional fingerprints later.
• Define a versioned data format and replication instructions.
• After those checks, run experiments and publish our data with a read-only website.

Possible later additions:

• Parameter presets and reuse of run configuration.
• Analysis and annotations about which runs can meaningfully be compared, possibly in another tool.
• External data deposits with attribution and review.
• An agent authoring skill and sandbox tests of its instructions.
• Dynamic credentials and declarations for credentials other than model access.
• Stronger execution isolation and a proxy that records requests.
• Pausing an execution and starting branches from it.

Choose inputs and run

Inspect an experiment’s manifest and bind its parameters in the terminal. Neither this nor the first-run example requires a checkout.

How do I use the command line?

Use the generated experiment launcher, which supplies the runner’s execution context. The command’s Nix form and source follow the command settings. A command using moving main runs that source’s current experiment; to repeat a historical configuration, select its recorded source revision and inputs.

For a model that needs credentials, configure a saved profile and select it with --credential SET=NAME when needed. An interactive terminal can prompt for missing built-in credentials; noninteractive runs require setup beforehand.

Ask the experiment for its manifest:

nix run .#inspect-hello -- --describe

Supply each declared parameter with --set. This keyless example uses the bundled mock model:

nix run .#inspect-hello -- \
  --set model=mockllm/model \
  --set limit=0 \
  --set epochs=1 \
  --set 'generate_args={}'

Every parameter is required on the command line, including those with an initial value in the manifest. If a parameter is missing, the runner prints an example command. Review its values before running it.

Values can be JSON, a bare string or @path to read a file. Quote a whole KEY=VALUE argument when it contains spaces or shell punctuation. For example, replace the last argument above with --set generate_args=@generation.json after saving a JSON object in that file. An explicit null is allowed only for a parameter declared nullable. See value syntax.

How do I check before executing?

Add --dry-run to a complete command. It checks parameter names and types and prints the condition and inputs without starting the experiment or resolving credentials. List-length bounds are checked only when executing. It does not test account access, model availability or an experiment’s external dependencies.

Remove --dry-run when ready. The runner prints a run ID and a store path. Each invocation executes one run.

How do I inspect the result?

Read run.json and events.jsonl in the printed run directory to inspect the saved metadata and events.

For terminal processing, add --json to the experiment command to stream event envelopes as JSON lines on the launcher’s stdout; its diagnostics go to stderr. This is distinct from the child experiment program’s stdout, which is captured as text events. Saved run files are still written. For unattended execution, also pass --non-interactive to disable prompts; required credentials and profile selections must be resolvable without input. Each envelope identifies its run.

Check event.state in each run.end envelope, or lifecycle.state in the saved run.json. The runner returns 0 when the run completes, 1 if it fails or times out, and 130 on interruption. A completed process can also report errors in individual evaluation items, so inspect the summary and relevant events. Missing run.end may mean execution is still active or stopped before recording its outcome. The terminal reading example shows how to inspect the saved record.

To interrupt a terminal run, press Ctrl-C. Inspect the retained record to see how far it progressed. To run the same configuration again, use the original command; repeat and compare runs explains seeds and source versions.

Configure model credentials

REPS stores model credentials and endpoints in named profiles on your machine. A run selects the credential sets referenced by its model inputs and passes the selected values to the experiment process. Exporting an API key in your shell does not configure a REPS run.

How do I configure a provider?

Use the standalone credential-management command below. Direct reps-runner invocation here configures profiles; experiment execution uses the named experiment app.

nix run .#reps-runner -- credentials set openai

Enter the requested values and profile name. Secret input is hidden. A bare set name offers default as the profile name; use an explicit name to edit or create a specific profile:

nix run .#reps-runner -- credentials set openai.research

The endpoint prompt offers the provider’s configured default where one exists. Use the endpoint expected by the adapter and provider, including its scheme and API path. For self-hosted models, configure vllm with VLLM_BASE_URL (for example, http://localhost:8000/v1) or ollama with OLLAMA_BASE_URL (for example, http://localhost:11434/v1). Neither has a default endpoint; VLLM_API_KEY and OLLAMA_API_KEY are optional.

List saved sets and locate the file with:

nix run .#reps-runner -- credentials list
nix run .#reps-runner -- credentials path

The store is normally ~/.config/reps/credentials.toml, or $XDG_CONFIG_HOME/reps/credentials.toml. REPS_CREDENTIALS_FILE overrides that path. REPS writes the file with mode 0600 and refuses regular credential files readable by group or others. The file contains plaintext credentials; keep it outside the run store and repository.

Which profile will a run use?

Add --credential openai=research to an experiment command to select that profile explicitly. Repeat the flag for runs using several credential sets. The selected set must be used by this run, and the profile must exist.

Without an explicit selection, the runner uses a remembered choice for this experiment and set, then a lone default profile. In an interactive terminal it offers a picker when there are other profiles, and offers first-use setup for an unconfigured built-in provider. With --non-interactive or non-terminal stdin it never prompts: it uses the remembered choice or default profile, and fails if a required selection is unavailable.

You can save a preference explicitly:

nix run .#reps-runner -- credentials remember inspect-hello openai research

Preferences are stored separately in $XDG_CONFIG_HOME/reps/preferences.toml, defaulting to ~/.config/reps/preferences.toml. An explicit --credential overrides them. Profiles are selected as a whole; missing fields are not filled from another profile.

How do model IDs select credential sets?

Usually the part before the first slash is the set: openai/MODEL uses openai, and vllm/Qwen/Qwen2.5-7B-Instruct uses vllm. Configure a self-hosted service with:

nix run .#reps-runner -- credentials set vllm.default

That prompt uses VLLM_API_KEY and VLLM_BASE_URL; use vllm/Qwen/Qwen2.5-7B-Instruct as the model ID. For Ollama, configure ollama.default and use a library tag such as ollama/qwen2.5:7b-instruct-fp16. Profiles within each set select the venue; for example, --credential vllm=cluster selects vllm.cluster.

Inspect-style IDs of the form openai-api/SERVICE/MODEL use SERVICE as the credential set. Which model-ID forms are accepted depends on the experiment’s adapter. An unknown prefix does not itself add provider support. Mock prefixes mock and mockllm require no credential set.

REPS discovers model IDs in llm-typed inputs, including nested lists and structures. It does not infer credentials from arbitrary string fields or free-form JSON objects.

How do I update, remove or script profiles?

Run credentials set SET.PROFILE again to edit a profile. Remove one profile or an entire set with credentials remove SET.PROFILE or credentials remove SET.

For automation, credentials set SET.PROFILE --json reads one JSON object from standard input. Each key is an environment-variable name and each value is a string; null removes an existing field. Empty strings and omitted fields retain existing values. Feed that input from your secret-management system rather than putting secret values in command arguments. credentials list --json returns masked inventory for tools. See command reference.

Run open-weight models

REPS talks to a self-hosted model the same way it talks to a hosted one: an OpenAI-compatible chat endpoint, selected by a credential profile, with the model ID naming the serving engine and the upstream repo (vllm/Qwen/Qwen1.5-72B-Chat-GPTQ-Int4, see model IDs). Where the engine runs is a venue detail that lives in the profile, never in the model ID or the records. This page is the venue-independent part; each venue has its own subpage with the choices that make its endpoint satisfy REPS:

• Vast.ai Serverless

What REPS needs from the endpoint

• A chat-completions route at one stable URL. The client appends /chat/completions to the profile’s VLLM_BASE_URL.
• The response’s model field must equal the repo id after vllm/; the first call of every run checks this and fails the run on a mismatch.
• A trailing assistant message is continued as a prefill. The client sends vLLM’s continue_final_message for vllm/ ids; nothing to configure server-side.
• vLLM rejects a request whose prompt plus max_tokens exceeds the model’s context. For models with contexts below the experiment’s token budget, set the budget to null where the experiment allows it (GovSim does; null omits it).

Profile

One credential profile per endpoint:

nix run .#reps-runner -- credentials set vllm.<key>

VLLM_BASE_URL is the URL the server mounts the OpenAI API under; the client appends /chat/completions itself. VLLM_API_KEY is whatever bearer the venue expects, and may be empty for a server that ignores auth. Profile names allow [a-z0-9_-] only, so qwen15-72b, not qwen1.5-72b. For scripting, the --json form reads {"VLLM_API_KEY": ..., "VLLM_BASE_URL": ...} from stdin. A run selects the profile with --credential vllm=<key> and the model with --set model=vllm/<REPO>.

Prove it before running

curl "$VLLM_BASE_URL/chat/completions" \
  -H "Authorization: Bearer $VLLM_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"<REPO>","messages":[{"role":"user","content":"Pick a number from 1 to 10."},{"role":"assistant","content":"Answer:"}],"add_generation_prompt":false,"continue_final_message":true,"temperature":0,"max_tokens":20}'

Two checks: model echoes the repo id, and the content continues after “Answer:” rather than restating it. Either failing means the first run would fail the same way, so fix it before anything bills.

Run

Match sweep concurrency to what the server can answer within any proxy or gateway timeout in front of it; the venue pages give numbers. One window per replicate (--split-replicates) avoids a serial tail at the end of a batch. Wait until the endpoint answers the probe before launching: a cold endpoint fails the first runs, which then need re-queueing. The sweep refuses to launch from a tree with modified tracked files, so commit or stash edits (a docs page in progress counts) before the endpoint is up and billing.

Finish

Audit the batch from its records (errors, retries, served model, counts per cell), then publish. If a batch was killed and relaunched, the killed cards stay running and the publisher reports them as not terminal; publish by run id (<stem>/<run>) instead of by stem. The publisher resolves the manifest catalog once per run with nix-build -A manifests, about seven seconds of evaluation each, so a 75-run batch spends ten minutes on that alone; build the catalog once and point REPS_MANIFESTS at it for the dry-run and the publish. Tear the venue down as soon as the last run completes; publishing reads only local records.

Vast.ai Serverless

The venue-specific part of running open-weight models for Vast.ai Serverless. It does not repeat Vast’s own documentation; it records the choices that make a Vast endpoint satisfy REPS and the operational facts learned while running GovSim’s open-weight models.

Facts that shape the setup

• The proxy is https://openai.vast.ai/<ENDPOINT_NAME> with no /v1; the /v1/... form returns 404. Put the bare URL in the profile.
• The proxy ignores the request’s model field: one endpoint serves one model. So one endpoint and one credential profile per model, vllm.<key>.
• The bearer token is a Vast API key. Two scopes are involved, and they differ: the key a run uses (stored in the vllm.<key> profile) only has to be accepted by the proxy; misc plus user_read and instance_read is enough and also lets a monitor list workers. The key that manages endpoints, workergroups and templates through the API needs misc, user_write (Vast files template creation under it, alongside API-key and SSH-key management), user_read and instance_read. Keep the manager key out of the profiles; the sweep never needs it.
• The proxy returns 504 for any request that takes longer than about a minute. The client retries 5xx, so slow models cost retries rather than runs; keep concurrency low enough that calls finish under a minute (see “Run”).
• A worker reports Ready only after the template’s benchmark finishes, a few minutes after vLLM logs “Application startup complete”. Hosts fail and get replaced; budget 10 to 30 minutes from creation to Ready.
• Stopped workers keep their disks and bill storage. Delete the endpoint when a batch is done; stopping it is not enough.
• Gated Hugging Face repos need HF_TOKEN (a read token) in the account’s environment variables, and the licence accepted on each repo page.

Template

Clone the stock “vLLM (Serverless)” template, keep it private, and change two environment variables:

• MODEL_NAME = the upstream repo id, exactly as it will appear after vllm/.
• VLLM_ARGS = --download-dir /workspace/models --host 127.0.0.1 --port 18000. The stock value carries Qwen3-specific reasoning and tool-call parsers; drop them for any other model. Tensor parallelism comes from the GPU count (AUTO_PARALLEL=true), so no --tensor-parallel-size.

Leave the port mapping and the SERVERLESS, BACKEND, MODEL_SERVER_URL, MODEL_HEALTH_ENDPOINT and MODEL_LOG variables alone; they wire vLLM to the worker process. The image does not count against the worker’s disk allocation; size the disk to the weights plus about 10 GB.

GPU filter

Base line, then adjust the three memory and count fields per model:

direct_port_count>=2 compute_cap>=750 cuda_max_good>=12.9 gpu_total_ram>=24000 gpu_ram>=24000 inet_down>500 num_gpus in ['1']

For GPTQ models use compute_cap>=800 compute_cap<=900 (Ampere to Hopper): the fast 4-bit kernels are built for that range and Blackwell cards fall back to a slow path. Restricting num_gpus matters because auto-parallel would otherwise shard a small model across every card it is given.

Endpoint

Advanced setup, not the one-click flow (its default keeps five workers). Minimum Workers 0 always, Minimum Load 1, Max Workers 1 or 2, Inactivity Timeout 1800. Minimum Load keeps one worker active while requests flow; Minimum Workers 0 plus the timeout lets the endpoint scale to nothing when idle instead of parking a worker whose disk bills storage. Name the endpoint after the model so a wrong profile is obvious. Workers bill per hour, so a second worker only pays for itself if the client keeps it as busy as the first: scale the sweep concurrency with the worker count (four concurrent runs per worker, see “Run”). Two workers at eight runs finish in about half the time for about the same money; two workers at four runs cost nearly double. Vast adds workers on its own up to Max Workers when the endpoint is saturated, so Max Workers is the real cost knob.

Probe and profile

Run the probe from the main page with VLLM_BASE_URL set to https://openai.vast.ai/<ENDPOINT> (no /v1) and VLLM_API_KEY to the Vast API key, then store exactly those two values in vllm.<key>. A 504 from the probe means no worker is Ready yet.

Concurrency

Four concurrent runs per worker keeps calls under the proxy’s 60-second cutoff for 70B-class models, so --max-active is four times Max Workers. Beyond that, calls are retried rather than lost, but every retry throws away a full generation.

Tear-down

Delete the endpoint as soon as the last run completes. Stopping is not enough: a stopped worker keeps its disk and bills storage. Confirm the account’s instance list is empty afterwards.

Doing it from the API

The console steps are three calls with the same Bearer key, then a poll:

1. POST /api/v0/template with the template fields above (env is the Docker flag string, extra_filters the GPU filter as JSON, private: true). Reading your own templates: GET /api/v0/template/?select_filters={"creator_id":{"eq":<id>}}.
2. POST /api/v0/endptjobs with endpoint_name, min_load 1, target_util 0.9, cold_mult 3, cold_workers 0 (Minimum Workers), max_workers. The create call ignores inactivity_timeout; set it with a PUT on the endpoint afterwards, which also accepts cold_workers and max_workers.
3. POST /api/v0/workergroups with endpoint_id, template_hash, the GPU filter as a JSON dictionary in search_params (the console-style string is mis-parsed: memory floors get scaled by 1000 and in ['1'] keeps its quotes), and the same scaling values. Read it back and check search_query before a worker is recruited. A PUT on a workergroup must include endpoint_id; it did not change cold_workers, so delete and recreate if the query is wrong.
4. GET /api/v1/instances/ until a worker is running, then the curl probe until it answers.
5. DELETE /api/v0/endptjobs/{id} when done; it removes the workergroup and its workers. Re-check the instance list.

Endpoint and workergroup objects returned by the API carry their own api_key fields; never print or record them.

Repeat and compare runs

A condition groups runs with the same experiment name, declared source identity and input values. Each execution gets a separate run ID. This lets you repeat a configuration while retaining each outcome and its evidence.

How do I repeat a configuration?

Keep the complete command generated by the form, including every --set. Run it again against the same source to create another run under that condition. In Jobs, rerun submits one new run using that job’s inputs and profile selections. It uses the local execution source available when the new job runs; it is not a replay from the original run’s saved source reference.

Each invocation executes one run with a new run ID. Invoke the command again for another run.

How do seeds work?

The recorded seed is the --seed argument, unchanged. It must be an integer from 0 through 2147483647. When omitted, the runner draws a random non-negative 31-bit seed. The same value is passed to the experiment as REPS_SEED and recorded in run.start and run.json’s inputs.seed.

This complete keyless command makes one run with seed 42:

nix run .#inspect-hello -- \
  --set model=mockllm/model --set limit=0 --set epochs=1 \
  --set 'generate_args={}' --seed 42

Repeating that command produces another run with seed 42 and a new run ID. Whether the resulting behavior repeats depends on the experiment, provider and external inputs.

To reuse a saved run’s seed, pass the number shown on its run page or stored as inputs.seed to --seed. The local job rerun action draws a new random seed.

How do I audit the first real run?

After the first live run of any experiment, verify its directory before launching more or publishing data. From the REPS checkout, run:

reps-runner verify /path/to/run
# With the packaged runner:
nix run .#reps-runner -- verify /path/to/run

You can also pass the printed run ID: nix run .#reps-runner -- verify RUN_ID --data-dir DIR. Run-ID lookup selects --data-dir, then REPS_DATA_DIR, then $XDG_DATA_HOME/reps (default ~/.local/share/reps). An explicit run-directory path is used directly.

The command scans every file, including the workspace, for known credential values and common secret patterns, validates every envelope through the experiment’s declared payload union, and compares every card section with its stream projection. It requires a terminal run.end, checks identity and sequence continuity, and exits non-zero on any finding. It reads without repairing files; diagnostics never print the matching credential or invalid payload. Failed or interrupted terminal runs can also be audited.

Known credentials come from the current environment and every local profile matching the run’s recorded provider endpoint origins. Keep the credentials used for the run available when verifying it; retired credentials cannot be matched by value once removed. Public shell settings such as KEYTIMEOUT are not credentials. Review the actual usage, reasoning content when the endpoint supplies it, and served model name in the first live stream as well; verification checks evidence integrity, not the scientific result.

The command uses REPS_MANIFEST when invoked through an experiment app (nix run .#govsim -- verify /path/to/run), or --manifest FILE / --catalog DIR for an existing build. Otherwise it builds the current checkout’s manifest catalog. The manifest must match the recorded experiment and schema version; its built interpreter imports the declared union. A missing interpreter or union fails verification rather than falling back to untyped custom records.

Verification also checks every llm.call’s served model against its requested name and each request seed when present against run.start.seed. It warns once per distinct model mismatch and exits non-zero on either kind of mismatch. Comparison uses the snapshot stem rule. The shared client checks its first successful response and stops a misrouted run after recording the response and an error log. The audit reports how many calls stopped with max_tokens or content_filter, counting calls rather than completion choices; those counts are informational, so inspect truncated or filtered replies before launching more.

Azure

Create an Azure OpenAI deployment named exactly like the requested model, for example gpt-5-nano, and use azure/gpt-5-nano as the REPS model ID. Identity names the model you asked for; the deployment is routing. A deployment label in identity would make conditions depend on an arbitrary label. A per-model deployment override is deferred until a deployment cannot be renamed.

Configure the profile locally, with the key entered at the hidden prompt:

nix run .#reps-runner -- credentials set azure.default

Supply AZURE_OPENAI_API_KEY and AZURE_OPENAI_BASE_URL: https://RESOURCE.openai.azure.com/openai/v1/. Neither has a default. The runner injects these from the selected profile; exporting them in a shell does not configure the launcher. Azure’s v1 API uses the ordinary OpenAI client, with the deployment name in model.

Start with one run of one round, then verify the printed run directory:

nix run .#govsim -- --non-interactive --credential azure=default \
  --set model=azure/gpt-5-nano --set max_rounds=1 --seed 42 \
  --set max_tokens=3000 --set reasoning_effort=low \
  --set temperature=null --set top_p=null --set embedder=hash
nix run .#reps-runner -- verify /path/to/run

The hash embedder makes this an endpoint audit; choose mxbai for the paper’s embeddings. GovSim’s default completion cap stays 8000 for upstream fidelity. For gpt-5-nano at low reasoning effort, start with 3000 and review the audit’s max_tokens stop count. Reasoning shares the completion budget.

Azure rate limiting estimates prompt tokens plus the requested maximum output, even when the reply is shorter. Budget approximately calls/minute × (prompt tokens + max_tokens) × concurrent runs against the deployment’s TPM. For example, ten calls/minute, 1000 prompt tokens and a 3000 cap need about 40,000 TPM per run; three runs need 120,000. Leave headroom for estimation and retries, and check RPM too. See Azure quota management.

Launch one runner process per run and stagger their starts. Audit the first run before increasing concurrency. Staggering reduces bursts but does not raise capacity.

Read a 429’s message: rate-limit or quota wording points to the deployment’s allocated TPM/RPM; temporary inability or high-demand wording points to shared capacity. A lower x-ratelimit-limit-tokens than configured TPM indicates a temporary shared-pool reduction. Honor the retry delay; reduce traffic or adjust quota for the former, back off for the latter. Low billed usage alone does not distinguish them. See Azure’s 429 troubleshooting table.

The shared client permits eight SDK retries, with unchanged SDK timeouts. SDK attempts are otherwise invisible: one llm.call contains the final outcome. The typed llm.call.retries field counts HTTP 429 and 5xx responses during that call, including the last rejected response when retries are exhausted. This is the only record of rate limiting; it also includes server errors and excludes connection failures, timeouts and other retryable statuses. The shared client records zero when none occur. Other producers may leave it None (omitted from JSONL) when unknown. For historical records, the Python property reads the old retry note, or infers zero from the old backend marker. Intermediate response bodies and headers are not retained. The card’s derived.served_models lists the endpoint-returned names across the run; the runs list shows them beside the requested model when they differ.

Sweeps

Pilot each configuration for one round and verify its saved run before starting a sweep. Review the served model, retries, stop counts and any provider rejection. Keep generation settings explicit; changing settings creates a different condition. Launch sweeps from a clean tree, including untracked files; dry-run inspection remains available while editing.

Use one runner process per run and stagger starts to spread provider traffic and local resource use. Invoke the same condition again for another run.

Evidence stays in each run directory: the event stream and its index card. Keep operational terminal logs separate from the evidence.

How do I keep the source version?

A run records source, the experiment’s declared content identity, and fetch_ref, a repository reference intended for fetching the source. The reference is omitted when no pinned clean revision is known. The launcher also records tree_hash, the packaging tree’s NAR hash, when available for either clean or dirty trees; a hash alone cannot recover unsaved working-tree content. Preserve your working tree separately when running local modifications.

For reproduction, select the recorded repository revision rather than moving main or a registry alias that can change. Working with Nix gives the source forms. Compare the resolved condition with --dry-run before executing.

For a clean, public revision built with its default Nix inputs, that revision also recovers the committed dependency locks and shared library code. The runner saves executable store paths and platform information in runtime so you can compare the selected builds; see run metadata. Keep the commit available in the public repository.

What does a matching condition tell me?

The experiment name, declared source content and parameter values match. It does not establish equal model behavior or complete equivalence of the execution environment.

The condition hash excludes seeds, credential profiles, endpoints, the shared runner, platform and fetch reference. An experiment author chooses the source paths included in identity. A model ID is included as an input string, but a provider can change what a moving model alias resolves to.

Self-hosted IDs name the serving stack: vllm/Qwen/Qwen2.5-7B-Instruct uses an upstream repo id, while ollama/qwen2.5:7b-instruct-fp16 uses an Ollama library tag. Runs across engines do not pool into one condition because engines differ in hidden sampling defaults, chat templates and quantization formats. The convention for vLLM is to serve under the upstream repo id without renaming it or quantizing at load time; choose a repo or tag that names the intended quantization. The venue (laptop, cluster or Vast) belongs in a credential profile, so moving the same model and engine between venues does not change the model ID. On vLLM a trailing assistant message is continued as a prefill rather than answered in a new turn.

The served name must equal the requested name ignoring case, except that an unqualified or -latest request also accepts the same name plus one date suffix (-YYYY-MM-DD, -YYYYMMDD, @YYYYMMDD, -YYYY-MM; -YYMM for mistral/ only).

Source identity includes the experiment’s declared files plus the source directories of the shared Python packages it uses. mkExperiment includes reps_events and reps_experiment by default; Inspect programs also include reps_inspect. These package directories pass through cleanImport, excluding development artifacts; their surrounding tests, READMEs and project tooling are outside the source list. Changes to shared execution code therefore change every dependent experiment’s source hash. A GovSim adapter edit changes GovSim’s hash without changing Concordia’s. The runner and documentation remain outside this hash. The lock is hashed without [package.metadata.requires-dev] tables, excluding repeated path-dependency test metadata while retaining runtime requirements. The filter is covered by scripts/check-experiment-identity.py and the synthetic-lock tests run by nix flake check (checks.<system>.identity-lock).

When comparing outcomes, inspect the declared parameters, source reference, instance completion counts, errors and available environment and model metadata. Decide whether those differences matter for your question.

Publishing

REPS publishes verified terminal runs to an S3 bucket at a target supplied by you. There are no built-in hosts, provider presets, saved remotes, or target environment variables. Under the selected prefix, the bucket contains only runs/ and, when the prefix declares a license, its license.json.

Set up an AWS profile

Create the bucket and configure a named AWS profile using your storage provider’s instructions. For Hugging Face Storage Buckets, follow its S3 client and AWS profile setup. For Amazon S3, follow the AWS credentials guide. Keep endpoint, region, addressing and checksum settings in your AWS configuration, and credentials in the provider’s supported credential source.

--profile research selects that AWS profile. Omit it to use boto3’s normal credential resolution. REPS creates a boto3 session and its S3 client without overriding endpoint, region or client configuration. Ambient AWS settings are not passed into the experiment process; credential sets explicitly selected for the experiment still are.

Publish when a run finishes

Add --publish and optionally an AWS --profile to the experiment command. This complete mock-model example executes one run and publishes it:

nix run .#inspect-hello -- --set model=mockllm/model --set limit=0 --set epochs=1 --set 'generate_args={}' --publish s3://my-bucket/reps-v1 --profile research

The run’s data directory still follows --data-dir, REPS_DATA_DIR, then the XDG default.

--profile NAME selects the AWS profile for publication. Model credentials use --credential SET=NAME; both flags can appear in the same command.

--license SPDX declares the license. The license is resolved before the run starts; if it cannot be resolved, the command exits 2 without running.

Publication requires a terminal state, a pinned clean provenance.fetch_ref, and a passing verify audit. An unpinned run is refused, including with --dry-run; a later release label cannot supply its missing revision. Failed and interrupted runs can publish when their evidence passes that audit. A publishing error is logged without changing the run state or exit code.

Publish saved runs

Run the batch command against a local data directory:

nix run .#reps-runner -- publish --to s3://my-bucket/reps-v1 --profile research --data-dir "$HOME/reps-data"

With no stems, it considers all runs. Add --experiment NAME to filter by the experiment recorded on each card, or supply one or more positional stems: <condition>-<experiment> for a whole condition, or <condition>-<experiment>/<run> for one run. Stems are relative to runs/. Each run is verified independently; a rejected run does not stop the batch. The command returns a nonzero exit code if any run fails to publish.

--dry-run verifies and prints the exact object keys and byte sizes, including compressed streams. It reads the license declaration and checks the bucket for existing run keys but writes nothing. Verification resolves the manifest in the same way as reps-runner verify: from REPS_MANIFEST, then the experiment’s file in REPS_MANIFESTS, or a manifest catalog built from the current checkout.

Publication checks both destination run keys with HEAD. If either already exists, it refuses to overwrite that run, including a partial upload. Other runs continue. Choose a new prefix when you need different immutable objects. The publisher uses only HeadObject, GetObject and PutObject: it reads the license declaration, then uploads each stream first and its card last.

Each run uploads exactly these two objects:

<prefix>/runs/<condition>-<experiment>/<run>/events.jsonl.zst
<prefix>/runs/<condition>-<experiment>/<run>/run.json

The stream is compressed at zstd level 19 in a single frame with Content-Type: application/zstd, preserving its original bytes when decompressed. The card is copied byte-for-byte with Content-Type: application/json. Workspaces are never uploaded. See the published layout.

Declare the license

A prefix declares at most one license for all runs under it, in <prefix>/license.json:

{"v":0,"spdx":"CC-BY-4.0"}

spdx is an SPDX license identifier. The experiment command (with --publish) and reps-runner publish both accept --license SPDX. Without it, an existing declaration applies. If the prefix declares none, a terminal shows this menu:

1) CC-BY-4.0 (default)
2) CC0-1.0
3) CC-BY-SA-4.0
4) CC-BY-NC-SA-4.0
5) CC-BY-NC-ND-4.0
6) No license
7) Other: an identifier from https://spdx.org/licenses/<id>.html

Enter a number or type an identifier; an empty answer selects CC-BY-4.0. Without a terminal, or with the experiment command’s --non-interactive, the command fails when the prefix declares none and --license is absent. Each publish logs the license in effect as LICENSE s3://BUCKET/PREFIX SPDX.

--license NONE, or option 6, publishes without a license. It writes no license.json and logs LICENSE s3://BUCKET/PREFIX undeclared, and the index records null for the store. If the prefix already declares a license, NONE warns (publish: WARNING: … declares SPDX; publishing under it) and the runs publish under that declaration. NOASSERTION is refused; use NONE. A hand-written license.json that names NONE or NOASSERTION declares nothing: the index records null with a warning, and the publisher refuses the prefix.

The declaration is written once, before the first run uploads, and never replaced. Repeating the declared identifier is accepted. A different identifier other than NONE fails before any run uploads: the batch publishes nothing, and the experiment command does not start the run. To change the license, choose a new prefix.

To declare the license of an existing prefix, publish from an empty data directory:

nix run .#reps-runner -- publish --to s3://my-bucket/reps-v1 --profile research --license CC-BY-4.0 --data-dir "$(mktemp -d)"

The index copies each store’s declaration into index.json.

Build and serve an index

Experiment buckets contain only runs/ and license.json. A separate, replaceable index lets a static site discover those runs. Write stores.yaml yourself; REPS does not infer a public URL from an S3 address. Store lists accept YAML (.yaml or .yml) or JSON. data.example.org below is an example store hostname.

v: 0
stores:
  - s3: s3://my-experiment-bucket/reps-v1
    profile: research
    url: https://data.example.org/experiments/reps-v1
    experiments:
      - govsim
    exclude:
      - runs:
          - <run-id>
        reason: Provider outage invalidated these results.

url must expose the same prefix over public HTTPS. Optional experiments, conditions and runs lists filter exact recorded values and intersect. Leave a filter out to include everything; an empty list includes nothing.

An optional exclude list removes runs even when the inclusion filters select them. Each entry requires a non-empty list of run IDs and a non-empty reason, because the store list is the site operator’s record of what it chose not to show. A run ID can appear in only one exclusion entry per store. Exclusions are logged with their reasons on every rebuild, including --dry-run; unknown IDs warn without failing the build. Exclusions change only the site’s index and never touch bucket objects.

Store profiles select source credentials. The index command writes to a local directory that the site deploys. The index reads only ListObjectsV2 and GetObject, using each store’s profile without endpoint, region or client-configuration overrides. Every rebuild reads each store’s license.json and logs LICENSE s3://BUCKET/PREFIX SPDX, or undeclared with a warning when the declaration is absent or malformed.

Run cards are cached under $XDG_CACHE_HOME/reps/index (or ~/.cache/reps/index), keyed by profile (default when omitted), bucket, full object key and the listed ETag. Every rebuild still lists the store, so the listing determines membership and changed ETags trigger a fetch. The cache does not affect the index output; built_at still records each rebuild’s time. Use --cache-dir DIR to choose another cache directory or --no-cache to bypass cache reads and writes. --dry-run can read existing entries but never writes them. Store summaries report cached and fetched cards before filters and exclusions.

Build the index with this repository’s public outputs:

catalog="$(nix build .#manifests --no-link --print-out-paths)"
nix run .#reps-runner -- index --stores stores.yaml --catalog "$catalog" --to site/index

Deploy the generated site/index directory alongside your site. The index command reads the stores with AWS profiles; clients read public run objects without those credentials.

--catalog DIR is required and points to the built manifest directory: <name>.json files and optional assets/<name>/ trees, as with verify --catalog. The site’s catalog is derived from the index, so an experiment with no published runs never appears.

--dry-run prints each store’s license and filtered run count and the files and sizes it would write, without creating or deleting anything. Every invocation rebuilds the whole index from complete run-object pairs. It deletes the output directory and rewrites it in full, writing index.json last. Shards for experiments no longer present are removed, together with their catalog entries and assets. Unreadable, malformed or disappeared cards produce warnings and are skipped, so healthy runs can still be indexed. Failure to list a store aborts the rebuild before replacing the destination.

The root index.json lists experiment names, their run counts, the total count, stores and built_at. stores has one {"url":"<public HTTPS base>","license":"<SPDX identifier>"} entry per store, in store-list order; license is null for an undeclared or malformed declaration. Each experiments/<experiment>/index.jsonl contains one row {"store":"<public HTTPS base>","card":<run.json as a JSON value>} per run. The card is compactly re-serialized with Unicode retained and key order preserved. This derived cache is never authoritative. Read the original card and compressed stream from the row’s store for the recorded bytes and decompressed lines.

The index includes catalog.json with current manifests, shared render hints and versioned experiment hints for indexed experiments only. README assets are copied to catalog/assets/<name>/, dereferencing the manifest directory’s symlinks. A missing manifest warns and leaves the experiment in the index through its runs, without a manifest or experiment assets.

Working with Nix

If you do not have Nix yet, follow the official installation instructions for your operating system. The default REPS commands do not require flakes to be enabled.

REPS supports classic Nix and flakes. Both entry points build the same package set and use the same flake.lock pins for nixpkgs and the Python build toolchain. Flake commands receive the inputs directly; classic commands fetch the locked sources through pkgs/locked-sources.nix. REPS declares Linux and macOS packages for x86-64 and ARM64; individual experiments can have additional platform or service requirements.

How do I override a dependency with classic Nix?

default.nix accepts nixpkgs, uv2nix, pyproject-nix, and pyproject-build-systems as source-path arguments. Omitted sources use their flake.lock pins:

import ./reps {
  uv2nix = /path/to/my/uv2nix;
}

The same arguments work on the command line:

nix-build -A reps-runner --arg uv2nix /path/to/my/uv2nix

Overriding nixpkgs selects the package set used by REPS unless you also pass pkgs. Pass pkgs directly when you already have an imported package set:

nix-build -A reps-runner --arg pkgs 'import <nixpkgs> {}'

How do I make commands match my setup?

Open command settings using the toolbar gear. Choose:

• From GitHub to fetch REPS without cloning, or local checkout (.) to use your current checkout.
• nix-build for stock Nix, flakes for nix run, or nix-run for the separate classic runner utility.
• The relevant installed/enabled options for your selected mode.

Commands throughout the book adapt to those choices. Authoring examples always target the local checkout. Preferences persist in this browser.

For GitHub sources, always fetch latest makes commands recheck the moving source using --tarball-ttl 0 or --refresh. This is useful during development; a pinned revision is more appropriate when repeating an earlier source version.

What does each mode execute?

All three modes below invoke the generated experiment launcher, which supplies the manifest and execution context to the runner. They do not invoke the raw experiment program. Use these named entry points rather than calling reps-runner directly to execute experiments.

In nix-build mode, REPS builds an exec.NAME output that points to that launcher, then invokes it. It needs neither flakes nor a globally installed REPS command.

$(nix-build --no-out-link -A exec.inspect-hello) --describe

In flakes mode, experiment names are app names. If flakes are not enabled globally, command settings add --extra-experimental-features 'nix-command flakes' to each command.

For example: nix run .#inspect-hello -- --describe.

In nix-run mode, the separate nix-run utility resolves the experiment package’s generated launcher. Experiment package attributes use the experiment- prefix; tools use their reps- names. If the utility is not installed globally, command settings wrap it in nix-shell -p nix-run --run ....

With the utility installed: nix-run . -A experiment-inspect-hello -- --describe.

Otherwise:

nix-shell -p nix-run --run 'nix-run . -A experiment-inspect-hello -- --describe'

How do I use the flake registry alias?

Register reps once if you prefer it to the full repository reference:

nix registry add reps github:antimemes/REPS \
  --extra-experimental-features 'nix-command flakes'

Select reps registry added in command settings. The alias is a convenience for the GitHub flake source; classic Nix uses the tarball URL directly.

How do I pin a source version?

For flakes, a GitHub reference accepts a commit after the repository name. Replace REV below with the recorded commit:

nix run github:antimemes/REPS/REV#inspect-hello \
  --extra-experimental-features 'nix-command flakes' -- --describe

For classic Nix, use the archive for that commit:

$(nix-build --no-out-link \
  https://github.com/antimemes/REPS/archive/REV.tar.gz \
  -A exec.inspect-hello) --describe

Git-generated archives carry a revision stamp used for the run’s fetch reference. An ordinary classic-Nix working-tree build has no expanded archive stamp and omits fetch_ref unless an repsRev is explicitly supplied to the package import. This is separate from the experiment content hash used for conditions.

How do I build or develop locally?

From the repository root, build all manifests without running experiments:

nix-build --no-out-link -A manifests

Enter the development shell with nix-shell, or nix develop if you use flakes. Both supply the project’s development tools. Package-specific Python dependencies come from their pyproject.toml and uv.lock through uv run.

A Git flake includes tracked files. Stage newly added source files before evaluating your new experiment through a local flake. Keep run data outside the checkout, and use the authoring guide for the contribution workflow.

Add or change an experiment

Experiments live under experiments/ in this repository. Each directory has a package.nix that returns one or more named experiments. The package registry imports those directories automatically; you do not need to add an entry to the root flake.

To add an experiment, follow the small keyless example below and check the complete integration before introducing external dependencies. To extend one already in the repository, go to change an existing experiment.

How do I prepare the checkout?

Clone the repository and enter its development shell:

git clone https://github.com/antimemes/REPS.git
cd reps
nix-shell

Flake users can enter the same shell with nix develop. Create experiments/example-count/ for the new experiment. Use a unique experiment name; the registry rejects duplicates across directories.

How do I declare the experiment?

Save this as experiments/example-count/package.nix:

{ reps, writeShellApplication, jq, reps-runner }:
let
  program = writeShellApplication {
    name = "example-count-program";
    runtimeInputs = [ jq reps-runner ]; # the runner's environment includes reps-events' CLI
    text = ''
      count="$(jq -er '.count')"
      reps-emit result --name count --value "$count"
    '';
  };
in
{
  example-count = reps.mkExperiment {
    name = "example-count";
    summary = "Record an explicitly supplied count.";
    src = ./.;
    inherit program;
    params = with reps.types; {
      count = param int {
        description = "Integer to record in the run results.";
        initial = 3;
      };
    };
    results = [
      {
        name = "count";
        type = reps.types.int;
        label = "Recorded count";
        description = "The supplied count echoed by the program.";
        details = "This checks the integration, not model performance.";
      }
    ];
  };
}

params supplies validation and command suggestions. initial supplies a suggested value; users must still bind the parameter explicitly when running from the CLI. results declares possible summary metrics and explains their meaning. See manifest reference for the result fields. Declaration list order sets the definitions order.

Give each result a readable label and a short, plain-language description explaining what the value means. Use optional details for the calculation, aggregation and interpretation caveats. Add unit when useful. Explain what a boolean means in the experiment, including whether it represents success or failure. Distinguish observed progress from configured limits, and explain special cases such as an equality score of 1 when every agent gained zero.

These definitions describe possible outputs, not required outputs: an absent metric is missing, not zero. Undeclared results remain in the stream, warn, and are excluded from the summary; repeated declared results warn and use the last value. The runner saves the definitions in run.json under definitions.results and in run.start as result_definitions, so readers retain the explanation used when the run began.

src declares the experiment’s identity sources. Include code, configuration and dependency locks that determine this experiment’s behavior. A path is usual; use a list when the experiment depends on several source trees. Changes anywhere in those declared inputs can change the condition identity, including a README inside a declared directory. Development artifacts such as .venv are filtered out. The identity reference gives the exact rule.

What must the program do?

reps.mkExperiment generates the named experiment launcher. Authors supply the underlying program; users invoke the named app. The generated launcher provides the manifest, source identity, and program path to the runner. Do not invoke reps-runner, build its execution environment by hand, or start the underlying program directly to create a run.

The runner starts the program in a fresh workspace, sends the complete parameter object as JSON on standard input, and provides the run directory and seed in environment variables. Use reps_events.emit() in Python or the reps-emit CLI in other languages. The shell example reads its input with jq and calls reps-emit to record a result matching its declared result name. The CLI belongs to reps-events; this shell example obtains it through the runner’s environment, and never invokes the runner itself. Ordinary stdout/stderr is captured as text, including printed JSON.

For a larger program, put the implementation beside package.nix and have the adapter invoke its packaged executable. Pin dependencies in the package definition and dependency lock. Python experiments can use reps.mkPythonEnv to build a pyproject.toml/uv.lock workspace, and reps-events for validated event emission. reps-experiment provides a shared parameter-reading scaffold and artifact helper; see its failure behavior before adopting it. Each call to reps.mkPythonEnv must declare python explicitly, for example python = pkgs.python313;; the helper has no interpreter default.

Both emitter APIs handle validation and transport. Do not write a socket client in an experiment.

Add standard events for the evidence a reader needs: model calls and results. Use custom events for messages and instance outcomes. Preserve useful native output as artifacts, write files under REPS_RUN_DIR/artifacts/, and emit pointers under an experiment-specific custom kind. Read REPS_SEED and pass it to supported random generators or backend settings. The process protocol defines the boundary in full.

Instrumenting an experiment

Enumerate everything upstream writes before writing the translator: logs, per-agent files, checkpoints and auxiliary data. GovSim’s ingestion module captures the environment log and each persona’s memory nodes, including checkpoints left by a failed simulation.

Keep foreign rows as open JSON dictionaries; never narrow them to the fields a current analysis uses. GovSim’s native row models retain unknown keys, nested values and explicit nulls while identifying known actions with typed custom kinds.

Mark each ingested source before its records with its relative path, byte size, SHA256 of the original bytes and emitted record count. GovSim’s file markers let readers check which checkpoint contributed each sequence of rows.

Leave out only what is recomputable, and state the inputs needed to reproduce it. GovSim’s adapter omits embeddings because its recorded node descriptions and the embedder named in govsim.config reproduce them.

Use upstream’s own actor IDs and declare a roster for display labels. GovSim’s config render hint maps persona_N to the configured name, so even a persona that never speaks has a label on its model calls and memory rows.

Attribute model calls to the component that made them. GovSim’s logger hook uses persona IDs for persona calls and framework/<component> for framework queries, preserving phase and query context as producer metadata.

Declare results and their meanings in the manifest, then calculate and emit them in the adapter. GovSim’s result declarations set their display order and definitions; its adapter computes the scientific quantities. Readers can use the emitted results without deriving experiment-specific metrics from native rows.

Run the reusable secrets scan from reps_events.secrets (also re-exported by reps_testing) on the entire run directory with fake credentials seeded into the environment. GovSim’s test_resolved_config_copies_no_secrets in test_adapter.py checks captured events and every workspace file, so a resolved config or raw request cannot silently copy those credentials into publishable data.

Upstreams without Python packaging

For an upstream with no packaging metadata, add it with a patch and install the package into the experiment’s venv. GovSim’s packaging.patch adds a minimal pyproject.toml that includes only simulation and its YAML configuration data. Its package declaration defines the pinned upstream in the overrides overlay and selects it with extraPackages. Keep this packaging inline until another experiment needs the same mechanism. Address installed data through importlib.resources or the framework’s package provider, never through a checkout path or an environment variable. Keep patches split: seed.patch edits existing upstream files and behavior; packaging.patch adds missing packaging files without changing behavior.

package.nix declares the experiment and never declares tests. Put checks in tests/default.nix, outside the experiment’s identity sources. Like nixpkgs’ testers.*, the shared reps.testers builders take test data and return ordinary derivations; the registry attaches them as passthru.tests and discovers them for nix flake check and task test:python.

{ experiment, reps }: {
  pytest = reps.testers.pytest { inherit experiment; tests = ./.; };
  smoke = reps.testers.smoke {
    inherit experiment;
    params = {
      experiment = "fish_baseline_concurrent";
      model = "mock/model";
      embedder = "hash";
      max_rounds = 1;
      max_tokens = 8000;
      threads = 2;
      reasoning_effort = null;
      temperature = 0.0;
      top_p = 1.0;
    };
  };
}

pytest selects the Python program’s dev group in its canonical Nix venv. Pass the test directory explicitly as tests = ./.; flags adds pytest arguments. Project test suites use pytest-xdist with -n auto; the Nix tester selects eight workers to bound concurrent numeric imports. Use uv run pytest -n 0 when debugging a hang, or uv run pytest -o addopts='' -p no:xdist to disable the plugin entirely. In a Nix check, flags = [ "-n" "0" ]; selects serial execution. Environment knobs such as HF_HUB_OFFLINE belong to the experiment: set env and preCheck in its tests/default.nix, like nixpkgs check hooks. overrideAttrs is the escape hatch. For example, Concordia’s program is a shell adapter, so its test declaration supplies an explicit Python test venv through nativeBuildInputs. Nixpkgs testers.* are also fine for checks the two REPS testers do not cover.

smoke takes the complete condition in params, runs the built launcher, and verifies its completed store. The oneliner IS the condition spec for a smoke run too: no defaults are hidden in the tester. Adding a parameter means adding it to the smoke check, just as to every oneliner and sweep script. Manifest initials are independent of the smoke condition. The tester does not seed credentials; testing whether an adapter copies environment secrets is an adapter unit-test concern. Use overrideAttrs check hooks for offline fixtures, as in inspect-hello’s tokenizer cache. For GovSim’s focused loop, run nix build .#checks.x86_64-linux.govsim -L (using your machine’s system).

How do I check the integration?

Keep the three test layers separate:

• experiments/*/tests uses reps-testing: event_capture and assert_run_has_no_secrets check that the adapter emits the right events, in order, without secrets. Experiment tests never import reps_runner.
• runner/tests uses small fixture programs to check that the runner executes anything that speaks the producer protocol.
• Each experiment’s passthru.tests.smoke runs its built launcher on a mock configuration, then reps-runner verify on the completed store. This is the check that the runner executes that registered experiment offline, writes conforming evidence and a matching card, and leaks nothing.

Keep smoke parameters in the experiment’s check, not in the shared declaration API. The smoke tester runs with --non-interactive, asserts that exactly one run completed, and passes that experiment’s manifest directly to reps-runner verify. Verification checks that reported result names equal the manifest’s declarations and reports missing or extra names. The separate completion check matters because verify can also audit failed or interrupted runs. See GovSim’s checks and Concordia’s test wiring. Keep seeded-credential scans in adapter tests; verify real runs against the real credentials before publishing. nix flake check -L discovers these checks through the registry; task ci uses that same entry point. No test-specific runner dependency belongs in an experiment’s development group.

For local flakes, stage the new files so Nix includes them:

git add experiments/example-count

From the repository root, inspect the schema and validate inputs:

nix run .#example-count -- --describe
nix run .#example-count -- --set count=3 --dry-run

Then execute the keyless example through its generated named app:

nix run .#example-count -- --set count=3

Inspect run.json and events.jsonl in the printed run directory and verify that the input and result are both 3. The terminal reading example shows how.

For a real experiment, test parameter rejection, event shapes, summary selection and failure reporting with mocks or small local fixtures. Run the affected package’s tests; after changing dependency declarations, update its lock and check dependent locks. task lock:check checks lock freshness, and task ci runs the repository’s broader checks. Build documentation separately with task docs:build when changing it.

How do I test Python emission without launching a full run?

Add reps-testing to your experiment’s development dependencies and regenerate its lock. From experiments/EXPERIMENT/pyproject.toml, the local source is:

[dependency-groups]
dev = ["pytest>=8", "pytest-xdist", "reps-testing"]

[tool.uv.sources]
reps-testing = { path = "../../lib/reps-testing", editable = true }

Merge these entries into existing sections. Pytest discovers the installed reps-testing plugin automatically; no conftest.py registration is needed. Tests that request event_capture receive the contract’s reference socket receiver and its connection environment; the fixture cleans up afterward. It is not autouse: parameter-validation tests that do not emit events need not request it. Do not implement a test socket server or patch emission to print JSON.

Call your adapter normally, then inspect event_capture.read(). For example, this shows the fixture interface; in an adapter test, replace the direct emit call with the adapter function being tested:

from reps_events import Result, emit


def test_recorded_score(event_capture):
    emit(Result(name="score", value=1))
    events = event_capture.read()
    assert any(e["type"] == "result" and e["name"] == "score" and e["value"] == 1
               for e in events)

read() returns and clears the captured payload dictionaries. Assertions on ordinary prints still use pytest’s capsys separately. Subprocess tests inherit the fixture’s environment; preserve it when constructing a child environment. This fixture checks emission and delivery, not run orchestration or stored summaries. Check the complete integration through the named app as above.

How do I change an existing experiment?

Find its declaration in experiments/ and the adapter that consumes its inputs. Read the experiment’s README and tests before editing.

For example, suppose an adapter currently calls its model with a hardcoded temperature=0.5. To let readers choose that value while retaining the previous behavior:

1. Capture the existing behavior with a small fixture that records the model request and emitted results. Keep its inputs and source revision so you can compare after editing.

2. Add a parameter to that experiment’s params declaration:

   temperature = param float {
     description = "Sampling temperature passed to the model.";
     initial = 0.5;
   };
   

3. Add the field to the adapter’s parameter schema, including any backend constraints, and replace the hardcoded argument with the supplied value. For a Python adapter with a validated params object, the call becomes temperature=params.temperature.

4. Update the README commands and fixtures to supply --set temperature=0.5 alongside every existing parameter. initial supplies a suggested value; it does not make this new CLI argument optional. Existing commands must be updated.

5. Run the before/after fixture with 0.5 and check that the model request and relevant results match. Then use a different value, such as 0.2, and assert that it reaches the backend. A mock that ignores temperature can check wiring but cannot establish how a real model responds. Check invalid inputs and inspect a small run’s parameters, events and summary.

Preserving behavior does not preserve fingerprints: changes to declared source content or the parameter set can change the condition ID. An explicit --seed stays unchanged across these edits. Use a fixed seed when testing behavior; see seeds.

If the change makes the experiment incompatible with its earlier meaning or behavior, give it a separate experiment identity and document the distinction. There is no agreed folder-suffix convention or general experiment-versioning system.

What belongs in the pull request?

Add a README.md in the experiment directory. Only Markdown is supported. Explain the research question, upstream source, required services, input meanings, result interpretation and a small runnable example. Keep experiment-specific usage there rather than adding a catalog page to this book.

Describe what the new experiment or change does, the evidence it records, and the checks you ran. Include updated locks and fixtures needed to reproduce those checks. Submit the code and documentation through an ordinary repository pull request.

Commands and settings

Run an experiment through its named app, such as nix run .#inspect-hello -- .... This is a generated launcher that supplies the manifest, source identity, and program path before starting the runner internally. Do not invoke reps-runner or the underlying experiment program directly to execute an experiment. REPS’s management tools use reps- names. Commands below follow the book’s Nix settings.

Experiment options

nix run .#inspect-hello -- --help

Pass these options to the named experiment app:

Each invocation executes one run. The runner returns exit code 2 for input, credential-resolution or license-resolution errors handled before execution. After execution it returns 0 if the run completed, 1 if it failed or timed out, or 130 if interrupted. Inspect run.end or saved run.json for the process outcome, and summaries and events for evaluation results.

How are parameter values read?

The value after = is parsed as JSON if possible, otherwise as a string. @path reads the file and applies the same JSON-or-text rule. Text-file contents are kept as the string value. A malformed value or file beginning with [ or { is rejected rather than treated as a string.

These are syntax examples; the keys must exist in the selected experiment. Unknown keys, omitted parameters and type mismatches are errors. Shell quoting is separate from JSON syntax: quote arguments containing spaces, braces or other shell punctuation.

Publish saved runs

reps-runner publish --to s3://<bucket>/<prefix> [STEM ...] [--experiment X]
  [--profile NAME] [--license SPDX] [--data-dir DIR] [--dry-run]

Stems select <condition>-<experiment> or <condition>-<experiment>/<run>. Without stems, all runs are considered, optionally filtered by experiment. --dry-run verifies, prints exact keys and sizes, and writes nothing. Verification resolves manifests as reps-runner verify does. Each run uploads only its compressed stream and card under runs/. Before any run uploads, the command writes the prefix’s license.json once, from --license or a terminal prompt. An existing declaration must match --license. --license NONE writes nothing and warns when a declaration exists. This also happens when no runs are selected. See Publishing and Declare the license.

Build a derived index

reps-runner index --stores FILE --catalog DIR --to DIR [--dry-run]

The store list provides source profiles, public URLs and optional filters. --catalog supplies the manifest directory (<name>.json and assets/<name>/); only indexed experiments’ manifests, hints and assets are exported. Each invocation deletes and rewrites DIR, writing index.json last, with each store’s declared license under stores. The site deploys this directory. Dry-run prints store licenses, filtered store counts, files and sizes without creating or deleting anything. See Build and serve an index.

Credential commands

Credential management is a standalone use of reps-runner; these commands configure profiles and do not execute experiments:

nix run .#reps-runner -- credentials --help

Profile names start with a lowercase letter or digit and continue with lowercase letters, digits, _ or -. new is reserved. See credential setup for selection order and model-ID routing.

Data and configuration settings

Experiment launchers, reps-runner publish, and reps-runner verify select run storage in this order: --data-dir DIR, REPS_DATA_DIR, then $XDG_DATA_HOME/reps (or ~/.local/share/reps when XDG_DATA_HOME is unset).

To audit a saved run by ID, use nix run .#reps-runner -- verify RUN_ID --data-dir DIR. verify also accepts a run-directory path directly; see run auditing.

Provider keys and endpoints come from the credential file. The runner does not forward arbitrary environment variables. Process protocol lists the variables passed to an experiment; local execution and storage covers terminal operation.

Package attributes

manifests builds a directory containing one manifest JSON file per registered experiment. Working with Nix explains fetching, registry aliases and revision pinning.

The public package outputs are manifests, reps-runner and experiment-<name>, with exec.<name> executable outputs in the classic entry.

Local execution and run storage

Use the generated experiment launcher for terminal runs.

Choose where results are saved

Experiment launchers, publishing and verification use the same data-directory convention.

For example, set this in every terminal used for the session before launching an experiment or inspecting its runs:

export REPS_DATA_DIR="$HOME/reps-first-run"

Changing the directory selects a different collection of runs; it does not move existing data. To inspect a run directly, use the store path printed by the terminal experiment. Under the chosen data directory:

• runs/<condition_id>-<experiment>/<run_id>/run.json records run metadata and outcome.
• runs/<condition_id>-<experiment>/<run_id>/events.jsonl contains the recorded events, one stream per run.
• runs/<condition_id>-<experiment>/<run_id>/workspace/ is the experiment’s working directory.

Condition identity, experiment, source and params live on each card and run.start. Readers derive conditions by grouping cards on condition.

Run the hello test in a terminal

Run this command as written for the default data directory, or set REPS_DATA_DIR first or append --data-dir DIR to choose another directory.

nix run .#inspect-hello -- \
  --set model=mockllm/model \
  --set limit=0 --set epochs=1 --set 'generate_args={}'

To run your checkout’s version of inspect-hello, select local checkout in the command settings and run from that directory. The named experiment app handles terminal execution. Every declared parameter must be supplied. This is a credential-free check using fixed mock replies. It executes one run with both samples (limit=0), one pass (epochs=1), and no generation overrides. Expect two completed samples, zero errors, and score 1.0. It needs no credentials or network access after the Nix build.

The runner prints a run ID and a store path. Read run.json and events.jsonl in that directory to inspect the run’s metadata and events.

Use --set KEY=VALUE repeatedly to change parameters, or --json to stream event envelopes to the launcher’s standard output while retaining saved run data. Add --non-interactive to disable prompts for unattended execution. --seed N sets the run seed unchanged, in 0..2147483647 (random when omitted). Each invocation executes one run. --dry-run prints the resolved configuration without executing, and --describe prints the experiment’s parameter schema.

How do I read the files without a browser?

Start with the store path printed by the runner. To locate an older run, look under runs/CONDITION_ID-EXPERIMENT/RUN_ID/ in your data directory. Its run.json identifies the experiment, condition, source reference and state. The card also contains inputs.params and provenance.source, copied from run.start. Readers derive conditions by grouping cards on condition; shared fields come from any member.

With Python 3, this example prints the metadata and reads result and completion events. Replace /path/to/run with that run’s directory:

python3 - /path/to/run <<'PYTHON'
import json
import sys
from pathlib import Path

run_dir = Path(sys.argv[1])
print(json.dumps(json.loads((run_dir / "run.json").read_text()), indent=2))
for line in (run_dir / "events.jsonl").read_text().splitlines():
    envelope = json.loads(line)
    event = envelope["event"]
    if event.get("type") in {"result", "run.end"}:
        print(json.dumps(envelope))
PYTHON

Use this example on a finished run; a live file can end with a line still being written. For live processing, use the runner’s --json stream. Preserve the envelope’s run ID and sequence number when combining records. Other event types contain model calls, progress, logs and experiment-specific observations.

The file reference and event reference define the fields. Check completion, errors and the evidence behind summary metrics before using a result in an analysis; matching condition IDs alone do not establish scientific comparability.

Experiment manifest

A manifest is generated JSON describing an experiment’s inputs and presentation. The runner uses it for validation, credential discovery and result descriptions.

Read a packaged manifest with:

nix run .#inspect-hello -- --describe

Top-level fields

The source identity and executable are wrapper settings, not manifest fields. reps.mkExperiment takes name, summary, params, program and src, with optional results, schema, schemaPython, sharedSrcs, env and links. Schema defaults to version 0 and reps_events:Payload; typed experiments supply their own union pointer and schemaPython = "${env}/bin/python" from the built program environment. Packaging imports that pointer and calls export_schema, failing the build if it cannot import. schema.json and the shared export sit beside the manifest, along with a python link to the schema interpreter used by reps-runner verify; schema.path is generated, not author-supplied.

The catalog packages the experiment directory’s images under assets/<experiment-name>/, preserving subdirectories such as figures/. Relative image paths in readme refer to these assets, copied to index/catalog/assets/<name>/ in the published site. Images are separate catalog assets, not embedded in the manifest JSON.

Parameter declarations

Each entry of params contains type plus optional fields:

LLM-typed parameters receive shared model suggestions during manifest generation. This also applies to LLM fields inside lists of structures. Experiment-specific suggestions are prepended to the shared suggestions.

Result declarations

results is an ordered list of declarations. Each declaration contains a unique name, a type descriptor and optional presentation fields:

In Nix, use results = [ { name = "count"; type = reps.types.int; label = "Recorded count"; description = "Count supplied to the program."; } ];. The manifest preserves the list and its presentation fields in that order. Map declarations and bare-type shorthand are not supported.

Results are possible outputs, not required outputs. A declared metric that was never emitted is absent from the summary; absence does not mean zero. Undeclared results remain in the stream but warn and stay out of the summary. Boolean results do not inherently indicate success or failure; their meaning comes from the declaration.

The runner snapshots the ordered declaration list in run.json under definitions.results and in run.start as result_definitions. These saved definitions preserve the run’s interpretation when the installed experiment changes. Result events have no unit field; units come from the declaration. Events do not override result labels, descriptions or details.

Type descriptors

Use listOf (struct {...}) for a list of structured values. A structure field can be a bare type descriptor or a wrapper containing type, description and suggestions. param type attrs attaches parameter metadata in Nix.

For example:

with reps.types; {
  agents = param (listOf (struct {
    name = str;
    model = param llm { description = "Model for this agent."; };
  })) {
    minLen = 1;
    description = "One row per agent.";
  };
}

What does validation establish?

The runner rejects unknown and missing parameter keys, then checks value types before hashing the condition. It checks realized values again before executing, including list-length bounds. Specified and realized parameters are currently identical; no distribution or sweep syntax is implemented.

Validation does not verify model existence, credentials, endpoint access or semantic constraints that only the experiment understands. Result type descriptors do not validate emitted metrics against the result declaration; standard result payloads have their own scalar-value rule.

Run files and identity

The local store keeps run metadata, the event stream and working files as ordinary files.

Where are runs saved?

Terminal experiments, publishing, and verifier run-ID lookup use --data-dir DIR, then REPS_DATA_DIR, then $XDG_DATA_HOME/reps, with XDG_DATA_HOME defaulting to ~/.local/share.

DATA_DIR/
  runs/
    CONDITION_ID-EXPERIMENT/
      RUN_ID/
        run.json
        events.jsonl
        workspace/

Readers derive runs/<condition>-<experiment>/ from the recorded (condition, experiment) pair. Conditions are groupings of cards, not stored objects. Each card and run.start carries condition identity, experiment, source and params; readers group cards on condition and take shared fields from any member. The experiment suffix is never parsed to recover either field; discovery reads run.json. Experiment names may themselves contain hyphens.

Readers accept exactly events.jsonl. Records are addressed by (run, seq). The runner flushes each event line, so another process can inspect a live run. Files from failed and interrupted runs remain in the store.

Schema exports come from the built manifests, identified by experiment and schema version. Schema files are not written to run directories.

How is a condition ID calculated?

condition_id = hex(sha256(JCS({experiment, source, params})))[:40]

JCS is RFC 8785 JSON canonicalization. The interface usually displays the first 12 characters of the condition ID. The hash uses values, not shell quoting or JSON object-key order.

source has the form content:sha256:HASH. Packaging computes it from the experiment’s declared src paths plus shared package sources (reps_events and reps_experiment by default; Inspect programs add reps_inspect), imported into the Nix store after filtering development artifacts. The filtered names are .venv, __pycache__, node_modules, dist, .direnv, .pytest_cache, .mypy_cache, .ruff_cache, result and names beginning result-.

The identity excludes the runner, docs, and other undeclared paths. It also excludes the packaging tree_hash, fetch reference, platform, seed, credentials and endpoints. See repeat and compare runs for the consequences.

Condition IDs are the first 40 lowercase hexadecimal characters of the SHA-256 hash (160 bits, matching the Nix store’s hash width). The same ID is recorded in run.start.condition and run.json.identity.condition and used in run directory names. Hash truncation belongs only to canonical.condition_id; readers use the recorded value.

A run ID has the form yyyymmddthhmmssz-<12 lowercase hex random>, for example 20260916t120000z-012345abcdef. The launching machine supplies the UTC clock label and six random bytes. Validation accepts exactly [0-9]{8}t[0-9]{6}z-[0-9a-f]{12}; the whole string is the ID everywhere. The date is a launch-time label, not evidence of when work occurred or of clock accuracy. Use capture timestamps and sequence numbers to interpret records. Replicates of a condition share its condition ID and have separate run directories.

What is in run.json?

The stream is evidence. The runner’s index card is a cache of that stream plus operational lifecycle state, used for listings and summaries. The stream wins on disagreement; no experiment or replay uses the card as an input. Tests compare its copies with run.start and recompute its derived block with read_events.

The seed is the --seed argument unchanged, or a random non-negative 31-bit seed when omitted. Each launcher invocation executes one run.

Provenance includes runtime platform, runner_python_version, cpu_model, cpu_count (logical cores available to the process), optional experiment_bin and runner_bin, and endpoints. Older records omit the hardware fields. Executable paths are recorded only under /nix/store/; endpoint values contain only scheme://host[:port], never userinfo, paths, query or fragment. A dirty or otherwise unpinned tree has no fetch_ref. tree_hash is included only where the launcher can compute it; absence carries no meaning. These copies do not include source code or build outputs.

derived.served_models is the sorted set of non-empty llm.call.output.model values returned by the endpoints. These may differ from the requested model; they do not change the condition identity.

derived.results is a map of the last result value per declared name. Undeclared results stay in the stream but do not enter the map. derived.usage contains input_tokens and output_tokens, summed across llm.call records; input totals include cache reads and writes, which Inspect reports separately. Missing counts contribute zero. derived.counts holds llm_calls, failed_calls, and the by_kind and llm_calls_by_agent maps. The latter counts only llm.call records, keyed by their recorded agent; calls without an agent still count toward llm_calls but have no map entry. The runner reads only stream records to derive the card. last_status copies the most recent status phrase when present; last_seq and last_event_at identify the last included record. These are observed totals, not a provider bill.

The runner folds records only after writing them, refreshes the derived block at every live heartbeat, and includes run.end in the final cache. The card is replaced atomically, so concurrent readers see a complete snapshot. A hard crash may leave a stale card; it never makes that card more authoritative than the stream.

Published runs

The final run path is runs/<condition>-<experiment>/<run>/ for both local and published runs. Under the selected prefix, the bucket contains only runs/ and license.json:

PREFIX/
  license.json
  runs/CONDITION_ID-EXPERIMENT/RUN_ID/
    run.json
    events.jsonl.zst

Publication copies run.json byte-for-byte and compresses events.jsonl to one level-19 zstd frame named events.jsonl.zst, with Content-Type: application/zstd. The workspace is local and never published. Indexes are derived, live outside experiment buckets, and are specified separately.

license.json is {"v":0,"spdx":"<SPDX identifier>"} with Content-Type: application/json: one license for every run under the prefix. The first publish with a license writes it; it is never replaced, and a publish naming a different identifier fails before any run uploads. A prefix can be undeclared: --license NONE writes no license.json.

Condition identity lives on the cards and run.start; conditions are groupings of cards on condition. Terminal runs must pass verification before upload. Both destination run keys are checked with HEAD; if either exists, the run is refused without overwriting it. A changed immutable object or license uses a new prefix. Compression does not change (run, seq) record addresses. See Publishing for commands and profile setup.

Derived indexes and static browsing

Indexes live outside experiment buckets and are rebuilt in full from complete runs/ object pairs. The root index.json has v: 0, experiments containing {name, runs} entries, total runs, stores, and built_at in the stream’s UTC timestamp format. stores has one {url, license} entry per store, in store-list order; license is null when a store declares none or its declaration is malformed or names NONE or NOASSERTION. Each experiments/<experiment>/index.jsonl contains rows of {"store":"<public HTTPS base>","card":<run.json as a JSON value>}. Card values are compactly re-serialized; the objects under runs/ remain byte-authoritative. Index objects are replaceable and never authoritative.

Supply stores.yaml (or JSON) with each S3 prefix, AWS profile, public HTTPS URL, filters and exclusions. Build the index with reps-runner index --stores stores.yaml --catalog DIR --to site/index. The command deletes and rewrites site/index in full, writing index.json last. AWS profiles are used only to read source stores.

Read original run objects from each row’s store. Manifests and versioned render hints are written to index/catalog.json. The catalog contains only indexed experiments; their README assets live under index/catalog/assets/<name>/.

See Build and serve an index for store configuration and index commands.

In an llm.call, model API evidence is stored in call.request and call.response; generation settings are read from the request. Inspect also saves every native transcript event as inspect.event with its JSON-mode dump as custom data. Each model event is immediately followed by the derived llm.call, even for cache hits. Sample identity and cache notes use producer-prefixed metadata keys; shared readers treat those keys as opaque. Harness bookkeeping, including generation config, remains available in the raw native record.

run.start omits the experiment field duplicated by its envelope and has no dirty flag. run.json.identity retains experiment because it is a standalone metadata file.

What do the states mean?

Terminal outcomes belong to run.end; run.json.lifecycle.state can contain any of the states above. Live stream state comes from the absence of run.end and the metadata heartbeat.

While active, the runner refreshes run.json approximately every ten seconds, including its derived block. Readers can use the file’s modification time as a heartbeat; no heartbeat timestamp is stored in the card. A hard crash can leave partial files, a stale active state and no run.end.

Experiment process protocol

This page describes the underlying experiment program passed as program to reps.mkExperiment. It receives one parameter object and emits event payloads. To start a run, use the generated named experiment app; launch commands describe the user-facing entry points. The program itself does not launch reps-runner. The runner supplies execution IDs, stores records and determines the process outcome.

What does the runner provide?

The runner invokes the executable without additional arguments, sets its working directory to the run’s fresh workspace/, writes the realized parameters as JSON on stdin, then closes stdin.

The child environment starts with only the host values of PATH, HOME, TERM, TMPDIR and DOCKER_HOST. The runner adds values from selected credential profiles, pins LANG and LC_ALL to C.UTF-8, then adds its run variables. Arbitrary exported keys and endpoints are not inherited. The experiment records its own interpreter, libc and effective locale in producer.python, separately from the runner’s runtime snapshot.

A fresh directory and restricted environment are not an operating-system sandbox. The program runs as the launching user and can access resources available to that user. Some experiments use their own Docker-backed tasks; their daemon and service requirements must be met separately.

What should the program emit?

Use the language-neutral reps-emit CLI, or Python’s reps_events.emit(model), to record structured events. The helpers handle the transport; experiment authors should not implement socket clients. For example, an experiment can invoke:

reps-emit result --name count --value 3

Both helpers validate the payload and wait for the runner to acknowledge it. The runner independently validates against ProducerPayload, adds the transport envelope, and records the event. Unknown types, missing required fields, undeclared fields, malformed payloads, and producer attempts to emit runner lifecycle events are rejected with an error response. They never become invalid records in the JSONL stream.

All nonempty stdout/stderr lines from the experiment program become stdout/stderr events with a line field, including lines that happen to contain JSON objects. Whitespace-only stdout lines are ignored. Ordinary prints cannot submit structured events. These are UTF-8 text channels: malformed bytes are replaced with � so capture continues. Store binary output in artifact files. An unexpected capture failure is recorded as an error and makes the run fail.

How does the run finish?

The runner drains captured output and waits for the process. Exit code zero produces completed; a nonzero code produces failed; a signal exit or handled interrupt produces interrupted. It then emits run.end and writes final process metadata. Readers derive results and usage from the event stream.

The state reports the process outcome. An adapter that catches errors and returns zero should emit metrics, logs or instance errors that make the research outcome clear. A hard runner crash may leave an active state and an incomplete stream.

How can Python experiments emit validated events?

Construct models from reps-events and pass them to emit:

from reps_events import CustomEvent, Result, emit

emit(CustomEvent(kind="govsim.record", data={
    "action": "utterance", "agent_id": "agent-1", "utterance": "I choose option A.",
}))
emit(Result(name="choices", value=1))

Models validate at construction: wrong types and unknown fields raise a Pydantic ValidationError. Use the declared metadata, meta, data, params, and raw call objects for open JSON metadata. They preserve their contents, including nulls. emit serializes once, validates that JSON with strict mode, and writes the exact same JSON. Validation uses the public ProducerPayload union and can reject the event but cannot rewrite its payload. Absent optional model fields are omitted; explicit nulls in open dictionaries are retained. Public producer classes and typed CustomEvent subclasses are accepted; other subclasses and runner lifecycle models are rejected.

Model calls expose their nested models directly:

from reps_events import (
    ChatMessageUser, ChatMessageAssistant, ChatCompletionChoice,
    LLMCall, ModelCall, ModelOutput, emit,
)

emit(LLMCall(
    agent="agent-1", model="provider/alias",
    input=[ChatMessageUser(content="Hello")],
    output=ModelOutput(
        model="provider-returned-model", completion="Hi",
        choices=[ChatCompletionChoice(
            message=ChatMessageAssistant(content="Hi"), stop_reason="stop",
        )],
    ),
    call=ModelCall(request={"model": "model-sent-to-sdk", "temperature": 0.2, "max_tokens": 128},
                   response={"system_fingerprint": "fp_123"}),
))

The chat and output data models are vendored from Inspect AI 0.3.263 in reps_events.inspect_chat; analysis needs only Pydantic. input contains typed messages and output retains every choice, usage, and output metadata. Raw SDK request/response evidence lives in call. Read generation settings from call.request; no separate projection is stored. The direct OpenAI client records effective SDK settings after adapter overrides. The Inspect adapter retains GenerateConfig only in its raw inspect.event record.

For data that does not fit a standard model, use the public custom container:

from reps_events import CustomEvent, emit

emit(CustomEvent(kind="my-experiment.resource", data={"remaining": 42}))

Experiments can subclass CustomEvent[DataModel] with a literal kind and strict Pydantic data. Their Payload union replaces the untyped custom arm with these kinds. Generic custom data still accepts JSON objects, arrays and nulls. Use prefixed custom kinds for per-item outcomes and attributed actions, and Log for diagnostics. The shared emission test fixture works for either form.

reps_experiment.scaffold.deposit_artifact writes a text artifact and emits its pointer using the required experiment-specific kind argument. Its experiment_main helper reads and validates parameters, reports validation or run-function exceptions to stderr, and returns 1 on those errors. A run-function exception also emits any supplied fallback metrics. Successful execution returns 0.

For other languages, invoke reps-emit, installed by reps-events and also included in the runner’s environment. It handles validation and delivery for you. The event reference describes that CLI and its schema output.

Reading runs in Python

Payload is the complete discriminated union, including runner lifecycle events. ProducerPayload is the subset experiments can emit. Envelope contains the transport fields and an event: Payload.

The runner’s internal record_payload function constructs the same typed Envelope that readers use. Socket submissions must validate before reaching it.

from reps_events import CustomEvent, LLMCall, read_events

for record in read_events("/path/to/run"):
    event = record.event
    if isinstance(event, LLMCall):
        print(record.seq, event.model, event.output.completion)
    elif isinstance(event, CustomEvent):
        print(event.kind, event.data)

read_events accepts a run directory or its events.jsonl file. The default shared reader needs no Inspect dependency or experiment imports. For typed GovSim data, pass payload=govsim_adapter.models.Payload; a Pydantic TypeAdapter is also accepted. For static type inference, construct TypeAdapter[GovsimPayload](GovsimPayload) and pass that adapter; bare runtime union aliases validate identically. Invalid records raise EventReadError with the file and line number, including unknown types and truncated JSON. Valid partial runs can be read without a run.end record. Files are never modified.

Use parse_event(json_text) for one event payload, parse_record(line) for one saved envelope, and read_events(path) for a file or run directory. parse_record decodes JSON, applies migrate_record(record) to the dict, then validates the current shape with Envelope; read_events does this for each line. Migrations read the written v, set it to VOCABULARY_VERSION, refuse missing, invalid or newer versions and leave saved files untouched. Both readers accept the optional payload union or adapter, validated after migration. Envelope accepts only the current v and defaults to it when constructed; use the readers for older saved records. EVENT_ADAPTER exposes Pydantic’s TypeAdapter[Payload] for bulk tooling. reps-emit schema --union exports the payload union’s JSON Schema; reps-emit schema --envelope exports the complete record schema.

Wire protocol reference for emitter maintainers

Experiment authors should use reps-emit or reps_events.emit(). This section documents the implementation boundary for maintaining those helpers and the runner; it is not an additional experiment integration step.

The runner creates a filesystem Unix stream socket in a private, short temporary directory before launching the experiment. This works on Linux and macOS. For each event, connect to REPS_EVENT_SOCKET, send one UTF-8 JSON object followed by a newline, and read one newline-terminated JSON response. Each connection carries exactly one event; concurrent emitters use separate connections.

The response is {"ok":true} after the event store has written and flushed the record, or {"error":"..."} on validation or recording failure. Acknowledgement does not promise an fsync to physical storage. Events are limited to 64 MiB, including the trailing newline. Socket operations time out after 30 seconds. The runner closes the listener and finishes active handlers before recording run.end, then removes the socket’s temporary directory.

Python emission raises EventTransportError on missing configuration, failed delivery, or rejection. reps-emit reports the error on stderr and exits with code 2; it writes nothing to stdout on successful emission. There is no automatic retry: a lost acknowledgement can leave receipt uncertain, and retrying could duplicate an event. The generated experiment launcher supplies the execution context; use the named experiment app to start a run. Schema commands work without a running experiment.

Events

If your experiment is written in Python, use the reps_events library:

from reps_events import Result, Status, emit

emit(Result(name="score", value=1))
emit(Status(detail="Starting evaluation"))

For experiments written in other languages, invoke the reps-emit CLI directly:

reps-emit result --name score --value 1
reps-emit status --detail "Starting evaluation"

Both APIs validate events, deliver them to the runner, and report delivery errors. To execute an experiment, use its named app (for example, nix run .#inspect-hello -- ...). The generated experiment launcher starts the runner and configures event delivery. Do not invoke reps-runner or the underlying experiment program directly to start a run, and do not implement a socket client in experiment code.

The experiment program’s stdout and stderr are captured as text, including printed JSON. The runner supplies the transport configuration automatically. See emission tools for CLI flags and Python emission for model construction.

Each run stores event envelopes in events.jsonl. The runner owns the envelope and lifecycle events. The following sections describe the saved format and event vocabulary.

Transport

{"v":1,"ts":"2026-01-01T12:00:00.000000Z","run":"RUN_ID","experiment":"example","schema":0,"seq":0,"event":{"type":"result","name":"score","value":1}}

Payloads use the closed set of types below. Unknown types, missing types, undeclared fields and malformed payloads are invalid. The runner rejects invalid socket submissions. All captured stdout/stderr remains text inside standard events, so saved records conform to the typed Python reader. Put experiment-specific JSON inside CustomEvent.data.

Lifecycle events

These types are generated by the runner. Experiments must not use the run. prefix for their own types.

run.start records the identity, inputs and run seed described in run files. params is the bound input passed to the process. Experiment identity is on the envelope, not duplicated in the start payload. There is no dirty field.

fetch_ref is a pinned clean repository revision; its Python value is None and its wire field is omitted when no such revision is known. Launchers reject fetch references containing URL userinfo. tree_hash is the packaging tree’s NAR hash, emitted whenever the launcher knows it, for clean or dirty trees. It is separate from experiment source identity.

runtime records platform, runner_python_version, cpu_model, cpu_count (logical cores available to the process), and endpoints. Hardware fields are absent on older records. Executable fields experiment_bin and runner_bin are recorded only for /nix/store/ paths; other paths are omitted. Endpoint keys are the configured provider sets used by this run; values are only scheme://host[:port], with no userinfo, path, query, or fragment. The child still receives the full configured URL. This is a runtime snapshot, not a process-environment dump.

result_definitions snapshots the manifest’s ordered list of named result declarations, including labels, descriptions, details and units when supplied. The same definitions are copied into run.json.definitions.results.

run.end contains only process facts: terminal state (completed, failed, or interrupted), duration_s, and exit_code. The exit code is the child’s return code; negative values identify signal termination. An incomplete stream may have no run.end. Consumers derive summaries from the last result per declared name and sum usage over llm.call records; the runner caches both in run.json.derived, while run.end stays free of derived values.

Producer toolchain

Python producers emit producer.python once per run, before their other events. reps_events.emit_producer() records the running interpreter; experiment_main and the Inspect entry point call it after validating params. Other Python entry points call it themselves. This is producer evidence, separate from the runner’s launch snapshot in run.start.runtime.

reps-runner verify checks that this event is unique and precedes other producer submissions. Asynchronously captured stdout/stderr may precede it. For Nix runs, it checks the executable’s store prefix against the launcher’s Python venv, following static wrapper references without executing them. It reports a skipped containment check when the venv cannot be recovered. Older runs without the event produce a warning and remain verifiable. Future language-specific producer.<language> tags belong to that language’s events library.

Status and logs

status describes experiment progress and does not change the lifecycle state. stdout and stderr are generated by process capture and have no inferred severity.

Results

A result has required name and scalar value (finite number, string or boolean). The name identifies a result declared in the manifest. Units and explanations come from the declaration, not from the event. An undeclared name warns and stays out of the summary. A repeated declared name warns and its last value wins. Missing results stay absent. Every event remains in the transcript. Series and per-item outcomes use experiment-specific custom events.

Experiment-specific observations

Model calls use the shared llm.call event across experiments. Other observations, including communication, use CustomEvent where we have not established a shared meaning. We are deliberately deferring further abstractions: similar-looking concepts can mean different things in different experiments.

Each experiment defines its own custom kind strings and JSON data objects. GovSim preserves its upstream rows as govsim.harvest, govsim.utterance, govsim.summary, and govsim.resource_limit; unrecognized actions remain govsim.record. A govsim.upstream_log marker identifies each ingested file by relative path, byte count, SHA256, and record count before its rows. Persona nodes.json rows become govsim.memory with a persona ID and the untouched node. Embeddings are omitted because node descriptions and the embedder named in govsim.config reproduce them. Those kinds do not define a communication convention for other experiments. The default reader uses generic CustomEvent objects. Pass an experiment Payload union to read_events to restore its typed custom data.

Event classes may declare render: ClassVar[RenderHint]. A hint has a Lucide icon name and optional dotted payload paths actor and actor_label. title and body accept a bare path or literal text with {dotted.path} substitutions; missing values become empty strings. Templates have no expressions, conditionals, functions, or defaults. format is text, markdown, json, or html-text; the last strips HTML tags and displays plain text, never foreign HTML. Optional fields are ordered dotted paths shown as an expanded definition list, omitting absent paths. A truthy badge path displays its last segment as a badge. The raw disclosure retains all original data. An optional actor_registry declares a path to a map of actor IDs to entries and a label field within each entry. export_schema(Payload) embeds hints under x-reps-render; they never enter event records. Literal-dependent hints use JSON Schema if/then clauses, for example log severity and the stdout/stderr channel.

Readers can resolve hints from the exported schemas. actor identifies an actor; actor_label supplies its display name, and registry events can provide labels for actors referenced elsewhere in the run. The original event payloads remain available regardless of presentation hints.

Round or step boundaries can be custom events in source order. Mark post-run replay segments explicitly to distinguish them from earlier live calls.

Model calls

An llm.call records one observed model API operation and its available outcome. SDK-internal retries may be invisible to the adapter; this is not necessarily one HTTP attempt. It is not an agent turn or a message sent between agents. Request messages record the input supplied by the caller, not proof of the provider’s complete internal context. Tool requests inside a response do not establish that those tools were executed.

llm.call retains boundary data from Inspect’s ModelEvent, using locally vendored Pydantic models from Inspect AI 0.3.263. The source version, archive hash, local changes and MIT license are recorded alongside reps_events.inspect_chat. The event library depends only on Pydantic. There is no tracing runtime.

SDK retries and empty-response retries are counted in the same retries field, so one record can exceed the per-layer cap: up to 8 attempts in the empty-response loop, each allowing 9 HTTP attempts (the initial request plus 8 SDK retries), for at most 72 HTTP attempts. Exhausted HTTP 429/5xx retries stop at the SDK budget; the client does not start a second cycle after those nine rejections.

Messages have system, user, assistant or tool roles. Content is text or typed blocks for text, reasoning, images, audio, video, documents, server-side tool use, or provider data. Tool calls, tool errors and citations retain their Inspect data fields relevant to the boundary. Tool error types are open strings; boundary producers use what the API reports (Anthropic’s error flag becomes "error"), and harness adapters can retain native error types. Parsing errors remain on ToolCall.parse_error; message source and tool viewer hints stay in raw harness records. Message IDs are preserved when supplied; the vendored models do not invent IDs. Provider-specific fields that do not fit these models remain in call.

output.usage keeps input/output/total tokens, cache-read/write tokens, reasoning tokens and cost. Inspect’s input_tokens excludes cached tokens; consumers add cache-read/write tokens when computing total input usage. Missing usage remains unknown. The shared usage model defaults absent token counts to zero, so the OpenAI-compatible converter leaves output.usage absent when required counts are missing, retaining all reported partial usage in call.response. output.model records the returned model; the SDK’s requested model remains in call.request when available.

The served name must equal the requested name ignoring case, except that an unqualified or -latest request also accepts the same name plus one date suffix (-YYYY-MM-DD, -YYYYMMDD, @YYYYMMDD, -YYYY-MM; -YYMM for mistral/ only).

A response with an empty model ID and a nonempty set of entirely content-filtered choices is exempt from the identity check. Verification warns once per run that model identity could not be checked on content-filtered calls. The client keeps its first-response identity check pending for the next response with a model name. Other empty or non-matching served IDs are mismatches. Version 0 records with empty choices and no error are read as failures with Empty choices for model '<model>' (version 0 record, not retried). Verification fails these runs, including when the empty response supplied a model ID.

Read generation settings, including temperature and token limits, from call.request. They are not projected onto separate fields. Inspect’s GenerateConfig is retained only in the raw inspect.event record. The OpenAI-compatible client converts SDK messages, tools, choices and usage at the capture boundary and keeps the full SDK request/response in call. The first well-formed <think>…</think> block in assistant text is reasoning (Inspect’s rule), provider reasoning_content is reasoning, and everything else is text, untrimmed. GovSim trims at its Pathfinder backend for parity with local generation and anchored parsing. call.response retains the SDK response and its unmodified content strings, including HTTP error bodies. An error body containing valid completion choices uses the same output conversion as a 200 response and has no error; new records leave call.error unset (None). HTTP errors without usable output retain the body with error set and call.error unset. call.error becomes true only when the request failed without a captured body, such as a connection failure or timeout. Non-object HTTP bodies are retained under call.response.body. The response hook only counts HTTP 429/5xx responses. HTTP errors without valid choices retain the body and SDK message and are raised immediately. Only 2xx responses with empty choices enter the empty-response retry loop; exhaustion records Empty choices with the model, attempt count and last ID. The SDK’s own retry policy is unchanged.

parse_event validates a bare event payload. parse_record decodes a saved JSON envelope, applies reps_events.migrate_record(record) to the dict, then validates it with Envelope; read_events calls parse_record for each line in a file or run directory. Both readers accept an optional payload union or adapter, validated after migration. Migrations interpret version 0 error-body reprs and silent empty responses under the version 1 rule and set v to VOCABULARY_VERSION. They read v from the record, rejecting missing, invalid or newer versions without rewriting saved files. An absent call remains absent. Recovering a version 0 error body clears its stale call.error; converting its choices to usable output sets that legacy flag to false. The silent-empty-response migration leaves call.error as written. For prefixed version 0 errors, a body that cannot be parsed as a Python literal and JSON-compatible data, or whose choices cannot be converted, raises MigrationError; read_events wraps it in EventReadError with the line number. Errors without the SDK prefix remain unchanged. Envelope accepts only the current v and defaults to it when constructed. derive_card projects the records supplied by the reader without applying migrations itself.

Verification uses migrated records for structure, order, identity, failed-call counts, model checks and the rebuilt card, failing on any remaining llm.call.error with the count and first message. Its experiment-union subprocess validates raw lines using the build’s own reps_events, including the v0 model in retained old builds. An older card is accepted only if it exactly matches the original projection built from the JSON lines, or the migrated projection. Verification does not compare record versions with the manifest: the runner checks vocabulary compatibility before launch and stamps the manifest’s version when writing. Historical inline reasoning strings remain unchanged.

File deduplication references (input_refs), tracebacks, streaming state (pending, working_start), and Inspect’s event identity/timestamps are omitted from the payload. The existing envelope supplies record identity and capture time; it has no span_id. Completion-time emission still cannot record a call if the process dies while waiting for the provider.

Custom events and deferred concepts

A custom event has a namespaced kind and JSON object data. GovSim schema 0 specializes these with strict models and a union discriminated by type, then kind:

{"type":"custom","kind":"govsim.state","data":{"round":1,"resource":42,"collected":{},"limit":5,"final":false}}

Read GovSim with read_events(run_dir, payload=govsim_adapter.models.Payload). Generic readers still use CustomEvent. Inspect and Concordia use their own prefixed kinds, including inspect.sample, inspect.artifact, and concordia.provenance. Every Inspect transcript event is emitted as inspect.event, with its native JSON-mode dump (omitting absent optional fields) as data, in transcript order. Each model event is followed immediately by its derived llm.call, including cached events. Those calls use the producer notes inspect.sample_id, inspect.epoch, and, when applicable, inspect.cache in metadata; raw config, roles, retries and cache bookkeeping stay available in the native record. Shared instance attribution is deferred.

Shared instance, artifact, agent.event, tool.call, span, and limit models are deferred. File pointers belong in custom data; producers write the file before emitting its pointer. reps_experiment.deposit_artifact requires the custom kind explicitly and writes text files under the run’s artifacts directory.

Emission tools

reps-emit validates a producer payload, sends it to REPS_EVENT_SOCKET, and waits for acknowledgement. The reps-events package installs it without depending on the runner. It is also available in the runner’s environment through that dependency. The following checkout command only inspects a schema; it does not launch an experiment:

uv run --project lib/reps-events reps-emit schema llm.call

In the experiment program, invoke reps-emit result --name score --value 1 to record a value.

reps-emit schema prints schemas for all standard payload types, while schema TYPE selects one. schema --union exports the discriminated payload union; schema --envelope exports the complete saved record. The helper offers these emission subcommands:

llm-call emits the llm.call wire type. JSON-valued flags take JSON text; --value accepts JSON or a bare string. llm-call also accepts a payload body object on stdin, with flags overriding fields from stdin. Invalid standard payloads return status 2 and print an error to stderr.

Python producers construct Pydantic event models and call emit(event); see process protocol. Public producer classes and typed CustomEvent subclasses are accepted by emit. Unknown constructor fields raise; explicit metadata objects preserve provider and experiment fields. Use CustomEvent for generic experiment data.