Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Repeat and compare runs

A condition groups runs with the same experiment name, declared source identity and input values. Each execution gets a separate run ID. This lets you repeat a configuration while retaining each outcome and its evidence.

How do I repeat a configuration?

Keep the complete command generated by the form, including every --set. Run it again against the same source to create another run under that condition. In Jobs, rerun submits one new run using that job’s inputs and profile selections. It uses the local execution source available when the new job runs; it is not a replay from the original run’s saved source reference.

Each invocation executes one run with a new run ID. Invoke the command again for another run.

How do seeds work?

The recorded seed is the --seed argument, unchanged. It must be an integer from 0 through 2147483647. When omitted, the runner draws a random non-negative 31-bit seed. The same value is passed to the experiment as REPS_SEED and recorded in run.start and run.json’s inputs.seed.

This complete keyless command makes one run with seed 42:

nix run .#inspect-hello -- \
  --set model=mockllm/model --set limit=0 --set epochs=1 \
  --set 'generate_args={}' --seed 42

Repeating that command produces another run with seed 42 and a new run ID. Whether the resulting behavior repeats depends on the experiment, provider and external inputs.

To reuse a saved run’s seed, pass the number shown on its run page or stored as inputs.seed to --seed. The local job rerun action draws a new random seed.

How do I audit the first real run?

After the first live run of any experiment, verify its directory before launching more or publishing data. From the REPS checkout, run:

reps-runner verify /path/to/run
# With the packaged runner:
nix run .#reps-runner -- verify /path/to/run

You can also pass the printed run ID: nix run .#reps-runner -- verify RUN_ID --data-dir DIR. Run-ID lookup selects --data-dir, then REPS_DATA_DIR, then $XDG_DATA_HOME/reps (default ~/.local/share/reps). An explicit run-directory path is used directly.

The command scans every file, including the workspace, for known credential values and common secret patterns, validates every envelope through the experiment’s declared payload union, and compares every card section with its stream projection. It requires a terminal run.end, checks identity and sequence continuity, and exits non-zero on any finding. It reads without repairing files; diagnostics never print the matching credential or invalid payload. Failed or interrupted terminal runs can also be audited.

Known credentials come from the current environment and every local profile matching the run’s recorded provider endpoint origins. Keep the credentials used for the run available when verifying it; retired credentials cannot be matched by value once removed. Public shell settings such as KEYTIMEOUT are not credentials. Review the actual usage, reasoning content when the endpoint supplies it, and served model name in the first live stream as well; verification checks evidence integrity, not the scientific result.

The command uses REPS_MANIFEST when invoked through an experiment app (nix run .#govsim -- verify /path/to/run), or --manifest FILE / --catalog DIR for an existing build. Otherwise it builds the current checkout’s manifest catalog. The manifest must match the recorded experiment and schema version; its built interpreter imports the declared union. A missing interpreter or union fails verification rather than falling back to untyped custom records.

Verification also checks every llm.call’s served model against its requested name and each request seed when present against run.start.seed. It warns once per distinct model mismatch and exits non-zero on either kind of mismatch. Comparison uses the snapshot stem rule. The shared client checks its first successful response and stops a misrouted run after recording the response and an error log. The audit reports how many calls stopped with max_tokens or content_filter, counting calls rather than completion choices; those counts are informational, so inspect truncated or filtered replies before launching more.

Azure

Create an Azure OpenAI deployment named exactly like the requested model, for example gpt-5-nano, and use azure/gpt-5-nano as the REPS model ID. Identity names the model you asked for; the deployment is routing. A deployment label in identity would make conditions depend on an arbitrary label. A per-model deployment override is deferred until a deployment cannot be renamed.

Configure the profile locally, with the key entered at the hidden prompt:

nix run .#reps-runner -- credentials set azure.default

Supply AZURE_OPENAI_API_KEY and AZURE_OPENAI_BASE_URL: https://RESOURCE.openai.azure.com/openai/v1/. Neither has a default. The runner injects these from the selected profile; exporting them in a shell does not configure the launcher. Azure’s v1 API uses the ordinary OpenAI client, with the deployment name in model.

Start with one run of one round, then verify the printed run directory:

nix run .#govsim -- --non-interactive --credential azure=default \
  --set model=azure/gpt-5-nano --set max_rounds=1 --seed 42 \
  --set max_tokens=3000 --set reasoning_effort=low \
  --set temperature=null --set top_p=null --set embedder=hash
nix run .#reps-runner -- verify /path/to/run

The hash embedder makes this an endpoint audit; choose mxbai for the paper’s embeddings. GovSim’s default completion cap stays 8000 for upstream fidelity. For gpt-5-nano at low reasoning effort, start with 3000 and review the audit’s max_tokens stop count. Reasoning shares the completion budget.

Azure rate limiting estimates prompt tokens plus the requested maximum output, even when the reply is shorter. Budget approximately calls/minute × (prompt tokens + max_tokens) × concurrent runs against the deployment’s TPM. For example, ten calls/minute, 1000 prompt tokens and a 3000 cap need about 40,000 TPM per run; three runs need 120,000. Leave headroom for estimation and retries, and check RPM too. See Azure quota management.

Launch one runner process per run and stagger their starts. Audit the first run before increasing concurrency. Staggering reduces bursts but does not raise capacity.

Read a 429’s message: rate-limit or quota wording points to the deployment’s allocated TPM/RPM; temporary inability or high-demand wording points to shared capacity. A lower x-ratelimit-limit-tokens than configured TPM indicates a temporary shared-pool reduction. Honor the retry delay; reduce traffic or adjust quota for the former, back off for the latter. Low billed usage alone does not distinguish them. See Azure’s 429 troubleshooting table.

The shared client permits eight SDK retries, with unchanged SDK timeouts. SDK attempts are otherwise invisible: one llm.call contains the final outcome. The typed llm.call.retries field counts HTTP 429 and 5xx responses during that call, including the last rejected response when retries are exhausted. This is the only record of rate limiting; it also includes server errors and excludes connection failures, timeouts and other retryable statuses. The shared client records zero when none occur. Other producers may leave it None (omitted from JSONL) when unknown. For historical records, the Python property reads the old retry note, or infers zero from the old backend marker. Intermediate response bodies and headers are not retained. The card’s derived.served_models lists the endpoint-returned names across the run; the runs list shows them beside the requested model when they differ.

Sweeps

Pilot each configuration for one round and verify its saved run before starting a sweep. Review the served model, retries, stop counts and any provider rejection. Keep generation settings explicit; changing settings creates a different condition. Launch sweeps from a clean tree, including untracked files; dry-run inspection remains available while editing.

Use one runner process per run and stagger starts to spread provider traffic and local resource use. Invoke the same condition again for another run.

Evidence stays in each run directory: the event stream and its index card. Keep operational terminal logs separate from the evidence.

How do I keep the source version?

A run records source, the experiment’s declared content identity, and fetch_ref, a repository reference intended for fetching the source. The reference is omitted when no pinned clean revision is known. The launcher also records tree_hash, the packaging tree’s NAR hash, when available for either clean or dirty trees; a hash alone cannot recover unsaved working-tree content. Preserve your working tree separately when running local modifications.

For reproduction, select the recorded repository revision rather than moving main or a registry alias that can change. Working with Nix gives the source forms. Compare the resolved condition with --dry-run before executing.

For a clean, public revision built with its default Nix inputs, that revision also recovers the committed dependency locks and shared library code. The runner saves executable store paths and platform information in runtime so you can compare the selected builds; see run metadata. Keep the commit available in the public repository.

What does a matching condition tell me?

The experiment name, declared source content and parameter values match. It does not establish equal model behavior or complete equivalence of the execution environment.

The condition hash excludes seeds, credential profiles, endpoints, the shared runner, platform and fetch reference. An experiment author chooses the source paths included in identity. A model ID is included as an input string, but a provider can change what a moving model alias resolves to.

Self-hosted IDs name the serving stack: vllm/Qwen/Qwen2.5-7B-Instruct uses an upstream repo id, while ollama/qwen2.5:7b-instruct-fp16 uses an Ollama library tag. Runs across engines do not pool into one condition because engines differ in hidden sampling defaults, chat templates and quantization formats. The convention for vLLM is to serve under the upstream repo id without renaming it or quantizing at load time; choose a repo or tag that names the intended quantization. The venue (laptop, cluster or Vast) belongs in a credential profile, so moving the same model and engine between venues does not change the model ID. On vLLM a trailing assistant message is continued as a prefill rather than answered in a new turn.

The served name must equal the requested name ignoring case, except that an unqualified or -latest request also accepts the same name plus one date suffix (-YYYY-MM-DD, -YYYYMMDD, @YYYYMMDD, -YYYY-MM; -YYMM for mistral/ only).

Source identity includes the experiment’s declared files plus the source directories of the shared Python packages it uses. mkExperiment includes reps_events and reps_experiment by default; Inspect programs also include reps_inspect. These package directories pass through cleanImport, excluding development artifacts; their surrounding tests, READMEs and project tooling are outside the source list. Changes to shared execution code therefore change every dependent experiment’s source hash. A GovSim adapter edit changes GovSim’s hash without changing Concordia’s. The runner and documentation remain outside this hash. The lock is hashed without [package.metadata.requires-dev] tables, excluding repeated path-dependency test metadata while retaining runtime requirements. The filter is covered by scripts/check-experiment-identity.py and the synthetic-lock tests run by nix flake check (checks.<system>.identity-lock).

When comparing outcomes, inspect the declared parameters, source reference, instance completion counts, errors and available environment and model metadata. Decide whether those differences matter for your question.