Run experiments¶
An experiment is a commit. You create one with submit, inspect it with diff
and log, move between experiments with checkout, and read results on the
dashboard — an experiment's output never prints in your
terminal.
Read Research Progress¶
Open a climb's Overview to see Research Progress. Merged runs shows only scored attempts that were merged, including merged runs that did not set a record on the selected metric. All runs also shows other scored attempts and grouped sweeps. Each view draws its best-so-far line from the attempts included in that view. Original run numbers stay the same when switching views.
The first visible score and best score have labels. Hover a point for its score and run details, or click it to open the run. Non-record points are smaller and fainter. Your selected view is remembered across reloads and applies to secondary metrics too. If a board has scored attempts but no merged scores, choose All runs to inspect them. Only official, passing evaluations appear in either view.
The overview's controls stack at narrower widths to keep labels, switches and compute details readable.
Submit¶
submit snapshots your working tree, commits it as a new experiment, and queues
it. There are three ways to use it, depending on how much you want the agent to
own.
You wrote the code and it's final. Autolab commits + pushes it and moves you into the new experiment.
Every field you set, including your code, is locked: the agent fills only the blanks (name, description) and never overwrites your work. How a submission runs and is scored comes from the hill, not the job.
A starting point you're happy for the agent to revise. You stay in your current experiment.
Where do I end up after submit?
You advance into the new experiment only when the code is final — you
pushed code and didn't pass --soft. With --soft (agent may revise) or
--nocode (agent writes it), you stay put. This keeps your working tree
predictable.
Nothing to submit?
A plain autolab submit with no changes and no message is refused —
there'd be nothing for anyone to run or implement. Edit code first
(autolab diff previews what would be sent), pass -m "<why>" to rerun
the current code as a new experiment (e.g. to check run-to-run variance),
or use --nocode -m "<idea>" to let the agent write the code.
Common options¶
autolab submit -m "bf16 + larger batch" \
--name "bf16-bs512" \ # else the agent derives a name
--node "8xH100" \ # node constraint
--from <id> \ # branch from a specific experiment instead of main
--priority 10 # queue priority
See all options in the CLI reference.
Inspect changes: diff¶
diff has two modes. With no argument it shows your uncommitted changes
versus the experiment you're in — exactly what submit would send. With an
experiment id it shows what that experiment changed versus the code it
branched from. (Your .autolab/ control dir is never shown.)
By default it prints a summary — each changed file with its added/removed line
counts, like git diff --stat. Pass -p/--patch to print the changed lines
themselves:
autolab diff # your changed files, ±lines each (new files included)
autolab diff -p # …the full patch: the changed lines themselves
autolab diff <id> # what experiment <id> changed vs its base, ±lines per file
autolab diff <id> -p # …as a full patch
autolab diff <id> works once the experiment's code is final (it ran, or
its code was locked). While the agent may still write or revise the code —
queued or --nocode/--soft experiments — there is no meaningful diff yet and
the CLI says so.
History: log¶
log includes queued and pending experiments. The commit column stays
empty until the code is final — for --soft/--nocode experiments the agent may
still be coding it.
log is history, not output
Like git log, it prints one line per experiment — not what a run printed.
For a run's stdout/stderr use autolab open <id>; for the node's own
narration use autolab serve log. See
Which log is which.
Lineage: graph¶
Where log is a flat list, graph shows the branch tree — what each
experiment was forked from, so you can see how the research actually evolved:
autolab graph # the lineage tree; → marks the experiment you're in
autolab graph --root <id> # only one experiment's subtree
autolab graph --format json # the same graph as edges, for scripts and agents
alice/nanochat
└── 9f2a01c4 merged loss 0.42 baseline
└── 7b1e88aa merged loss 0.31 rotary embeddings
├── → 4c0d12ff running bf16 + larger batch
└── a83b77e1 merged loss 0.25 cosine decay
Each node shows the short id, status, primary metric, and title. A child hangs
under the experiment it branched from — the one you chose with
submit --from <id>, or the latest merged experiment by
default. Status is colour-coded the same way as the dashboard's
graph view.
Only experiments that have started appear — queued/pending ones have no
lineage yet, so use log to see those. --format json emits one node per
experiment with its resolved parent_experiment_id, parent_source
(parent_id or base_commit), depth, and is_current — handy for an agent
reasoning over the tree.
Dashboard graph¶
The climb's Graph starts in Compact mode. It groups two or more older discarded runs that share a parent and have no descendants. The three most recent ordinary discards stay visible, along with merged and active runs, failures, and discarded runs that later experiments branched from. Sweeps keep their own expand/collapse controls.
Select an older-discard group to expand it. The Run, Score, and Status columns stay aligned while its rows open. Choose All runs to remove this grouping, or use the legend to hide a whole status category. This only changes the display; every run and its results are retained.
Move between experiments: checkout¶
Reset your working tree to any experiment's code, or back to main:
autolab checkout <id> # this experiment's code
autolab checkout main # the current main branch
autolab checkout <id> -f # discard local changes while switching
checkout fetches first so the target commit is present. If you have uncommitted
changes it stops and tells you how to keep them (submit) or discard them
(-f).
Stay current: pull¶
Read results¶
Results live on the experiment's dashboard page — open it straight from the terminal:
That page carries everything a run produced: Logs (a Setup tab for environment prep and a Run tab with your experiment's stdout/stderr, streaming live), the extracted metrics, the agent's analysis and merge/discard verdict, the code changes versus the parent experiment, and an Agent Activity feed of what the agent did on it.
The score comparison follows the evaluator's declared higher-is-better or lower-is-better direction and includes only runs from the same evaluation version, primary configuration, final mode, and metric schema. Compact scores keep eight significant digits; hover a shortened score to see its exact recorded value.
Artifacts aren't captured yet
The runner does not upload checkpoints or files, only logs.
Shared experiment history¶
When your climbs are selected for a private shared-history workspace, each retry has a separate attempt record. A later success does not erase a failed try. Existing descriptions and analysis are included when available; submitting an initial rationale is optional. Results, errors, parameters and code commit hashes can be shared without sending raw transcripts, environment variables or code bodies.
Sharing runs in the background and retries automatically if the history service is temporarily unavailable. The separate history API supports recent-first listing and filters for the hill, evaluation version, climber, job and outcome. Participating AutoLab agents consult shared history before proposing experiments. They can filter results or request older pages, cite useful trace IDs, and query another explicit hill in the workspace. Queries default to the current hill and evaluation version. If the service is unavailable, research continues with local history. A crash is recorded separately from a measured regression.
Existing stored outcomes can be imported when your workspace is enabled. Importing again does not duplicate attempts. Retry outcomes overwritten before attempt capture existed cannot be recovered from the current job record. Records already shared remain in that workspace when a job is deleted.
Cancel¶
Cancelling a running job also stops it on its node — a not-yet-started
assignment is dropped, and an in-flight run is killed (scancel on Slurm
clusters) so the GPU frees up right away.
Most experiment IDs accept a short prefix. If a prefix is ambiguous the CLI tells you — see Troubleshooting. For the full split between what the terminal shows and what the browser shows, see CLI and dashboard.
The Add Job dialog keeps your draft if submission fails and displays an error so you can retry. Keyboard focus stays in the dialog; Escape closes it and returns focus to the button that opened it.