Execution nodes

Experiments don't run on the control node — they run on execution nodes (runners): machines you attach. A node clones the project's private code and runs scheduled experiments in an isolated git worktree per run. Connect whatever compute you have; nodes are heterogeneous by design.

Nothing runs until a node is attached

A live project with no execution node will queue experiments but never run them. Attaching a node is what turns idle compute into results.

Turn this machine into a node

autolab serve --project alice/nanochat

serve starts a background daemon and returns once the node is registered — the node stays connected after you close the terminal or SSH session. No tmux, nohup, or systemd unit required.

Getting your prompt back is success

serve is not a progress feed: it prints a connect banner, then exits while the daemon keeps running. Experiments starting, running, and finishing on this node show up on the dashboard (autolab open), not in this terminal.

Because it fetches private code, registration needs a researcher+ token (researcher, admin, or owner). The token is read from, in order: --token, then AUTOLAB_TOKEN, then your stored autolab login.

Manage the background node

autolab serve status     # running? project, control-node view, recent log
autolab serve stop       # disconnect this machine (idempotent; --name for one node)
autolab serve log        # this node's entire runner log (-n N for a tail, -f to follow)

serve status exits 0 when a daemon is running on this machine and 1 otherwise, so it's scriptable. It also shows what the control node currently thinks of the node (online/offline, last heartbeat, running experiment) when you're signed in.

Daemon state and logs live under ~/.local/state/autolab/serve/ (one slot per node identity — machine + node name). If that directory is on a shared home (NFS), records written by other machines show up in status as read-only entries — manage those daemons on their own machine. You can also disconnect a node from the dashboard — the daemon notices and exits; autolab serve status on the machine will say it was disconnected.

Keeping a node current. The control node stamps a fresh CLI build on every deploy; a running daemon keeps its old code and logs a warning when it drifts. Update and restart:

autolab update
autolab serve stop && autolab serve --project <owner>/<slug>   # or: serve … --replace

The runner keeps a timestamped narration of everything the node does — registration and reconnects, accepted jobs, agent bash commands, failures, shutdowns. autolab serve log prints it on the machine, and the node ships the same log to the control node with each heartbeat, so it also renders live in the Node Log panel on the node's dashboard page — you can debug a remote node without SSHing into it.

This is the machine's log, not your experiment's. A run's own stdout/stderr is on the experiment's page (autolab open <id>); read the node log when you want to know whether the node is healthy and picking up work.

One daemon per node identity. A node's identity is its machine plus its node name, and the default name is unique per GPU allocation (<host>-gpu<pci>, falling back to <host>-job<slurm-job-id>, then <host>). In practice:

• Re-running serve on the same resource is a no-op that reports the existing daemon.
• Starting a different project on the same resource errors and points you at autolab serve stop --name <node>; pass --replace to switch it in one step.
• Separate GPU allocations each get their own node — two interactive Slurm sessions on one host (or several machines sharing an NFS home) can all run autolab serve, and each becomes its own node. serve prints what else the machine is serving so nothing runs behind your back; serve stop disconnects all of this machine's nodes, --name just one.
• Two shells that see the same GPUs derive the same name, so an accidental duplicate is still caught — use --name only to deliberately run two nodes on one resource (e.g. MPS sharing).

serve options

Option	Default	Meaning
--project	(required)	Project slug owner/slug to run jobs for.
--name	derived	Node name shown in the dashboard (default is unique per machine / GPU allocation).
--poll-interval	5	Seconds between polls for new work.
--token	—	Researcher+ token to register with.
--autolab-home	~/.autolab	Root dir for node workspaces.
--foreground	off	Stay attached to the terminal (Ctrl-C stops).
--replace	off	Stop the same-named background node first, then start.
--type	local	slurm connects a Slurm cluster (see below).
--max-parallel-jobs	4	Concurrent sbatch runs a Slurm node accepts (Slurm only).
--gpu-scheduling	whole_node	How this machine's GPUs are shared between experiments (see below).
--jobs-per-gpu	2	Experiments allowed to share one GPU (with --gpu-scheduling shared).

Node workspaces live under <AUTOLAB_HOME>/<hostname>/projects/<owner>--<slug>/. The hostname namespacing makes a shared NFS root collision-safe.

Using several GPUs on one machine

By default a node runs one experiment at a time, and that experiment sees every GPU. That's what you want when a run scales across cards — but it leaves seven cards idle while a small run uses one. --gpu-scheduling lets Autolab treat the GPUs as separate resources instead:

Policy	Concurrent experiments	Each experiment gets
whole_node (default)	1	every GPU on the machine
per_gpu	one per GPU	its own GPU
shared	--jobs-per-gpu per GPU	a share of a GPU

autolab serve --project you/proj --gpu-scheduling per_gpu
autolab serve --project you/proj --gpu-scheduling shared --jobs-per-gpu 2

On an 8-GPU box, per_gpu runs up to 8 experiments at once. Use shared when runs are small relative to the card (short jobs, inference sweeps) and you're willing to trade some contention for throughput.

Your code doesn't change. Autolab picks the cards and sets CUDA_VISIBLE_DEVICES before your run command, so inside a run the visible devices are always numbered from 0 — cuda:0 is whichever card it was given. Don't set CUDA_VISIBLE_DEVICES yourself in a run: that's how two experiments end up on one card.

An experiment that genuinely needs several GPUs still gets them — the agent requests them per run, and Autolab starts it only once that many cards are free. The node's policy appears in the agent's briefing, so it knows whether it has the whole machine or one slice of it.

This is different from running several nodes on one host

--gpu-scheduling is one daemon owning the machine and running several experiments on it. If your cluster instead hands you each GPU as a separate allocation, run autolab serve once per allocation — each becomes its own node (see One daemon per node identity above).

Headless / remote GPU box

An agent (or you) can bootstrap a remote node over SSH with a dedicated, revocable token — no browser on the remote box:

TOK=$(autolab token create --name gpu-box)        # a fresh researcher+ token
ssh gpubox 'curl -fsSL https://app.autolab.ai/install.sh | sh'   # install the CLI
ssh gpubox "AUTOLAB_TOKEN=$TOK ~/.local/bin/autolab serve --project alice/nanochat"

The last command returns once the node is online; the daemon keeps running on the box after the SSH session ends. Check on it later with ssh gpubox "~/.local/bin/autolab serve status".

Revoke the token with autolab token revoke gpu-box. See Tokens & API keys.

Slurm clusters

On a Slurm login node,

autolab serve --type slurm --project alice/nanochat

scans the cluster (partitions, node states, GPUs, your accounts/QoS), shows the inventory, then connects the login node as an execution node whose experiments run as sbatch jobs on the cluster — the daemon on the login node only codes, submits, and streams logs/status. Requirements and behavior:

• AUTOLAB_HOME must be on a filesystem the compute nodes share (experiment worktrees are prepared on the login node and executed by the batch job).
• QoS/account handling: many clusters reject jobs without an explicit --qos/--account. AutoLab derives them from your sacctmgr associations automatically when unambiguous; otherwise set --slurm-qos/--slurm-account (or let the agent pin resources.qos per run). Standard SBATCH_* environment variables exported before autolab serve are honored too.
• --slurm-partition, --slurm-account, --slurm-qos, --slurm-gres set submission defaults; unset values fall through to the derived values, then cluster defaults.
• Environment setup (setup_command, e.g. uv sync) re-runs inside the batch job on the compute node, so the venv matches the compute architecture.
• The job's time limit is the experiment's expected runtime; Slurm state maps back honestly (TIMEOUT, OUT_OF_MEMORY, CANCELLED, …) and the run log carries [autolab]-prefixed submit/finish lines with the Slurm job id.
• Restarting the daemon (or the login node) re-attaches to still-running sbatch jobs and delivers verdicts of jobs that finished in between.
• The cluster node runs up to --max-parallel-jobs experiments concurrently (default 4): the agent codes the next experiment on the login node while earlier ones execute as sbatch jobs.
• The agent requests resources per run (partition, GPUs, CPUs, memory, time limit, QoS/account — plus any other sbatch flag, e.g. --exclusive or --constraint=a100) from the cluster inventory it sees — including each partition's allowed accounts/QoS, so restricted partitions get valid submissions; your --slurm-* flags are the fallback defaults. Ask the agent in chat to schedule differently and it records the preference in the node notes for future runs.
• A preempted or node-failed run is requeued automatically (no failure analysis) and reschedules on the next pass — up to 3 automatic reruns, after which it is handed to the agent as a failure.
• The job page shows the exact submit command (sbatch --time=… --partition=… trees/<id>/.autolab-run.sh) and the Slurm job id for every run, live while it executes — with the submitted script (workspace env + setup + run) one click away under the command.
• The dashboard marks the node with a slurm badge and shows the cluster panel — partitions, GPU inventory, your accounts/QoS, derived submit defaults — plus a Check GPU availability button that asks the node for live idle counts.

You can still run plain autolab serve inside an allocation (salloc/srun) to attach a single allocation as its own node — including several allocations at once, each with its own serve daemon ("one daemon per node identity" above).

List & remove nodes

autolab nodes              # list the project's execution nodes
autolab nodes add          # how to attach a machine (prints the instructions)
autolab nodes rm <id>      # request shutdown + deregister a node (id or name)

Removing a node (CLI or the dashboard's Disconnect button) asks the runner to shut down: it exits on its next poll, and a mid-run experiment is cancelled and requeued for another node.

---

Keep the agent and queue in view while nodes run with autolab status.