Grade by Running: A Complete Tutorial for swarm-gym, the Local-Model Swarm Testbed
Mihai Perdum
Author
5 min readJuly 3, 2026
Key takeaways
swarm-gym is a Python harness (python -m harness ...) that drives goose local-edition's coding swarm, then grades what it built by RUNNING it, not by trusting the model.
Two modes: benchmark (a frozen 3-spec suite for reproducible, variant-paired A/Bs) and exploratory (a brain invents work and vibes follow-ups, with idle-node monitoring and auto-tuning).
The brain is pluggable — a Claude judge, a local Qwen, or the operator (you) via a plain req/resp file handshake with no API key.
Verification is seven dimensions across four sources: a deterministic golden-check core, cluster/fleet health, an AI judge, and static diagnostics.
It is a flywheel: measure, diagnose a fleet or drift problem, flip a swarm flag or tune a knob, then re-run the same frozen ruler.
A small local model is a confident liar. Ask a 27B-class coder running on your own hardware to build a CLI app and it will happily report "All tests pass, the feature works" — while the binary it wrote crashes on the first real command, or its own unit tests pass around a feature that was never implemented. If you grade a coding swarm of these models by reading its final message, you are grading its confidence, not its software.
swarm-gym is the answer. It is the self-driving test harness for goose local-edition — the fork of goose whose swarm runs a fleet of small local models as one coordinated coding team. This tutorial walks you through the whole thing: set it up, run both of its modes, read every output it produces, and use what you learn to tune the swarm. By the end you will be able to benchmark a model build, drive an open-ended session as the brain yourself, and read a verdict down to the individual failing check.
Success
The one principle everything follows from: the harness never trusts the model's self-report. It re-runs the produced app against golden values committed before the run, and grades on what the software actually does. A green test suite the model wrote is not evidence; the built binary printing the right number is.
Setup
The harness lives under evals/swarm-gym/ in the goose local-edition repo and runs as a Python module. You need three things: the Python environment, a built swarm binary, and a brain.
1
Create the environment
cd evals/swarm-gym, then python3 -m venv .venv && . .venv/bin/activate && pip install -r requirements.txt. Everything is invoked as python -m harness <subcommand>.
2
Build the binary under test
the harness shells out to swarm.binary, which defaults to ../../target/debug/goose. Build it with cargo build -p goose-cli from the repo root. This is the thing being measured, so it must exist.
3
Pick a brain
the component that invents tasks and (optionally) judges them. claude (default) uses the Anthropic SDK and reads ANTHROPIC_API_KEY from the environment; local points at an LM Studio endpoint and needs no key; operator lets you be the brain with no key and no SDK. More on all three below.
4
(Optional) MCP secrets
if you want workers to use tools like context7 (library docs) or web-search, list them under swarm.mcp and set the matching secret env vars. Secrets always come from the environment or a local .env, never from a committed file.
Note
Nothing about the harness config holds a secret. config.yaml names models and endpoints; ANTHROPIC_API_KEY, CONTEXT7_API_KEY, WEBSEARCH_BEARER and friends live in your shell or .env.
The core idea: grading by running
Every spec the harness runs carries deterministic golden checks — commands run against the finished app whose output must equal a specific, pre-decided value. The model's reasoning, its claimed test results, the tidiness of its code — none of it enters the score. Only this: you run the command, and you either get the golden value back or you don't.
Here are three real checks, verbatim from the frozen benchmark suite, one per archetype:
thebash
1# crud / invtrack — an inventory store. Add two items, then the total value must be 25.2python3 -m invtrack --db bench.db item add A1 --name Bolt --qty10--price2&&\3python3 -m invtrack --db bench.db item add A2 --name Nut --qty5--price1&&\4python3 -m invtrack --db bench.db report value |grep-qE'(^|[^0-9])25([^0-9]|$)'56# compute / rdcalc — a recursive-descent calculator. ^ is right-associative, so 2^3^2 = 512, not 64.7python3 -m rdcalc eval'2^3^2'|grep-qE'(^|[^0-9])512([^0-9]|$)'89# txn / txkvbench — a transactional KV store. A nested ROLLBACK must undo only the inner scope.10python3 -m txkvbench --db bench.db exec'SET x 1; BEGIN; SET x 2; GET x; ROLLBACK; GET x; COUNT'\11|tr'\n'' '|grep-qE'2.*1.*1'
That last one is the whole philosophy in a line. 2 1 1 is the only correct output: the inner GET x sees 2, the ROLLBACK discards it, the outer GET x sees 1 again, COUNT is 1. A model can write a beautiful transaction class with a passing unit test and still print 2 2 1 here because it merged the savepoint the wrong way. The harness catches that; the model's self-graded suite does not.
The two modes
The harness runs in two families, tagged on every ledger record so you always know which one produced a result.
Benchmark
Exploratory
Prompts
FROZEN suite — byte-identical every run
Invented fresh by the brain each session
Question it answers
"Is variant A better than variant B?"
"Where does the swarm break, and can we fix it?"
Grading
Deterministic golden checks only
Golden checks + fleet health + an AI judge
Needs an API key
No (grading is model-free)
Only if the brain is Claude
Output
BENCHMARK.md + paired CSVs
Per-session report.html + monitor findings
Reproducible
Yes, that is the point
No, that is the point
7 rows × 3 columnsHeader row enabled
Use benchmark mode to measure one model build against a fixed bar and compare variants. Use exploratory mode to discover where the swarm breaks, seed apps, exercise MCP tools, and auto-tune the fleet.
Tutorial part 1 — your first benchmark
Benchmark mode replays the frozen suite through graduated tiers, tagged with a --variant label. Start with a smoke run:
benchmarkbash
1python -m harness bench --tier smoke --variant mlx # 1 spec, 1 rep — "did I wire it up right?"2python -m harness bench --tier medium --variant mlx # 3 specs × 5 reps = 15 runs, a real signal3python -m harness bench --tier medium --variant gguf # same suite, same checks, the other build4python -m harness bench-report # head-to-head + writes runs/BENCHMARK.md5python -m harness bench-csv # export benchmark-runs.csv + benchmark-summary.csv
The tiers scale the same three specs from a sanity check to a paired A/B you can trust:
Tier
Specs × reps
Total runs
Use
smoke
1 × 1
1
did I break the harness?
light
3 × 1
3
one pass over every archetype
medium
3 × 5
15
a real per-variant signal
high
3 × 10
30
a paired A/B you can trust
extreme
3 × 10
30
high, under harder-spec pressure
6 rows × 4 columnsHeader row enabled
The runs interleave — spec 0, spec 1, spec 2, spec 0, … — so a campaign cut short still covers all three archetypes evenly. The suite is graded entirely by the deterministic golden checks, so benchmark mode needs no brain and no API key.
Warning
The frozen suite is frozen on purpose: editing a prompt or a golden check silently breaks every prior comparison paired against it. New coverage goes in a new suite. Note too that bench accepts --turns, --no-judge, and --tweak but ignores them — they only do something in exploratory mode.
Reading the benchmark output
bench-report regenerates runs/BENCHMARK.md from the ledger (so MLX and GGUF accumulate into one file) and the two CSVs give you the raw numbers:
benchmark-runs.csv — one row per run: variant, tier, spec, app_slug, session_id, wall_secs, functional, overall, checks, swarm, cluster, diagnostics, tasks_done, tasks_failed, exit_code. The functional column is the honest bottom line: 1 only if the checks dimension passed — i.e. the app actually did the golden thing.
benchmark-summary.csv — per variant×spec plus an ALL roll-up: pass percentages and wall-clock median/mean/p90/stdev/min/max.
Tutorial part 2 — an exploratory session
Exploratory mode is where a brain invents work and reacts like a real user. The simplest entry point is one session:
exploratorybash
1python -m harness once --archetype heavy-spec --turns6# a densely-specced new app + follow-ups2python -m harness once --archetype minimal-spec # a terse one-liner; score the gap-filling3python -m harness once --archetype continue-existing # extend a previously-green app, with regression checks4python -m harness loop --n3# cycle the three archetypes, seeds 1000, 1001, 10025python -m harness report # print the last 30 sessions from the ledger
once takes --archetype (default heavy-spec), --persona, --seed (default 1), --turns (falls back to the config's default_turns, 6), --no-judge, and --tweak. loop runs --n sessions (default 3), cycling the archetypes and bumping the seed each time.
The three exploratory archetypes
Each archetype tells the brain to invent a different kind of task, chosen to stress a different swarm weakness:
Archetype
What the brain invents
Stresses
heavy-spec
a new app with a dense, explicit spec — features, file/CLI surface, acceptance criteria — buildable in one shot
satisfying a fully-specified brief without drift
minimal-spec
a terse one-liner a lazy user would type, with the real engineering pushed into hidden requirements the swarm never sees, plus an expected MCP tool
inference from underspecification + tool use
continue-existing
a feature request on top of an existing green app, with regression requirements that its old behavior still works
code comprehension + non-regression
4 rows × 3 columnsHeader row enabled
For continue-existing, the substrate is a real app kept on disk from an earlier green session (the ledger tracks which apps are still passing). From turn two onward the brain issues follow-ups drawn from a fixed vocabulary — feature, fix, tests, refactor, mcp-feature, change-direction — or sets done to end the session early.
What one session actually does
Whether benchmark or exploratory, a session moves through six steps on the same evolving app workspace (apps/<slug>/, kept on disk).
1
open
the brain invents the opener: an app slug, language, prompt, visible and hidden requirements, and the deterministic checks it will be held to.
2
run
the harness shells out: goose swarm run "<prompt>" --output-format json in the app workspace, appending --mcp <server> for any expected tools.
3
collect
it joins three sources of truth: the swarm's JSON result, the structured .swarm/run-<id>.jsonl event log, and every worker's full session trace (opened read-only from goose's session DB by its logged session_id), plus a file snapshot of the workspace.
4
verify
the seven-dimension stack runs (next section). This is where the golden checks execute against the built app.
5
next_move
for another turn, the brain reacts like a real user and amends the same app; repeat to the turn budget or until it says done.
6
tweak
if a systemic fleet problem showed up, propose a scope-guarded knob change, apply it, re-run, and record the before/after.
Each turn is written to runs/<session_id>/turn-<n>.json, and the whole session to session.json + a report.html you can open in a browser.
The brains, and being the brain yourself
The brain is pluggable via SWARMGYM_PROVIDER (which overrides the config without editing it):
claude — the default. Uses the Anthropic SDK; the generator/vibe model defaults to Sonnet and the judge to Opus, because the verifier should be the strongest model in the loop.
local — an LM Studio endpoint running a local model such as Qwen 27B. It forces a large token budget because reasoning models spend tokens "thinking" before they answer.
operator — no key, no SDK: the operator driving the session (for me, Claude Code) is the brain, through a plain file handshake.
The operator handshake is the mode I lean on most — a capable brain steering the exploration at no API cost. It works through one directory:
runs/operator/text
1req-<n>.json # the harness writes a request, then blocks
2resp-<n>.txt # you write the answer here to unblock it
3PENDING.md # a running checklist of open requests
Each req-<n>.json carries { id, role, system, user, max_tokens, ts } — the full system and user prompts the brain would have received. You read it, produce the completion it asks for (which must contain exactly one JSON object of the requested shape), and write that to resp-<n>.txt. The harness polls every three seconds and returns your text verbatim; an empty file does not unblock it, and it gives up after thirty minutes. Kick one off with:
explore is the exploratory driver built for this: --n sessions (default 3), an optional --archetype (otherwise it cycles), and --tweak to let it auto-tune. It prints each session's verdict plus any monitor findings, and flags idle or starved nodes inline.
The verify stack — seven dimensions, four sources
No single signal is trusted. The verdict composes up to seven dimensions from four independent sources, and the run is only as healthy as the weakest dimension that matters. A run is fail if any high-severity finding or any fail dimension appears; partial if anything is partial or warned; otherwise pass.
Dimension(s)
Source
What it establishes
swarm
synthesized
did the run exit cleanly with nothing failed
checks
deterministic
the golden checks — build, run, tests, files, and tool-called, run against the real app
cluster
fleet log
per-device work distribution, starved nodes, retries, MCP calls
diagnostics
static scan
zero-tool "narration" tasks, and code smells (NotImplementedError, breakpoints, leftover TODOs)
requirements, code_quality, bugs
AI judge
are all requirements met, is the code sound (1–5), are there subtle bugs
6 rows × 3 columnsHeader row enabled
checks is the floor — the model-free core that cannot be argued with. command_succeeds is how build, run, and tests are graded; file_exists/file_contains/file_matches assert the shape of the tree; tool_called confirms an MCP tool actually fired.
cluster is the dimension whose findings decide whether the tweaker runs. A run can pass every check and still reveal that two of three machines sat idle — a throughput bug worth fixing.
diagnostics catches the local-model tell: a task that reported "done" with zero tool calls (it narrated instead of acting), or a breakpoint() and a stubbed NotImplementedError left in the shipped source.
the judge reads the actual code for what a grep can't see — requirement coverage, correctness, subtle bugs — and is deliberately kept honest by the deterministic floor beneath it. Turn it off with --no-judge when you only want the model-free signal.
Tip
The reason all of this exists is the failure mode local models specialise in: passing tests around a broken feature. The deterministic golden check on the real advertised command is the one lens that mechanically refuses to be fooled, so it is the floor and everything else builds on top.
Tutorial part 3 — tuning the swarm from what you find
Exploratory mode does not just score the swarm; it can improve it. Two model-free components handle this.
The monitor watches how work was spread across the fleet each session. It flags a node as starved (got zero tasks), idle (under 5% of its fair share), or underused (under half its fair share), and marks the session unbalanced if any of those hold. Those findings appear inline in explore output and in the report.
The tweaker turns a fleet finding into an action, but only within a hard scope guard — it will only ever touch swarm knobs, never core:
Allowed knob prefix
Example
How it applies
pool.
pool.workhorse.weight
runs goose swarm pool weight/enable/disable
planner_model
planner_model
proposed, applied manually
worker_max_turns
worker_max_turns
proposed, applied manually
max_attempts
max_attempts
proposed, applied manually
env.GOOSE_LOCAL_CONTEXT_CAP
env.GOOSE_LOCAL_CONTEXT_CAP
set as an env override on the re-run
env.GOOSE_MAX_BACKGROUND_TASKS
env.GOOSE_MAX_BACKGROUND_TASKS
set as an env override on the re-run
7 rows × 3 columnsHeader row enabled
Anything outside that list is rejected outright. When --tweak is on and a cluster finding appears, the tweaker proposes a change with a rationale, applies the ones it safely can (pool weights and enables, plus the two env knobs), re-runs the swarm, and records the before/after per-device distribution in the turn — a self-contained A/B you can read back later.
The ledger and every artifact
Every session lands in the ledger under ledger/: an append-only sessions.jsonl (the full record) and a queryable ledger.sqlite. A record carries the mode tag, the archetype and app slug, the turn count, the overall verdict, wall time, the per-dimension statuses, and — for exploratory runs — the idle-node report. python -m harness report prints the last thirty.
Per session, under runs/<session_id>/:
report.html — the turn-by-turn view: each turn's kind and overall badge, the prompt, every dimension badge, the diagnostics per-model table, a colored list of findings each with a fix hint, and a blue banner when a tweak ran.
session.json — the complete session including every turn.
turn-.json — the task, verdict, and run result for one turn.
The harness measures the effect of the swarm's own feature flags — the knobs on the goose local-edition side that change how the fleet builds. Flip one on, re-run the frozen benchmark, and read whether it moved the numbers. The load-bearing ones:
Flag
Default
What it does
GOOSE_SWARM_SMOKE
off
a post-build smoke oracle (does the entry point run) with one corrective fix attempt
GOOSE_SWARM_CONTRACTS
off
freezes signature-only module interfaces fleet-wide before build, to kill cross-module drift
GOOSE_SWARM_PREREVIEW
on
idle-node correctness pre-review of completed tasks
GOOSE_SWARM_JUDGE
on
an idle-model judge that re-reviews and re-dispatches a stuck task
GOOSE_SWARM_GOALS
off
distills app-level pillars at plan time and injects them into every worker so modules cohere
GOOSE_SWARM_SPLIT
off
runtime splitting of a too-big task into parallel children
GOOSE_SWARM_SINK_CAP_SECS
off
a wall-clock cap that cleanly finalizes the integrate-verify step
GOOSE_SWARM_REVIEW
off
a model-free AST wiring/drift review with a post-run wire-fix
9 rows × 3 columnsHeader row enabled
The flywheel
That is the whole point. A local-model swarm will always tell you it's done. swarm-gym is how you find out whether it actually is — and then, because every finding has a fix hint and every fix is a flag or a knob, how you close the gap and prove it on the same frozen ruler. Measure, diagnose, tune, re-run. It is that loop, run over and over, that moves the ceiling of what a fleet of small local models can actually ship.
thebash
1python -m harness bench --tier medium --variant mlx # measure2python -m harness bench-report # read the head-to-head3GOOSE_SWARM_GOALS=1 python -m harness bench --tier medium --variant mlx # flip a flag, re-measure4python -m harness bench-report # did it move?
