MLX vs GGUF on Apple Silicon: Benchmarking the Same Local Model Two Ways

You've pulled a local coding model onto your Mac and you're staring at two downloads of the same weights: a GGUF build and an MLX build. Which one do you actually run? The folklore says MLX is faster on Apple Silicon but you pay for it in quality. We got tired of guessing — so we built a harness that makes local coding agents produce real software, grades that software by running it, and then pointed the whole thing at the exact same model in both formats. This is what it found, and how it works.

Why you can't just ask the model how it did

Local coding models are very good at declaring victory. The tests pass, the summary says done, the diff looks plausible — and the actual feature is quietly broken. If you grade a model on its own report, you are measuring its confidence, not its competence. So the first design decision was the most important one: everything is graded by running it. Build an inventory store, add two items, ask for the total, and check the number against a golden value. If the program doesn't run, or prints the wrong answer, it doesn't count — no matter what the model claimed.

The harness: swarm-gym

The harness is called swarm-gym, and it ships inside goose local-edition — our open-source fork of the goose agent, hardened for local models and extended with a multi-node build swarm. One run of swarm-gym does five things in sequence:

• Plan — a planner model decomposes the spec into a small DAG of subtasks (core module, CLI entry, tests, an integrate-verify step) with explicit file ownership.
• Execute — worker agents build those subtasks in parallel across the fleet, each confined to its own files, seeing the signatures of its siblings.
• Judge — a semantic judge watches each worker's live activity and intervenes only on real trouble (a worker looping on a failing test, or shipping a stub).
• Integrate-verify — a final sink task wires the modules together and runs a smoke gate: does the package import, does its entry point run.
• Grade — deterministic verification runs the built app against golden values and records everything to a ledger.

The whole run is captured — a structured event log, every worker's tool calls, per-device dispatch counts — so a result is never just a number; it's traceable back to what each agent actually did.