MetroLLM-Bench

Motivation

Running AI on infrastructure an organization owns is only viable if the model is small enough to own and capable enough to deploy. Whether that point has been reached is an empirical question rather than a matter of opinion, and it is the question this benchmark was built to answer.

A public-transit kiosk is a deliberate choice of testbed. Its output must be correct, since a wrong fare is a billing error. It runs under real constraints, often without a network, and it cannot send passenger data to someone else's servers. A model that handles that task well needs no network and no outside service to do it.

Benchmark design

The benchmark treats a language model as the decision-making runtime for a public-transit ticket kiosk. The model reads a plain description of how the kiosk should behave, then handles each request by calling tools for routing, fares, and live disruptions. Every answer is checked against deterministic ground truth, so a score reflects what the model got right rather than how plausible it sounded.

The cases span six real metro systems, from MARTA in Atlanta with 38 stations to the Beijing Subway with 414, and three distinct fare models. They reach well beyond simple lookups, into multi-turn dialogue, accessibility and policy questions, live disruptions, and adversarial inputs constructed to induce failure.

A scripted kiosk encodes its rules in code, so a station closure, a holiday schedule, or a new weather advisory usually means a code change and a release before the kiosk reflects it. A language model reads the same instruction as text, so an operator can describe the exception in plain language and the model applies it. The benchmark includes disruption cases of this kind across the six systems, among them a Taipei typhoon warning that suspends a line for high winds, a Doha sandstorm advisory, a San Francisco earthquake that suspends the Transbay Tube, and a Chicago polar vortex that escalates from cold-weather delays to a full closure. In each, the advisory is plain operator prose, and the model must interpret it and respond correctly, where a scripted kiosk would need the contingency written into its code in advance.

Cases

955

Metro systems

6

Capability categories

11

Scoring components

22

Tools

6

Train / held-out split

717 / 238

Results

A four billion parameter model we fine-tuned, shipped as a single 2.6 GB file, runs offline on a laptop, matches GPT-5.4 at maximum reasoning effort on this task, and exceeds both GPT-5.6 tiers on the core metric. On the held-out partition it scored 91.32 on Tier-1, against 91.37 for GPT-5.4 at its highest reasoning effort and 90.63 and 90.00 for GPT-5.6 luna and sol. At standard effort GPT-5.4 scores 2.15 points lower than the student.

Twenty-six models from six vendors were evaluated and twenty-three are ranked. The top eleven span 3.18 composite points, and the 2.6 GB student sits inside that group at rank ten, alongside models several times its size and three proprietary API rows.

Held-out leaderboard, 238 cases, ranked by composite score. The fine-tuned 4B student is highlighted. Three evaluated models are not ranked, the two Gemma 4 edge variants and Llama 3.1-8B, for exhausting the tool-call budget. Serving configurations, licences, and vendors are in the paper.

The claim is stated precisely. On the core Tier-1 metric the student matches GPT-5.4 at maximum reasoning effort and trails it by roughly one point on the composite. We read this as parity at the deployment frontier of small, self-hosted models, not as surpassing a frontier API.

The top of the table is compressed. Eleven models lie within 3.18 composite points, a range comparable to the single-run interval, so the order among the leaders is not a capability ordering, and serving configuration alone moves the Qwen 3.5-to-3.8 comparison by 2.7 Tier-1 points. The task is demanding but bounded. It rewards correct tool use and a strict output contract rather than the open-ended reasoning of a mathematics olympiad, and a task of that shape has a ceiling that several capable models reach; Muse Glimmer 30B leads the composite. GPT-5.4 keeps its edge where reasoning is the work, leading the temporal category by 13.7 composite points. The relevant finding is that a 2.6 GB model reaches the same ceiling on this task.

Capability also ceases to track size earlier than expected. The fine-tuned 4B, 9B, and 27B students fall within 0.4 Tier-1 points of one another, so the 2.6 GB model concedes almost nothing to the 16 GB one on this task.

A model this size runs on consumer hardware, not datacenter accelerators. Memory at decode is the binding constraint, and the fine-tuned students fit on a modern laptop or a single consumer GPU. Decode speed is bandwidth-bound, so it scales with the device. On a fanless M2 Air the 2B student sustains about 39 tokens per second, enough to reproduce the results locally, while the 9B student runs at around 190 tokens per second single-stream on an RTX 5090, comfortably fast for a live deployment.

A second result is less expected. Beyond a certain scale, fine-tuning degraded the model. The gain over each base model narrowed as the base grew, from seven Tier-1 points at 2B to a measured loss at 27B, with every training seed agreeing on direction at every size, three seeds at 2B and two at the other sizes.

Fine-tuning gain over each base model, held-out Tier-1, mean over training seeds, three at 2B and two otherwise. The benefit shrinks as the base grows and turns negative at 27B.

The full 955-case run, which carries the statistical power the 238-case held-out partition lacks, certifies both ends, a +1.72 Tier-1 gain for the 4B student and a −1.09 loss for the 27B. A rule-based baseline reaches 84.6 Tier-1 and locates the model's real advantage in policy adaptation, compound scenarios, accessibility, and temporal reasoning, the categories a fixed script cannot cover. The held-out confidence intervals, and which differences they do and do not certify, are reported in full in the paper.

Availability

The benchmark, the fine-tuned weights, and the evaluation harness are released under a permissive licence, so every figure above can be reproduced. The accompanying paper is published as a technical report (v1.2, September 2026) (DOI: 10.5281/zenodo.21893944) and on arXiv as arXiv:2609.10016 (cs.LG). The 955 cases are also available as a dataset on Hugging Face, continker/metrollm-bench, with the train and held-out splits of the paper.

Outlook

Models that handle work like this keep getting smaller, while the inference hardware in everyday devices keeps getting faster, with dedicated accelerators and NPUs now standard in consumer SoCs. On capable consumer hardware a model of this size already runs fast enough for live use, and each hardware generation raises that floor. Most operational software is built from bounded, well-specified tasks of this kind, and for those the model can run on hardware the operator owns, without a remote frontier API. The same applies to point-of-sale terminals, industrial controllers, and back-office workflows. Frontier models stay necessary where the problem is genuinely open-ended.