Structured Decisions on the Edge: a 27B on Jetson Thor Scores 0.958 to a Cloud Model's 0.883 — and Still Loses on Calibration
Structured decisions — sorting a piece of text into one of a few options and attaching a confidence to it — are becoming their own model category. TypeSafe calls it System One, and its flagship model Jev does nothing else: you pass state and a typed question, you get a typed answer plus probabilities. No prose.
Which raises an obvious question: does this work actually need the cloud? Can an open model on an edge box, under constrained decoding, do the same job?
I benchmarked a Jetson AGX Thor against Jev to find out. The results are more interesting than I expected, but the most valuable part is not the scoreboard — it is the three traps I hit along the way, each of which lets constrained decoding fail silently while the benchmark data looks perfectly healthy.
Setup
| Cloud | Edge | |
|---|---|---|
| Model | TypeSafe Jev 1.13.0 | Qwen3.5-27B / Qwen2.5-7B-Instruct |
| Hardware | vendor serverless | Jetson AGX Thor, 122 GB unified memory |
| OS | — | JetPack R38.2.2, Linux 6.8.12-tegra |
| Serving | official HTTP API | vLLM 0.19 (official Jetson Thor image) |
| Precision | undisclosed | BF16 |
| Billing | $0.042 / 1M input tokens, output free | hardware up front plus electricity |
The task set is 24 industrial fault reports to be routed to one of three maintenance teams (mechanical / controls / safety). Ground truth was labeled by hand by the domain owner, not generated by consensus among large models — the latter measures "who agrees with the big model", which is a different question.
Three things were done for fairness:
- The local models receive the criteria descriptions verbatim as Jev receives them. A local model given only the bare question would be losing on prompt content, not capability.
- Every item is repeated 3–5 times, and flip rate is reported next to accuracy.
- Cloud latency includes the network round trip and says so. It is the honest number for someone choosing between the two.
Calibration always uses the probability mass on the predicted option, never the vendor's confidence field. They are different quantities: on a near-tie Jev returned a top probability of 0.46 and a confidence of 0.19. Mixing them compares a heuristic against a distribution.
Headline results
| acc | flip | ECE | Brier | p50 | |
|---|---|---|---|---|---|
| Jev 1.13.0 (cloud) | 0.883 | 0.042 | 0.099 | 0.164 | 1141 ms |
| Qwen3.5-27B (Thor) | 0.958 | 0.000 | 0.149 | 0.134 | 873 ms |
| Qwen2.5-7B (Thor, guided) | 0.625 | 0.000 | 0.367 | 0.725 | 295 ms |
| Qwen2.5-7B (Thor, rank classification) | 0.625 | 0.000 | 0.195 | 0.574 | 415 ms |
The 27B on the edge takes accuracy, Brier and latency, and loses ECE.
That split says something. Calibration is TypeSafe's stated selling point, and it holds up: Jev is right less often, but it knows better when it is unsure. The edge model is more accurate and more likely to be confidently wrong.
What that means in practice:
- If you route on confidence (escalate low-confidence cases to a human), calibration matters more than accuracy, and the cloud still has an edge.
- If you take the argmax and act on it, the 27B on Thor is simply better.
Jev is not random, but its variance concentrates at the decision boundary: twelve calls on an unambiguous item returned twelve identical probability vectors; twelve calls on a near-tie returned eleven distinct ones, with the answer flipping between two options. Single-shot accuracy hides this entirely.
Trap 1: vLLM silently drops guided_choice
This was the expensive one.
vLLM 0.19 renamed the constrained-generation parameter from guided_choice to structured_outputs. The old spelling is accepted and ignored. No error, no warning.
The decisive test is simple — ask for options no model would produce on its own:
# guided_choice (legacy)
{"guided_choice": ["zorblax", "quixnar"]}
# → 'Thinking Process:\n\n1. **Analyze' constraint not applied
# structured_outputs (current)
{"structured_outputs": {"choice": ["zorblax", "quixnar"]}}
# → 'quixnar' applied
The danger is that failure looks exactly like success. As long as the model obeys the prompt on its own, you get valid option words, 100% schema compliance, and plausible accuracy. No check on output format can catch it.
I ran a full pass on Qwen2.5-7B before noticing. Re-running with the correct parameter:
| acc | ECE | Brier | |
|---|---|---|---|
| constraint not applied (legacy) | 0.625 | 0.366 | 0.723 |
| constraint applied (current) | 0.625 | 0.367 | 0.725 |
Essentially identical. Qwen2.5-7B is not a reasoning model; told to answer with one option name, it does, constraint or no constraint.
Which gives a rule that generalizes: you cannot verify constrained decoding with a small, obedient model. This bug only surfaced once a verbose reasoning model was in the loop.
My harness now sends a sentinel request with nonsense options before any guided run, and aborts if the response is not one of them, rather than recording data that would look valid and measure nothing.
Trap 2: thinking mode collides with a first-token constraint
The model that exposed trap 1 immediately produced trap 2.
Qwen3.5-27B is reasoning-capable, and its chat template puts it in thinking mode by default. Constrained generation demands that the very first token be an option word. Those two things fight: the model is set up to open a reasoning span and is forced to emit a verdict instead.
The cost:
| 27B configuration | acc |
|---|---|
| guided, thinking on (default) | 0.583 |
guided, enable_thinking: false | 0.958 |
One flag, 37.5 accuracy points. Omitting it raises no error.
That 0.958 has independent corroboration: the same model through ollama (free-text parsing, think: false) also landed on 0.958. Two unrelated paths, one number.
The debugging path is worth recording, because I killed two wrong hypotheses first:
- "The constraint removes reasoning space." No. The ollama run produced its 0.958 in three output tokens — it was not reasoning either.
- "I downloaded a base model by mistake." No. The chat template and generation config are both instruct-tuned.
- Thinking mode colliding with the first-token constraint. Verified on a single item, confirmed on a full re-run.
If you use constrained decoding with a reasoning-capable model, disable thinking explicitly. Otherwise you will conclude the model is bad at your task.
Trap 3: ollama silently ignores format on some models
The third variant of the same problem. ollama's format parameter works on qwen2.5:7b and is completely ignored on qwen3.5:9b — no error, free-text output.
qwen2.5:7b + format → {"department": "sales"} honored
qwen3.5:9b + format → "Here are a few options for..." ignored
Constrained-decoding support is per model, not per framework. Re-verify when you change models.
Rank classification carries a class prior
The other way to get probabilities is rank classification: append each option to the prompt, score the full sequence log-likelihood with echoed prompt logprobs, and softmax across options. In theory this yields a true P(option | state), semantically closest to Jev's probabilities. I started the project believing it was the more correct path.
Measurement said otherwise.
Across 24 tasks and 120 inferences, this method never once predicted safety — 0 of 6 safety items.
Three fixes, all failures:
| fix | acc | safety predicted |
|---|---|---|
| bare option (baseline) | 0.625 | 0 / 24 |
| length-normalized | 0.625 | 0 / 24 |
| option scored together with its description | 0.542 | 1 / 24 |
| clarified safety criteria | 0.542 | 2 / 24 |
The third fix not only failed, it moved the bias from controls to mechanical and raised latency 3.7x, to 1538 ms — above the cloud, erasing the edge's only advantage.
The mechanism is structural: every criteria description sits in the shared prefix, so the only text distinguishing the scored candidates is the bare option word. Improving the descriptions never reaches the comparison. The method under-uses the schema by construction.
So the conclusion inverts: use the engine's native constrained generation on the edge. Its probabilities are reconstructed from generated tokens rather than exact, but a clean probability attached to a systematically wrong answer is worthless.
Schema text is a precision/recall knob
The original safety category was defined by device domain (safety circuits, interlocks, light curtains, E-stop chains), while mechanical and controls were defined by failure mechanism. That makes "a mechanical failure on a safety device" ambiguous by construction — a guard-door switch with a bent actuator, for instance.
Widening the safety criteria to explicitly cover any fault on a safety-rated device, regardless of mechanism:
| original | clarified | delta | |
|---|---|---|---|
| Jev | 0.883 | 0.903 | +2.0 |
| Qwen2.5-7B guided | 0.625 | 0.708 | +8.3 |
| Qwen2.5-7B rank classification | 0.625 | 0.542 | −8.3 |
It fixed the targeted item and pulled one neighbouring controls item into safety.
So the accurate statement is: widening a category recalls its misses and absorbs its neighbours. It is not a free win. And it does not reach rank classification at all, for the reason above.
Cost: there is no break-even
This is where most edge-versus-cloud comparisons go wrong.
The cloud bills input tokens, and each call carries roughly 344 tokens of fixed overhead plus about 30 per additional question. The correct way to use it is therefore to pack several decisions into one call:
| cloud packing | cloud $/1M decisions |
|---|---|
| 1 question per call | $15.71 |
| 3 questions per call | $6.08 |
| 10 questions per call | $2.70 |
On the edge, one Thor serves 98,969 decisions/day at 873 ms. At $3499 over three years, $0.10/kWh and a measured 78.1 W, a fully loaded board costs $34.18 per 1M decisions — the edge's theoretical floor.
Even at 100% duty the edge is 2.2x the cloud's worst configuration and 12.7x its best. More volume buys more boards at the same rate, so the gap never closes. There is no break-even.
The bottleneck is neither electricity nor hardware price. It is per-board throughput.
Charging the cloud one call per decision — the edge-favourable accounting — overstates its cost by 6x and manufactures a break-even that does not exist.
The edge case is latency, privacy and offline operation. It is not price. That is a more honest conclusion than constructing a cost win, and a more useful one.
Energy: an axis the cloud does not have
Measured on a quiet machine with Jetson's tegrastats, running the 27B guided with thinking disabled:
- idle baseline 21.6 W
- under load 78.1 W
- 64.6 J/decision including idle draw
- 46.8 J/decision marginal
One sampling caveat: 59.6 seconds yielded only 47 samples, an effective interval of ~1.3 s rather than the requested 200 ms, because tegrastats buffers through an ssh pipe. The mean is usable; an instantaneous power curve is not.
There is no cloud counterpart, so this is reported as an edge-only axis, not a column in the shared table.
Other operational traps on Thor
Everything here was actually hit:
- Unified memory is not released promptly when a container exits.
freereported 96 GB used against 3 GB of actual process RSS, and the next container failed its startup memory check. Allocating a large buffer and freeing it forces reclaim (25 GB → 60 GB available). - vLLM's startup memory profiling breaks when co-tenants release memory mid-profile, raising
AssertionError: Error in memory profiling. Retrying usually works. - A weight shard can be truncated and only show up at engine start. A local Qwen2.5-7B copy had one 246 MB shard where 3.86 GB was expected, surfacing as
SafetensorError: incomplete metadata. Check shard sizes againsttotal_sizeinmodel.safetensors.index.json. - Watch out for benchmarking the wrong endpoint. Thor's ollama binds to localhost, so it needs an SSH tunnel — and my Mac was running its own ollama on 11434 plus a shim on 11435 that answers with
gpt-4oin its model list. Both respond convincingly. Verify the model list before trusting a run. - A local HTTP proxy will swallow LAN requests to the box and return 502. Bypassing proxies in the client is the simplest fix.
Conclusion
Back to the original question: can the edge do System One?
Yes, depending on what you care about.
- Accuracy: 0.958 on Thor at 27B, against the cloud's 0.883.
- Latency: 873 ms vs 1141 ms, before accounting for network variance.
- Calibration: 0.149 vs 0.099 ECE — the cloud wins. A model trained specifically for structured decisions really is better at knowing what it does not know.
- Cost: the cloud wins, with no break-even at any volume.
- Privacy / offline: something a cloud API cannot offer at all.
If your system gates on confidence, calibration is a hard requirement and the cloud still has a place. If you take the argmax and care about latency or keeping data on site, a 27B on Thor is already good enough, and then some.
But the thing to take away is different: constrained decoding has several silent failure modes, and each one leaves your benchmark data looking completely normal. I hit two of them in this project, and only cross-validation caught them. Before you ship, verify with sentinel options no model would produce on its own that the constraint is genuinely in effect.
What this does not cover
No quantization was tested. The original research question — whether quantization destroys probability calibration — was never reached, and the BF16 27B result is the reference a quantization sweep would be measured against.
That trade was deliberate. Measuring quantization levels on top of a silently broken constrained-decoding path would produce noise, not findings. Getting the measurement right came first.