Jex 1.0, Part 3: The Audit Tax
Previously: Jex 1.0, Part 2, The Gate.
The one speedup the audit allowed, and the parity work that paid for it: Route B, batch invariance, and the 55-minute dry run that turned a projection into a schedule.
Route B
The 72-hour card has 259,200 seconds in it. The fast pass over the full suite eats about 9.4 hours of that, roughly 33,840 seconds, leaving 225,298 for reasoned rows. Against a measured ~9.16 s amortized per reasoned row, that pins coverage to a small window:
| coverage | reasoned rows | amortized budget per row | cut needed vs 9.16 s |
|---|---|---|---|
| 5% | 12,022 | 18.7 s | none (current stack ≈ 40 h total) |
| 15% | 36,066 | 6.25 s | −32% |
| 25% | 60,110 | 3.75 s | −59% |
To clear 61.21 we need 10–15% coverage. That means cutting per-row cost by a third, or making tokens arrive faster. Five research lanes went to look (the local hot path; serving engines and speculative decode; reuse, caching and token budget; determinism and batch invariance; test-time compute allocation ) and came back as PERF_RESEARCH.md. This part is what survived.
The early-exit idea, and its null result
The most attractive Tier-1 idea was deterministic early exit. Every m tokens, read the option margin with a short cache-continued forward; stop when the argmax has been stable and the margin exceeds τ for k consecutive checks. The papers are good: DEER, Answer-Convergence and EAT all report 19–48% token cuts at preserved accuracy. So we built a checkpoint probe: 60 rows, positions up to 512, 1,481 checkpoints. The cheapest possible version first, because it costs zero extra forwards: a free rank gate that checks whether the </think> token is already the model’s top prediction at a checkpoint. If it is, the model is announcing that it’s done, and you can stop for free.
On the frozen Gemma-4-31B, the free gate fired once in 1,481 checkpoints. 0.07%. It saved one token.
The calibrated checkpoint readouts were too weak to be the alternative: choice accuracy 0.25–0.6, with heavy loss of already-correct rows. The lane closed with evidence. There was no token-cut lever on this model, which is a sentence we did not want to write after reading six optimistic papers. But it is better to write it now than to build a 37-hour run on top of it.
If you can’t cut tokens, move them faster
The decode-economics lane measured the exact card: single-stream bf16 vLLM about 22 tok/s, FP8 about 42, INT4-AWQ about 64, and batched FP8 380 average / 548 max tok/s aggregate. But the two obvious ways to use that are both closed. Weight quantisation modifies the frozen artifact, which the rules gate. Plain batching changes answers, because of the reduction-order problem.
That problem is the whole second era in one paragraph. The non-determinism at temperature zero is not atomics. It is the order in which RMSNorm, matmul and attention kernels reduce their partial sums, and that order depends on the batch shape. One independent study gave batch-invariant kernels 1,000 out of 1,000 identical outputs where the default produced 80 unique, at a ~20% slowdown. Another measured the argmax flip rate from batch noise as zero, but our own HF batch-8 experiment had a flip that cascaded into a different answer. The audits don’t care about your average flip rate.
So the decisive question was narrow: is there a shipped serving path that is batch-invariant on Gemma-4-31B and sm_120? The research said vLLM V1 with VLLM_BATCH_INVARIANT=1 was the only shipped candidate, with a fussy recipe: TP=1, TRITON_ATTN (forced by Gemma-4’s heterogeneous head dimensions), cudagraph_mode=FULL_DECODE_ONLY (the default piecewise mode breaks BI), prefix caching off, no speculative decode, bf16 KV, max_num_seqs 8/16/32. Overhead on sm_120 was unknown; elsewhere the brackets were 1.2–2.1×.
The reason to care is not elegance. The maintainers re-run one request at a time. An engine whose answers depend on cross-request batch shape is a reproducibility hazard, full stop. BI deletes that class of risk, and nothing else on the menu did. Route B: vLLM V1 plus the flag, under the audit recipe.
The traps were catalogued before we hit them, which saved a couple of evenings. vLLM 0.27.x plus transformers 5.15 crashes on gemma-4-31B (pin transformers==5.14.1 or take a newer vLLM with the fix). Tensor parallelism above one breaks BI. The default CUDA-graph mode breaks BI, hence FULL_DECODE_ONLY. BI might want a newer torch than our 2.8.0. And Gemma-4 had its own bug reports on record (FP8 KV garbage, hybrid KV sizing, prefix-cache misses), which is why the plan demanded pinned versions and a golden-set replay at max|Δp| ≤ 1e-4 before any scale-out. The final provenance block reads: vLLM 0.30.0, torch 2.13.0+cu130, transformers 5.14.1, triton 3.7.1, CUDA 13.0, VLLM_BATCH_INVARIANT=1, VLLM_ENABLE_V1_MULTIPROCESSING=0, TRITON_ATTN, FULL_DECODE_ONLY, prefix caching off, max_num_seqs=32. Every one of those settings is load-bearing. The provenance file records them so a re-run is a reproduction and not an improvisation.
Building the bake-off harness was its own afternoon. harness/vllm_bakeoff.py generates a fixed set of divergence-prone prompts, runs the BI ladder at batch 1/8/16/32 with repeats, runs a no-BI control on the same warm VM, and computes token-level diffs against the batch-1 reference with first-divergence positions. The first smoke used 64-token generations and we labelled it, correctly, low-power. The bake-off raised it to 256 tokens across 32 prompts, which is what turned a clean-looking null into a powered positive. A determinism test is only as strong as its longest generation. Divergence cascades, and short sequences hide it beautifully.
There was a quiet warning in the single-stream numbers too. BI at batch-1 ran ~10 tok/s against HF’s ~18–20. The invariant path is slower exactly where an auditor’s one-request-at-a-time re-run lives. That is a tax on the audit, paid once per re-run, not on the admission latency: the maintainers measure the engine-level median, and the BI build is not the latency-bound part of the pipeline. Still worth knowing.
The first smoke proved nothing, and we said so
The first vLLM BI smoke (2026-10-02 17:46Z) came up clean: TRITON_ATTN accepted, FULL_DECODE_ONLY accepted, no kwargs dropped. The BI arm showed batch-1 == batch-8, run-to-run identical. Then the --no-bi positive control also matched. The 64-token generations were too short to expose divergence, so the whole smoke was underpowered. It proved the stack boots. It did not prove BI does anything. We wrote that down and queued the bake-off.
Throughput was already talking: BI batch-8 at 87 tok/s against 178 for no-BI, about 2.1×, and BI batch-1 sitting at ~10 against HF’s ~18–20.
The powered proof
Thirty-two divergence-prone prompts, 256-token generations, batch sizes 1, 8, 16 and 32, each repeated, with a no-BI control on the same warm VM. The result was exactly what the hypothesis needed:
- BI: batch-1 == batch-8 == batch-16 == batch-32, run-to-run identical, 0 mismatches.
- No-BI: batch-8 diverged by 1,965 token mismatches against batch-1, batch-16 by 2,490, batch-32 by 2,033. First divergence early: prompt 4 step 54 at batch 8; prompt 3 step 17 at batch 16 and 32.
We ran the same experiment twice, once in the vllm-smoke/vllm-bakeoff pair and once refreshed in the runs/vllm-spec pair after a zombie-GPU-memory problem, and the verdict held both times.
Say it plainly: without the flag, changing the batch size changes roughly two thousand tokens out of about eight thousand generated on a small adversarial set. With the flag, zero. BI is load-bearing, and the ~2× is the price of audit safety.
The ladder, and the wall that moved
With BI proven, the scaling numbers arrived. The BI ladder over batch 1/8/16/32: 10.2 / 81.3 / 156.8 / 311.0 tok/s (vLLM 0.30.0, TRITON_ATTN, FULL_DECODE_ONLY). The no-BI control hit 506.7 tok/s at batch 32. That is the tax, and its size is the premium you pay for reproducibility. For scale, HF’s batch-1 was ~16–20 and batch-4 ~57.
At BI batch-16, 156.8 tok/s, a 437-token reason row takes about 2.8 s. The 15% coverage gate, ~36k rows, is ~28 hours plus a ~9.4-hour fast pass, about 37 hours, inside the 72-hour card. The budget wall from Part 2 had moved. The new question was not performance. It was whether the vLLM engine’s answers matched the HF reference the banked run used. A different engine can be internally consistent and still be a different function. That is a separate test, and it has its own part.
The graveyard we added to
The second research cycle closed a few more doors, and it is worth naming them so nobody reopens them. HF CUDA graphs worked after the inference-mode fix, for a +6–15% gain, and every accelerated path changed tokens versus eager (static 65 mismatches at batch-4, graph 223, graph_compiled 243). Deprioritized: not drop-in, dead on arrival for the audit without an answer-level validation we did not have. Cross-request prefix caching: order-dependent, incompatible with BI. Weight quantisation: modifies the frozen artifact. FA2: unavailable on sm_120, so SDPA is the practical default. Multi-GPU: not a submission path.
The one that survived was the engine itself, under the recipe. Which is the actual point: the campaign did not find a clever speedup. It found the speedup that was allowed, and then rebuilt the operation around the constraint. The risk moved from throughput to parity, and parity is where Part 3 lives.
Batch invariance earned the right to batch; it did not earn the right to be the engine that scored. Moving the running pipeline onto vLLM changed which function computed the answer, and a maintainer’s re-run compares us to us. So before the wide gate could be spent, both halves of the pipeline had to be checked against the artifact they replaced, and checked on the maintainer’s side of the wire as well as our own.
Paying the Tax
Batch invariance makes an engine consistent with itself. It does not make it the engine that scored.
That distinction sounds like pedantry until you write it as a sentence a maintainer would. They re-run the submitted engine and compare answers. If we moved the fast pass and the reasoning pass onto vLLM and the numbers drifted, the drift doesn’t matter because we can’t reproduce ourselves; it matters because we no longer match the artifact we’re submitting. Those are two different tests, and we had only run the first one. So we ran two parity checks against stored references, one per half of the pipeline.
The fast pass: 98.92%
The fast pass is the one-pass label-slot readout over the panel: 14,669 questions, six catalogs. Reference on one side, the vLLM BI engine on the other.
| metric | value |
|---|---|
| common questions | 14,669 |
| argmax agreement | 14,510 / 14,669 = 98.92% |
| flips | 159 |
| mean |Δp| | 1.1% |
| p95 |Δp| | 3.3% |
| max |Δp| | 0.998 (one row) |
159 questions changed their answer. That sounds bad until you look at where they changed. The flips are near-ties (rows sitting on the decision boundary), and the probability shifts are tiny: a mean of about one percentage point, a 95th percentile around three. The disagreement is symmetric and small, not a systematic bias. We recorded the careful version of the conclusion: the baseline shift from moving to vLLM is small and near-tie-symmetric, and the production engine will be self-consistently vLLM, with run-to-run readout stability to be confirmed in the integrated dry run.
That last clause is the one that actually matters. A 98.92% agreement against HF says the move didn’t silently break the readout. It does not say vLLM reproduces vLLM. The maintainer re-run doesn’t compare us to Hugging Face. It compares us to us. Both numbers are needed and they answer different questions.
The distinction is not academic, and it is worth spelling out because it governed the last two days. The HF reference is a fixed artifact, so the 98.92% is a one-time sanity check we compute once and file. The vLLM-to-vLLM check is not a one-time anything. It has to hold on the maintainer’s hardware, at their batch size, in their request order, on a day we don’t control. We cannot run it for them. All we can do is make the engine’s numerics independent of the things they might change (which is what the invariance flag is for) and then be honest in the notes about whatever residual remains. That residual was the 159 flips, and it was smaller than anything we were going to do with the extra throughput.
The reasoning pass: 119 out of 120
Generating is harder to compare than reading a slot, because the outputs are long and greedy drift cascades. Reference: the HF arm’s B1 run over 735 questions. Candidate: a 120-question slice under vLLM with the exact recipe. The first attempt at this died when its SSH stream dropped after the fast pass; no usable artifact. We re-ran it on a fresh VM with a two-hour cap.
119 of 120 agreement. The single mismatch is worth the whole part:
run_id: 25:GPQA-Diamond:GPQA-Diamond:test:98
hf_choice: B (correct)
vllm_choice: D
gold: B
pre_choice: D
pre_margin: 0.24999999999999978
hf_post_margin: 0.625
vllm_post_margin: 1.375000014901161The fast pass already preferred D over the correct B by a margin of 0.25 in log-odds, a coin flip. The thought pass pushed both engines further toward D, but by different amounts, so one near-tie item landed differently. The record notes the same item flipped in both engine runs, which is the tell that this is a genuinely ambiguous question and not an engine defect. One disagreement in 120 coin-flips is the expected cost of changing the numerical substrate, and we will take it.
The runner needed two small fixes on the way. logprobs=len(ids) is required by vLLM 0.30’s validation rule. And the report serializer needed default=str, because vLLM config objects aren’t JSON serializable. The fast-pass run had already died once on TypeError: Object of type CompilationConfig is not JSON serializable, which produced an unreadable report and a false failure. The tooling is part of the experiment. That is the tax, and it never shows up in the paper.
Screwing the halves together
With both halves validated in isolation, we built the thing that actually runs: harness/vllm_pipeline.py. One pass does the fast pass, applies the coverage gate, generates a rationale for the gated rows, re-reads the options, folds the post-thought answer into the probabilities, and applies the full-cal temperature transform. It grew an opt-in --panel-catalogs flag (default = the pilot six, unchanged) so the same code could take a full manifest. A bootstrap wired the environment and the upload path.
Its provenance block carries the entire audit recipe (vLLM 0.30.0, torch 2.13.0+cu130, transformers 5.14.1, triton 3.7.1, CUDA 13.0, VLLM_BATCH_INVARIANT=1, VLLM_ENABLE_V1_MULTIPROCESSING=0, VLLM_USE_V1=1, VLLM_ATTENTION_BACKEND=TRITON_ATTN, TRITON_ATTN, FULL_DECODE_ONLY, prefix caching off, max_num_seqs=32) so a re-run reads as a reproduction. The first integrated run was a detached nohup on the VM. A 400-row GSM8K smoke passed the fast pass, the full panel started, and an earlier rerun promptly died after its own fast pass when the SSH stream dropped. The recurring failure mode. The v2 run landed.
The dry-run fold
The dry run is the closest thing we had to a production number by 2026-10-03. The integrated pipeline, the pilot panel, 5% coverage:
| metric | value |
|---|---|
| panel questions | 14,669 |
| gated | 736 |
| reasoned | 736 |
| folded | 736 (0 fold-missing) |
| fixed | 430 |
| flipped | 47 |
| fix rate | 93.9% (430 / 458 baseline-wrong-gated) |
| panel-scale delta | +3.252 |
| full-index delta | +0.498 |
| wall time | 3,322 s (~55 min) |
Every error counter was zero: 0 context refusals, 0 key-fallbacks, 0 unsupported, 0 readout errors, 0 batch errors, 0 mechanism failures, 0 unmatched rows. And the full-index delta of +0.498 is the pilot’s B1 result reproduced end to end on the production-shaped pipeline. The integration did not lose the effect.
Fifty-five minutes for the panel is the number that turns a projection into a schedule. The same code at 5% over the full edition, then at 10–15%, is arithmetic against the throughput ladder. The dry run also confirmed the full-cal transform composes with the fold without touching choice-scored benchmarks, because temperature is argmax-invariant, exactly as advertised.
The per-catalog shape is worth keeping, because it shows the pipeline’s gain is the same shape as the pilot’s and not a different effect that happened to land nearby:
| catalog | n | gated | baseline acc | policy acc | fixed | flipped | Δskill |
|---|---|---|---|---|---|---|---|
| 25 GPQA | 198 | 24 | .505 | .510 | 6 | 5 | +0.007 |
| 30 GSM8K | 2,638 | 218 | .827 | .874 | 129 | 4 | +0.057 |
| 32 MuSR | 756 | 37 | .634 | .640 | 11 | 6 | +0.011 |
| 43 CRUXEval | 570 | 33 | .793 | .826 | 21 | 2 | +0.053 |
| 44 CLadder | 5,000 | 152 | .745 | .759 | 86 | 19 | +0.027 |
| 58 BBH | 5,507 | 272 | .762 | .792 | 177 | 11 | +0.044 |
The reasoning-heavy catalogs gain; the knowledge-bound ones barely move, and GPQA’s tiny net gain is six fixes against five flips. Same profile as the pilot. The +0.498 is computed with the kit’s weights, chance levels and area structure, clustered by question. The pre-registered machinery: a 2,000-draw bootstrap with fixed seed 20260929, clustered by question rather than benchmark (n=6 benchmarks makes benchmark clustering wrong), with panel share asserted at 0.153 and verified at 0.15318 before any delta was reported. The same machinery that produced the pilot’s intervals survived the move unchanged.
What the tax actually bought
None of this added a point of score. The score was already known from the pilot. What the parity work bought was the right to spend: to run a wide gate fast enough to matter, on an engine a third party can reproduce.
The chain by this point is short enough to say in one breath: adaptive thinking is worth ~+1.5–1.9 index at modest coverage; the ceiling grows with coverage; coverage is bounded by throughput; throughput is bounded by the audit, which forbids batch-dependent numerics; and the only shipped path that makes throughput safe is vLLM BI. Every link is measured. The links that remain are the ones with deadlines attached: budget, clock, and the kind of thing that happens at 3 a.m. when nobody is watching.
Which is where the story is, as it happens.
Next: Jex 1.0, Part 4, The Rewrite A sleeping laptop burns twenty dollars. The wrapper can’t recognize success and leaves VMs running. The production bundle gets built anyway, the calibration starts, and the case for the rewrite (and its ceiling) gets made.
