• AI

Jex 1.0, Part 2: The Gate

Previously: Jex 1.0, Part 1, The Frozen Engine.

A gate curve that promised +20 index, a latency wall at 40% coverage, and a pilot built to catch us fooling ourselves: the reasoning hypothesis, measured under a ten-euro cap.

The Gate Curve

A gate curve is a dangerous document, because it makes the fantasy look like a plan.

We built one offline, over the 285,242 stored choice questions from the banked run. The x-axis is how many questions you let the model think about. The y-axis, the tempting one, is the index you would gain if the gate selected exactly the right errors and every one of them got fixed. Here it is, priced at the time’s single-stream rate (256-token rationale, ~14.7 s per reasoned row, ~18 tok/s, $0.95/h):

gated questions	τ	rows reasoned	error value caught	share of all errors	hours	$	median	rows < 1 s
2%	1.88	3.9%	+2.44	7.4%	19.7	19	148 ms	89.9%
5%	4.13	9.2%	+5.53	16.8%	49.2	47	151 ms	85.5%
10%	6.38	16.9%	+10.41	31.7%	98.3	93	206 ms	78.9%
15%	8.50	24.1%	+14.30	43.5%	147.5	140	209 ms	72.3%
20%	10.63	30.9%	+17.46	53.1%	196.7	187	246 ms	65.9%
25%	12.63	37.4%	+20.15	61.3%	245.8	234	329 ms	59.8%
30%	14.44	43.8%	+22.46	68.3%	295.0	280	523 ms	53.6%
40%	17.31	57.6%	+26.07	79.3%	393.1	373	14.8 s	40.2%

A 5% gate with perfect repair is +5.53 index. Past the whole target. The ceiling climbs all the way to +26. The document is intoxicating and it is almost entirely fiction. Real gains are the ceiling times a measured fix rate, and our prior on that fix rate, from the literature and from our own gating, was 20–40%. So the 5% row is really +1 to +2, the 15% row is +4 to +6, and the whole thing depends on a number nobody had measured yet.

There is also a wall at the bottom of the table. The maintainers measure the engine-level median, not the per-reasoned-request time. The baseline median is 141 ms. It holds under 1,000 ms while fewer than roughly half of rows contain a reasoned question. At 25% coverage, 37.4% of rows are touched and the median is 329 ms. At 30%, 43.8% and 523 ms. At 40%, 57.6% of rows and the median explodes to 14.8 seconds. The practical maximum is somewhere around 25–30% coverage, and even that assumes we can pay the bill, which at single-stream rates we absolutely cannot.

The obvious move is a trap

Look at that curve and your first instinct is to build a better uncertainty detector. We tried. On all 240,442 stored questions we tested margin, top-1 probability, entropy, the top1−top2 gap, rank combinations. They came out statistically identical: AUC 0.762 to 0.765, and at 20% coverage they all caught 52.5–53.4% of the error value. An independent ablation from the field reached the same place: single-pass logit signals are near-monotone transforms of each other, and the ranking flips by model. Signal engineering is not a lever. We kept margin because it is scale-free across option counts, and moved on.

The actual lever is fixability, and it is a much less comfortable thing to measure. Every common gate answers “how likely is this wrong,” not “how likely is this error repairable.” The literature’s self-correction number is around 20%; retrieve-verify is 41–47%. Our prior was 20–35%. And note what that does to the wide gates: the +20 ceiling at 25% coverage needs a ≥40% fix rate to be worth its own cost. Cold, on a 31B, that is optimistic. The whole reason we built a breadth arm was to replace that assumption with a measurement.

We did get one clean design decision out of this. The gate should be a pre-registered coverage target (a quantile), not an absolute τ. A quantile is scale-free and survives the trip to v0.3’s private held-out set; an absolute cutoff is a property of this sample and a liability. So the pilot would run at 5%, 10% and 15%, with hard exclusions for ForecastBench, the noul probability catalogs, and any question with more than 26 options.

The external evidence was a push, not a win

Adaptive thinking was not our idea. It is Rune’s and GLiDE’s pattern: fast pass, read the option letters, and if the answer is uncertain, generate a short rationale, re-read with the rationale in context, fold it back in. We just wanted a uniform version (one gate, one prompt, no per-benchmark tuning) because anything fitted to the public set is a v0.3 liability.

The prior we attached to it was P(decidable positive) ≈ 20–35%, which tells you how much we trusted the idea going in. The evidence really does cut both ways. GLiDE (Fastino, closed, API-only, therefore ineligible) reported a 64.81 self-run on edition 0.2.1 against Jev 57.91, with CRUXEval +31, CLadder +32, GSM8K +24 skill. Rune, same Gemma family, reported +1.5 self-run on edition 0.2 (about +1.9 extrapolated onto 0.2.1) with 4.9 s median thinking. Both are exactly this cascade.

And then the warning label: Rune’s ForecastBench fell by 9.8. “A thought makes probabilities more extreme.” The forecasting literature agrees; CoT optimisation inflates Brier and ECE. The mechanism helps accuracy-scored decisions and hurts probability-scored ones.

That would be a deal-breaker on a suite full of Brier benchmarks. Ours has exactly one: catalog 48. Which turns the hazard into a one-line rule: never fold a thought into ForecastBench, or into the noul probability channels. Keep the one-pass probability there. We already had the temperature fix covering that catalog, so the two mechanisms could coexist without a fight.

The mechanics we had to get exactly right

Two research lanes pinned down how to build the thought pass so it was more than a plausible-looking generator. These are the details that separate a working arm from a demo that scores flat.

Generate first, then re-read. Never score before generating. Rune generates the whole thought and then reads the option letters at the next position. If you instead append “think step by step” and read the label logits before the model has thought, accuracy collapses: one study had ScienceQA dropping 80.8% → 45.5%, with a 93.5% first-slot bias. Our B1 arm copies the working design.

The Gemma-4 template was verified locally, on CPU, before any GPU spend. The cached chat_template.jinja (dated 2026-07-09) showed enable_thinking=true injects <|think|> at the top of the first system turn, thinking renders as <|channel>thought … <channel|>, and thinking-disabled prompts carry an empty thought block as suppression. <|think|>, <|channel> and <channel|> are all in tokenizer.json. Rune’s thought path is scoped to Gemma-4, the same family, so its template mechanics transfer. We could build and dry-run the prompts without renting anything.

Length is intermediate. Most of the gain sits in the first ~350–800 tokens, and sometimes the shortest chains beat the longest. We capped at 512 with a forced close.

One Gemma-specific caveat: an independent study reported that corrupted reasoning chains can raise the probability of a correct token in this family. That is a chain-corruption check we noted for the pilot if it was cheap.

Determinism recipe: one GPU model, one serving build, temperature=0, seed=0, VLLM_BATCH_INVARIANT=1 online or fixed ordering plus VLLM_ENABLE_V1_MULTIPROCESSING=0 offline, no spec decode or fused MoE kernels unless reproduced, and a golden-set replay with max|Δp| logged. That list looks like paranoia until you remember the maintainers re-run you one request at a time. It is also a preview of Route B.

Two measurements that changed the budget

At single-stream prices the gate curve was unaffordable. 5% coverage: ~$47. 15%: ~$140. 25%: ~$234. Plus it would take too long. The pilot had a hard cap of €10, with an abort above €12, and every spend needed sign-off.

Then we measured. On the actual card, Gemma-4-31B in vLLM ran about 22 tok/s single-stream bf16, 42 in FP8, 64 in INT4-AWQ, and (this is the number that mattered) 380 average / 548 max tok/s aggregate when batched in FP8. Re-priced, the widest gate (60k–72k questions × 256 tokens) cost roughly $10–30 and 8–20 GPU-hours, not $230–280. Cost is not the binding constraint. Determinism and engineering are.

The second measurement killed our prettiest optimization. The plan assumed the re-read was expensive, because it re-prefills 2–3k tokens per question. An A/B on 80 rows: fresh readout 0.242 s/row. KV-cache continuation would save about 0.061 s/row including its failures. On a ~9.5 s row, that is 2%. Decode is ~97% of the cost and prefill is nearly free on this card. Dropped. Where alignment held, 44 of 45 readouts agreed; the single flip was an exact probability tie. The budget would have to come from generating fewer tokens or generating them faster.

The control arm that could catch us fooling ourselves

The pilot spec went through an adversarial review and came out as arms: B0 the fast pass; B1 gated thought plus re-read; B2 a token-length-matched neutral filler; B3 a two-threshold variant; B4 two-stage escalation; B5 a P(True) self-check; W a breadth arm over all GPQA + CRUXEval plus a 200-question GSM8K slice. The decision rule was pre-registered and priced in full-index units: ADOPT at ≥ +0.20 with the interval clear of zero, INCONCLUSIVE if the upper bound stays above +0.20 without clearing it, REFUSE if the upper bound falls below +0.20. Power was stated honestly: the test can separate “≥40–50% repair” from “nothing,” and cannot resolve small gains.

The detail I am proudest of is B2. Its job is to replace the rationale with a neutral filler of the same token length, so B1−B2 isolates the content of the reasoning and B2−B0 exposes plain re-conditioning. We built it because the first interim read looked absurd: gated pre-accuracy 0.366 → post 0.888, 233 fixed against 19 flipped. Two things about that smell. Every thought that finished before the cap was correct (162/162), and the post readouts were saturated at a median top-p of 1.000. Maybe the model genuinely stops when it has solved the problem. Maybe it is echoing an answer it read inside its own rationale. Without B2 you cannot tell those apart, and you will believe whichever one makes you look smarter.

So: a real prior, a real ceiling, a cost that only closes if the serving changes, a latency wall near 30%, and a filler arm built specifically to catch us. That is where we were the night before the pilot. Then the wrapper dropped the SSH connection mid-run, and the bill came due.

The bill came due on a cap that had never been checked against the plan, and that is where the pilot picks up. The smoke that followed set a card on fire, the reset exposed a runaway, and the verdict, when it finally arrived, came with a control arm built specifically to tell us whether we had fooled ourselves.

The Pilot

The first GPU smoke of the reasoning harness died with the GPU holding 93 GiB it had no business holding.

torch.OutOfMemoryError: CUDA out of memory.
Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 95.60 GiB
of which 2.00 MiB is free.

Ninety-three gigabytes, on a 96 GB card, for two questions. The cause was our own greedy_generate: it ran without torch.inference_mode(), so every decode step kept its autograd graph alive through the whole KV-cache chain. The fix was two lines (wrap the loop in inference_mode, flip requires_grad_(False) at load) and the GPU went back to using a sane amount. It is the same lesson from the engine-repair week, arriving again: the mock path and the real path are different animals, and only one of them is on fire.

We had done the responsible thing first, mind you. The harness (harness/reason_pilot.py and friends) was built and CPU-validated before any GPU time: reference-rendering body byte-identical to the engine, a coverage gate, the thought prompt with enable_thinking=True, greedy generation to <channel|>, generate-then-re-read, and a B2 filler hook. Fourteen tests in the venv, stdlib-only tests passing without dependencies, --dry-run validated on actual gated rows. All of that, and it still OOM’d on the first GPU call.

The corrected smoke ran end to end: two gated questions, mean generation 15.75 s, 256 tokens, 16.25 tok/s. Both hit the cap and flipped A→B. That is n=2, so it proves the plumbing moves and nothing else. Then the speed matrix (8 prompts, cap 128, two repeats each) picked the generation configuration:

config	eligible	peak VRAM	tok/s	agreement vs batch-1
baseline batch-1	yes	59.1 GB	16.1	1.000
batch-4	yes	61.3 GB	56.8	1.000
batch-8	yes	65.2 GB	96.9	0.907
assisted (Gemma-4 assistant)	yes	60.2 GB	50.3	0.985
static / static+compile	yes (after python3-dev)	59.3 GB	7.2 / 14.6 (slower)	0.907

Read the last column carefully, because it decided the next three days. Run-to-run repeatability was 1.000 for every config; the drift is deterministic per stack, not random noise. Batch-8 is 70% faster than batch-4 and it changes one near-tie token at position 32 of one prompt, which then cascades. Batch-4 had zero flips over 1,024 tokens. Assisted had one flip at token 107. A second host reproduced the same flips at the same tokens, which is somehow worse: the divergence is deterministic, so you cannot average it away. Static and compile only became eligible after installing python3-dev, and then they were slower than baseline and still flipped tokens. Closed as a regression.

We integrated fixed batch-4 into the pilot (3.5×, token-identical on the sample) and kept the readouts at batch-1. That table contains the seed of the entire second era. Batch-8 was fast and unsafe; the maintainers compare answers, so batch-dependent speed is worthless. The only way to use big batches is a serving stack whose numerics ignore the batch shape. We did not have one. We would go get one.

The reset, and the number we were not supposed to see

The official pilot went up on Nebius (uk-south2, ubuntu24.04-cuda13.0, a preemptible RTX PRO 6000 under the project’s service account). Seven arms: B1/B2 at 5/10/15% coverage, plus W. About 9.8k generations.

Mid-arm, an SSH reset dropped the wrapper. The remote process tree survived, which was lucky, because it let us look inside. What we found was worse than a dead connection.

The seven-arm plan priced at roughly 20 hours and more than €20. Our cap was €10.

Nobody had re-priced the v3 coverage arms against the final arm list. The cost discipline was written into the spec and never checked against the thing we actually launched. This is the same failure mode as everything else in this project (an assumption that survived because nobody executed it), except this one was measured in dollars against a hard limit the user had set. The corrective was a smaller scope: keep the in-flight B1@5% (735 questions), add B2@5%, the W breadth arm (968 questions), and the determinism slice (16 rows). About 4.5 more hours, ~€4.4, campaign total near €7. The 10% and 15% coverage arms were deferred to their own pre-registered decision, where they belonged.

We relaunched as a detached continuation on the VM, with a local supervisor polling for a completion marker, fetching the arms, deleting the VM. One more correctness note surfaced during recovery: the official chance for catalog 30 (GSM8K) is the kit spec’s 0.175, mixed 4/10-choice, not the legacy 0.25 sitting in scores.json. The analysis treats the kit spec as authoritative. Small thing. It would have biased a benchmark that carries real weight.

The verdict

The arms came back to runs/reason-pilot in our private capture, and the pre-registered rule sorted them without any argument from us:

arm	coverage	gated	full-index delta	95% CI	decision	fixed	flipped	fix rate
B1 (thought + re-read)	5%	735	+0.496	[2.197, 4.294] (panel scale)	ADOPT	408	35	0.897
B2 (length-matched filler)	5%	735	+0.093	[−0.235, 1.533]	INCONCLUSIVE	168	123	0.369
W (breadth: GPQA+CRUXEval+GSM8K)	–	968	+1.038	[4.968, 8.544]	ADOPT	164	18	0.675

B1 fixed 408 and flipped 35. B2 (the same length of neutral text, no reasoning) fixed 168 and flipped

1. The B1−B2 content contrast is +0.403. The reasoning itself is doing the work; the rest is

re-conditioning, and the control arm caught the difference. Leakage was measured too: B1 leaked an answer pattern on 6.8% of rows (50/735), and excluding those left +0.482, still ADOPT. W leaked 4.9% and left +1.029, still ADOPT. B1’s untruncated rows fixed at 1.000 (172/172, zero flips); its truncated rows at 0.834.

And the cross-host determinism gate passed: 16 of 16 answers identical between Nebius and HF, rationale max_abs = 0.00. On the scratch harness. We wrote down that the pass would have to be re-earned on the shipping engine, because the scratch harness was not the shipping engine. That note would come back in Part 3.

Chasing the “too good” signatures

The raw numbers looked too clean, so we spent real effort trying to falsify them.

We patched the harness to persist rationale text and added a leak classifier, then peeked at 88 records. The rationales were genuine step-by-step derivations. A GSM8K chain walked from “Toni = 16” to “Shondra = 9” to “maps to option (H).” GPQA records had actual chemistry steps. 88 of 88 carried text, and only 2 of 88 had a naive answer-pattern leak. Head fixed/flipped/unchanged was 52/2/34.

The “too good” signatures had a mundane explanation: completion-selection. The model stops generating once it has solved the problem and rambles to the cap when it hasn’t. That is consistent with the truncation data. On B1, 60% of rows hit the 512-token cap (438/735); the untruncated ones were 100% correct, and the truncated ones still fixed at 83%. If the easy wins were an artifact of early stopping, the truncated majority would not have fixed that well. They did. The saturated post-readouts are just renormalisation over letter tokens, a consequence of how the readout is built. Nothing was waved away; we checked the boring explanations first and they held.

The per-catalog shape was a sanity check too. The gains land where a derivation should help: GSM8K, BBH, CRUXEval, CLadder, and flatten where the difficulty is knowledge-bound, GPQA and MuSR. A uniform lift would have been suspicious. This looked like reasoning.

Two more decisions from this stretch were made on evidence rather than taste. First, the gate cut. The B0 re-run reproduced the stored reference choices, and 5% coverage landed at a margin of 2.25 (the offline distribution’s fifth percentile), selecting 736 of 14,669 panel questions. Because that cutoff is a quantile, it is scale-free and transports to v0.3, a design choice made before the run, not after. Second, the cap. The decode smoke’s kill criterion was explicit: abort if throughput fell below 15 tok/s or mean length exceeded 300 tokens. The corrected smoke cleared it at 16.25 tok/s, and we then chose cap 512 over 256 precisely because 60% of rows were truncating. If the model rambles when it hasn’t solved the problem, a longer leash gives the hard items a chance to finish, and the truncation split gets reported instead of hidden. The rationale stats (median 509, mean 434) are the receipt.

And one guard deserves credit for never firing. ForecastBench (catalog 48), the noul channels, and any question over 26 options were hard-excluded from the thought pass. The suite has exactly one Brier-scored benchmark, and we kept the thought away from it. That is why the later full-cal temperature fix and the reasoning fold could compose instead of fighting.

One flag we recorded honestly: a parallel W2 arm on a separate performance VM read +1.031 against the official W’s +1.038. One record out of 968. A cross-VM instability of about 0.1%, small enough to ignore and exactly the size of thing a reproducibility audit is built to find.

What the pilot actually told us

The pilot is real but not sufficient. Adaptive thinking on the frozen 31B is worth +0.496 at 5% coverage, +0.403 of it content. The breadth arm at 3.4% of the suite’s GPQA+CRUXEval questions is worth +1.038. full-cal adds +0.31–0.35. Stack them and you get roughly 58.09 + 1.9 ≈ 60.0.

That is not 61.21. To clear Clef we need a wider gate (10–15% coverage) with the fix rate holding, or a cheaper way to pay for the gate we already have. The W arm’s 0.675 says the fix rate decays with breadth but stays comfortably above the 40% the wide-gate arithmetic demands. The engineering count is the problem: the pilot generated at fixed batch-4 in the HF stack, where 15% coverage is tight and 25% is impossible.

So the next era is the hunt for throughput that doesn’t break the audit. Which is a short way of saying we went looking for a serving stack whose answers don’t care how many requests you hand it at once.

---

Next: Jex 1.0, Part 3, The Audit Tax Two parity checks. The fast pass loses 159 argmaxes out of 14,669. The reasoning pass loses exactly one, and the one it loses is a GPQA coin-flip. Then the whole pipeline runs end to end in 55 minutes.