What Modulum does, across every variable we measured.

The cleanest possible isolation of platform contribution.

Same Gemma-4-31B-Q4 weights, same 50 prompts (idx 0..49) per cell, vanilla served via raw llama.cpp /completion on Hypernym's mirror endpoint, Modulum served via the full platform stack. p-values via two-sample Wald test for difference of proportions.

Mean Δ Platform across 9 cells = +11.78 pp. Three cells reach p<0.05 (qa1 64k, qa2 32k, qa2 128k). The qa2 32k lift (+28 pp at p=0.0031) is the load-bearing single piece of evidence the platform does real architectural work. The qa3 32k −6 pp is not significant; could be sample noise on N=50.

Headline accuracy with exact Wilson 95% CIs.

These cells use Modulum's full sample budget (N=100–500), not just idx 0..49. They represent the absolute capability ceiling of Modulum on a 31B-Q4 workstation model at each context length and task.

128k average across qa1/qa2/qa3 = 46.00 %. qa3 128k is the cell with the tightest CI (N=500): the 27 % accuracy holds at ±3.9 pp. The 3 errors on qa1 128k are 503-busy retries that recovered. All other cells: zero backend errors.

Where Modulum's qa3 slope is best-in-panel.

OLS linear fit of accuracy on log₂(context tokens) across cells with N≥20 and context≥8k. Lower magnitude = flatter decay = better long-context preservation. Modulum's headline slope advantage is on qa3, not qa1.

Critical engineering note: the qa3 slope advantage is partly base-model — vanilla Gemma-4 already decays at −4.0 pp/2× on qa3 (matching Opus 4.6). Modulum extends that by +1.5 pp/2× to −2.5 pp/2×. The platform contribution on qa3 is in both the intercept (absolute level lifted ~+10 pp) and the slope.

The compute cost basis of the accuracy lift.

Tokens-per-second medians across all 9 mask cells, both sides. The llama.cpp timings block exposes prefill_ms and decode_ms natively for Modulum + Vanilla; frontier API models do not expose these so the comparison is platform-only.

Decode tokens / second (output generation rate)

The decode story is non-uniform. Modulum is 19–30 % SLOWER on qa1 retrieval at short/mid context, but 17–22 % FASTER on qa3 at every context length. The compute cost basis of the qa3 accuracy lift is therefore negative — Modulum is both more accurate and faster on the hardest task.

Prefill tokens / second (context ingest rate)

qa1 short-context prefill is an outlier — Modulum runs 54 % slower than vanilla at 32k and 64k qa1. This anomaly is isolated to qa1 32k/64k; on every other cell prefill rates converge to ~3 % of vanilla. Hypothesis: the qa1 short-context cells were the earliest experimental runs (phase-1 with 78 backend 503-storms) and the timing data may include retry latency. Worth re-measuring on a fresh run.

Production-deployment signal: does accuracy degrade run-to-run?

Each cell's samples split into 3 equal slices by sample_idx (early / mid / late) and accuracy measured per slice. Cells with N≥100 are shown below.

Two cells show production-concerning drift: qa1 64k (−23 pp end-to-end, N=100) and qa3 128k (−10.5 pp, N=500 across two phases). Both are real — same direction across independent measurements. The qa3 32k cell shows the opposite pattern (improving with samples), which suggests the platform's behavior is sample-order-dependent on temporal-reasoning prompts.

The honest answer, finding by finding.

1. It really does help base Gemma-4 on the BABILong matrix — but unevenly.

   Average lift of +11.78 pp across 9 mask cells. Three cells reach p<0.05 at N=50 (qa1 64k, qa2 32k, qa2 128k). The other 6 trend positive but are below the noise floor at this sample size. The lift exists; the average is the right direction; individual cell claims need higher N to formalize.

2. The strongest single finding is qa2 at short context (+28 pp, p=0.003).

   qa2 32k Modulum 56 % vs vanilla 28 %. This is the load-bearing piece of evidence the platform does real architectural work — a 2× absolute accuracy doubling on 2-fact reasoning at the easiest context length, well outside sampling noise.

3. On qa3 multi-fact temporal reasoning, Modulum has the flattest decay slope of any tested stack (−2.5 pp / 2× context).

   Better than Opus 4.6 (−4.0), GPT-5.5 (−9.0), Gemini 3.1 Pro (−7.6), Grok 4.3 (−8.2). The "First, Not Lost" thesis is strongest here. Caveat: vanilla Gemma-4 already has a −4.0 slope on qa3 — Modulum extends it by +1.5 pp/2×; the rest of the qa3 slope advantage is base-model inherited.

4. Modulum decodes 17–22 % FASTER than vanilla on qa3 across every context length.

   qa3 32k: 49.5 t/s vs 40.7 t/s (+21.6 %). qa3 64k: 45.9 vs 38.0 (+21.0 %). qa3 128k: 40.2 vs 34.4 (+16.9 %). Hypothesis: the attention conditioning produces more focused, shorter outputs on multi-fact reasoning. The qa3 lift comes with no compute penalty.

5. The trade-off: 19–30 % slower decode on qa1 short-context retrieval.

   qa1 32k: 35.1 t/s vs 50.4 t/s (−30 %). qa1 64k: 33.6 vs 41.5 (−19 %). This is the platform overhead on the easiest task. Speed converges to vanilla by 128k.

6. Sustained-run drift on two cells is a production-deployment blocker.

   qa1 64k drops 23 pp end-to-end across 100 sequential samples (87.9 % → 64.7 %). qa3 128k drops 10.5 pp across 500 samples (32.5 % → 22.0 %, confirmed across two independent phases). Likely KV-cache state accumulation or attention drift. Needs diagnosis before partner deployment.

7. The qa3 32k regression claim is not significant.

   Modulum 22 % vs vanilla 28 % = −6 pp, p=0.49 — well above any significance threshold at N=50. Could be sample noise. Within-run data adds a second caveat: the same cell shows +25.9 pp improvement from early to late tercile, suggesting sample order strongly affects Modulum's behavior on short-context temporal prompts.

8. Modulum trails current-generation frontier by 38–43 pp on absolute 128k accuracy.

   128k task averages: Opus 4.6 88.7 %, GPT-5.5 84.0 %, Gemini 3.1 Pro 65.6 %, Opus 4.7 65.3 %, Modulum 46.0 %, Grok 4.3 21.1 %. The value of Modulum is not absolute-capability parity with hyperscaler-served frontier; it is workstation-scale qa3 slope preservation on an open-weight base.

9. We cannot evaluate Modulum at 256k+ context yet.

   Hypernym's Modulum endpoint is documented as capped at 128k. We did not run probe requests above 128k against Modulum to obtain a direct 4xx confirmation in SQLite. The strongest possible "Modulum holds where frontier collapses" claim is currently untestable. Important: the prior v1/v2 "Gemini 1M collapses to 0 %" claim is RETRACTED — those 50/50 failures were HTTP 429 spending-cap errors, not Gemini-model-context failures. Codex audit caught this; we have no published 1M data either direction.

10. Modulum's published deployment profile is workstation-scale single-GPU.

    Quoted to us by Hypernym; not independently measured in this bench. The previously cited specific figures (16 GB / 50–100× / 250–640 GB serving fleet) are not evidenced in our canonical data and have been softened or removed.