CHAPTER 10 / THE REGISTER
Fourteen people measured Jev.
Here is what each one measured.
In the six days after TypeSafe launched Jev, at least fourteen independent parties published a calibration measurement of it — none commissioned, most one day and one run, scattered across GitHub, a blog and a forum. This page collects them. It does not re-score them, rank them, or average them: it records what each measured, on how many items, on which field, with how many bins, and whether the rule was written down before the run. Every number below was read from the author’s own results file, not from a summary, on September 20–21. Where a figure is not in the author’s file, the cell says so.
THE ONE CORPUS SEVERAL PEOPLE SHARED
Banking77, four ways.
Same public corpus, four independent runs, four different slices and setups. Read across, not down: the numbers are not comparable in size, and one of them is on a different field.
| Who · date | n | Banking77 ECE | Field · bins | Frozen first? |
|---|---|---|---|---|
| ASSAY-001 (ours) · 09-18 re-scored blind on a second base model | 3,080 | 0.0936 | chosen-option probability · 10 | yes — protocol hashed, pre-registered |
| AbdelStark · 09-17 the earliest we found | 100 | 0.054 | probability · Brier 0.179 | yes — git tag before inference |
| Running-Dolphins · 09-20 | 500 | 0.099 | p_max · accuracy 78.2% | not stated |
| FirasSX914 / calibre · 09-19 | 500 | 0.130 [0.102, 0.164] | the confidence field, not the probabilities · 10 bins · Brier 0.352 | frozen before reviewing results |
Every Banking77 measurement of the probabilities is above 0.05 (the convention for “calibrated”); the two CLINC150 measurements — ours 0.0204 (n=5,496), Running-Dolphins 0.028 (n=500) — are below it. That split is the one thing this register can say in one sentence.
EVERYTHING ELSE
Other corpora, other findings.
Each row is one author’s file. “Frozen first” means the author says the rule was written before the run; “hash” means they published one; “no” means they say it wasn’t; “not stated” means the file doesn’t say.
| Who · date | What they measured | n | Result (verbatim figures) | Frozen first? |
|---|---|---|---|---|
| scienthoon · 09-19 | Public QA sets vs synthetic support tickets; refit temperature per question type | 500 / 1,221 / 2,000 / 900 | OBQA 0.024 · CSQA 0.032 · HellaSwag 0.029 · tickets 0.107 vs noise floor 0.024; choice T 3.29 and score T 3.40 (over), boolean T 0.66 (under); 15 bins | not stated |
| AHTOOOXA · 09-20 | English vs Russian, XNLI and MASSIVE | 600 + 600 | XNLI EN 0.032 [0.019, 0.056] → RU 0.096 [0.069, 0.126]; MASSIVE EN 0.066 / RU 0.071; 10 bins; 1,000-resample bootstrap | yes — SHA-256 in PREREG |
| fstandhartinger / jevbench · 09-19 | 242 routing/judging/policy/intent/ordinal/extraction decisions; nine systems | 242 | acc 96.3% [93.7, 98.4] · ECE 0.027 · Brier 0.056 · schema validity 100% strict and at 2% band · rerun 16 min later changed 3 of 242 (1.2%) | mappings and cost bases pre-registered |
| Gaurav-Gosain · 09-16 | Prompt injection; vulnerable-code pairs | 662 / 200 | injection ECE 0.0588, AUC 0.9927 · code ECE 0.1868, AUC 0.794 | not stated |
| anisselbd · 09-17 | Phishing emails, vs Claude Haiku 4.5 | 2,000 | Jev acc 62.6% [60.5, 64.7], AUROC 0.689, ECE 0.154 · Haiku 81.3%, 0.837, 0.097 · $0.038 vs $0.462 per 1k · two passes | not stated |
| AnthusAI | Constructed sentiment; post-hoc recalibration | 8,801 (5,280 fit / 3,521 test) | raw ECE 0.117, Brier 0.162; conf 91.4% vs acc 76.1% (Choice); Platt → 0.052; isotonic → 0.008 | no — says so; multiple framings on the same split |
| ickma2311 · 09-18 | Jev vs nano LLM vs GPT-5.6 Terra vs a supervised encoder, as an escalation gate | 208 / 200 | B77 acc 0.832 vs 0.793 / 0.875 / 0.933 (encoder); C150 0.870; AUROC 0.853 / 0.734; no ECE measured; verdict “AMBIGUOUS” | yes — kill/go criteria written first |
| manjunathshiva · 09-19 | Four small tasks vs frontier models | 200 | Jev 72.5%, ECE 0.161 · Fable 5.1 84.0%, ECE 0.064 · cascade Jev→Claude 82.5% at 37% of Claude’s cost | not stated |
| primeline / Robin · 09-18 | Four public sets + two production jobs | ~9,750 calls | at ≥0.9: “correct about 92% of the time… covers 73%”; ECE yes/no 0.012 · pick-one 0.086 · rating 0.254 | pass/fail bar written first; no hash |
| Mike Moore · 09-20 | Agent tool-call risk, four classes | 60 | 91.7%; ECE 0.0505–0.0712; “never returned 1.000 and was wrong” (40/40 at 1.000 correct) | not stated |
| Shuffle probe (ours) · 09-20 | Option-order invariance, Banking77 slice | 300 + 50 control | shuffled: 16 flips (5.3%); same order: 0 of 50; every flip below 0.9 both sides; |Δp| > 0.05 on 28.5% of stable items | rule hashed first; not an engagement |
WHAT THE REGISTER SHOWS
Three things, and no fourth.
One. Calibration is task-dependent and the sign flips by question type: in-distribution multiple choice sits near the noise floor; out-of-distribution choice, ordinal scores, emotion labels, non-English text and adversarial code go overconfident by two to seven times; yes/no goes underconfident. Two. “0.9” is not one number — accuracy above that line spans 81.6% to 99.8% across tasks (Running-Dolphins), so a threshold set on one corpus does not carry (calibre: 0.67 on Banking77 became 0.37 on Web of Science). Three. Post-hoc recalibration works when you have labels (isotonic: 0.117 → 0.008), which is another way of saying the number needs your data before code should act on it. There is no fourth finding: nothing here says Jev is “broken,” and several rows say it is fast, cheap, and type-safe, which nobody has disputed.
WHAT THIS PAGE IS NOT
Not verification. A register.
We did not re-run any of these, re-score any of them, or check their code. We read each author’s results file and copied the figures as written; where a cell says “not stated,” the file didn’t say. Sample sizes range from 60 to 9,750. Bin counts differ (5, 10, 15). Two rows score a different field than the others. Most are one day and one pass, and Jev itself changes about 1–3% of answers between runs, so a figure’s second decimal is weather. An author who finds their row misstated can open an issue on the ASSAY-001 repository and we will correct it and say so here. Our own rows are included at the same weight and with the same caveats.
Three writers said on September 16–18 that calibration was “the one claim nobody outside the company had tested.” Fourteen people had, or were about to. This page exists so that the next person doesn’t have to find them one at a time.