ASSAY-001 / INDEPENDENT VERIFICATION
Jev, checked.
Half of it holds.
On September 15 TypeSafe AI launched Jev and wrote that it returns calibrated probabilities and never makes type errors. We froze a protocol before we had access, ran it once, had it re-scored blind on a different model, and are publishing exactly what came back: calibrated on one of two corpora; zero type errors in 8,576 responses. Nobody asked us to. TypeSafe had no edit and no veto.
THE VERDICT
Two claims. Two corpora.
Human-labeled, third-party intent corpora, pinned by revision hash before the first query. One run per corpus, no retries, label names only, no prompt tuning.
| Claim (verbatim, typesafe.ai) | Banking77 77 intents · n=3,080 | CLINC150 150 intents + out-of-scope · n=5,496 |
|---|---|---|
| “Calibrated: higher confidence means higher accuracy.” — pass ECE ≤ 0.05 | ECE 0.0936 — does not hold overconfident in every bin below 0.9 · accuracy 0.7977 · Brier 0.3088 | ECE 0.0204 — holds within 2.5 points of the diagonal in every bin · accuracy 0.8812 · Brier 0.1846 |
| “The model never makes type errors.” — pass 0 violations | 0 in 3,080 — holds | 0 in 5,496 — holds |
Contract: choice is the highest-probability option | 2 exceptions, both within 0.01 of a tie | 1 exception, within 0.01 of a tie |
| Observed latency, end-to-end from central Texas (ms) | median 382.3 · p95 553.7 · p99 852.1 | median 385.9 · p95 615.3 · p99 845.1 |
The sentence that travels, whole: on CLINC150 Jev’s probabilities were calibrated (ECE 0.0204); on Banking77 they were not (ECE 0.0936, systematically overconfident). Across 8,576 responses there were zero type errors. Accuracy is reported, not judged — both corpora are public and old and may be in Jev’s training data; calibration on this distribution is what was measured.
RELIABILITY
What “overconfident” looks like.
Each point is a confidence bin: how sure Jev said it was, against how often it was right. The dashed line is a perfectly calibrated model.
| Banking77 bin | n | mean prob | accuracy | gap |
|---|---|---|---|---|
| (0.1,0.2] | 1 | 0.14 | 1.0 | 0.86 |
| (0.2,0.3] | 6 | 0.275 | 0.1667 | 0.1083 |
| (0.3,0.4] | 40 | 0.3668 | 0.2 | 0.1668 |
| (0.4,0.5] | 92 | 0.4652 | 0.337 | 0.1283 |
| (0.5,0.6] | 176 | 0.5526 | 0.358 | 0.1946 |
| (0.6,0.7] | 181 | 0.654 | 0.5083 | 0.1457 |
| (0.7,0.8] | 206 | 0.755 | 0.5874 | 0.1677 |
| (0.8,0.9] | 249 | 0.8614 | 0.6667 | 0.1947 |
| (0.9,1.0] | 2129 | 0.9857 | 0.9272 | 0.0585 |
| CLINC150 bin | n | mean prob | accuracy | gap |
|---|---|---|---|---|
| (0.2,0.3] | 14 | 0.2743 | 0.2143 | 0.06 |
| (0.3,0.4] | 59 | 0.3751 | 0.3898 | 0.0147 |
| (0.4,0.5] | 162 | 0.4603 | 0.4753 | 0.015 |
| (0.5,0.6] | 278 | 0.5526 | 0.5612 | 0.0086 |
| (0.6,0.7] | 279 | 0.6533 | 0.6667 | 0.0134 |
| (0.7,0.8] | 332 | 0.7589 | 0.756 | 0.0029 |
| (0.8,0.9] | 510 | 0.8605 | 0.851 | 0.0095 |
| (0.9,1.0] | 3862 | 0.9863 | 0.9614 | 0.0249 |
METHOD
Sealed before we had a key.
Protocol frozen 2026-09-17 (SHA-256 f8ce3f5b…a09b48) with the claims, corpora, metrics and pass criteria fixed — before any query. Corpora: Banking77 test split from the PolyAI GitHub at commit 57ec275d; CLINC150 plus test split from Hugging Face at revision 155b9c71; every file hashed first. One Choice question per item, option descriptions null, one fixed instruction per corpus, Jev defaults. Every request and response logged verbatim and sealed (b5c96bc1…, 5f5c58b2…). Positive controls — uniform-random, always-confident, oracle, broken-schema — had to go red and green before a single Jev number was computed. Then a second party on a different base model re-scored the sealed responses from the written spec, blind to our numbers, and matched every field exactly. No model anywhere in the scoring.
AMENDMENTS — NUMBERED, NOT SILENT
Two, and what they changed.
Amendment 1 (before any query): which field “calibrated” refers to — the returned probabilities are pass/fail; TypeSafe’s derived confidence is reported, not gated. And the harness was built by the auditor because the build lane wasn’t present at midnight, so an independent re-score became a condition of publication.
Amendment 2 (after the run): Jev rounds probabilities to two decimals, so on 513 responses they summed to 0.99. The frozen scorer called that a type error; it is rounding. Tolerance widened to 0.02, every item scored, both scorer outputs published. Verdicts unchanged under either rule.
Also disclosed: 4 CLINC items unscored because DNS failed on the auditor’s laptop mid-run (ours, not theirs; not retried). A 3-item connectivity check per corpus, excluded from scoring. Banking77’s max calibration error (0.86) is a single-item bin; ECE is the gated field.
THE LIMIT
What this is not.
A result applies to the artifacts and criteria examined — jev-latest on 2026-09-18, Choice questions with null option descriptions, these two corpora. It is not a statement about Jev on any other task, corpus, or day, and it is not a comparison to any other model. TypeSafe is invited to reproduce it; the harness, protocol, raw responses and both scorers are in the receipt.
Cost of the run: about fifty cents of input tokens at TypeSafe’s published price. Output tokens were free, as advertised.