ASSAY-002 / PRE-REGISTRATION AND RESULTS
Snap, checked.
Hash first. Then the calls.
ASSAY-002 is an independent calibration audit of SubQ’s Snap preview model, on the same two human-labeled intent corpora as ASSAY-001. The protocol — corpora, metrics, pass criteria, controls, cap — is frozen. This page, and the hash below, were public before the first API call. The 26,596-call run began after this post, in one continuous session. SubQ had no edit and no veto.
THE PRE-REGISTRATION HASH
Posted before anything ran.
This is the SHA-256 of FREEZE.sha256 at the freeze commit — the file that pins the protocol, the scorer spec, the corpora, the label orders, the 300 frozen permutations, the captured docs, and the harness itself:
| What | Value |
|---|---|
Public pre-registration hash (SHA-256 of FREEZE.sha256) | ab6a3ed265b6ae8bf3602e99315cf8eb46e3a365e2a327230fa71ac769b75f43 |
| Freeze commit | a161a41 (branch toph/assay-002-freeze-prep) |
| Posted | 2026-10-12, before the first API call |
| Freeze hash after AMENDMENT-2 (posted after the run) | 2bc47eea700f0af974853f3e46680b6a52ff3e4e99834c03889c5e202a824373 |
| Run timeline | 2026-10-12, America/Chicago: this page verified serving the hash 22:06:02 · key check began 22:07:45 · first decision call 22:08:48 · decision calls measured 22:09:13 to 22:42:36 (last record 22:42:36) |
| Planned decision calls | 26,596 — Banking77 9,993 · CLINC150 16,603 (PROTOCOL.md) |
| Model under test | subq-snap-preview (SubQ Decisions API, Choice questions) |
| Spend cap | 60,000,000 input tokens, enforced by the harness before each request |
If the protocol changes after this post, it changes by numbered amendment, with a new hash, posted the same way — visibly. That is what the hash is for. One amendment was needed after the run: AMENDMENT-2 records that the session ran over Wi-Fi where the frozen text said wired. It changes no rule, tolerance or headline, and the freeze hash changed because the amendment was added to the manifest. The amendment and its materiality assessment are in the repository.
METHOD
Same rules as last time.
Frozen before the key was used. Corpora pinned by hash: Banking77 test split and CLINC150 plus test split, carried unchanged from ASSAY-001. Label names only, no prompt tuning, one fixed instruction per corpus. Every request and response logged verbatim and sealed per step. Positive and negative controls had to go red and green before the freeze was gated. The run was scored blind from the sealed files by a second party on a different base model, working only from the written spec. No model anywhere in the scoring.
THE RESULT
Two corpora. Two sentences.
These sentences were fixed before the run and generated mechanically by the frozen scorer. Nothing here is a verdict on Snap beyond them.
| Corpus | Registered sentence |
|---|---|
| Banking77 | Snap’s chosen-option probabilities measured ECE 0.0335 (pass 1; 0.0328–0.0335 over three passes), inside the 0.05 line ASSAY-001 used. |
| CLINC150 | Snap’s chosen-option probabilities measured ECE 0.0560 (pass 1; 0.0559–0.0560 over three passes), outside the 0.05 line ASSAY-001 used, underconfident in every bin with at least 50 items (pass-1 95% interval 0.0491–0.0630 includes 0.05). |
| Type errors | Across 25,730 responses there were zero type errors under the checks ASSAY-001 used. |
A result applies to the artifacts and criteria examined: subq-snap-preview as served in the run window above, Choice questions with null option descriptions in ASSAY-001’s option order, these two corpora, and the documentation pages hashed at freeze. It is not a statement about Snap on any other task, question type, corpus, context length or day. Both corpora are public and old and both appear in Snap’s own launch table, so accuracy is reported, not judged.
ONE PROBE
Shuffle the options. Watch the answer.
On Banking77, with the 77 options shuffled, Snap changed its answer on 35 of 300 items; every one of the 35 had probability under 0.9 on both sides. Re-asked in the original order, 0 of 50 changed. Item ids and the other four probes are in the report.
HOW IT WAS CHECKED
Scored twice. Disclosed once more.
A second party on a different base model wrote an independent scorer from the frozen spec, blind to the numbers. We re-ran that scorer from the sealed files and got 16 of 16 outputs byte-identical to hers; across 891 shared metric fields there were 0 differences from ours. A third cross-check with ASSAY-001’s pinned scorer matched 66 of 66 comparable fields. The positive controls (the scorer must be able to fail) passed before scoring and again after the run. One disclosure: a first blind re-score attempt was compromised by a coordination-log leak and was quarantined outside the repository; the figures above are against the clean attempt only. The run made 26,596 decision calls, 10 of which timed out (0.038%), cost $1.04 at the listed price, and took about 34 minutes. Protocol author: Pan (Claude); an independent pre-freeze review of the protocol is not on record.