CHAPTER 09 / A PROBE, NOT AN ENGAGEMENT
Shuffle the options.
Five percent of the answers move.
A decision model takes a question and a list of options and returns one, with a probability. The order of the list should not matter. We took the first 300 Banking77 items from ASSAY-001, asked Jev again with the 77 options in a seeded random order, and compared every answer to the sealed one from September 18. Sixteen changed. Then we asked fifty of them again in the original order, the same day: none changed. The flips come from the order. Every one of them sat below a 0.9 probability on both sides. Rule written and hashed before the run; one pass; every request and response published.
THE RESULT
Two arms. One cause.
Same request shape as ASSAY-001 (jev-latest, same instruction, null option descriptions). Treatment: options shuffled with random.Random(20260920), a fresh shuffle per item. Control: original order, re-asked.
| Arm | n | answers that changed vs the sealed 09-18 answer |
|---|---|---|
| Options shuffled | 300 | 16 — 5.3% |
| Options in the original order, re-asked (control) | 50 | 0 — 0.0% |
The sentence that travels, whole: on the first 300 Banking77 items, reordering the options changed 16 of Jev’s answers (5.3%); re-asking in the original order the same day changed none of 50; all 16 flips had a chosen-option probability below 0.9 on both sides.
What flipped
Of the 16: the sealed answer was right on 9, the re-asked answer on 4, both wrong on 3. Mean probability of the chosen option, 0.59 before and 0.55 after. The dominant pair is card_arrival against card_delivery_estimate — labels a human would hesitate between too. These are the coin-flips, and the model’s own number says so.
Zero flips with both sides at or above 0.9. A floor at 0.9 — refuse to answer below it — would have refused all sixteen. On this slice, the floor alone catches every order flip.
What moved without flipping
On the 284 items whose answer did not change, the probability did: by more than 0.05 on 81 of them (28.5%), by as much as 0.38 on one. In the same-order control, 6 of 50 (12%). The number a program would act on is not order-invariant even when the choice is. That is a calibration statement, and it belongs in any receipt that carries the number: ask twice, print the spread.
Accuracy on the stable 284: 83.8%. Elsewhere this week, another evaluator reported 8.7% order flips on a different slice and setup; direction agrees, magnitude differs, neither is pre-registered.
WHY THIS IS A PROBE
What was fixed first, and what wasn’t.
The rule — corpus slice, seed, treatment, control, metrics, cost cap — was written to PROBE.md and hashed (97bf59c2…) before the first call. It was not pre-registered as an ASSAY engagement: one pass, 300 items, no second party re-scoring it, no positive controls run again. It took 350 calls, 41 seconds, and about ten cents. It is here because the finding is new, the method is honest, and the site’s rule is that a result ships with its limits attached. If it matters to you, run it: the harness, the seed, and every response are in the repository.
THE LIMIT
What this is not.
Three hundred items from one corpus, one day, one seed. Not a statement about Jev on any other task or about other decision models. Not a measurement of accuracy — ASSAY-001 is. The 0.9 floor is Banking77’s number, set from ASSAY-001’s published bins, not a constant; on other tasks it would be set on other labels. And “refuse the coin-flips” has a cost this page does not hide: at 0.9 the same floor refuses about 31% of Banking77 answers.
Cost of the probe: about ten cents of input tokens at TypeSafe’s published price.