Skip to content
The Canon Is a Prior

Level II · the same question, asked of something else

Given the same ambiguous evidence, how do a person and a model move?

Not which is smarter. Not which is more human. Just: what happens to each interpretation as the evidence arrives.

Same ambiguous scenarios, to people and to a model, under the same constraint that the numbers total a hundred. Two predictions, written before any data existed:

  • A language model asked for the same distributions will report lower entropy than the human group average — it will commit harder.

    Wrong if — Model entropy matches or exceeds the human mean, or varies enough across prompt versions that no stable comparison exists.

  • Additional context will reduce the model's entropy more than it reduces human disagreement.

    Wrong if — Human disagreement falls at least as fast as model entropy, or the model's entropy rises with context in scenarios where the added fact is genuinely ambiguous.

Epistemic status: hypothesis. Stated in advance, not yet tested.
1 of 34

Two friends have been trying to arrange an evening for a while. One of them sends a message in the afternoon: “Are you coming tonight?”

The reply, forty minutes later, is a single word: “Sure.”

Written for this experiment — no real message was used

What does the reply mean?

Spread one hundred points across the readings. Raising one necessarily lowers the others — that constraint is the point.

Assign probabilities to each interpretation. The values always total one hundred percent: raising one lowers the others.

25%
25%
25%
25%
Total100%

Total 100 percent. Enthusiastic agreement 25, Reluctant agreement 25, Neutral agreement 25, Not really agreement 25.

Optional. Confidence in the spread itself, not in any one reading.

It starts flat. Leaving it flat means every reading is equally plausible to you.

Three things this cannot establish.

Not calibration. That needs a right answer. These scenarios have none, by construction.

Not understanding. This tests what numbers come out when you ask for numbers.

Not a result. One run is one sample from a stochastic process. Runs are stored and shown separately, because the spread between them is part of whatever the finding turns out to be.

Prompt wording changes what a model reports as confidence, so every run is pinned to a prompt version and versions are never pooled. Counts are on the method page.

Contribute a model run

This half needs a key. It does not need mine.

If you have an Anthropic API key, you can run a few scenarios and add the results to the dataset. It costs a few cents of your own credit and takes about a minute.

Every run records the model identifier and a prompt version, and runs from different prompt versions are never pooled. What gets stored is on the ethics page.