Skip to content
The Canon Is a Prior

Level III · justification

More evidence is not the same as better evidence.

Two explanations. Both account for everything. The question is not which one you like — it is what you would have to go and look at.

Epistemic status: hypothesis. Stated in advance, not yet tested.
  1. Two models
  2. Prefer
  3. Design a test
  4. Predict
  5. Outcome

A student who was talkative in the first four weeks of a seminar has said almost nothing since.

  • The change is abrupt — week four to week five, not a gradual decline.
  • Their written work over the same period is unchanged in quality and length.
  • They still arrive early and sit in the same seat.
  • In week four, a contribution of theirs was corrected by another student, at some length.

The correction did it

Being corrected in public made speaking costly. They still prepare, still attend, and now they weigh contributing against the risk of that happening again.

Something outside the seminar

Something changed in week five that has nothing to do with the seminar. Attendance and written work are unaffected because those are habits; speaking takes energy that is currently going elsewhere.

Both of these account for every line of evidence above.

Neither is wrong yet. Nothing you have been given separates them.

Why this is the hardest instrument here

Everywhere else on this site the measurement is how far a belief moved. That is Level II, and Bayes’ rule describes it completely — given a hypothesis set and a likelihood, the update is forced.

This page is Level III. It asks what would make one of two surviving models better supported than the other, when nothing you currently have separates them. No updating rule answers that, because the answer is not in the evidence you have. It is in the evidence you would have to go and get.

What this can measure

A proposed test either separates the two models or it does not, and that is a fact about the test rather than a judgement about the person. So unlike almost everything else on this site, there is a gradable response here.

Several of the non-discriminating options were written to be vivid, specific, and clearly relevant — because that is what makes more-evidence feel like better-evidence. If people reliably pick those, that is a finding, and it is the one this instrument exists to look for.

Where the cases are weak

Two cases is not a study. The candidate tests are written by one person who knows which answer is correct, which is the classic route to an instrument that measures its author’s intuitions. And a stipulated outcome is not an observation — nothing was run; the world is a paragraph I wrote.

What survives all that is narrow and still worth having: whether people distinguish confirming evidence from discriminating evidence when both are on the same screen. The page that attacks this project is the right place to press on it.