Level III · the protocol
You think you’ve found a pattern. Now what?
Eight questions. Seven have no right answer. One does, and it is the one almost everybody gets wrong.
There is no score at the end and there will not be one. A number would be the exact error this project is about, wearing the costume of its cure.
01 · observation
What did you actually see?
Not what it means. The part a camera would have caught.
An example of each
Weak — Fanfiction writers understand characters differently.
Better — Two stories use the same scenes and describe the same character as cruel and as frightened.
Every step came from somewhere on this site.
None of it was designed in the abstract. Each question is a place where one of these experiments went wrong, or nearly did.
- Fanfiction →Same evidence, different assumption, different person. Step 02 is this.
- Icarus →The events held still and the premise moved. Step 03 is this.
- Two models →Both fit everything; nothing available separated them. Step 04 and 05 are this.
- The seventh specimen →The options themselves were the wrong shape. Step 06 is this.
- The claims audit →Three of my own connections were fabricated and passed. All of it is this.
Does this actually help? Nobody knows.
A protocol that feels rigorous is not the same as one that works, and the genre it belongs to has a bad record. Most critical-thinking interventions that report success have moved a threshold rather than improved discrimination — people become warier of everything, reject more of both the good and the bad, and accuracy on the bad rises as a side effect.
So the study attached to this separates two quantities. Sensitivity: can you tell a fabricated connection from a defended one at all. Bias: how willing you are to accept anything. A protocol that raises the first is useful. One that only lowers the second has made you harder to convince without making you better at judging, which is worse than nothing.
I expect the null. If it comes back that way, that is a finding about a whole genre of thinking tools, and it stays on the findings page marked as one.
How it will be tested
- 1. Judge a set of structural claims cold, on the audit. Some are defended, some fabricated.
- 2. Half of sessions then work through this protocol. The other half spend comparable time on an unrelated task — so practice and fatigue are controlled rather than assumed away.
- 3. Judge a second, matched set.
- 4. Compare the change in sensitivity and the change in bias across the two arms.
Within-subject and pre/post, because between-subject arms on a site people choose to visit differ by who chose to arrive.
The confound I cannot fix from here
The fabricated claims and the defended ones share an author, and the author knew which was which while writing them. So “people accept the fabrications” might measure nothing about reasoning and everything about one person writing fluent fakes.
Partial remedy: participants also rate each claim on whether it names a mechanism and whether they could say what would falsify it. If acceptance tracks those surface features equally for real and fabricated claims, the confound is doing the work, and the analysis says so rather than reporting the headline.
The real fix is a second author writing controls blind. That is not built, so it is not claimed.