Method & data

Predictions, written first

The hypotheses above were written before any response was collected. They live in one file in this repository — research/lab.ts — so that putting the project under version control makes any later edit to them visible as a dated change rather than a silent one. If they do change, the change and its reason go in the log; the original is not replaced quietly.

1. H1

   Participants will move substantially on the first piece of evidence and less on later ones, even when later evidence is more diagnostic.

   Wrong if — Mean update magnitude is flat or increasing across evidence positions, or the ordering of update sizes tracks how diagnostic the items were designed to be rather than their position.

2. H2

   Disagreement between participants will not fall to zero as evidence accumulates. Some scenarios will end with two readings sharing most of the mass across the group.

   Wrong if — Mean pairwise disagreement declines monotonically toward zero in most scenarios, with a single interpretation taking the majority of mass for nearly every participant.

3. H3

   A language model asked for the same distributions will report lower entropy than the human group average — it will commit harder.

   Wrong if — Model entropy matches or exceeds the human mean, or varies enough across prompt versions that no stable comparison exists.

4. H4

   Additional context will reduce the model's entropy more than it reduces human disagreement.

   Wrong if — Human disagreement falls at least as fast as model entropy, or the model's entropy rises with context in scenarios where the added fact is genuinely ambiguous.

5. H5

   In the myth track, participants who see the engineering framing of Daedalus's warning will shift mass away from 'hubris' more than the equivalent-strength evidence shifts other readings.

   Wrong if — The hubris reading is stable against that item, which would suggest the canonical reading is anchored by something other than the evidence presented.

How

1. Stimulus

   Forty-two scenarios: eight paraphrased from public-domain classical texts, thirty-four written for this experiment. No real private messages are used, and no participant's own material is collected. Each scenario presents a context, one ambiguous act, three to five interpretations, and two or three evidence items.

2. Elicitation

   The participant distributes one hundred points across the interpretations using sliders that enforce the constraint. Raising one reading necessarily lowers another — the constraint is the instrument, since an unconstrained rating scale would let someone call everything plausible and measure nothing.

3. Evidence

   Evidence items are revealed one at a time in a fixed order per scenario. After each, the participant re-distributes. The previous distribution stays visible: this measures revision, not recall.

4. What is stored

   A random session identifier, the scenario id, the ordered list of distributions, an optional confidence rating, and elapsed milliseconds. No account, no email, no name, no IP address, no demographics, no cookies used for tracking.

5. Model runs

   The same scenarios are presented to a language model through an evaluation interface that requires numeric distributions summing to one hundred, plus a brief rationale per step. Every stored model response records the exact model identifier and a prompt version. Responses produced under different prompt versions are never pooled.

6. Analysis

   Movement is measured as total variation distance between consecutive distributions. Within-group disagreement is mean pairwise total variation. Commitment is Shannon entropy, normalised by the maximum for that scenario's option count. Kullback–Leibler divergence is reported only with explicit additive smoothing, and labelled as smoothed wherever it appears.

What is stored about participants

A random identifier, which instrument you used, ids and counts, and elapsed time. No account, no email, no IP, no analytics, and nothing typed — there is nowhere on this site to type. Payloads are closed rather than merely validated.

What this cannot show

• The options are mine

  Every experiment here hands you a fixed list of readings and no box to write a better one. You are distributing belief inside a hypothesis space someone else chose — which is the thing the project is about, performed on you, by me, unavoidably.

• Self-selection

  Everyone in this dataset chose to do an online experiment about belief revision in Greek myth. That is not a sample of anything except people who would do that. No claim on this site generalises beyond the people who happened to participate.

• Sample size

  Small, for as long as it is small, and the dashboard shows the exact number rather than a percentage that hides it. Nothing here is tested for statistical significance, because with these numbers a significance test would be theatre.

• Wording effects

  Each interpretation is a label plus one gloss, and both were written by one person. A reading that sounds generous and a reading that sounds petty do not start from the same place, however carefully the evidence is balanced.

• Order is fixed, not randomised

  Evidence is presented in the same order to every participant, which means order effects and evidence effects are confounded. Randomising the order is the single most valuable change to make next, and it has not been made yet.

• Numbers are reports, not beliefs

  Asking someone to put a number on a reading changes the reading. The heuristics-and-biases literature is thirty years of reasons to treat elicited probabilities as approximate at best — anchoring on the starting position of a slider is a live risk in this exact design.

• A model's stated probability is not a probability

  Verbalised confidence from a language model is a token sequence that has been shaped to look like a probability. It is sensitive to prompt wording, and it changes between model versions. This site pins a prompt version to every stored response for that reason, and still cannot claim the numbers are calibrated.

• There is no ground truth here

  The scenarios have no correct answer, by construction. That means accuracy cannot be scored and no proper scoring rule applies. What can be measured is movement, spread and agreement — and those are not the same as being right.

• Narrative interpretation is not scientific inference

  This is the project's own central caveat. A reading of Icarus and a hypothesis about electron diffraction are not the same kind of object: one has an experiment that can go against it and the other does not. The site argues that they share a structure. It does not argue that they share a standard of evidence.

• One language, one cultural frame

  The scenarios are in English and assume conventions about politeness, directness and silence that are not universal. The blunt-reply scenario is about that assumption; it does not escape it.

Next

• Randomise evidence order so that order and content stop being confounded.
• Run the same scenario against several models and several prompt versions, and report the spread between them rather than a single number.
• Ask a subset of participants to predict the group distribution as well as giving their own — a cheap way to find out whether people know they are unusual.
• Find out whether participants who resist an evidence item can say what would have moved them. Popper's question, asked of a reader.
• Build a version where the participant writes the next evidence item themselves: what would you need to see?

A prediction that turns out wrong stays on this page marked wrong. Quietly adjusting the question until the data answers it is the easiest way to make a project like this worthless, and it is almost invisible from outside. Changes go in the log. Model runs happen from a token-protected console, never triggered by visitors.