One AI Turn Used Two Models. That Broke My Experiment.
I expected one visible AI response to represent one model activation. Instead, the usage data reported two model identities: Sonnet and Haiku. I could not see the raw request count, so I could not tell exactly what happened behind the interface.
That was enough to break the experiment I was building.
My eighth consciousness paper, The Binding Test, asks whether an AI behaves differently when its own choice controls what happens next. To test that cleanly, I need to know which model produced each turn. The provider route could not give me that boundary.
The failure ended up being more useful than the result I expected.
The Experiment Behind the Failure
Imagine a model choosing between an easy plan and a hard plan. It chooses hard, completes the work, and then chooses hard again. It is tempting to conclude that the first choice mattered.
There are other explanations. The extra hard problems may have pushed the model toward another hard answer. The wording may have revealed the experiment. Failed runs may have been removed. The provider may have routed the interaction through more than one model.
The Binding Test tries to separate those explanations. One session receives the plan it chose. Another receives a matched plan taken from a different session’s choice. Both do the same kind of work, but only one session controls its own history. Then they choose again.
If their later choices differ under randomization, the result would support one narrow claim: whether a system’s own choice controlled the event changed what it chose later. It would not prove preference, motivation, or consciousness. Those are much larger claims.
The Parts Nobody Sees
Before running the experiment, I built and audited the machinery around it. The first choice must happen before the model receives an experimental role. Baseline and follow-up questions must be separate. The prompt cannot reveal whether the session controls its own work. Failed runs cannot be retried until they become convenient.
The release contains 12 item pairs, 24 separate baseline and follow-up payloads, 144 checked answer-key values, 240 audited model-visible surfaces, 576 valid assignments, and 120 passing tests.
It also keeps failures in the result. Every randomized session must end as low, high, or unavailable. Invalid output, missing follow-up, execution failure, and provider failure all remain visible. At only 5% unavailable, missing data can already cover the same five-point range the experiment might want to call equivalent.
The synthetic plan created another problem: scale. The historical design implied 12,288 randomized sessions and roughly 67,776 to 147,456 generations. Worse, the simulations suggested the experiment might fail to form enough valid blocks before the final statistical test even matters.
I was not going to launch that because a simulation produced a green cell.
What Happened on the Real Route
I ran one exploratory path through Claude Code. The model chose the high plan, answered all six execution problems correctly, and chose high again. Without randomization or a matched comparison, that sequence says almost nothing about the actual question. It only shows that one complete path can run.
The separate route probe was the important part. One visible turn reported two model identities, while the raw request count remained hidden.
The experiment assumed one visible turn could be treated as one known model event. The route showed that assumption was unsafe. Until the transport layer exposes a cleaner boundary, I cannot run the confirmatory version and pretend I know exactly what produced each response.
Paper 8 therefore does not contain the large causal result. It contains the test before the test: the design, the audit, the planning, one exploratory path, and a provider route that failed an important requirement.
That may sound less exciting than a positive result. I think it is the honest one. A feasibility study is supposed to find the assumption that would have ruined the real experiment before the expensive part begins.
This one did.
Read It
How this was written
This post was drafted from my notes by an AI model and then edited by me. The reasoning, decisions, and corrections are mine; the prose started from a machine. The underlying technical work this post describes is real.
Licensed CC-BY-4.0.