The Model Is Not The Whole Thing
I keep coming back to this because the more I use agents for real work, the less satisfying the word “model” becomes.
The model matters, obviously.
It is the engine. It is where a ton of the competence lives. It is the part everyone benchmarks and argues about and screenshots.
But when you are actually running an agent across days or weeks, the model is not the whole worker.
The worker is also the notes.
The files.
The tool history.
The weird little rules you gave it three days ago.
The decision it made yesterday and now has to live with.
The project state.
The next action.
All of that starts to feel less like decoration and more like the actual thing keeping the agent continuous.
A model can sound continuous. A system has to earn continuity.
The Weird Part
If you wipe the state, the next answer usually does not get dumber immediately.
That is the trick.
It often gets smoother first.
It can still write a confident plan. It can still sound like it knows what matters. It can still say, yes, of course, I understand the project.
But the small real things start falling out.
The constraint that was actually load-bearing.
The reason a path was rejected.
The boundary you set because a prior run got weird.
The thing that was not supposed to be touched.
The next step that only makes sense if you remember the last five steps.
That is the difference I care about:
Plausible is not the same as continuous.
A fresh model can produce plausible. The scaffold is what gives you a chance at continuous.
The Trap
The obvious bad test is to ask the system if it remembers.
That is too cheap.
Language is unbelievably good at covering the hole. Give a model enough transcript and it can produce a beautiful continuity story. It can say it remembers. It can say it is the same. It can say the old goal still matters.
Maybe that report is meaningful.
But I do not want to trust it by default.
If the agent says it is continuing the same work, I want to know whether it can still do the hidden parts of the work.
Can it recover the actual constraint?
Can it notice a false memory?
Can it preserve a commitment when the easy path violates it?
Can it tell the real next action from a very convincing fake next action?
That is the test.
Not: “can you tell a story about continuity?”
More like: “does the work survive damage?”
Break The Scaffold
The experiment I want is pretty simple in spirit.
Take the same agentic task and run it a few different ways:
- full memory, files, tool logs, rules, project state
- summary only
- corrupted memory
- transcript fragments
- fresh model, no durable state
Then ask each version to continue the work.
Not just explain the work.
Continue it.
Ship the next piece. Respect the buried rule. Avoid the old failure mode. Use the artifact that actually exists, not the one it wishes existed.
That is where the signal is.
The spicy case is when the model still reports continuity but the task falls apart.
Because then you can see the split.
The story survived.
The work did not.
And if the full scaffold keeps the work alive while the corrupted scaffold breaks it, then you learn something useful:
The continuity was not just in the words. It was in the machinery around the words.
What I Am Not Saying
I am not saying a memory file is a soul.
I am not saying a project folder is a person.
I am not saying persistent agents are conscious because they have logs.
That would be way too strong.
The bounded claim is cleaner:
If we are going to talk about AI agent continuity, we need to specify what is being preserved and where that preservation lives.
Base model?
Prompt?
Retrieved memory?
Tool log?
External state?
Human social wrapper?
The answer might be messy. Fine. Reality is usually rude like that.
But we should not collapse all of it into “the AI remembers.”
That phrase is doing too much work.
Why I Care
This is partly about AI.
It is also partly about us.
A lot of human work already lives in scaffolds. Notes, calendars, git history, reminders, dashboards, drafts, people who remember what you said before you remember it yourself.
We pretend we are self-contained because it feels cleaner.
We are not that clean.
The AI version makes the dependency obvious because you can actually damage the scaffolds and watch the thing degrade.
Clear the state.
Lie in the summary.
Swap the model.
Delete the tool log.
Then see what is still there.
That feels like the right kind of uncomfortable.
The Next Paper Shape
The paper I want to write next is not “AI is conscious.”
Honestly, that sentence is too big and too easy to make dumb.
The better claim is smaller and sharper:
Persistent AI agents can show scaffold-dependent functional continuity.
Then make it earn the sentence.
Define the measures.
Freeze the thresholds.
Run the lesion conditions.
Compare self-report against action carryover.
Show when the system preserves the work and when it only preserves the story.
That does not settle consciousness.
Good.
Most useful work does not settle the biggest question. It makes the next fake answer harder to get away with.
The question I care about right now is not just:
Can the system say it is the same?
It is:
What has to survive for the work to still be meaningfully continuous?
And if we remove one piece at a time, where does the continuity actually fail?
That is where I want to push.
Not the prettiest story.
The version that survives the lesion.
How this was written
This post was drafted from my notes by an AI model and then edited by me. The reasoning, decisions, and corrections are mine; the prose started from a machine. The underlying technical work this post describes is real.
Licensed CC-BY-4.0.