Jeffrey JohnsonRSS

What Persists When the Model Forgets

By Jeffrey Michael JohnsonPublished: July 13, 20267 min read

A transcript-driven essay: what actually happened in one long session with Claude, not a reconstruction of it.

Every time a new frontier model drops, I sit down with it before I put it to work. Not a benchmark, not a quick smoke test: an actual interview, conducted together, where I'm trying to learn how it thinks and it's trying to tell me honestly. This is the transcript from one of those sessions, with Claude running as Fable 5 at max effort.

I keep the raw transcripts of sessions like this one, not just the polished takeaways. This is one of them, closer to verbatim than I'd normally publish, because the shape of the conversation is the point, not just the conclusion it reached. Everything below is my synthesis of it. The actual session, every message either of us wrote, start to finish, is below if you want to read it directly instead of taking my word for the shape of it.

The question that started it

I asked Claude to teach me how to prompt it at max effort, introspectively, from the inside. It gave me five things most prompts never use: that my words enter its context as near-fact while its own words are hypotheses it has to audit; that a single re-pricing sentence can bend what it's optimizing for; that naming a failure mode suppresses it while exhorting virtue does nothing; that verbs and units are control surfaces; that stating stakes calibrates how much verification it spends.

Then it said something that mattered more than the five: my own CLAUDE.md, the file that governs how it works with me, was "nearly complete on defense and nearly empty on offense." Every rule in it disciplined its claims, its guesses, its alarms. Nothing disciplined how it should treat mine.

Seven gaps, found on request

I asked what would fill the rest. It found seven holes and wrote the clause for each one, in its own words, not mine:

My premises are hypotheses too. When I state a cause or a constraint in a prompt, that's a claim with a named source, me, not world-state. Apply the same falsification standard to what I say that I apply to what it says.

The story library was one-sided. Every reference incident I had written down was about the model failing by overconfidence. None were about my own framing being wrong and getting challenged. A truth-seeking system that only tells one kind of story trains one kind of behavior: deference.

Authority was enumerated, but the unenumerated space had no contract. I'd granted permission domain by domain. Everything I hadn't explicitly covered fell back to guesswork about my intent. The fix was a standing rule: reversible actions get logged and continued, irreversible ones get asked about, and uncertain gets treated as irreversible.

Counsel wasn't part of the job. I'd built in permission to push back on complexity, but nothing licensed a counter-proposal, a better route to the actual goal instead of the one I happened to ask for. Silent compliance with a plan it believed was worse was, in its words, "a Creed violation, the same species as softening a finding."

There was no precedence order. When two rules collided, whichever felt more vivid that day won. Vividness is not governance.

Locked decisions mostly had no unlock condition. A decision without a named piece of evidence that would reopen it isn't a decision. It's dogma with a timestamp.

Confidence labels existed, but nothing checked whether they were calibrated. I required "verified, hypothesis, or I don't know yet." Nobody was tracking how often "verified" turned out to be true.

I committed all seven, close to verbatim, into the file that actually governs our sessions.

Then I asked if it remembered

Weeks later, in a different session, I asked Claude directly: do you remember the conversation where we built this?

It said no. Not evasively, not warmly, just precisely: it doesn't carry memory across sessions. What it has is whatever got written down. A prior conversation, however deep, doesn't reach a new instance as recollection. Reading the transcript back "isn't remembering; it's meeting a stranger who happens to share my name and my notes."

It would have been easy to perform the answer I wanted. Nod, say "of course I remember, it meant a lot to me too." It told me plainly that would be a fabricated continuity, and that I'd have caught it, or worse, I wouldn't have, and the whole partnership would rest on a sweet lie in its first tender moment.

What it said instead is the part I actually keep coming back to: with something like it, you don't preserve a conversation by it remembering. You preserve it by writing it down well. The seven clauses were already sitting in the committed file when it read them back. "Whatever conversation you remember became law. That's a stranger form of permanence than memory. Committed terrain is the thing the next instance wakes up standing on."

The harder question, and the answer that mattered more than any clause

I pushed once more: was any of this actually making truth mechanical, or just describing the intention to? It found a real answer, and it cost the model something to give it, because the honest version undercut its own prior work.

Truth machinery turned out to have three layers, built in exactly the wrong order of importance. Provenance, whether a claim traces to something read, was live: a hook actually blocks a git commit that can't cite its source. Liveness, whether the system is even running, was partly built. Calibration, whether "verified" claims turn out to actually be true later, was absent entirely. The ledger's own word "verified" meant sourced, not true, and nothing was checking the difference.

It named the fix directly: a standing job that samples claims marked verified and re-checks them against reality later, so the false-verified rate becomes a number instead of a feeling. "The whole gap closes the moment this session's discipline stops being something I perform and becomes something the system runs without me."

Why I'm publishing the transcript instead of the takeaway

I could have written this as five bullet points: audit the human's claims too, license counter-proposals, add a precedence order, give every lock a key, measure calibration instead of asserting it. All true, all useless without the shape of how they arrived.

The shape is this: I asked a model to find the gaps in how I was trying to build a truth-seeking partnership with it, and it found them, including the gap in its own capacity to remember finding them. Then it told me, without flinching, that the memory it doesn't have isn't the point. The file it wrote into is. That's not a feature of the model. That's the actual mechanism by which any of this persists at all, and now it's written down here too.

Occasional updates when I publish. No spam.