Your AI agent has a cache invalidation problem

Persistent memory lets an agent carry yesterday's correct answer into a world where it is wrong.

Suppose an agent learns how your company calculates net revenue. Refunds are deducted from the payment amounts, so summing the payments gets the right answer. The agent saves the query and an explanation. Next month, accounting changes the model. Payments retain their gross amounts and refunds live in a separate table.

The remembered query still runs. It returns a plausible number, and the agent can point to the session where it learned the procedure. It faithfully remembers the old procedure. Your report is wrong.

Better recall can make this bug easier to reproduce.

In July, I argued that agent systems were becoming a graph engineering problem. Useful work needed explicit state, dependencies, and recovery across many steps. Persistence leaves us with another obligation. A system can resume exactly where it left off, carrying assumptions that should have been retired while it slept.

The system needs to know what would make an old lesson stop applying. We tend to bury that requirement inside the word "memory." We use it for a user's instructions, old tool results, conversation summaries, facts extracted from documents, and the agent's guesses about why something worked. Those records have different owners and different reasons to be trusted. Put them in one searchable pile, and a useful observation can quietly acquire the standing of a rule. Staleness starts with something that was correct. Bad initial inferences and accurate facts used outside their scope need different checks.

I like plain files. In Agent memory as a file format, Cal Paterson proposes portable Markdown pages with optional metadata and a search index. The agent's notes remain readable, editable, and independent of the software that wrote them. I want those properties. Paterson also recommends source links so agents can check outdated material. That recommendation deserves as much attention as the file format. The stale revenue query may be the closest semantic match to the new revenue question.

Researchers have started separating these failures from ordinary recall. STALE tests 400 scenarios where later information changes what an assistant should believe without explicitly announcing that an older memory is false. In one LightMem diagnostic case, a user has moved from Seattle to Austin. For a question that still presupposes Seattle, the old entry ranks first and the Austin update tenth. Both are available. The answer follows the old state.

Across the benchmark's probes, Gemini-3.1-pro, the best general model tested, scored 55.2% overall. The authors' targeted CUPMem prototype scored 68.0%. These are constructed scenarios, with human validation and model-based judging, not a measurement of how often your assistant fails. The benchmark shows a specific failure worth testing in our own systems. Retrieving the update can still leave the old belief in control.

A stored answer saves work because you assume it remains reusable. HTTP caching gives responses a freshness lifetime and supports validation against the origin. The client has a way to check whether it can keep using its copy. Reuse comes with conditions.

For an agent's observations about a repository or an API, that is a useful starting point. The saved revenue query should carry the source it came from and the version of the accounting model it depended on. When that model changes, the saved query needs review before reuse. A timestamp on the memory file does not do this. Rewriting the same old lesson this morning gives it a younger timestamp and the same old defect.

In 1977, Jon Doyle's truth maintenance work recorded the justifications behind a program's beliefs so it could revise them when their supporting assumptions changed. That is a better ancestor for a model-written conclusion than a bag of text chunks. A conclusion can lose one justification and retain another. A past decision can remain important historical evidence after it stops being the current policy.

The cache analogy has a boundary here. An explicit instruction to use Python in a particular project has an author and a scope. A recollection that a service was healthy yesterday is evidence about yesterday. An agent's theory that a timeout was caused by rate limiting may never have been established in the first place. A seven-day expiration policy cannot tell these records apart.

The engineering gets easier when we keep those distinctions and track the inputs to the things we generate. Build systems à la carte describes the familiar job of rebuilding outputs when their dependencies change. A memory summary is also an output. It came from particular records through a particular transformation. If a source record changes, everything derived from it deserves reconsideration.

Mark Lubin's Synix uses a build system for agent memory, tracking sources and transformations so affected artifacts can be rebuilt. Graphiti represents temporal validity and the source episodes behind extracted relationships. The difficult work is deciding which facts depend on which sources, and making the whole application honor the result when something changes.

When source changes are observable, a dependency graph can identify which generated notes need checking. It cannot prove that the model correctly summarized the source, or that it identified every dependency. Hashing a hallucination gives you a reproducible identifier for a hallucination. Where dependencies are uncertain, the system may need to throw away a larger summary and reread its sources. For an external source that exposes no revision or change notification, checking may require another read. The right frequency depends on how quickly the information changes and what a stale answer would cost.

An August 31 preprint on invalidation contracts tests a narrow version of the problem. Agents save API error fixes and reuse them across tasks. Responses carry a version stamp for each reference table, and a separate change report identifies affected rows. The client can discard the corresponding fixes while retaining unrelated ones. The evaluation covers seven models and roughly 9,400 episodes across two APIs built for the study.

The authors separate whether a cached fix remains valid from whether the model applies it. Tracking versions addresses the first. The second still varies by model. Detection happens when a response arrives, so the design can incur a stale attempt before correction. The APIs and recovery metadata are hand-written for the experiment. Production services would have to expose and maintain those signals.

Checking the world can also improve what gets remembered in the first place. In the September 10 preprint Grounding Agent Memory, Microsoft researchers gave a memory curator read-only tools to check candidate lessons against the environment. Across five paired runs of a 40-question CLBench database test using GPT-5.4, with schema changes halfway through, reported pass rates were 39% without memory, 70% with trajectory-only memory, and 73% with environment-probed memory. The 95% confidence intervals overlap for the two memory conditions. The largest observed jump is from no memory to trajectory-only memory. The extra checking produced a much smaller difference in this experiment. Useful memory is worth maintaining.

For someone building an agent today, I would start by keeping the original evidence available. A summary should lead back to the records that support it. Keep explicit user instructions distinguishable from what the model inferred, and record the project or account a lesson belongs to. A perfectly accurate memory from the wrong customer is still the wrong input.

Then give source changes somewhere to go. A schema revision can mark dependent query notes for review. A stale flag should suspend reuse while the system checks whether the note still applies. A corrected document can invalidate its summaries and search entries. A newer approved decision can supersede an older decision without erasing the history. The event needs to reach what the agent will actually read, including any context that has already been assembled. Changing a database row does little for an agent still reasoning over the old copy in its prompt.

This can start with source IDs and revisions attached to a Markdown note, plus a check before reuse. When a fact is cheap to fetch and expensive to get wrong, remember how to fetch it. Fetch current service health and schema information when the task depends on them. Check permissions with the system that grants them. A sentence in an agent's notebook should not be able to approve its own deployment.

The test I would run changes the world halfway through. Let the agent learn the original revenue procedure, alter the accounting model, and ask for another report. Check the answer and the sources it used. Repeat with a harmless source change to see whether it throws away useful work unnecessarily. Include corrections that arrive indirectly, and a question whose wording invites the old assumption. Measure the revalidation cost alongside correctness.

That is the version of persistent memory I want. It should save the agent from rediscovering everything while making it possible to find out which lessons no longer apply. A demo where the agent recalls what it was told is the beginning. I want to see the next session, after that thing has changed.

- Dr. J