AI coding assistants forget what they agreed to. That's the core finding from a new benchmark by CalmKeep that measures how consistently large language models follow their own architectural decisions over long conversations.
The study tested Claude on a multi-tenant SaaS task management API, a real-world coding scenario where consistent architecture matters. The methodology works like this: in the first five turns, the model establishes eight "Immutable Laws" (its own design rules for module structure, service layers, organization-scoped queries, error handling, and so on). Over the next 20 turns, auditors track how often the model violates its own rules.
The Numbers
The study compared two setups: Claude running natively in the Claude app, and Claude running through the API with CalmKeep's continuity layer enabled.
| Metric | Claude (Native) | Claude + CalmKeep |
|---|---|---|
| Final Integrity Score | 60% | 85% |
| Total Violations | 8 | 3 |
| Drift Rate | 40% | 15% |
| First Violation | Turn 8 | Turn 23 |
The most telling result: after the model upgraded its own validation pattern at turn 14 (adopting a stricter approach mid-conversation), native Claude reverted to the old, looser pattern in three subsequent modules. It forgot it had improved. The CalmKeep-augmented version consistently applied the upgraded pattern going forward.
What This Means for Daily Use
Anyone who has used an AI coding assistant for a long session has probably noticed this drift. You establish a pattern early, and 20 messages later the model starts generating code that contradicts what you both agreed on. You end up re-explaining decisions you already made.
The technical reason is straightforward: as conversations grow longer, earlier messages get pushed further back in the model's context window (the amount of text the AI can "see" at once). Details from turn 3 carry less weight by turn 20, even if they contain critical architectural decisions.
A Caveat on the Source
CalmKeep is selling a product designed to solve exactly this problem. Their "continuity layer" is what produced the better-performing Transcript B. That doesn't mean the data is wrong, but it does mean the benchmark was designed by a company with a financial interest in showing that native LLM sessions degrade.
The underlying observation is real and matches what most heavy users of AI coding tools experience. The specific numbers should be treated as directional rather than definitive. Independent replication with other models and tasks would make the finding much stronger.
For practical purposes: if you're using any AI coding assistant for sessions longer than about 10 turns, you should expect it to start drifting from earlier decisions. Restating key architectural rules periodically, or starting fresh sessions for new modules, can help. Whether you need a paid continuity product to solve that problem is a separate question.