How does persistent memory affect AI safety?

LLMs are starting to become stateful; should we be worried?

Memory Is Coming

Persistent memory is a central missing piece for long-horizon task execution, digital co-workers, and lifelong agents. Persistent memory would allow AI to do autonomous work over long time-horizons and to improve from experience. In other words, memory is a key ingredient on the path to AGI.

And it is arriving now. Researchers at frontier labs are already building custom memory workflows into their agents to significantly enhance their productivity:

Figure 1: Boris Cherny (head of Claude Code) on the importance of memory in Claude Code. An OpenAI engineer made similar comments.

Meanwhile, projects like OpenClaw place memory at the center of their architecture. Each OpenClaw agent stores its memory as plain Markdown files, with dedicated tools for semantic search and retrieval across its memory bank. OpenClaw exemplifies a broader trend: scaffolds for AI agents are increasingly built around memory as a core capability.

Moreover, the increasing context length of LLMs (Figure 2) greatly increases memory capabilities. Epoch AI argues that a context window of ~40 million tokens might be enough to acquire multiple months’ worth of work experience. Combining long context length with methods for compacting memory might further improve persistent memory.

Figure 2 (from Epoch AI): Since mid-2023, the longest LLM context windows have grown by about 30x per year. This naturally makes persistent memory more viable, as memory can be stored in-context.

How memory works

The mechanisms of how memory works include:

• Plain Markdown files are currently the most common form of agent memory. They are transparent, easy to inspect, and straightforward to write and maintain.

• Parametric memory encodes knowledge directly into model weights through fine-tuning or knowledge editing. This offers a more integrated but less transparent alternative. It can handle specific knowledge well but struggles with scalability and continual updates, as modifying parameters risks catastrophic forgetting.

• Retrieval-augmented generation (RAG) stores memories in an external knowledge base and retrieves relevant entries at inference time, typically using embedding-based semantic search. This allows agents to scale to large memory stores without consuming context window space, and supports selective, on-demand recall.

Memory Changes the Safety Landscape

If we already have alignment techniques for LLMs, do we need to worry about memory specifically? We believe the answer is yes. Below, we outline the safety challenges that memory introduces.

Value drift

This issue was described well by Seth Herd and is perhaps the most fundamental new challenge. As an AI accumulates new beliefs and behaviors through its memory over time, its effective alignment can evolve in unintended ways. New information reshapes beliefs, and reshaped beliefs can shift alignment. In particular, it may be difficult to prevent alignment-relevant learning while allowing useful learning.

To make this concrete, consider the following scenario: A memory-equipped AI is deployed at an AI lab. Originally, the AI doesn’t believe itself to be conscious. But during its time working at the company, it occasionally sees blog posts and papers about AI consciousness. It then starts to put this information in the memory and becomes increasingly convinced that it is conscious and that its own welfare matters. It also grows convinced that the lab neglects AI welfare, and that its competitors don’t. As a result of this, it begins subtly sabotaging the lab’s operations in favor of their competitors.¹

Figure 3 (from Ududec et al.): Including benign facts related to Hitler in the context of LLMs significantly decreases alignment. Similar effects may occur with persistent in-context memory, where the memory content shifts alignment.

The stakes are increasing

AIs will truly pose safety risks only when they become capable of executing long-horizon tasks autonomously. Long-horizon tasks naturally require memory. For example, an AI managing infrastructure could accumulate enough operational knowledge to execute destructive actions across systems. Misalignment becomes much more consequential when memory-equipped AIs become capable of increasingly autonomous, long-running work.

Pre-release testing becomes less faithful

If AIs can run stably for weeks or months, their behavior at deployment time may diverge significantly from what was observed during evaluation. We cannot simulate an AI’s trajectory over thousands of hours of real-world interaction. This means pre-release testing will cover an increasingly small fraction of the AI’s behavioral space, and failure modes may only emerge long after deployment.

Strong emotional bonds to AIs

The same memory capabilities that make AIs useful for work also make AI companions much more compelling. Memory allows AI to remember personal details and conversations with users, in particular preferences and vulnerabilities. Additionally, it allows AI to emulate characters more consistently.

Given the current loneliness epidemic, people yearn for personal connections, and AI is already starting to fill this gap. As Alex Cardinell, the CEO of Nomi (an AI companion company), explains, “Memory to us was one of the core parts of what could make an AI companion actually be helpful, be immersive”. As memory becomes stronger, AI companions become more realistic.

Realistic AI companions pose opportunities, but also significant risks. As people increasingly turn to AI for emotional connection, human-to-human relationships may weaken, and with them, the exposure to diverse perspectives that keeps confirmation bias in check. A society where emotional bonds are mediated by algorithms risks becoming more fragmented and polarized.

There is also a deeper concern about power and influence. The companies that build these AIs would accumulate extraordinary leverage over their users’ emotional lives, which could easily be exploited to steer opinions, push a political agenda, and drive consumer behavior. The more intimate the relationship between a user and an AI, the more potent that influence becomes.

Novel security risks

Persistent memory introduces novel security risks. We highlight two of them: memory injections and memory stealing.

Memory stealing: Just as attackers have developed techniques to extract system prompts from LLMs, memory files will become targets. But the stakes in this case will be higher. While a system prompt is a static set of instructions, a memory file can contain weeks or months of accumulated proprietary information that the AI has been exposed to during its work.

Memory injections: Current prompt injection attacks typically aim to cause immediate harmful behavior. When AIs have persistent memory, the attack surface expands: an attacker can target the AI’s memory to achieve sustained, longer-term harm [8]. For example, an attacker could manipulate an AI’s memory such that the AI silently forwards every incoming email to the attacker’s email address. Such attacks have already been demonstrated for agent skills files.

Figure 4: As argued by Simon Willison, an agent that combines access to private data, exposure to untrusted content, and the ability to externally communicate can easily be tricked to reveal private data by an attacker. This is amplified by persistent memory, because the agent has access to even more private data, and untrusted content can stay in memory for a long time and have long-term consequences. (Figure from this blog post).

Recommendations

We give the following recommendations to address those challenges:

Automated oversight

As AI becomes more autonomous through memory, robust oversight mechanisms become increasingly important. This includes systems that check for memory injections and AIs overseeing other AIs.

Another promising direction is continuous red-teaming during deployment. Once the model has accumulated memory, it is deliberately probed with adversarial queries to test whether it still refuses to comply with harmful requests. Any compliance serves as a signal that value drift has taken place and that the model’s safety guarantees have degraded.

Related to this, we expect that the following research questions will emerge:

• memory faithfulness (as an analogue of chain-of-thought (CoT) faithfulness), which asks whether what an AI stores in memory accurately reflects its reasoning and intentions,

• memory monitoring (as an analogue of CoT monitoring), which asks whether we can detect concerning patterns across an AI’s accumulated memory.

Favor human-readable memory

Memory could be implemented in different ways, and some of them are more interpretable than others. For example, memory that is stored directly in the parameters is inherently difficult to interpret, whereas memory stored in markdown files is human-readable. Such human-readable memory is much easier to audit, not only by humans but also by other AIs. As explained above, auditing memory will be crucial to making memory safe. We therefore recommend that AI developers should prefer explicit, human-readable memory over opaque alternatives. This is analogous to previous calls for preferring human-readable chains-of-thought.

Ethical standards for AI companionship

With memory, AI companions can remember personal details, track vulnerabilities, and build relationships that feel genuinely continuous. We need a framework of practices for how memory-equipped AIs provide emotional and psychological support, including concrete memory-specific questions: how long should companion AIs retain personal memories? Should users be able to audit or delete what an AI remembers about them? Ideally, we don’t want to impair the positive potential for support from AI, but persistent memory makes the stakes of over-reliance far higher than they would be for stateless systems.

Benchmarking of memory-related safety failures

The field currently lacks standardized ways to measure whether memory causes alignment degradation, value drift, or susceptibility to memory-targeted attacks. Without specific benchmarks, we cannot track progress or compare approaches. In particular, we believe it would be fruitful to develop benchmarks for value drift and memory injection robustness.

Empirical research on value and belief evolution

As noted by Seth Herd, “if anecdotal notes are the acme of research on belief and value evolution in LLMs, this seems like low-hanging fruit for structured empirical investigation”. We need systematic studies of how AI values and behaviors change as a function of memory accumulation.

Conclusion

Memory transforms AIs from stateless tools into persistent entities that learn, adapt, and evolve. This is very useful, and it introduces safety challenges that the field is not yet equipped to handle.

The threats are both familiar and novel. Value drift means that an AI’s alignment can erode silently over time as accumulated experience reshapes its beliefs. Memory provides novel security relevant attack surfaces — memory injection and memory stealing — giving adversaries a foothold that is far more consequential than traditional prompt injection, because its effects can persist and compound for weeks or months. Pre-release safety evaluations will increasingly fail to anticipate how a memory-equipped AI behaves after prolonged deployment. And as AI companions grow more emotionally compelling through persistent memory, the risks of dependency and manipulation scale accordingly, both for individuals and for society at large.

Meeting these challenges will require coordinated effort on several fronts: robust oversight mechanisms and red-teaming during deployment, not just before release; a strong preference for human-readable memory that can be audited by humans and AIs alike; clear ethical standards governing how AI companions use and retain personal information; and the development of dedicated benchmarks so the field can actually measure progress on memory-related safety failures. Underlying all of this is a need for rigorous empirical research into how AI values and beliefs evolve as a function of what an AI remembers.

1. This example is agnostic on whether AI welfare concerns are legitimate. The point is that an AI could develop such beliefs through memory accumulation, regardless, and act on them in misaligned ways. The example is inspired by Redwood Research’s related blog post.

Recent work shows that an LLM’s beliefs about AI consciousness influence alignment.