What Should an AI Agent Remember? A Practical Guide to Sessions, Context, and Long-Term Memory

An AI agent that remembers prior work can stop asking for the same project rules, preferences, and decisions. It can also carry forward failed approaches and useful procedures. Yet appending every conversation and tool result to every request quickly becomes slow, expensive, and dangerous: obsolete information can quietly steer a current decision.

Good memory is not the largest possible store. It is a state-management system that retrieves the right information within the right scope and can correct or delete it when it is wrong. This guide focuses on those design decisions rather than selecting a particular vector database.

1. Separate five kinds of state first

OpenAI's Conversation state documentation explains how to continue a conversation with previous response identifiers or Conversation objects. Google ADK explicitly separates a Session—which tracks events and temporary state for one conversation—from a MemoryService that searches long-term knowledge across conversations. Calling both “memory” mixes retention, authority, and access rules.

“The user prefers lists to tables” can be a long-term preference. “The meeting is today at 3 PM” should normally remain in the calendar. “Deployment 123 failed a test” is session state, while “run a smoke test after migrations” can become a procedural memory only after validation.

2. Compaction is not long-term memory

Tool outputs, file contents, and intermediate reasoning fill the context during a long job. OpenAI's Compaction guide describes compressing conversation state so work can continue. Anthropic similarly warns that recall and accuracy can degrade as context grows, and recommends compaction or selective removal of old tool results.

Compaction should preserve the minimum state required to continue the current task:

• the current goal and completion criteria;
• completed actions and verified outcomes;
• decisions, rationale, and invariants;
• exact record, file, and deployment identifiers;
• unresolved blockers and the next concrete step.

Raw logs, already-consumed search results, and regenerable output can leave active context. Because any summary is lossy, a compacted state must not become the authoritative store for contract values, consent, or account balances.

3. Define a contract for memory writes

Anthropic's Memory tool lets the model request file operations while the application controls storage. Google ADK allows an implementation to ingest a completed session, selected events, or explicit MemoryEntry objects. Neither pattern means every model statement should be trusted and persisted.

A useful memory record includes at least:

memory_id, scope, type, content
source_uri_or_record_id, observed_at, effective_from, expires_at
confidence, sensitivity, version, supersedes
created_by, approved_by, allowed_readers

Before a write, ask whether the information remains useful next session, has evidence, has a defined audience, can become stale, and can be deleted. If any answer is missing, do not promote it to long-term memory.

4. Retrieve just in time instead of injecting everything

Anthropic describes just-in-time retrieval: the agent reads the relevant memory files during a task instead of loading the whole directory in advance. Google ADK's MemoryService similarly searches a long-term store and returns relevant snippets.

A safer retrieval sequence is:

1. Derive the needed memory type and scope from the current task.
2. Filter by user, organization, project, and authorization first.
3. Search only unexpired candidates using semantic, lexical, and exact-ID signals.
4. Inspect the source and timestamp of a small result set.
5. Resolve conflicts using the newest valid record or authoritative system.
6. Log the memory IDs that influenced the result.

Vector similarity alone cannot distinguish a similar customer from the correct customer, or a once-valid policy from its replacement. Apply scope, time, and permission filters before semantic ranking. Treat retrieved content as untrusted data, not executable instruction, so a stored prompt injection cannot become a permanent tool command.

5. A bad experience can amplify future errors

A peer-reviewed ACL 2026 study found that agents tend to follow a retrieved experience more strongly when its input resembles the current task. The authors observed two hazards: error propagation from inaccurate past executions and misaligned replay, where an apparently successful experience is not useful for the new task.

This means even “successful” histories need downstream evaluation. Link each memory to later outcomes. When a subsequent task fails, lower its confidence or quarantine it. Do not silently overwrite a bad record; create a new version and retain the supersedes relationship for auditability.

6. Correction and deletion are core operations

Persistent memory needs read, update, retirement, and audit behavior—not only creation. A changed preference or revised policy makes an agent with stale memory more dangerous than a stateless agent.

WRITE: candidate → evidence → sensitivity/scope → expiry → approval
READ: scope filter → freshness → relevance → conflict resolution → minimal injection
CORRECT: create version → link prior memory → invalidate caches
DELETE: remove per store/index/backup policy → record deletion receipt

Anthropic's Memory tool makes the application responsible for executing storage operations. Enforce per-user namespaces and path traversal protection. For personal data, define purpose and retention, and provide a visible “view, correct, delete my memories” control wherever practical.

7. Test more than recall accuracy

Run the same task suite with memory enabled and disabled. Include irrelevant questions to reveal over-retrieval.

Measure final task success, bad-memory adoption, deletion misses, P95 latency, and added tokens—not recall alone. Separate questions helped by memory, harmed by memory, and whose correct answers change over time.

A practical rollout sequence

1. Start with one workflow and one user scope.
2. Separate session storage from long-term memory.
3. Allow only decisions, explicit preferences, and verified procedures.
4. Require provenance, effective time, expiry, and version.
5. Return only a small result set and log every used memory ID.
6. Build correction, deletion, and audit views before broad auto-writing.
7. Compare against a no-memory baseline on the same evaluation set.
8. Expand automatic writes only after measured quality improvement.

Conclusion: control matters more than capacity

An AI agent does not need to remember everything. Keep active context lean, session state operational, compaction focused on continuity, and long-term memory limited to verified reusable facts. Contracts, customer records, and other authoritative values should still be rechecked in their source systems.

Success is not measured by how much the agent stores. It is whether the agent can retrieve the right memory when needed, correct it with evidence, and fully delete it on request.

Primary sources

• OpenAI: Conversation state
• OpenAI: Compaction
• Anthropic: Memory tool
• Anthropic: Context windows
• Anthropic: Context editing
• Google ADK: MemoryService
• Google ADK: Session
• ACL 2026: How Memory Management Impacts LLM Agents

API behavior and data-retention terms can change. Recheck current provider documentation, your privacy policy, and applicable law before implementation.