AI Agent Memory vs RAG: How to Separate Long-Term Memory, Session State, Checkpoints, and Knowledge Bases

AI Agent Memory vs RAG: How to Separate Long-Term Memory, Session State, Checkpoints, and Knowledge Bases

An agent may remember that a user prefers Objective-C, continue an unfinished workflow, and retrieve the latest API documentation. All three behaviors look like “memory,” but they belong to different data paths in a production system.

Collapsing them into one vector store is attractive in a demo: save chat history, preferences, tool results, documents, and workflow snapshots, then let similarity search decide what goes back into the prompt. Long-running systems expose the problems quickly—stale facts override newer facts, tenant boundaries become fuzzy, execution recovery conflicts with personalization, deletion becomes incomplete, and the model cannot tell whether retrieved text is evidence, state, or a durable constraint.

The first question should not be “which vector database should we use?” It should be:

Who produced this data, how long should it live, who is it valid for, can it change, must execution recover from it, and should the model treat it as evidence, state, or a durable constraint?

This page owns that boundary question. For retrieval, reranking, freshness, and conflict resolution inside long-term memory, see AI Agent Memory Retrieval in Practice. For a concrete coding-agent memory evaluation, see Funes Agent Memory Tested. To see how memory, state and RAG fit inside an agent that can take cross-app actions, continue with Personal AI Agent Architecture.

The short answer: do not collapse these five layers

Layer	Question it solves	Typical data	Typical scope	Primary read path
Session / Thread State	What is happening in this conversation now?	messages, current step, temporary variables, tool results	session / thread	deterministic read
Checkpoint	How do we resume after interruption or failure?	graph snapshot, node progress, pending interrupt	thread + checkpoint	checkpoint restore
Long-term Agent Memory	What should still matter later?	preferences, project constraints, decisions, experience	user / project / agent	deterministic + retrieval
RAG	What external evidence is needed now?	docs, code, policies, manuals	corpus / tenant	retrieve + rerank
Knowledge Base	What knowledge assets exist?	source docs, structured knowledge, graphs, indexes	org / product / corpus	indexing and versioning

The same PostgreSQL cluster, object store, full-text engine, or vector database may support several layers. Shared infrastructure does not imply shared semantics, lifecycle, or access policy.

LangGraph’s persistence documentation makes this distinction explicit: checkpointers persist thread-scoped graph state, while stores persist application-defined data outside graph state for cross-thread memory. Google ADK similarly separates Session, State, and MemoryService.

Different frameworks use different terms, but the architectural principle is consistent: current execution state and cross-session memory are different data surfaces.

Why Agent Memory and RAG are easy to confuse

The read path can look almost identical:

query
→ embedding
→ metadata filter
→ vector / hybrid search
→ rerank
→ context assembly

If you only inspect retrieval, agent memory can look like “RAG over chat history.”

Memory, however, has a write and governance path that standard RAG usually does not own:

interaction / tool result / user correction
→ decide whether it deserves persistence
→ extract a durable fact or experience
→ attach identity, scope, provenance, and authority
→ compare with existing memory
→ ADD / UPDATE / SUPERSEDE / DELETE / NOOP
→ persist
→ recall later under authorization

A practical distinction is:

• RAG: which external evidence do I need to answer this request?
• Memory: which past facts, constraints, decisions, or experiences should continue to change future behavior?

An API reference belongs in a knowledge base. “This repository must not use dependency X” is closer to project-scoped memory. Both may be retrieved, but they should not have the same authority inside the prompt.

Session State: the working surface for the current thread

Session state fits data that is changing inside the current conversation or task:

• which form fields have already been collected;
• which workflow step is active;
• the latest tool result;
• pending actions;
• temporary mode or routing decisions.

Google ADK describes session.state as a scratchpad inside a Session and provides multiple scopes. LangGraph keeps short-term memory in graph state and persists it with a checkpointer so the same thread can continue.

A critical distinction:

Persistent session state is still not automatically long-term Agent Memory.

A value can be stored durably in PostgreSQL while still being valid for only one thread. Persistence answers “will this survive a process restart?” Scope answers “where should this be allowed to affect behavior?” Those are separate decisions.

Checkpoints: resume execution, not user identity

A checkpoint exists to preserve recoverable execution state.

In LangGraph, a checkpointer supports:

• continuing the same thread;
• resuming human-in-the-loop flows;
• interruption recovery;
• fault tolerance;
• inspecting or restoring prior graph state.

That is different from remembering a user’s durable preference.

A checkpoint might contain:

{
  "thread_id": "support-781",
  "current_node": "awaiting_approval",
  "retry_count": 2,
  "pending_tool_call": "refund_order",
  "messages": ["..."]
}

Long-term memory might contain:

{
  "subject": "user:42",
  "scope": "project:mobile-app",
  "kind": "explicit_preference",
  "content": "Use Objective-C for iOS examples.",
  "source": "user_statement",
  "valid_from": "2026-09-01",
  "status": "active"
}

The first answers “where did this run stop?” The second answers “what should still constrain a future related task?”

Also avoid a common oversimplification: a checkpoint does not necessarily disappear when a business workflow is considered complete. Retention and deletion depend on the persistence implementation and the application’s data policy. Thread-scoped does not mean automatically deleted.

Knowledge Base: the asset; RAG: a runtime retrieval pattern

A knowledge base may contain:

• product documentation;
• API references;
• company policies;
• source code;
• PDFs;
• support tickets;
• database records;
• a knowledge graph.

RAG is the runtime process that finds evidence from those assets for a specific request.

A useful model is:

Knowledge Base = what you know
RAG = how you retrieve evidence now
Agent Memory = what should continue to affect this agent/user/task over time

Implementations can overlap. Google ADK’s MemoryService can ingest past sessions or other information, and its RAG-backed memory implementation retrieves conversation data through a vectorized knowledge service. That overlap is exactly why the application still needs explicit semantics.

Long-term Memory: “long” is less important than “changes future behavior”

Do not define long-term memory as “data kept for more than one day.”

A better admission question is:

If this information is not provided next time, could the agent violate a durable constraint, repeat a solved failure, or make a materially worse decision?

Good candidates include:

• explicit, stable user preferences;
• project or repository constraints;
• confirmed business facts that are not cheaper to fetch live;
• important decisions and their rationale;
• reusable failure-resolution experience;
• a small amount of state required to continue work across sessions.

Poor default candidates include:

• greetings;
• one-off numeric values;
• unverified model inferences;
• raw multi-kilobyte tool logs;
• expired intermediate results;
• facts that can be fetched from an authoritative source of truth;
• sensitive data without a clear retention purpose.

Choose a read pattern before choosing a vector store

Agent Memory does not automatically require a vector database.

Deterministic reads

Use them for:

• language preference;
• subscription plan;
• project tech stack;
• tool authorization;
• time zone;
• account and tenant scope.

These have stable keys such as:

tenant_id + user_id + project_id + key

A relational, document, or key-value store is often a better fit.

Semantic or hybrid retrieval

Use it for:

• prior incident-resolution experience;
• similar project decisions;
• long unstructured historical events;
• fuzzy recall across multiple past interactions.

Only then does vector or hybrid retrieval become the primary read path.

Recent memory-system research also argues against assuming one substrate is universally best. A 2026 evaluation across multiple memory substrates found task- and scale-dependent trade-offs, including cases where excessive retrieval can hurt sequential decision-making by distracting from action-critical context.

A practical production layering model

At minimum, separate six data surfaces:

1. Invocation Context
   └─ one call only

2. Session / Thread State
   └─ current conversation, progress, recent messages

3. Checkpoint / Durable Execution
   └─ recoverable execution snapshots

4. Long-term Memory
   ├─ user facts
   ├─ preferences
   ├─ project constraints
   ├─ decisions
   └─ experience

5. Knowledge Base / RAG
   └─ docs, code, policies, external facts

6. Audit / Trace
   └─ debugging, risk, and audit data; not injected by default

The dangerous shortcut is to merge layers 4, 5, and 6 because all of them “live for a long time.”

An audit log may be retained for years and still should not become long-term memory. Retention duration does not determine whether data should influence model behavior.

One more boundary: memory is not current world state

The October 2026 memory discussion exposes another production failure mode. Even after separating RAG, long-term memory, and session state, a system can still treat something that used to be true as something that is true now.

For example, “this project defaults to Objective-C” can be durable project memory; “the current build is failing” belongs to current task state; “production is on version 3.2.1” should come from the deployment source of truth when available; and “the API returned 500 yesterday” is a historical observation, not proof that the API is still failing.

Type	Question	Typical source	How it should be used
Durable Memory	Which historical constraints, preferences, or decisions should still apply?	explicit user input, project rules, confirmed decisions	may influence behavior across sessions
Current State / Belief State	What does the system currently believe about the world?	live tools, task state, recent verification	used for the current decision with freshness and provenance
Historical Observation	What happened in the past?	traces, events, old tool output	not treated as current truth without revalidation

Belief state here should not mean free-form model guessing. It should be an explicit, inspectable representation of current operational state: source, verification time, expiry risk, and whether a live source of truth can override it.

This prevents a common failure: semantic retrieval finds a highly relevant old record, but its factual validity has expired. Vector similarity cannot solve that. Provenance, validity, and source-of-truth policy must.

The full long-term-memory lifecycle

A system with only insert() and search() is not a production memory system.

1. Admission

Decide:

• will this matter later?
• who produced it?
• is it explicit user input, a tool observation, or model inference?
• is it sensitive?
• what scope owns it?
• how long should it remain valid?

2. Normalization

Store governance metadata, not only raw text:

{
  "memory_id": "mem_...",
  "tenant_id": "t1",
  "user_id": "u42",
  "scope": "project:recalai-ios",
  "kind": "project_constraint",
  "content": "Use Objective-C for iOS examples.",
  "source_kind": "explicit_user",
  "source_ref": "conversation:...",
  "confidence": 1.0,
  "valid_from": "2026-09-01T00:00:00Z",
  "valid_to": null,
  "status": "active",
  "supersedes": null
}

3. Reconciliation

New information should not always append. Support:

• ADD
• UPDATE
• SUPERSEDE
• INVALIDATE
• DELETE
• NOOP

A new project-scoped preference may coexist with a global preference, while a direct correction may supersede an older value.

4. Retrieval

Do not query long-term memory for every request. Pure formatting, calculation, translation, and self-contained summarization often need no persistent user context.

Use long-term recall only when history, durable preferences, project rules, or prior experience can change the answer.

For the detailed retrieval pipeline, see AI Agent Memory Retrieval in Practice.

5. Forgetting and deletion

There are several forms of forgetting:

• logical invalidation;
• TTL-based expiration;
• superseding an older memory;
• physical deletion where required.

Do not promise “100% one-click erasure” unless every storage and retention layer actually provides that guarantee. Use a deletion ledger across:

primary store
vector index
cache
derived summaries
backup / retention boundary
audit record

Then pair deletion with read-back verification where the underlying system supports it.

XBSTACK’s Google ADK delete_session() memory test demonstrates one concrete boundary: deleting a Session does not imply that information already copied into a separate long-term MemoryService has been deleted.

Multi-user systems: identity and scope must apply before recall

The most dangerous memory failure is not forgetting. It is remembering someone else’s data.

Hard constraints commonly include:

tenant_id
user_id
agent_id
project_id / workspace_id
memory status
authorization scope

Prefer enforcing these constraints inside the database or retrieval query rather than retrieving a global candidate set and filtering it afterward.

Keep these concepts separate:

• identity: who owns the record;
• authorization: may this agent use it;
• scope: where is it valid;
• authority: explicit user statement or model inference;
• lifecycle: active, expired, superseded, or deleted.

Vector similarity answers only “how similar is this?” It cannot make those governance decisions for you.

Memory evaluation needs more than Recall@K

RAG evaluation often focuses on retrieval metrics such as Recall@K or Precision@K.

Memory adds additional failure modes:

• Useful Recall;
• Stale Memory Rate;
• Conflict Error Rate;
• Cross-user Leakage Rate;
• Unnecessary Recall Rate;
• Write Precision;
• Correction Success;
• Deletion Verification;
• Prompt Token Cost;
• P95 Retrieval Latency.

Recent research is increasingly framing long-term memory as an evolving state-management problem, not merely a storage or embedding problem. That is why increasing embedding dimensions or top_k rarely solves the hardest reliability issues.

First-party RecalAI architecture: why knowledge, memory, and current state stay separate

The same boundary appears in my RecalAI local personal knowledge-base work. A strong retrieval pipeline does not mean the knowledge base should become the user’s long-term memory system.

RecalAI’s ingestion path preserves the source, runs OCR/parsing and normalization, chunks the text, makes it immediately searchable through FTS, then adds embeddings, summaries, tags, and image understanding in the background. Querying combines FTS and query embeddings, performs chunk-level hybrid retrieval, RRF/deduplication, reranking and evidence selection before local or cloud generation. That is a Knowledge Base + RAG data surface: it answers what material has been saved and which evidence is relevant now.

If RecalAI adds full personal memory, user preferences, durable project constraints, current task state, and PDF chunks should not become one undifferentiated index. Documents and chunks remain knowledge assets; explicit user preferences and durable project constraints enter a separately governed Memory Store; current progress belongs to Session / Task State; authentication, subscription, and device status come from operational sources of truth; and context assembly decides per task whether Memory, RAG, or both are needed.

That implementation experience makes the distinction concrete: Memory vs RAG is a lifecycle, authority, and access-control boundary, not a naming debate. Sharing SQLite, FTS, embeddings, or even a vector index does not imply shared write policy, deletion semantics, or authority.

Decision table: where should this data go?

Example	Best home	Why
last 10 messages	Session / Thread State	current continuity
workflow completed 4 of 7 nodes	State + Checkpoint	execution recovery
“answer me in Chinese by default”	Long-term User Memory	durable cross-session preference
“this repository must not use Swift”	Project-scoped Memory	durable but scoped
official API documentation	Knowledge Base + RAG	external evidence
prior deployment failed because of a missing environment variable	Episodic / Experience Memory	reusable experience
raw 500 error log from this call	Trace / Audit	should not become a durable behavior constraint by default
current subscription tier	live business source of truth	authoritative value should be fetched live
“the user probably likes minimalist design”	do not write by default	inference must not masquerade as explicit preference

A safer context-assembly order

A production request can assemble context in this order:

1. identity / authorization
2. current invocation inputs
3. session/thread state
4. active checkpoint state
5. deterministic user/project constraints
6. optional long-term memory retrieval
7. RAG / external knowledge retrieval
8. live tool results / source-of-truth data
9. prompt budget + conflict resolution
10. model inference

Steps 6 and 7 should be conditional, not mandatory.

This separation also improves debugging: when the answer is wrong, you can ask whether the failure came from bad evidence, stale memory, lost state, or authorization—not just inspect an anonymous vector search result.

Mapping the model to LangGraph, Google ADK, and custom agents

LangGraph

A practical mapping is:

• graph state + checkpointer → thread-scoped short-term memory / durable execution;
• Store → cross-thread long-term memory;
• external retriever / vector store → a concrete RAG or memory-retrieval implementation.

A Store does not automatically solve admission, permissions, conflicts, or deletion. Those remain application responsibilities.

Google ADK

A practical mapping is:

• Session → one conversation thread;
• State → dynamic data inside that session;
• MemoryService → searchable information across sessions;
• Memory Bank / RAG Memory → different implementations of long-term memory.

The ADK documentation itself is a useful warning: “Memory” can mean extracted evolving memory in one backend and vector retrieval over raw conversation data in another. Your application still has to define which semantics it expects.

Custom agents

Do not begin with “we need a memory module.”

Define:

data class
→ scope
→ authority
→ source of truth
→ write policy
→ retention
→ read policy
→ conflict policy
→ delete policy
→ evaluation

Then choose frameworks and databases.

The goal is not to remember more

Good Agent Memory is not unlimited history. It is a controlled system that lets the right information influence the right action, within the right scope, at the right time and authority level—and stop influencing behavior when it is no longer valid.

In a production architecture:

• RAG answers “what external evidence do I need now?”;
• Session State answers “what is happening in this thread?”;
• Checkpoints answer “how do I resume?”;
• Long-term Memory answers “what historical information should still matter?”;
• Knowledge Bases answer “what knowledge assets exist?”;
• Audit / Trace answers “what happened and how can we investigate it?”

Once those responsibilities are explicit, storage technology becomes the easier decision.

Further reading

• AI Agent Memory Retrieval in Practice
• Funes Agent Memory Tested
• Google ADK delete_session() and Long-Term Memory
• Context Engineering: Retrieval, Tool Search, and Memory
• AI Agent RAG Integration
• AI Agent Tool Authorization: Policy Gates, HITL, and Per-Call Access Control

Topic path / LangGraph

Continue through the production LangGraph learning path

The LangGraph hub organizes state isolation, checkpointing, human approval, retries, observability, supervisors, subgraphs and memory into one reviewable path.

Open LangGraph hub → Start with state isolation Subscribe