01Why an agent needs memory at all

A large language model is stateless: by itself it remembers nothing from one call to the next. The only thing it can "see" is whatever text you hand it in the current request. So when an agent appears to remember your name, a decision from three turns ago, or a fact it looked up yesterday, that memory isn't inside the model — it's a system built around the model that feeds the right information back in. Surveys describe agent memory along a simple cycle: encode (turn an experience into something storable), store (keep it somewhere), and retrieve (pull it back when it's relevant).

• The model is stateless — memory is plumbing the system adds around it, not a property of the model itself.
• A useful mental model is encode → store → retrieve: capture an experience, keep it, and bring it back when it matters (per memory surveys).
• Without memory an agent can't carry context across turns, learn from past attempts, or personalise to a user over time.

02Short-term vs. long-term memory

The first split to learn is short-term versus long-term. Short-term (working) memory is the information held in the model's context window during a single session or conversation thread — it's right there in front of the model, but it disappears when the session ends and it's strictly finite. Long-term memory is information deliberately persisted in an external store and recalled across sessions and threads. Production frameworks make this concrete: in LangGraph, short-term memory is handled by thread-scoped checkpointers, while long-term memory lives in cross-thread stores. The everyday metaphor — popularised by the MemGPT line of work and its framework Letta — is a computer: the context window behaves like RAM (fast, limited, wiped on restart) and the external store like disk (slower, durable, large).

• Short-term: the context window, scoped to one thread/session; immediate but finite and lost when the session ends.
• Long-term: an external store (often a database or vector store) that survives across sessions and threads.
• LangGraph maps this directly: checkpointers (short-term, per-thread) vs stores (long-term, cross-thread).
• The RAM-vs-disk analogy comes from the MemGPT/Letta "agent-as-operating-system" framing.

03The context window is the agent's working memory

Whatever sits in the context window is what the model can actually reason over right now — its working memory. The catch is that the window is finite, and Anthropic and others document that accuracy and recall tend to degrade as it fills up, a pattern sometimes called "context rot." So once a conversation grows past the budget, the agent has to do something. Common moves: OpenAI's (now-deprecated) Assistants threads auto-truncate older messages when history exceeds the model's context length; LlamaIndex keeps a short-term FIFO queue with a token limit and flushes the oldest messages into long-term memory when it overflows; and Anthropic recommends compaction and context editing — curating the working set of tokens rather than letting it grow unchecked. The skill of deciding what to keep, summarise, or drop is often called context engineering.

• The context window holds everything the model can reference for the current response — its working memory.
• It's finite, and recall/accuracy can degrade as it fills ("context rot," per Anthropic).
• When it overflows, agents truncate, summarise/compact, or flush older content out to long-term memory.

04See it work: a memory-flow simulator

Step through a short conversation and watch where each piece of information goes. New turns land in short-term memory (the context window). Important facts are also written to long-term memory (a vector store). When the context window fills, older turns are summarised and flushed out to make room. On a later turn the agent retrieves a relevant fact from long-term memory by similarity. Then try the switch: turn long-term memory OFF and replay — when the window evicts an early fact, there's nowhere to recall it from, and the agent forgets.

• Write: every turn enters short-term memory; salient facts are also copied to long-term memory.
• Summarise / flush: when the window is full, the oldest turns are compacted out (LlamaIndex FIFO flush; OpenAI truncation; Anthropic compaction).
• Retrieve: on later turns, relevant items are pulled back from long-term memory by similarity search.
• Forgetting: with long-term memory OFF, anything evicted from the window is simply gone.

05Four kinds of memory agents borrow from cognitive science

The CoALA framework (Cognitive Architectures for Language Agents) gives agent memory a useful vocabulary borrowed from how psychologists describe human memory: alongside short-term working memory, there are three flavours of long-term memory — episodic, semantic, and procedural. A note of caution: frameworks implement these labels loosely and inconsistently, so treat them as a thinking tool, not standardised definitions. Switch between them to see what each one stores and a grounded example.

How items get into and out of these stores varies. Generative Agents retrieve by combining recency, importance, and relevance scores, and periodically reflect to synthesise raw observations into higher-level insights. Some systems deliberately forget: MemoryBank reinforces or decays stored items using a model inspired by the Ebbinghaus forgetting curve. And a growing pattern is a dedicated memory layer (e.g. Mem0) that sits between the agent and storage to manage the extract/store/retrieve lifecycle — though any cost or latency reductions such vendors cite are vendor-reported and worth treating with caution.

06Check your understanding

07Take it with you & go deeper