AI Agent Memory: Short-Term, Long-Term and Retrieval

Models are stateless: memory is your design

A large language model does not remember anything between calls. Each request contains everything the model knows about the situation: instructions, the conversation so far, retrieved documents and tool results. When people say an agent "remembers" a customer, what actually happens is that the application stored something earlier and chose to put it back into the prompt.

That makes memory an engineering decision with four questions attached: what to capture, where to keep it, when to bring it back into context, and how long to keep it. Get these right and the agent stops asking returning customers the same questions, picks up a WhatsApp thread from yesterday, and stays within a sensible token budget. Get them wrong and you get bloated prompts, stale facts presented as current, or personal data resurfacing where it should not.

Memory is one layer in the wider design described in AI agent architecture. This guide goes deeper on that layer.

How memory flows through an agent

The memory types

Conversation (short-term) memory

The simplest form: keep the recent messages of the current session and resend them with each turn. It is stored in a cache or session table keyed by conversation ID, usually with a sliding window of the last N turns or last N tokens. It is what makes "and what about the blue one?" work.

Short-term memory fails in two ways. Windows that are too small lose details the user gave at the start, such as a booking reference. Windows that are too large increase cost and latency on every turn, and very long contexts can reduce accuracy because important facts get buried. Pin critical slots, such as the verified customer ID, the order being discussed and the language, in structured state rather than hoping they stay inside the window.

Summarised memory

When a conversation grows, older turns are compressed into a rolling summary: "Customer is asking about a delayed order 88213; already confirmed address; wants a refund rather than reshipment." The prompt then carries the summary plus the most recent turns. Summaries are generated by a model call, typically a smaller and cheaper model, either every few turns or when the buffer crosses a token threshold.

The trade-off is fidelity. A summary can drop a detail that matters later or phrase an assumption as a fact. Mitigate by summarising into a structured template (goal, facts confirmed, open questions, actions taken), keeping key identifiers in structured state, and retaining the raw transcript for a limited period so that a human handoff can see the original.

Long-term user and profile memory

Profile memory stores durable facts about a person or account across sessions: preferred language, dietary preference for a restaurant, room preference for a hotel guest, the child's class for a school parent, the account tier. It is best kept as structured fields in a normal database, with a source and a timestamp on each fact, rather than as free text. Structured fields are easy to show to the user, correct, expire and delete.

Often the best long-term memory is not a new store at all. If your CRM already holds the customer's plan and last order, have the agent read it through a tool instead of copying it into a memory database that will drift out of date.

Episodic vs semantic memory

These terms come from cognitive science and are useful for design discussions.

• Episodic memory is what happened: "On 2 August the customer reported a damaged item; we issued a replacement; they rated the resolution poorly." It is time-stamped, tied to a user, and retrieved when relevant, for example when the same customer contacts support again.
• Semantic memory is what is true: facts about the user ("prefers Hindi") and facts about the world or business ("the return window is 30 days"). User-level semantic facts belong in the profile. Business-level semantic knowledge belongs in the knowledge base behind retrieval.

Keeping them apart avoids a common bug: an old episode ("we offered a 20% goodwill discount") being treated as a standing fact.

Retrieval memory via a vector database

When there is too much history or content to load directly, it is embedded and stored in a vector index, and the agent retrieves the few most relevant items per turn. This suits long interaction histories, past tickets, meeting notes, and of course business documents. Retrieval memory should always be filtered by user and tenant inside the query, combined with recency where it matters, and limited to a fixed number of items. Many teams start with a vector extension in their existing Postgres database and move to a dedicated vector store only when scale requires it. The pipeline design is covered in production RAG architecture.

Comparison table

What to store and what not to store

Every memory record is personal data you now hold, secure, disclose on request and delete. Our usual approach is to start from the question "what will the agent do differently because it remembers this?" and store nothing that does not have a clear answer.

Usually worth storing

• Preferences that change the experience: language, channel, communication times, accessibility needs the user has chosen to share
• Stable identifiers needed to serve the user, stored as references to your system of record
• Outcomes of past interactions: issue type, resolution, open follow-ups
• Explicit instructions from the user: "don't call me, message me instead"

Usually not worth storing, or never

• Full card numbers, CVVs, passwords, OTPs and government ID numbers. These should never enter memory; mask them before logging and before model calls.
• Sensitive categories, such as health details, religion or financial hardship, unless the service genuinely requires them and you have a lawful basis and explicit consent
• Inferences the user did not state, such as guessed income or relationship status
• Data about third parties mentioned in passing
• Raw transcripts kept indefinitely "just in case"

Retention and the right to erasure

Data protection laws in many markets, including the GDPR in Europe and India's Digital Personal Data Protection Act, give individuals rights to access and erase their data and expect purpose limitation. For agent memory that translates into concrete requirements:

• Retention per memory type. For example: session buffers expire within hours or days, summaries and transcripts after a defined support period, profile facts reviewed or expired if unused. Set the actual periods with your legal team.
• Everything keyed to a user. Memory rows, summaries, embeddings, trace logs and analytics copies all carry a user identifier so erasure can find them.
• Deletion that reaches the index. Deleting a database row but leaving its embedding in the vector store is a common gap. Delete by ID in every store and verify with a test.
• Transparency. Tell users what the agent remembers, and ideally let them view and clear it ("forget my preferences").
• No personal data in fine-tuning. Information baked into model weights cannot be selectively erased. Keep personal memory in stores you can delete from.

Memory is also a security surface: it must be tied to verified identity so one person's facts are never loaded into another's session. See AI agent security for tenant isolation and data leakage controls.

Memory on WhatsApp and voice

WhatsApp

WhatsApp conversations do not have clean session boundaries. A customer may message on Monday, reply on Thursday and send a photo a week later, all in one thread. Design implications:

• Identity is a phone number, not a verified person. Family members share phones and numbers change hands. Load low-risk preferences on the number alone, but require verification, such as an OTP or a booking reference, before loading account details or order history.
• Use time-based session breaks. After a gap of hours or days, start a new logical session and load the thread summary rather than the raw backlog.
• Respect platform messaging rules. Proactive follow-ups outside the customer service window need approved templates, so memory of open follow-ups should feed a template-based notification flow rather than free-form messages.
• Handle media. Store references and extracted text for images, voice notes and documents with the same retention rules as text.

Voice

• Latency is the constraint. Retrieve the profile and last-interaction summary once, at call start, based on the caller, then keep per-turn memory lookups to a minimum.
• Caller ID is not authentication. It is easy to spoof. Verify before discussing account details.
• Transcription errors propagate. Names, numbers and addresses heard wrongly can be written into memory. Confirm critical values aloud before storing them.
• Summarise after the call. Write the episodic record and any profile updates asynchronously once the call ends, not during it.

For more on voice-specific design, see our AI voice agent development service.

Cost and latency trade-offs

Memory affects running cost in three places: tokens added to each prompt, background model calls for summaries and extraction, and storage and retrieval infrastructure. Token prices change often, so plug your own figures into the LLM and RAG cost calculator rather than relying on a fixed number.

Techniques that keep memory affordable: set an explicit token budget for each memory type in the prompt; run summarisation and extraction asynchronously with a small model; use provider prompt caching for stable prefixes such as system prompts and tool descriptions; cache profile reads for the session; and skip retrieval when a classifier says the turn does not need it. Memory quality should also be part of your evaluation set, with tests that check the agent uses remembered facts correctly and does not invent ones it was never told. Our guide to testing and evaluating AI agents covers this, and AI agent ROI shows how running costs feed into payback.

Common mistakes

• Storing everything. Memory full of small talk makes retrieval noisy and increases compliance exposure.
• Free-text profiles. A paragraph of "notes about the user" is hard to correct, expire or delete. Use fields with sources and timestamps.
• No conflict handling. The user moved city; now memory holds two addresses. Prefer newest facts, and confirm before acting on anything older than a set age.
• Memory without identity checks. Loading account history on an unverified channel identity is a data leak waiting to happen.
• Duplicating the system of record. Copying order data into memory creates stale answers. Read live data through tools.
• Erasure that misses the vector index, logs or backups. Test the whole path.

Browse more guides on the AI agent resources hub, or see how we build production agents on our AI agent development page.

Frequently asked questions

Do LLMs remember previous conversations on their own?

No. A model call is stateless: it only knows what is in the context you send with that call. Every form of agent memory, from the last few messages to a customer profile, is data your application stores and chooses to include in the prompt.

What is the difference between episodic and semantic memory in an AI agent?

Episodic memory records what happened in specific past interactions, such as a complaint last month and how it was resolved. Semantic memory holds general facts, either about the user, such as a preferred language, or about the business, such as policies and product details. Episodic memory is usually searched by time and similarity; semantic memory is usually looked up directly or retrieved from a knowledge index.

Do we need a vector database for agent memory?

Not always. Short-term buffers, summaries and structured profile fields work well in an ordinary database or cache. A vector index becomes useful when you must search large volumes of unstructured history or documents by meaning. Many teams add vector search to an existing Postgres database before adopting a separate vector store.

How do we handle a right-to-erasure request for an agent's memory?

Key every memory record, summary, embedding and log entry to a user identifier so you can find it. Deletion must cover the primary store, the vector index, caches, backups according to their rotation policy, and any analytics copies. Test the deletion path before launch, and avoid using personal conversations for fine-tuning, because data inside model weights cannot be selectively removed.

Does long-term memory make an AI agent more expensive to run?

It can, mainly through extra tokens in every prompt and background summarisation or extraction calls. Well-designed memory often lowers cost instead, because a compact summary and a few retrieved facts replace resending a long conversation history on every turn. Set a token budget for memory in each prompt and measure it.