Memory Systems for AI Agents: What's Actually Worth Building
Every agent project hits the same wall eventually. The model is smart, the model is fast, and the model knows nothing about your codebase, your documents, or what it did yesterday. So you go looking for a memory system, and the field obliges with an alphabet of options: RAG, agentic RAG, GraphRAG, wiki memory, cognitive memory. Vendor pages make all of them sound mandatory.
None of them are mandatory. The expensive ones are not reliably better, either. I have run most of these patterns in production and in real life. The rule I keep coming back to is that the value of a memory system tracks how well you can maintain it; sophistication mostly raises the maintenance bill. I want to walk that ladder one rung at a time, because where you stop is the real decision.
Simple RAG is the workhorse
A query goes in, it gets embedded, a vector database (or a hybrid vector plus keyword index) returns the nearest chunks, the model answers from them. That is the whole mechanism, and it is a decade-old pattern by now.
It handles semantic search over documents that change slowly: FAQs, internal docs, manuals, product catalogs. “How do I rotate the API keys” retrieves the right page even when the page says “regenerate credentials”, because the embeddings care about meaning, not spelling.
The limitation is equally simple: the query is used as-is. Whatever the user typed is what gets embedded. Vague questions retrieve vaguely relevant chunks. Questions that need three facts get the chunks for one of them. Nobody reformulates anything, because there is nobody in the loop to reformulate.
That is fine more often than you would think. Most questions people ask an assistant are single-hop lookups over text. For those, simple RAG with a decent hybrid index is the correct amount of machinery.
Keep grep in the stack
Here is the one that gets left out of every “AI-native memory” pitch: plain keyword search, sometimes literal grep. It is the other half of the 80%.
Not everything shares semantic space. Code symbols, error strings, IDs, config keys, function names, log lines. If an agent needs to find every call site of processRefund, or the exact error string a job threw at 3am, embeddings are the wrong tool. They will blur the exact match into a neighborhood of similar-looking strings and miss the one that matters. Grep does not miss it. Grep cannot miss it; matching the literal string is the entire job.
Every coding agent worth using reaches for search tools before embeddings, for exactly this reason. The lesson generalizes past code: when your corpus is full of exact identifiers, you want exact retrieval, and being boring about it is a feature. It also costs nothing to run and nothing to maintain beyond the files themselves, which puts it in a small club of infrastructure that never pages anyone.
Agentic RAG: the loop is the product
Agentic RAG’s move is to think about the search before doing it. An agent decomposes the question, formulates queries, retrieves, looks at what came back, decides whether it is sufficient, rewrites, retrieves again, and synthesizes across everything it gathered. The retrieval step itself is unchanged; the intelligence lives in the planning loop wrapped around it.
That loop is genuinely useful for vague, multi-hop, research-shaped questions. “Why did our checkout conversion drop after the redesign” is a whole research project, and query planning is the difference between a synthesis and a shrug.
It is also where the bill lives. A recent ACL industry-track evaluation (arXiv:2601.07711) compared well-tuned simple pipelines against agentic ones and found agentic RAG ran up to 3.6 times more expensive, burning 2.7x to 3.9x the input tokens and about 1.5x the latency. And the part nobody puts on the landing page: the optimized simple baseline matched or exceeded agentic RAG on most of their benchmarks. Agentic won in narrow, well-structured domains where understanding user intent is the hard part, and lost on general, noisy ones.
Add it up: 3 to 4 times the cost, half again the latency, better answers precisely when the question is ambiguous and multi-hop. If your users ask research questions, that can be a bargain. If they ask lookups, it is a tax.
Graph RAG: rarely worth the squeeze
GraphRAG stores documents plus typed relationships between them in a knowledge graph, so retrieval can follow edges: X depends on Y, A caused B, this service owns that table. In theory it enables the relational reasoning that flat chunk retrieval cannot do.
The theory is fine. The maintenance is brutal. You now have two things to keep current, content and relations, and every document update churns both. An edge that was true yesterday is stale today, and unlike a stale chunk, a stale edge does not look wrong; it confidently returns outdated relationships. On a dynamic corpus, the graph decays continuously, and re-extracting relations with LLM calls at write time is exactly as expensive as it sounds.
For a small, stable, deeply relational corpus (a compliance map, an ontology that changes quarterly), it can pay off. For everything else I have seen it applied to, the answer quality gain over a good hybrid index was marginal and the upkeep was not. The juice is rarely worth the squeeze, and I say that as someone who wanted it to work.
The LLM wiki: a graph you can read
My default for curated knowledge now is the pattern Andrej Karpathy sketched: skip the graph database, keep a wiki of interlinked markdown pages that the agent maintains itself. When a source comes in, the agent compiles it once: updates the relevant entity pages, adds links, notes contradictions, files the answer. At query time it reads pages the way a person would.
You get most of what GraphRAG promises, relations included, with none of the infrastructure. Links are just text. Pages are human-readable, diffable in git, and editable by hand when the agent gets something wrong. My own notes run this way, and it is the first memory pattern I have not had to babysit.
Two honest caveats. First, markdown links have no referential integrity: rename a page and every link to it breaks silently, so stale references accumulate until you lint for them. Second, it caps out. Past a couple hundred curated sources, asking the agent to keep every page current on every ingest stops scaling, and you end up wanting RAG over the raw sources as a fallback. The ceiling is real, but for a curated base it is a long way up.
Memory about the user is a different animal
Everything above remembers facts about a corpus. The last category remembers the person: their preferences, their projects, how they like things done. It keeps getting bundled into the same “agent memory” bucket, and it should not be.
Two systems worth studying here. Honcho from Plastic Labs is memory infrastructure built specifically for modeling users, agents, groups, and projects as they change over time, rather than retrieving similar chunks of past conversation. And CrewAI rebuilt their memory system from scratch around the principle that “memory is cognition, not storage”: on write, it extracts atomic facts, infers scope and importance, and consolidates contradictions instead of appending forever; on read, it ranks by relevance, recency, and importance, and it can deliberately forget. Their founder’s line that most agent memory systems make agents worse is only half a joke. Half-remembered context injected into every prompt is a cost you pay in attention.
If personalization is the product, this category matters as much as retrieval. It is also a big enough topic to deserve its own post, so I will leave it there.
What I actually build
The decision rule I use, in order:
- Start with hybrid simple RAG plus grep. Together they cover roughly 80% of real questions, cost cents, and need almost no maintenance.
- Add the agentic loop only for question flows that are vague or multi-hop by nature, and check the cost math first. A 3.6x token multiplier at your query volume is a number you can know in advance.
- Consider a wiki instead of a graph when the corpus is curated and you want humans able to read and fix the memory. Reach for an actual graph database only when the corpus is small, stable, and the questions are relational at their core.
- Treat user memory as its own project; build it only when personalization is the point.
The meta-rule underneath all four: dynamic content is the enemy of every structured approach. The faster your corpus changes, the lower you should sit on the ladder, because the bottom rungs re-index cheaply and the top rungs rot.
Start at the bottom. Move up one rung at a time, and only when a real failure at the current rung demands it. The best memory system is the one your future self will still be maintaining six months from now.
