Eleven agent memory systems lost to plain RAG over raw history in 90 of 99 tests

Agent memory has turned into its own product category. Systems such as A-Mem, MemOS and MemoryOS promise that an assistant which extracts, summarizes and links what it has seen will recall it better later. A new paper from researchers at UIUC, Tsinghua and several other institutions tests that promise against a baseline the field mostly skips: chunking the raw conversation history, embedding it, and retrieving from it.

The baseline comes out ahead most of the time. The authors ran eleven open-source memory systems on three long-term memory benchmarks (LongMemEval-s, PersonaMem-v2 and MINT-Eval), each with three backbone models (Qwen3.5-4B, Qwen3.6-27B and GPT-5.6 Luna at low effort). That makes 99 system-benchmark-backbone combinations. In 90 of them the memory system scored lower than two-round RAG over the raw history. In the nine where it scored higher, the margin was at most 3.9 points. The worst loss was 40.4 points.

Where memory helps and where it doesn't

On LongMemEval-s, the most widely used of the three benchmarks, no memory system beat RAG on any backbone. With Qwen3.6-27B, RAG reached 81.4% and the best memory system, A-Mem, 79.4%. With GPT-5.6 Luna, RAG reached 90.0%, ahead of every memory system (A-Mem and SimpleMem tied at 84.0%), and ahead of simply putting the entire history in the context window (84.0%).

The wins cluster on the other two benchmarks, which the authors describe as asking for more aggregation and preference inference. On PersonaMem-v2, MemOS beat RAG with both Qwen models, by 1.0 and 1.5 points, and Memory-R2 by 2.0 with the 4B model. On MINT-Eval, the trained memory policy Mem-T and the graph-based PlugMem came out ahead, mostly with GPT-5.6 Luna (by 3.9 and 3.3 points). No system won consistently. MemOS, which beat RAG on PersonaMem-v2, lost by 13.4 points on MINT-Eval with the same backbone.

The authors call this gap the "memory tax", and they go looking for where it comes from. Part of it is information that simply doesn't survive construction. In a hand-annotated sample of facts, OpenViking, from ByteDance's cloud unit Volcengine, kept 39.4% intact and produced no memory unit at all for 38.0%. MemoryOS and MAGMA preserved almost everything, though, and still lost to RAG. For those, the paper's explanation is that evidence becomes harder to find: once facts are split up, merged with unrelated material or rephrased, they match the question less well and need more retrieval rounds. The authors also find that many memory systems are more sensitive than RAG to swapping the model that reads the memory.

The bill for building it

Building the memory also costs a lot. On the 100-conversation LongMemEval-s subset run with GPT-5.6 Luna, RAG's only offline step, embedding the history, took 0.11 hours. A-Mem took 117 hours of end-to-end building and MemoryOS 126.5 hours, roughly a thousand times longer, to reach 84% and 80% against RAG's 90%. Building and maintaining memory consumed between 5 million tokens (LightMem) and 232 million (Mem-T), against RAG's zero. These are wall-clock times from the authors' own runs, so better parallelism would shrink them, but the token counts don't depend on scheduling.

Was RAG given an unfair edge?

When a baseline wins this broadly, the first question is whether the comparison was rigged in its favor. The setup points the other way. The authors tuned each memory system's top-k upward from its default so that it saw at least as large a share of its store as single-round RAG. In the GPT-5.6 Luna run, OpenViking was shown 6.7% of its memory against two-round RAG's 2.75% of the raw history, and still finished 27 points behind. All systems, RAG included, share the same small embedding model, Qwen3-Embedding-0.6B.

The paper also checks how often the original papers made this comparison. Of the systems tested here, plus a few influential ones such as Mem0, Zep and MemGPT, only LightMem, PlugMem and Mem-T report a RAG baseline at all, each with its own chunking and retrieval setup. That may be the most useful finding for anyone choosing a memory layer: the comparison that matters most is often the one that was never run.

The caveats

There are real limits. Two of the three benchmarks were subsampled for cost: 20% of PersonaMem-v2 (934 questions) and 10% of MINT-Eval (1,557). The GPT-5.6 Luna LongMemEval-s run covers 100 conversations, so one question moves the score by a point. Answers were graded by Qwen3.6-27B, which is also one of the tested backbones. The authors report 95-100% agreement with human graders on samples. Mem0 and Zep, two of the most widely deployed memory products, are cited but not tested. And the benchmarks are static, offline and English-only, which the authors acknowledge rewards accurate recall more than the long-running, shifting use that memory systems are designed for.

Reproducibility is partial. The project repository still lists "Release reproduction codes of each baseline" as unfinished, and the harness folders are empty placeholders. The raw outputs are on Hugging Face, though: the Qwen3.6-27B result set alone has more than 17,000 files. So the grading can be audited even though the runs can't yet be repeated.

The authors' proposed fix is modest: keep a path back to the raw history and retrieve from it alongside the memory store, reranking both. That is, in effect, an admission that the baseline should be part of the system. In experiential memory for agents, where past task traces are distilled into strategies, the picture is more mixed: distilled workflows beat RAG on web tasks but fell behind it on tool use. For conversational memory, the paper's advice is simpler. Before adopting a memory system, measure it against RAG on the raw logs.