Three Papers Say Your Agent's Memory Loses to Plain RAG

What gets lost when an agent rewrites its history into memory, and a laptop test of where it happens

Between 21 and 26 October, three separate groups posted the same uncomfortable result. Build an elaborate memory for your agent (fact extraction, graphs, hierarchies, learned policies), then compare it against plain retrieval over the raw chat history. Plain retrieval often wins.

I've built memory layers like this, so I read all three. Then I ran the cleanest test from one of them on my laptop, to see where the loss actually happens.

💡 TL;DR & Key Takeaways:
Three recent papers demonstrate that elaborate agent memory systems often underperform plain RAG over raw chat history, a phenomenon attributed to a "memory tax" caused by fact loss during ingestion, retrieval fragmentation, and model fragility. The author validates these findings with a local reproduction and outlines specific scenarios where memory remains valuable, such as when using weak reader models or when the task requires distilled know-how rather than simple recall.

* **The Memory Tax:** He et al. identify three sources of performance loss compared to RAG: facts lost during the write/ingestion phase, evidence that survives but becomes harder to retrieve due to fragmentation or rephrasing, and increased fragility when the model used for reading differs from the one used for building the memory.
* **When Memory Wins:** Memory systems are still beneficial when the reader model is too weak to process long raw contexts, when the task requires distilled workflows rather than raw recall, or when a single history is queried enough times to amortize the high initial build cost.
* **Practical Recommendations:** To mitigate the tax, authors should report naive RAG baselines alongside memory results, maintain a retrieval path to the raw history for reranking, treat the extraction model as a hard accuracy ceiling, and test memory systems with the specific models intended for production use.

What does agent memory add over plain RAG?

Every memory system does the same three things. It ingests the conversation history and turns it into something: extracted facts, summaries, a graph, tiered stores. It retrieves some of that at question time. And a model answers from what came back.

Naive RAG skips the first step. It chunks the raw history, embeds it, and retrieves chunks. That makes it the honest baseline: whatever a memory system scores above it is what the ingestion bought you. Whatever it scores below is what the ingestion cost.

Diagram: naive RAG retrieves from the raw history directly. A memory system first rewrites the history into a store, and both the writing step and the retrieval over rewritten units can lose evidence that RAG would have found.

He et al. at UIUC and Tsinghua call that cost the memory tax, and they ran the comparison at scale: 11 open-source memory systems, three benchmarks, three backbones, all sharing the same embedding model, with retrieval budgets aligned so that nobody wins just by reading more tokens.

How often does memory actually beat RAG?

On LongMemEval-s, never. Here's their Table 1 with the Qwen3.6-27B backbone:

None of the eleven beats two-round RAG on that benchmark with any of the three backbones. On the harder PersonaMem-v2 and MINT-Eval a few systems do pull ahead in some cells (Mem-T is +3.9 on MINT-Eval with GPT-5.6 Luna, for example). So the honest version of the headline is "on these benchmarks", not "memory is useless".

The other two papers land in the same place from different directions. Bao et al.'s RAW-RAG, dense retrieval plus reranking over raw history, ranks first in six of seven benchmark-model settings, 0.13 to 5.53 points above the best agentic memory system. Peng's MemoryTax: The Bitter Lesson of Agent Memory is a small study, one LoCoMo conversation and one run per cell, but it adds cost: Mem0's build alone cost more than a whole full-context run with the same model ($25.88 against $17.84 with Claude Opus 5.5), and both ended at 95%.

Where does the memory tax come from?

This is the part I found most useful, because it tells you what to check in your own system. He et al. split the tax into three sources.

Facts lost at write time. They traced sampled facts from the raw history into each system's store. The share that survived with no loss ranges from 98.4% for MemoryOS down to 39.4% for OpenViking, and 38.0% of OpenViking's sampled facts had no memory unit at all. If the extractor didn't write it down, no retriever will find it.

Evidence that's there but harder to reach. Even when the fact survives, memory construction can fragment it across units, dilute it with unrelated content, or rephrase it until it no longer matches how the user asks. They measured how many retrieval rounds it takes to recover all the gold evidence, and memory stores recover fewer queries than raw chunks in the first round.

More fragility. Build the memory with one model and read it with another, and many memory systems lose more accuracy than RAG does under the same swap. Long-lived agents change models, so this one bites in production.

Peng adds a sharper version of the first point: the model that writes the memory sets a ceiling. On a Zep graph written by GPT-4o-mini, readers from GPT-4o-mini up to Claude Opus 5.5 score 43%, 43% and 45%. A stronger reader can't recover what a weaker writer left out.

What happens when I run the test myself?

When is building memory still worth it?

None of the three papers says never. Read together, they point at three cases:

• The reader is weak. In Peng's study, Mem0 beat full context only with GPT-4o-mini (78% against 74%). A model that can't use a long raw context benefits from someone pre-chewing it.

• The task needs distilled know-how, not recall. In He et al.'s experiential-memory tests, distilling past traces into workflows beat naive RAG on web tasks and lost on tool-use tasks.

• Many questions hit one history. Mem0's cheaper per-question cost would repay its build after 372 to 596 questions on the same history in Peng's numbers. LoCoMo asks 152.

The benchmark matters too. He et al. show that a bare IDF-keyword baseline already scores above 50% on some memory benchmarks, and RAG-Hybrid around 70%. When retrieval alone solves most queries, there's little room for memory to add anything, and any loss it introduces shows up clearly.

Rules I'm taking from it

My reading, not measurements:

• Report naive RAG over raw history next to every memory result. Without it, you can't tell what the memory bought.

• Keep a path back to the raw history. He et al.'s simplest fix, retrieving from the source alongside memory and reranking both, took SimpleMem on LongMemEval-s from 73.6% to 82.8% and cut its tax from 12.0 to 5.2 points.

• Treat the extraction model as part of your accuracy budget. A cheap writer caps every reader after it.

• Test memory with the model you'll swap to, not only the one you built with.

• Count the build cost per question you actually expect, not per benchmark run.

I hit the same shape in a different domain a few weeks ago: on Vietnamese legal retrieval, BM25 beat the RAG frameworks on the retrieval metrics. Two unrelated benchmarks, same lesson. The cheap baseline is a real baseline, and it has to be in the table.

All three papers are English-only, and He et al. list that as a limitation. I don't have a Vietnamese long-term-memory benchmark to run this on, so I'm not going to claim anything about it. If you know of one, tell me.

How to check this yourself

from __future__ import annotations

import argparse
import collections
import json
import os
import random
import sys
import time
import types
import torch  # noqa: E402
from transformers import AutoModelForCausalLM, AutoTokenizer  # noqa: E402

HERE = os.path.dirname(os.path.abspath(__file__))
DATA = os.path.join(HERE, "..", "data", "longmemeval_oracle.json")
RESULTS = os.path.join(HERE, "..", "results")

TYPES = ["single-session-user", "single-session-assistant", "multi-session",
         "knowledge-update", "temporal-reasoning"]

BUILD_PROMPT = """Below is one past conversation between a user and an AI assistant, dated {date}.
Write the memory an assistant should keep from it: a list of short, self-contained facts.
Cover facts about the user (events, possessions, plans, numbers, names, dates, preferences)
and any concrete information the assistant gave. Resolve relative dates against the session
date when you can. One fact per line, each starting with "- ". No commentary.

Conversation:
{conv}"""

ANSWER_PROMPT = """You are a personal assistant with memory of past conversations with this user.
Use only the context below to answer. The current date is {qdate}.

Context:
{ctx}

Question: {q}
Answer concisely."""

# Adapted from the LongMemEval judge prompts (Wu et al., 2025).
JUDGE_PROMPT = """I will give you a question, a correct answer, and a response from a model. Answer yes if the response contains the correct answer, or is equivalent to it, or contains all the intermediate steps needed to get it. If the response contains only a subset of the required information, answer no.{extra}

Question: {q}
Correct answer: {a}
Model response: {r}

Is the model response correct? Answer yes or no only."""

EXTRA = {
    "temporal-reasoning": " Do not penalize off-by-one errors in counts of days, weeks or months.",
    "knowledge-update": " If the response contains old information alongside the updated answer, it is still correct as long as the updated answer is what it gives.",
}


class LM:
    def __init__(self, model_id, device):
        self.id = model_id
        self.tok = AutoTokenizer.from_pretrained(model_id)
        self.model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to(device).eval()
        self.device = device
        self.kw = {"enable_thinking": False} if "qwen3" in model_id.lower() else {}

    @torch.no_grad()
    def __call__(self, prompt, max_new):
        text = self.tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False,
                                            add_generation_prompt=True, **self.kw)
        enc = self.tok(text, return_tensors="pt").to(self.device)
        out = self.model.generate(**enc, do_sample=False, max_new_tokens=max_new,
                                  pad_token_id=self.tok.pad_token_id or self.tok.eos_token_id)
        text = self.tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True).strip()
        if self.device == "mps":
            torch.mps.empty_cache()
        return text


def device():
    return "mps" if torch.backends.mps.is_available() else ("cuda" if torch.cuda.is_available() else "cpu")


def pick(n, seed):
    data = json.load(open(DATA))
    data = [x for x in data if x["question_type"] in TYPES and not x["question_id"].endswith("_abs")]
    rng = random.Random(seed)
    by = collections.defaultdict(list)
    for x in data:
        by[x["question_type"]].append(x)
    per = n // len(TYPES)
    return [x for t in TYPES for x in rng.sample(by[t], per)]


def conv_text(session):
    return "\n".join(f"{t['role']}: {t['content']}" for t in session)


def jl_read(path):
    if not os.path.exists(path):
        return []
    return [json.loads(ln) for ln in open(path)]


def out_dir(tag):
    d = os.path.join(RESULTS, tag) if tag else RESULTS
    os.makedirs(d, exist_ok=True)
    return d


def cmd_build(a):
    path = os.path.join(out_dir(a.tag), "memory_units.jsonl")
    done = {r["question_id"] for r in jl_read(path)}
    lm = LM(a.builder, device())
    t0 = time.time()
    qs = pick(a.n, a.seed)
    with open(path, "a") as f:
        for j, x in enumerate(qs):
            if x["question_id"] in done:
                continue
            units = []
            for date, sess in zip(x["haystack_dates"], x["haystack_sessions"]):
                out = lm(BUILD_PROMPT.format(date=date, conv=conv_text(sess)), a.max_new)
                facts = [ln[2:].strip() for ln in out.splitlines() if ln.strip().startswith("- ")]
                units.append({"date": date, "facts": facts})
            f.write(json.dumps({"question_id": x["question_id"], "builder": a.builder, "units": units}) + "\n")
            f.flush()
            print(f"build {j + 1}/{len(qs)} {sum(len(u['facts']) for u in units)} facts {time.time() - t0:.0f}s", flush=True)


def contexts(x, mem):
    gold_turns = []
    for date, sess in zip(x["haystack_dates"], x["haystack_sessions"]):
        ts = [f"[{date}] {t['role']}: {t['content']}" for t in sess if t.get("has_answer")]
        gold_turns += ts
    sessions = "\n\n".join(f"[Session on {d}]\n{conv_text(s)}"
                           for d, s in zip(x["haystack_dates"], x["haystack_sessions"]))
    units = "\n".join(f"[{u['date']}] {fct}" for u in mem["units"] for fct in u["facts"])
    return {"gold_turns": "\n".join(gold_turns), "gold_sessions": sessions, "memory_units": units}


def cmd_answer(a):
    path = os.path.join(out_dir(a.tag), "answers.jsonl")
    done = {(r["question_id"], r["context"]) for r in jl_read(path)}
    mem_tag = a.tag if a.memory_from is None else a.memory_from
    mems = {r["question_id"]: r for r in jl_read(os.path.join(out_dir(mem_tag), "memory_units.jsonl"))}
    qs = [x for x in pick(a.n, a.seed) if x["question_id"] in mems]
    lm = LM(a.answerer, device())
    t0 = time.time()
    with open(path, "a") as f:
        for j, x in enumerate(qs):
            for name, ctx in contexts(x, mems[x["question_id"]]).items():
                if (x["question_id"], name) in done:
                    continue
                r = lm(ANSWER_PROMPT.format(qdate=x["question_date"], ctx=ctx, q=x["question"]), 128)
                f.write(json.dumps({"question_id": x["question_id"], "type": x["question_type"],
                                    "context": name, "answerer": a.answerer, "ctx_chars": len(ctx),
                                    "response": r}) + "\n")
                f.flush()
            print(f"answer {j + 1}/{len(qs)} {time.time() - t0:.0f}s", flush=True)


def cmd_judge(a):
    path = os.path.join(out_dir(a.tag), "judged.jsonl")
    done = {(r["question_id"], r["context"]) for r in jl_read(path)}
    gold = {x["question_id"]: x for x in json.load(open(DATA))}
    lm = LM(a.judge, device())
    with open(path, "a") as f:
        for r in jl_read(os.path.join(out_dir(a.tag), "answers.jsonl")):
            if (r["question_id"], r["context"]) in done:
                continue
            x = gold[r["question_id"]]
            v = lm(JUDGE_PROMPT.format(extra=EXTRA.get(x["question_type"], ""), q=x["question"],
                                       a=x["answer"], r=r["response"]), 4)
            r = dict(r, judge=a.judge, verdict=v, correct=v.lower().startswith("yes"))
            f.write(json.dumps(r) + "\n")
            f.flush()


def cmd_report(a):
    rows = jl_read(os.path.join(out_dir(a.tag), "judged.jsonl"))
    acc = collections.defaultdict(list)
    for r in rows:
        acc[(r["type"], r["context"])].append(r["correct"])
        acc[("ALL", r["context"])].append(r["correct"])
    ctxs = ["gold_turns", "gold_sessions", "memory_units"]
    print("| Question type | n | " + " | ".join(ctxs) + " | tax (turns - units) |")
    print("|---|---|" + "---|" * (len(ctxs) + 1))
    for t in TYPES + ["ALL"]:
        cells = [acc[(t, c)] for c in ctxs]
        if not cells[0]:
            continue
        pct = [100 * sum(c) / len(c) for c in cells]
        print(f"| {t} | {len(cells[0])} | " + " | ".join(f"{p:.1f}%" for p in pct) +
              f" | {pct[0] - pct[2]:+.1f} pp |")
    # per-query overlap, as in the paper's Venn plot
    by = collections.defaultdict(dict)
    for r in rows:
        by[r["question_id"]][r["context"]] = r["correct"]
    both = sum(1 for v in by.values() if v.get("gold_turns") and v.get("memory_units"))
    only_raw = sum(1 for v in by.values() if v.get("gold_turns") and not v.get("memory_units"))
    only_mem = sum(1 for v in by.values() if not v.get("gold_turns") and v.get("memory_units"))
    print(f"\nPer-query: both {both}, raw-only (tax) {only_raw}, memory-only (benefit) {only_mem}, n={len(by)}")
    mem_tag = a.tag if a.memory_from is None else a.memory_from
    mems = jl_read(os.path.join(out_dir(mem_tag), "memory_units.jsonl"))
    nf = [sum(len(u["facts"]) for u in m["units"]) for m in mems]
    print(f"Memory units per question: mean {sum(nf) / max(len(nf), 1):.1f}")
    lens = collections.defaultdict(list)
    for r in rows:
        lens[r["context"]].append(r["ctx_chars"])
    print("Mean context chars: " + ", ".join(f"{c} {sum(v) / len(v):.0f}" for c, v in lens.items()))


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("cmd", choices=["build", "answer", "judge", "report"])
    ap.add_argument("--n", type=int, default=60)
    ap.add_argument("--seed", type=int, default=0)
    ap.add_argument("--builder", default="Qwen/Qwen3-1.7B")
    ap.add_argument("--answerer", default="Qwen/Qwen3-1.7B")
    ap.add_argument("--judge", default="Qwen/Qwen3-4B")
    ap.add_argument("--max-new", type=int, default=400)
    ap.add_argument("--tag", default="", help="results subfolder; empty = results/")
    ap.add_argument("--memory-from", default=None,
                    help="tag whose memory_units.jsonl to read (writer/reader swap); default = --tag")
    a = ap.parse_args()
    {"build": cmd_build, "answer": cmd_answer, "judge": cmd_judge, "report": cmd_report}[a.cmd](a)


if __name__ == "__main__":
    main()

Share your thoughts in the comments — I'd love to hear how this technology is impacting your industry.

Follow me: LinkedIn | GitHub