Skip to content

Benchmarks — Overview

Popoto's memory retrieval is evaluated against published datasets. Every page in this section is auto-generated at build time from the committed benchmark artifacts under tests/benchmarks/results/ — a new benchmark run only needs to commit its refreshed _latest artifact and these pages update on the next deploy, with no hand-edited tables to drift.

Metric families are not convertible

Popoto reports retrieval recall (any-hit Recall@k / MRR): did the correct evidence appear in the top-k retrieved memories? The widely-cited "LoCoMo leaderboard" systems (Hindsight, Backboard, Dakera, Memori, ByteRover, RGMem, Mem0, Zep) report LLM-as-judge answer accuracy: did a language model produce a correct final answer? These are different metric families and are not convertible in either direction, so Popoto's recall is never tabulated beside a judge-accuracy percentage. (Note: Dakera advertises "88.2% recall", but their methodology is judge-scored: it is not retrieval recall despite the name.)

Popoto's own judged answer accuracy is published, with its N, its interval, and its protocol, in Benchmarking → Judged-Answer Accuracy.

LongMemEval-S is the headline

LongMemEval-S is the retrieval benchmark we lead with — MEMTIER (arXiv:2605.03675) argues it is the more appropriate retrieval benchmark, and the hybrid win over the agentmemory reference is a like-for-like comparison in the same metric family on the same dataset.

Benchmark Recall@1 Recall@5 Recall@10 MRR
LongMemEval-S — Lexical (BM25) Retrieval 0.8560 0.9520 0.9780 0.8987
LongMemEval-S — Hybrid Retrieval 0.8920 0.9860 0.9920 0.9307

LoCoMo — read in the retrieval regime

Correction (2026-08-07): every LoCoMo number below was re-measured after a scoring defect that consulted the answer key when collapsing retrieved turns to result IDs was fixed (#514). The corrected figures are lower. All four arms (lexical, hybrid, graph, judged) now run under gold-blind scoring (#530), but their coverage differs and is not uniform: lexical is the full 1986 questions, hybrid is a 250-question stratified sample (a full hybrid pass measured ~5.2 h), graph is the full 282-question multi-hop slice, and judged is 100 questions sampled from a 2-dialogue subset. Each page states its own sample mode and limit in the run header; read that before comparing two pages.

MEMTIER anchor (arXiv:2605.03675)

MEMTIER is the nearest published work that measures LoCoMo as retrieval rather than as judged answers, which makes it the right anchor for latency and for the shape of the regime. It reports hybrid-RRF retrieval at 96.7 ms/query on comparable hardware. Its own LoCoMo retrieval scores are baselines its authors describe as uninformative, so they are not used here to place Popoto on a scale.

Variant: this is the LoCoMo variant with 10 dialogues, 5 categories (including adversarial), 1986 QA pairs (as described in Omni-SimpleMem, arXiv:2604.01007) — not the common leaderboard variant of 1,540 QA pairs / 50 dialogues / 4 categories (adversarial excluded). The two are easy to cross-read; they are not comparable.

Benchmark Recall@1 Recall@5 Recall@10 MRR
LoCoMo — Lexical (BM25) Retrieval 0.2981 0.5302 0.6017 0.4005
LoCoMo — Hybrid Retrieval 0.3400 0.5120 0.5880 0.4172

Category-5 (adversarial) caveat

Adversarial is historically the hardest category industry-wide (the original LoCoMo paper reports humans ≈89 F1 vs LLMs ≈2 F1). Popoto's category-5 scoring comparably to the other categories (corrected lexical Recall@1 0.3341) is presented here as a factual observation, not a strength: it may indicate the harness matches populated evidence spans rather than exercising refusal behavior. An evidence-matching audit is filed separately; until it resolves, read cat-5 numbers with this caveat.

Deterministic CSR regression gate

The CSR page reports the deterministic Constraint Satisfaction Rate harness — a per-PR CI regression gate whose scores are bit-identical every run (no LLM judge, no embeddings, no Redis module). Its numbers are report-only regression signals, not a leaderboard metric.

Further reading