LoCoMo — Lexical (BM25) Retrieval¶
Retrieval regime — read the Overview first
Read this as retrieval recall, not answer accuracy. The nearest published reference point measured the same way is MEMTIER (arXiv:2605.03675), whose own LoCoMo retrieval baselines its authors describe as uninformative, so treat it as evidence that LoCoMo retrieval numbers run low across systems, not as a ranking to place Popoto within. This variant is the 10-dialogue / 5-category (adversarial included) / 1986-QA-pair LoCoMo, not the 1,540-QA / 4-category leaderboard variant.
Corrected 2026-08-07 — these numbers replace an inflated set
The pre-correction run reported Recall@1 0.2986 / Recall@5 0.5534 /
Recall@10 0.6400 / MRR 0.4124. Its scoring consulted the answer key
when collapsing retrieved turns to result IDs, so gold turns kept
their own rank slot while non-gold turns shared one — 20 retrieved
turns became 13.2 rank slots on average, lifting gold. Scoring now
ranks turn IDs for every record alike
(#514); the
superseded artifact stays committed as locomo_20260708.json.
Run date: 2026-08-07
Retrieval mode: lexical
Ranking unit: turn (gold-blind, first occurrence wins)
Python: 3.12.13
Platform: macOS-26.5.2-arm64-arm-64bit
Sample mode: stride
Seed: 0
Limit: all
Summary¶
| Metric | Value |
|---|---|
| Questions evaluated | 1986 / 1986 |
| Errors | 0 |
| Skipped | 0 |
| Recall@1 | 0.2981 |
| Recall@5 | 0.5302 |
| Recall@10 | 0.6017 |
| MRR | 0.4005 |
| Latency p50 (ms) | 7.01 |
| Latency p95 (ms) | 11.64 |
By question_type¶
| question_type | n | Recall@1 | Recall@5 | Recall@10 | MRR |
|---|---|---|---|---|---|
| 1 | 282 | 0.1312 | 0.2979 | 0.4220 | 0.2175 |
| 2 | 321 | 0.3489 | 0.6199 | 0.6729 | 0.4588 |
| 3 | 96 | 0.1042 | 0.2708 | 0.3333 | 0.1818 |
| 4 | 841 | 0.3377 | 0.5719 | 0.6397 | 0.4408 |
| 5 | 446 | 0.3341 | 0.5897 | 0.6502 | 0.4453 |
Leaderboard-parity slice¶
Categories excluded: 5 (LoCoMo cat-5 'adversarial' — see docs/benchmarks.md for the evidence audit and caveat). Re-aggregated from the per-category breakdown; comparable to the no-adversarial leaderboard variant.
| Slice | n | Recall@1 | Recall@5 | Recall@10 | MRR |
|---|---|---|---|---|---|
| Full (lexical) | 1986 | 0.2981 | 0.5302 | 0.6017 | 0.4005 |
| Parity (lexical) | 1540 | 0.2877 | 0.5130 | 0.5877 | 0.3875 |
Notes¶
- Retrieval mode: lexical — ContextAssembler.assemble() is the primary path; effective mode resolves to 'lexical'.
- Lexical uses BM25 (query-sensitive) only; no vector/embedding signal is fused in this mode.
- Ranking unit: turn — Every retrieved record is collapsed to its turn ID before scoring — gold and non-gold alike. The unit is fixed by the dataset's ground-truth granularity and resolved before retrieval, so the answer key affects only the final metric (issue #514).
- LoCoMo: image-only turns skipped (text-only evaluation).
Reference Numbers¶
agentmemory BM25+Vector (all-MiniLM-L6-v2) on LongMemEval-S: - Recall@5: 95.2%, Recall@10: 98.6%, MRR: 88.2%
Popoto BM25-only baseline on LongMemEval-S (any-hit, #438): - Recall@5: 95.2%, Recall@10: 97.8%
This run used lexical retrieval (BM25 only). Re-run with --retrieval-mode hybrid to fuse the all-MiniLM-L6-v2 vector signal (RRF, k=60) and compare against the agentmemory reference.