LongMemEval-S — Hybrid Retrieval¶
Headline result — like-for-like win over the reference
Hybrid retrieval (BM25 + all-MiniLM-L6-v2 vector fused via Reciprocal Rank Fusion, k=60) reaches Recall@1 0.892 / Recall@5 0.986 / Recall@10 0.992 / MRR 0.931, beating the agentmemory BM25+Vector reference (Recall@5 0.952 / Recall@10 0.986 / MRR 0.882). Both use the same retrieval-recall metric family on the same dataset, so this comparison is like-for-like and real.
Full 500 questions, re-confirmed 2026-08-07, not a sample
This page is the complete LongMemEval-S question set, no sampling. It replaces a run that predated the #457 weighted fusion change and whose only post-#457 evidence was a 100-question sample (#530). Recall@1 moved 0.894 → 0.892 (one question of 500), Recall@5 and Recall@10 are unchanged, MRR moved 0.9317 → 0.9307. LongMemEval-S was never affected by the #514 scoring correction: its ground truth is session IDs, which the old rule already emitted on both branches.
Run date: 2026-08-07
Retrieval mode: hybrid
Ranking unit: session (gold-blind, first occurrence wins)
Python: 3.12.13
Platform: macOS-26.5.2-arm64-arm-64bit
Sample mode: stride
Seed: 0
Limit: all
Summary¶
| Metric | Value |
|---|---|
| Questions evaluated | 500 / 500 |
| Errors | 0 |
| Skipped | 0 |
| Recall@1 | 0.8920 |
| Recall@5 | 0.9860 |
| Recall@10 | 0.9920 |
| MRR | 0.9307 |
| Latency p50 (ms) | 57.01 |
| Latency p95 (ms) | 71.91 |
By question_type¶
| question_type | n | Recall@1 | Recall@5 | Recall@10 | MRR |
|---|---|---|---|---|---|
| knowledge-update | 78 | 0.9359 | 0.9872 | 1.0000 | 0.9634 |
| multi-session | 133 | 0.9023 | 0.9925 | 0.9925 | 0.9373 |
| single-session-assistant | 56 | 1.0000 | 1.0000 | 1.0000 | 1.0000 |
| single-session-preference | 30 | 0.7000 | 0.9667 | 0.9667 | 0.8111 |
| single-session-user | 70 | 0.9000 | 0.9857 | 1.0000 | 0.9326 |
| temporal-reasoning | 133 | 0.8496 | 0.9774 | 0.9850 | 0.9019 |
Notes¶
- Retrieval mode: hybrid — ContextAssembler.assemble() is the primary path; effective mode resolves to 'hybrid'.
- Hybrid fuses BM25 (lexical) + vector (all-MiniLM-L6-v2, 384-dim, in-process numpy cosine) via Reciprocal Rank Fusion (k=60).
- Ranking unit: session — Every retrieved record is collapsed to its session ID before scoring — gold and non-gold alike. The unit is fixed by the dataset's ground-truth granularity and resolved before retrieval, so the answer key affects only the final metric (issue #514).
- LoCoMo: image-only turns skipped (text-only evaluation).
Reference Numbers¶
agentmemory BM25+Vector (all-MiniLM-L6-v2) on LongMemEval-S: - Recall@5: 95.2%, Recall@10: 98.6%, MRR: 88.2%
Popoto BM25-only baseline on LongMemEval-S (any-hit, #438): - Recall@5: 95.2%, Recall@10: 97.8%
This run used hybrid retrieval (BM25 + all-MiniLM-L6-v2 vector fused via RRF, k=60). Compare Recall@5/Recall@10 above against the BM25-only baseline and the agentmemory hybrid reference.