Benchmarks
A real, reproducible number —not a marketing claim.
Public memory-system benchmarks have a credibility problem: competing products have published wildly different scores on the same LoCoMo benchmark without disclosing matching methodology — one vendor claimed 84%, another "corrected" that to 58.44%, a third countered with 75.14%. Neither side fully agreed on what the other actually measured. So instead of a headline percentage, here's the full method, exactly what was found along the way, and what this does and doesn't prove.
50.2%
Overall accuracy — all 10 LoCoMo conversations, 1,986 questions, real GPT-4.1, scored with LoCoMo's own unmodified scoring code.
LongMemEval
A second, independent benchmark — real run in progress. Results land here once scoring is complete, on the same honest terms as the LoCoMo number above.
| Category | Questions | Accuracy |
|---|---|---|
| Multi-hop | 282 | 39.6% |
| Single-hop | 321 | 38.0% |
| Temporal | 96 | 23.1% |
| Open-domain | 841 | 52.3% |
| Adversarial (abstention) | 446 | 67.7% |
Method
cmd/hupi-bench, a real benchmark harness in this same repo — real gateway.Handler, real nightly consolidation, real retrieval, against LoCoMo's own published data (snap-research/locomo, pinned commit). Scored with LoCoMo's own, completely unmodified F1/EM scoring code — no reimplemented metric logic anywhere in this repo. Answer/consolidation model: GPT-4.1. Embedding model: text-embedding-3-small.
Is this comparable to other products' published numbers?
Not directly, and that's an honest "no," not a hedge. Competing vendors' publicly claimed figures were never confirmed to use the same category inclusion, the same scoring code, or even the same metric — an LLM-as-judge tends to score far more generously than literal word-overlap F1/EM, which is what this number uses. This score is real and reproducible on its own terms: same input data, same unmodified scoring code, full method disclosed above. Anyone can re-run the scoring script against the raw predictions and get the same number back.
What we found along the way
Getting from a first real run (24.9%) to the final number (50.2%) surfaced five real product bugs — not benchmark-only tuning — all fixed on the main branch: a consolidation bug that could fail an entire day's memory processing over one malformed relationship, a missing retry/backoff that let a single rate limit fail an entire run, a soft prompt instruction the model routinely ignored, a database table of specific extracted facts that was being written but never read back at retrieval time, and a hardcoded context budget many times smaller than this codebase's own documented target. Full write-up, including a no-memory baseline comparison and every caveat, is in the repo.
Read the full write-up on GitHub →