Benchmarks
A real, reproducible number -not a marketing claim.
Public memory-system benchmarks have a credibility problem: competing products have published wildly different scores on the same LoCoMo benchmark without disclosing matching methodology — one vendor claimed 84%, another "corrected" that to 58.44%, a third countered with 75.14%. Neither side fully agreed on what the other actually measured. So instead of a headline percentage, here's the full method, exactly what was found along the way, and what this does and doesn't prove.
50.2%
Overall accuracy — all 10 LoCoMo conversations, 1,986 questions, real GPT-4.1, scored with LoCoMo's own unmodified scoring code.
52.1%
LongMemEval overall accuracy — a 48-instance stratified sample, real GPT-4.1 answering, real GPT-4o judge, scored with LongMemEval's own unmodified code.
LoCoMo — by category
| Category | Questions | Accuracy |
|---|---|---|
| Multi-hop | 282 | 39.6% |
| Single-hop | 321 | 38.0% |
| Temporal | 96 | 23.1% |
| Open-domain | 841 | 52.3% |
| Adversarial (abstention) | 446 | 67.7% |
LongMemEval — by question type
| Question type | Questions | Accuracy |
|---|---|---|
| Single-session (user) | 8 | 100% |
| Single-session (assistant) | 8 | 100% |
| Temporal reasoning | 8 | 50% |
| Multi-session | 8 | 37.5% |
| Knowledge update | 8 | 25% |
| Single-session (preference) | 8 | 0% |
single-session-preference scores low because that category's expected "answer" is a personalized-recommendation rubric, not a fact to recall — an answer-style gap, not a memory-recall failure. Full explanation in the write-up below.
Diagnostic evaluation — EvalMem
EvalMemis a different kind of evaluation: instead of one opaque accuracy score, it decomposes a wrong answer into where it actually failed — encoding (was the fact ever stored?), retrieval (was it surfaced?), or generation (could the model use it once given?). Ran EvalMem's real three-probe pipeline against one full LoCoMo conversation (19 sessions, 199 questions), real GPT-4.1 answering, real GPT-4o-mini judge —199/199 questions, 0 errors.
82.2%
Abstention accuracy — correctly declines to answer when nothing relevant is in memory.
29.1%
EvalMem's own strict, LLM-judged final_accuracy —not comparable to the F1/EM accuracy numbers above (a different, harsher rubric, on one conversation rather than the full published set).
The genuinely useful signal from a diagnostic run is where the misses trace to, not the raw score: retrieval and generation defects dominated over encoding defects, meaning most facts asked about were actually stored in memory — the gap is more often in surfacing or using them well, not losing them outright. Two real bugs were found and fixed while running this: one in EvalMem's own strict-mode judging logic, and one in HUPI's own answer tool, which was missing the same answer-style tuning the LoCoMo/LongMemEval harness already uses (fixing it nearly tripled the score above, from 9.5% to 29.1%).
Read the full EvalMem integration write-up on GitHub →Method
cmd/hupi-bench, a real benchmark harness in this same repo — real gateway.Handler, real nightly consolidation, real retrieval, against LoCoMo's own published data (snap-research/locomo, pinned commit). Scored with LoCoMo's own, completely unmodified F1/EM scoring code — no reimplemented metric logic anywhere in this repo. Answer/consolidation model: GPT-4.1. Embedding model: text-embedding-3-small.
Is this comparable to other products' published numbers?
Not directly, and that's an honest "no," not a hedge. Competing vendors' publicly claimed figures were never confirmed to use the same category inclusion, the same scoring code, or even the same metric — an LLM-as-judge tends to score far more generously than literal word-overlap F1/EM, which is what this number uses. This score is real and reproducible on its own terms: same input data, same unmodified scoring code, full method disclosed above. Anyone can re-run the scoring script against the raw predictions and get the same number back.
What we found along the way
Getting from a first real run (24.9%) to the final number (50.2%) surfaced five real product bugs — not benchmark-only tuning — all fixed on the main branch: a consolidation bug that could fail an entire day's memory processing over one malformed relationship, a missing retry/backoff that let a single rate limit fail an entire run, a soft prompt instruction the model routinely ignored, a database table of specific extracted facts that was being written but never read back at retrieval time, and a hardcoded context budget many times smaller than this codebase's own documented target. Full write-up, including a no-memory baseline comparison and every caveat, is in the repo.
Read the full write-up on GitHub →