Benchmarks
Agentics memory on LoCoMo, LongMemEval-S and ConvoMem, run through the same public API your agents use.
Status
Full runs are pending
- No benchmark has been run in full yet, so this page has no scores.
- We publish a number only from a committed run report. Each one will sit here with its configuration, judge prompt, cost and raw outputs.
- We will not set other vendors' self-reported numbers beside ours. Any comparison here will come from runs we made with the same harness and the same settings.
Benchmarks
What we will run
- LoCoMo: long multi-session conversations, with questions about what was said and when. Not run yet.
- LongMemEval-S: long chat histories, with questions on single facts, updates, time and abstention. Not run yet.
- ConvoMem: conversational memory across many short sessions. Not run yet.
Method
How each result is produced
- The harness is memorybench at commit
e8d6e1f40e8d, with the Agentics provider. It writes one memory per conversation session throughPOST /v1/memories, waits for each ingest job, and searches withPOST /v1/memories/search. - We report two numbers separately. Retrieval recall: did the memories that answer the question come back in the top results. QA accuracy: given those memories, did a model answer correctly, as judged by a second model.
- Answering model:
gpt-4o-mini. Judge:gpt-4o-2024-08-06. Both run at temperature 0, which each run checks and records in its log. - The judge prompts are memorybench's own, unmodified. LoCoMo uses Mem0's published judge prompt (source, SHA-256
62395dd312a631dfd9355026a0b69cc936018274c3198b6365b5c2a5c9bca9e0). LongMemEval-S and ConvoMem use memorybench's defaults by question type (default prompt SHA-2562f4df52cb6dd110e91cdaa51f433628f21edb6981511d13250dc1260cfaefb61). Every hash is in the run configuration.
Cost
Budget
- All judge and answering-model calls go through the Agentics gateway, under a total cap of $150 for every run combined.
- Before a run starts it is charged a ceiling of $0.04 per question, an upper bound at list prices. A run that would go over the cap does not start.
- What each run was charged is in the committed spend ledger. That is the ceiling, not a measured cost.
Reproduce
Run it yourself
- The provider, the installer that pins memorybench, and the runner are in benchmarks/agentics.
- With an org API key,
benchmarks/agentics/run.sh smokeruns ten LoCoMo questions andrun.sh full locomoruns the whole benchmark. - Run logs and raw reports are committed under benchmarks/agentics/runs.