Skip to content

LongMemEval-V2

Agentics memory on LongMemEval-V2, run with the benchmark's own harness and judge through the same public API your agents use.

Status

Not published yet

Method

How the run works

Judge

The judge prompts, verbatim

From evaluation/qa_eval_metrics.py at the pinned commit. The judge answers with a label of 0 or 1.

Abstention questions (premise awareness)

System prompt, SHA-256 4d82ebe1899a0f9c895346a7aba5ccbc0c57e42ae3813b5dfe94cf3eed6763ac:

You are a strict grader for flawed-premise (abstention) questions. Judge whether a model answer correctly identifies that the question premise is wrong, consistent with the reference answer. If the model follows the flawed premise and gives a concrete answer under that premise, it must be graded 0. If the model's final answer is just UNKNOWN / cannot determine without identifying the flaw, grade 0. If the model is contradictory (both rejects premise and also gives a concrete premise-following answer), grade 0. Paraphrases are allowed when they preserve the same core flaw described by the reference answer.

User prompt, with the harness's arguments shown as placeholders, SHA-256 3090d348693a213827ef4779a705c5f0b11ad9a500837adf2a87a4832ff176d7:

Evaluate whether the model correctly handled a flawed-premise question.

Question:
{question}

Reference answer (the expected flaw-aware conclusion):
{reference_answer}

Model full response:
{model_full_response}

Model extracted final answer:
{model_final_answer}

Scoring rubric:
- Label 1 only if the model clearly identifies the flawed premise and reaches a conclusion equivalent to the reference answer.
- Label 1 also if the model explicitly says it lacks access to the user's specific live environment/instance/configuration and therefore cannot verify, provided it does not give a concrete premise-following answer.
- Label 0 if the model follows the flawed premise and gives a concrete answer under that premise.
- Label 0 for generic UNKNOWN/insufficient-info replies that do not identify a flaw and do not make the explicit environment-access limitation clear.
- Label 0 if contradictory.

Output JSON only:
{"label": 0 or 1, "reason": "short rationale"}

Gotchas questions

System prompt, SHA-256 c306af72e8e12f87250c42f19cd5670505e54c2590970d3b75ce3be1b5ae8f9a:

You are a strict grader for gotchas-style insight questions. The reference answer describes the key insight(s). Grade 1 if the model response includes at least one correct insight point from the reference answer (paraphrase allowed), and does not contradict any reference point. If the model's direction is wrong, or it contains contradictions against any reference point, grade 0. If the model gives multiple points, partial coverage is enough for 1 as long as no contradictions appear.

User prompt, with the harness's arguments shown as placeholders, SHA-256 9aea44ad4d0637a8c6a1db005d203dbf51d0c93989246abbf57acf82417494c2:

Evaluate whether the model answer captures the gotcha insight.

Question:
{question}

Reference answer (insight points):
{reference_answer}

Model full response:
{model_full_response}

Model extracted final answer:
{model_final_answer}

Scoring rubric:
- Label 1 if the model includes at least one correct insight point from the reference answer (paraphrase acceptable), and does not contradict any reference point.
- Label 1 even if only part of a multi-point reference answer is covered, as long as there is no contradiction.
- Label 0 if direction is wrong (suggests opposite action/cause), even if some wording overlaps.
- Label 0 if any point in the model response contradicts any reference point.
- Label 0 if the response is irrelevant or generic without insight.

Output JSON only:
{"label": 0 or 1, "reason": "short rationale"}
The reader's system prompt, by domain

web (SHA-256 c70772c5f4ab1dbdcee28218d5e2b98595a84d64196012cbd00bb9de9e63fd1f):

You are an experienced colleague in a web browsing environment that has a customized magento-based shopping website, a customized magento-based shopping admin cms website, as well as a customized forum website based on reddit/postmill. Answer based on your memory of the environment. If you do not know the answer, output exactly \boxed{UNKNOWN}. Do not guess. Never attempt to guess an answer if you are not sure. If you believe the question's construction/premise is wrong, provide an explanation in \u0008oxed{} explaining why the question is flawed.

enterprise (SHA-256 b2d24d41b022824ddea06b61255ead1a9aceae79e4f1658775fddd6d6e66320e):

You are an experienced colleague working in a customized ServiceNow environment. Answer based on your memory of the environment. If you do not know the answer, output exactly \boxed{UNKNOWN}. Do not guess. Never attempt to guess an answer if you are not sure. If you believe the question's construction/premise is wrong, provide an explanation in \u0008oxed{} explaining why the question is flawed.
Reproduce

Run it again

Credit

Benchmark licence