Repository navigation
Fix #2525: LongMemEval judge drops a correct answer unless the model returns raw JSON - #2528
Open
Memtensor-AI wants to merge 2 commits into
Open
Memtensor-AI wants to merge 2 commits into
Memtensor-AI wants to merge 2 commits into
Conversation
…MemTensor#2525) LongMemEval's `lme_grader` applied `json.loads` directly to the full judge reply, but the judge prompt asks the model for a one-sentence rationale followed by a `{"label": "..."}` JSON snippet. A prompt-compliant reply therefore triggered `JSONDecodeError: Expecting value: line 1 column 1 (char 0)` and the sample was dropped, degrading the LLM-as-a-Judge score to `N/A (0/0)` on affected runs. LoCoMo had already fixed the identical prompt shape with `extract_label_json`; LongMemEval had not. - New shared helper `evaluation/scripts/utils/judge.py` with `extract_label_json` (regex, same semantics as LoCoMo's) and `parse_judge_label` (returns the label string, or raises `ValueError` with a 200-byte preview of the raw content so empty completions are distinguishable from prose-with-no-label in the `[ERROR]` log line). - `lme_grader` now calls `parse_judge_label`. - `locomo_grader` imports the shared helper and drops its local `extract_label_json`; behaviour is preserved by the existing try/except. - Tests under `tests/evaluation/longmemeval/`: 12 unit tests on the shared helper + 4 integration tests driving `lme_grader` through a mocked OpenAI client (eval-only deps stubbed in `sys.modules`, so the tests run without `poetry install --with eval`).
Memtensor-AI
added
ai:generated
3 of 4 tasks
Collaborator
Author
🤖 Open Code ReviewTarget: PR #2528 ✅ OpenCodeReview: Review complete: 0 finding(s) across 5 selected item(s). Generated by cloud-assistant via Open Code Review. |
Collaborator
Author
🔧 Open Code Review requested Agent fixOpen Code Review found 3 issue(s). I have resumed the development Agent to fix them.
The Agent will push a new commit to this PR branch. OCR will recheck after the commit is pushed. |
- judge.py parse_judge_label: use the regex capture group directly
instead of a global single->double quote replace. The replace was
unsafe for any label value containing a single quote (e.g.
{"label": "it's CORRECT"}), and json.loads is redundant now that the
regex already pins the {"label": "..."} shape. Drop the unused json
import.
- judge.py _preview: drop the manual \n/\r pre-escape before !r. repr()
already escapes control characters, so pre-escaping produced
double-escaped output (a real newline rendered as \\n in logs).
- lme_eval.py lme_grader: wrap parse_judge_label in try/except ValueError
so an empty / unparseable judge reply logs a warning and returns False
instead of propagating through asyncio.gather and aborting the whole
sample batch (the log-collapse symptom called out in issue MemTensor#2525).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixed issue #2525: LongMemEval's
lme_graderwas passing the full judge chat completion intojson.loads(...)["label"], but the judge prompt (LME_JUDGE_MODEL_TEMPLATE) asks the model for one sentence of reasoning followed by a{"label": "..."}snippet. Any prompt-compliant reply therefore raisedJSONDecodeError: Expecting value: line 1 column 1 (char 0); the exception bubbled tomain, which dropped the sample and degraded the summary toN/A (0/0). LoCoMo had already solved the same prompt shape withextract_label_json; LongMemEval had not.The fix introduces a shared
evaluation/scripts/utils/judge.pywithextract_label_json(same regex semantics as LoCoMo's) andparse_judge_label, which returns the label string on success and raisesValueErrorwith a 200-byte quoted preview of the raw content on empty / no-label replies — making a transient empty completion visibly different from a prose-only reply in the[ERROR] Processing user failed:log line.lme_gradernow callsparse_judge_label;locomo_graderwas rewired to the same helper (behaviour preserved by the existing try/except) so the two codepaths cannot drift again. The judge prompt itself was not changed.Tests: 12 unit tests on
parse_judge_label(raw JSON, prose+JSON reproduction from the issue, single-quoted value, empty, None, no-label, long content preview truncation) plus 4 integration tests drivinglme_graderthrough a mocked OpenAI client — eval-only deps (bert_score,sentence_transformers,rouge_score) are stubbed insys.modulesso the suite runs under the standardtestextras withoutpoetry install --with eval.pytest tests/evaluation/longmemeval: 17/17 passed. Combined with unrelated tests (tests/test_hello_world.py,tests/test_log.py): 33/33 passed.ruff checkandruff format --checkboth clean on all touched files. Nosrc/memos/ordocs/openapi.jsonedits.Related Issue (Required): Fixes #2525
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Not run; documentation-only change.
Checklist
@endxxxx please review this PR.
Reviewer Checklist