Repository navigation
test(bench): commit-localization benchmark from git history - #106
Merged
Merged
Conversation
…from git Every earlier panel was authored by us, so "tuned to the test" was a fair objection. This one cannot be authored: the question is the commit message verbatim, the gold is `git show --name-only`, and the corpus is a worktree at a frozen base commit B = 7c2c7a3 (2026-09-06), so the fix is not in the index. 174 commits in (B, HEAD]; 66 reference files created after B and are unsolvable by construction (dropped and printed, PLAN Amendment 2); 108 solvable, split 71 dev / 37 sealed by date. Sealed ran once. hit@1 hit@5 any@8 dev (n=71) sealed (n=37) tool (default) 12 19 23 0.268 0.189 tool + neighbours 24 38 44 0.535 0.568 search channel alone 11 22 29 0.310 0.135 grep (floor) 0 2 6 0.028 0.027 P1 FAIL (default 0.268 < 0.35), P2 FAIL (default is 3 below plain search), P3 PASS, P4 PASS (dev/sealed within 0.10 for both arms). The cause is measured, not argued: 36 of 52 default misses are `abstain` with zero rows. The commit message names things the commit introduces, the gate finds no occurrence, and it withholds the eight neighbours it already computed. With `include_neighbours=true` the same tool gains 19 of 71 (dev) and 14 of 37 (sealed) - 33 tasks of the 108 solvable. That is a policy decision for the owner (honesty vs recall), not a ranking fix, so nothing is changed here. The third-party data gate blocked the first commit: one message quoted a contributor email. Messages are redacted inside the generator, before the manifest is hashed (89940dd8 -> 71395d45), and both splits were re-run on the redacted panel rather than inheriting the old numbers. Instrumentation: the sealed process died twice (after 16 and 22 tasks) with no traceback; the runner now appends every row and resumes by id, and the report discloses that the first 22 per-task booleans were visible before the crash. Arms were frozen before the run; no code or metric changed afterwards.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test(bench): commit-localization benchmark — questions and gold come from git
Every earlier panel was authored by us, so "tuned to the test" was a fair
objection. This one cannot be authored: the question is the commit message
verbatim, the gold is
git show --name-only, and the corpus is a worktree at afrozen base commit B = 7c2c7a3 (2026-09-06), so the fix is not in the index.
174 commits in (B, HEAD]; 66 reference files created after B and are unsolvable
by construction (dropped and printed, PLAN Amendment 2); 108 solvable, split 71
dev / 37 sealed by date. Sealed ran once.
tool (default) 12 19 23 0.268 0.189
tool + neighbours 24 38 44 0.535 0.568
search channel alone 11 22 29 0.310 0.135
grep (floor) 0 2 6 0.028 0.027
P1 FAIL (default 0.268 < 0.35), P2 FAIL (default is 3 below plain search),
P3 PASS, P4 PASS (dev/sealed within 0.10 for both arms).
The cause is measured, not argued: 36 of 52 default misses are
abstainwithzero rows. The commit message names things the commit introduces, the gate finds
no occurrence, and it withholds the eight neighbours it already computed. With
include_neighbours=truethe same tool gains 19 of 71 (dev) and 14 of 37(sealed) - 33 tasks of the 108 solvable. That is a policy decision for the owner
(honesty vs recall), not a ranking fix, so nothing is changed here.
The third-party data gate blocked the first commit: one message quoted a
contributor email. Messages are redacted inside the generator, before the
manifest is hashed (89940dd8 -> 71395d45), and both splits were re-run on the
redacted panel rather than inheriting the old numbers.
Instrumentation: the sealed process died twice (after 16 and 22 tasks) with no
traceback; the runner now appends every row and resumes by id, and the report
discloses that the first 22 per-task booleans were visible before the crash.
Arms were frozen before the run; no code or metric changed afterwards.