Repository navigation
test(search): measured file-dedup - removes repeats, moves no score - #112
Merged
Merged
Conversation
The external benchmark said the tool answers with chunks while grep ranks whole
files, and 22-29% of the answer rows repeated a file already shown. This tests
whether that is the gap: one row per file first, leftover rows fill the tail,
behind MSCODEBASE_FILE_DEDUP=file, off by default.
Frozen spec, written before the code: experiments/file_dedup_2026_11/frozen/PLAN.md
sqlalchemy, 40 commits hit@5
flag off 11 neighbours 19 · search 18 · grep 20
flag on 11 (bar >=14: FAILED)
B1 dev, 71 commits 19/71 both ways (bar 19: PASS)
usage probes (P3) 5/20 kept (bar 3 of 4 groups = 15/20: FAILED)
flag off, 10 answers sha256-identical to HEAD (P4: PASS)
The flag did work: on the same 20 commits duplicate rows went 25 -> 0, and 10 of
those 25 sat at ranks 1-5. The swaps were real and neutral. What limits the tool
is which distinct files rank where, not that one file filled several slots.
The cost is real for "where is X used": once 8 distinct files are found the
second pass never runs, so same-file call sites collapse to one row.
Side finding, recorded rather than fixed: 6 of 20 sqlalchemy commits get an empty
answer (0 rows) with and without the flag.
Verdict: negative result. The switch stays experimental and off and is documented
as such in all three CONFIGURATION.md files. "Dedup by file" is not the
explanation for the grep gap and must not be shipped as one.
P4 needed a second run: the first pair was measured while the sqlalchemy
benchmark held the same SQLite index, one baseline answer carried
"degraded: impact:InterfaceError", and the two files differed - the harness, not
the code.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
 |
added 2 commits
November 11, 2026 00:43
…t-rate numbers
The report's "6 of 20 empty answers" and "22-29% repeated rows" came from
duplicate_positions.py, which at the time sampled raw commits from git log. That
sample includes docs-only commits with no .py gold - a different population than
the benchmark's 40-task panel, where the bar is measured.
Re-measured on the benchmark's own task list, flag off:
rows=216 duplicate_rows=70 (0.32) empty_answers=13
duplicates at ranks 1-5: 34 of 70
The E72 verdict does not change (the flag removes every duplicate and hit@5
stays at 11/40), but the motivation is now stated on the right denominator, and
the 13 empty answers point at E73 rather than at "a small-repo effect".
duplicate_positions.py now applies the same task filter as the benchmark and
exits 2 when the population size does not match, so the share cannot be printed
against a different denominator again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test(search): measured file-dedup — removes repeats, moves no score
The external benchmark said the tool answers with chunks while grep ranks whole
files, and 22-29% of the answer rows repeated a file already shown. This tests
whether that is the gap: one row per file first, leftover rows fill the tail,
behind MSCODEBASE_FILE_DEDUP=file, off by default.
Frozen spec, written before the code: experiments/file_dedup_2026_11/frozen/PLAN.md
The flag did work: on the same 20 commits duplicate rows went 25 -> 0, and 10 of
those 25 sat at ranks 1-5. The swaps were real and neutral. What limits the tool
is which distinct files rank where, not that one file filled several slots.
The cost is real for "where is X used": once 8 distinct files are found the
second pass never runs, so same-file call sites collapse to one row.
Side finding, recorded rather than fixed: 6 of 20 sqlalchemy commits get an empty
answer (0 rows) with and without the flag.
Verdict: negative result. The switch stays experimental and off and is documented
as such in all three CONFIGURATION.md files. "Dedup by file" is not the
explanation for the grep gap and must not be shipped as one.
P4 needed a second run: the first pair was measured while the sqlalchemy
benchmark held the same SQLite index, one baseline answer carried
"degraded: impact:InterfaceError", and the two files differed - the harness, not
the code.