test(search): measured file-dedup - removes repeats, moves no score

AGENT_DIARY.md

@@ -57,6 +57,16 @@
   Пункты «RU→EN словарь» и «concept→symbol словарь» сняты: их заменяет контракт.
 - **Урок:** не учить сервер языкам, которые знает хост; отдавать `hints` и честно
   просить их, когда в вопросе нет code-имени.
+- **E72 (file dedup):** `MSCODEBASE_FILE_DEDUP=file` убрал все повторы файлов
+  (25→0 на 20 коммитах), но hit@5 остался 11/40 на sqlalchemy и 19/71 на B1 dev.
+  На решаемых 40: 34 из 70 повторов стояли в рангах 1–5, то есть замена была
+  реальной и всё равно нейтральной. Цена: «где используется X» теряет строки одного
+  файла (5/20 проб). Флаг оставлен экспериментальным; дедуп не объясняет разрыв.
+- **Урок знаменателя (E72):** долю нельзя брать с одного сэмплера и сравнивать с
+  планом на другом. «6 из 20 пустых» пришло из скрипта без фильтра задач (там
+  docs-only коммиты, им отвечать нечего); на решаемых 40 пустых 13, и это уже другой
+  вывод. Guard: `duplicate_positions.py` теперь применяет фильтр бенчмарка и падает
+  с кодом 2 при несовпадении популяции.
 - **E71 (Change-Id):** хеш из Gerrit занимал первый слот идентификаторов, гейт считал
   `0` вхождений и отказывал на каждом коммите sqlalchemy (0/40). Фильтр хешей и
   трейлеров поднял tool до 11/40, B1 dev не изменился (19/38/22/2).

EXPERIMENTS_LOG.md

@@ -1,4 +1,42 @@
 # EXPERIMENTS_LOG.md — Audit Verification (2026-07-22)
+## [2026-11-11] — E72: file-level dedup changes the shape, not the score
+
+**Why:** on external repos the tool answers with chunks while grep ranks whole
+files, and 22-29% of answer rows repeated a file already shown. Hypothesis: that
+is the gap. Frozen spec written before the code:
+`experiments/file_dedup_2026_11/frozen/PLAN.md`.
+
+**Change under test:** `MSCODEBASE_FILE_DEDUP=file` — one row per file first,
+leftover rows fill the tail; off by default.
+
+```
+sqlalchemy, 40 commits      hit@5
+  flag off                     11        neighbours 19 · search 18 · grep 20
+  flag on                      11        (>=14 was the bar: FAILED)
+B1 dev, 71 commits        19/71 both ways (bar 19: PASS)
+usage probes (P3)         5/20 kept (bar 3 of 4 groups = 15/20: FAILED)
+flag off, 10 answers      sha256 identical to HEAD (P4: PASS)
+```
+
+The flag did work: on the same 20 commits duplicates went 25 -> 0
+(`experiments/file_dedup_2026_11/duplicate_positions.py`). 10 of those 25 repeats
+sat at ranks 1-5, so the swaps were real and still neutral: 11/40 either way.
+The limiting factor is which distinct files rank where, not that one file filled
+several slots.
+
+**Side finding, corrected:** 13 of the 40 solvable sqlalchemy commits get an
+empty answer (0 rows), with and without the flag. E73 traced that to a false
+refusal, not to an absent subject.
+
+The first draft said "6 of 20". That count came from the unfiltered raw-commit
+sample in `duplicate_positions.py`, which includes docs-only commits whose empty
+answer is correct; the script now applies the benchmark task filter and refuses
+to print a share on a different denominator.
+
+**Verdict:** negative result. The switch stays experimental and off; it is
+documented as such in all three `CONFIGURATION.md` files. "Dedup by file" must
+not be shipped, or claimed, as the explanation for the grep gap.
+
 ## [2026-11-10] — E71: a Gerrit `Change-Id` made the gate abstain on every commit
 
 **Found by** the sqlalchemy run of B2: tool 0/40 at hit@5 while neighbours got

docs/en/CONFIGURATION.md

@@ -58,6 +58,7 @@ are gitignored.
 | `MSCODEBASE_EVIDENCE_GATE` | `true` | Verify each `investigate` row (definition/ast/fresh/graph) and print a deterministic verdict; `0` restores the pre-gate answer |
 | `MSCODEBASE_ASK_REWRITE` | `smart` | `on` — ask for an identifier whenever a Cyrillic question has no code token; `smart` — only when it also names a proper-noun-like Latin word (XERJ N03); `off` — return neighbours with a verdict. Measured: `experiments/xerj_2026_11/evidence_gate/REPORT_REWRITE.md` |
 | `MSCODEBASE_FUSION` | `weights` | merge formula for `investigate`: `weights` — channel score + hand weight (shipped); `rrf` — reciprocal-rank sum over the channels that found a row; `norm` — both terms on a [0, 1] scale. Unknown or empty falls back to `weights`. This is an experiment switch, not a tuning knob: `experiments/xerj_2026_11/fusion/` |
+| `MSCODEBASE_FILE_DEDUP` | *(off)* | answer ordering for `investigate`: `file` — one row per file first, the leftover rows fill the tail; off — the shipped order. Unknown or empty is off. Measured in E72: no hit@5 change on `sqlalchemy` (11/40) or B1 dev (19/71), and "where is X used" lost its same-file rows (5 of 20 probes kept) — an experiment switch, not a default: `experiments/file_dedup_2026_11/` |
 | `MSCODEBASE_SCRIPTS_DEMOTE` | `true` | Penalise `scripts/`, `examples/`, `benches/`, `fuzz/` rows for code questions, like tests and docs. Measured: XERJ P05 top-1 recovered, no `any` lost (`experiments/xerj_2026_11/probe_scripts_demote.py`) |
 
 `EXPERIMENTS/**` is excluded from indexing by `FileGuard` regardless of these flags:

docs/ru/CONFIGURATION.md

@@ -61,6 +61,7 @@
 | `MSCODEBASE_EVIDENCE_GATE` | `true` | Проверять каждую строку `investigate` (definition/ast/fresh/graph) и печатать детерминированный вердикт; `0` возвращает ответ до гейта |
 | `MSCODEBASE_ASK_REWRITE` | `smart` | `on` — просить идентификатор для любого кириллического вопроса без code-токена; `smart` — только если в нём есть proper-noun-подобное латинское слово (XERJ N03); `off` — возвращать соседей с вердиктом. Замер: `experiments/xerj_2026_11/evidence_gate/REPORT_REWRITE.md` |
 | `MSCODEBASE_FUSION` | `weights` | формула слияния для `investigate`: `weights` — score канала + ручной вес (шипится); `rrf` — сумма обратных рангов по каналам, нашедшим строку; `norm` — оба слагаемых в [0, 1]. Неизвестное или пустое значение откатывается к `weights`. Это переключатель эксперимента, а не ручка настройки: `experiments/xerj_2026_11/fusion/` |
+| `MSCODEBASE_FILE_DEDUP` | *(выкл.)* | порядок строк ответа `investigate`: `file` — сначала по одной строке на файл, остаток добивает хвост; выкл. — шипящийся порядок. Неизвестное или пустое значение = выкл. Замер E72: hit@5 не изменился ни на `sqlalchemy` (11/40), ни на B1 dev (19/71), а вопрос «где используется X» терял строки одного файла (сохранились 5 из 20 проб) — переключатель эксперимента, не дефолт: `experiments/file_dedup_2026_11/` |
 | `MSCODEBASE_SCRIPTS_DEMOTE` | `true` | Понижать строки из `scripts/`, `examples/`, `benches/`, `fuzz/` на code-вопросах, как tests и docs. Замер: XERJ P05 вернулся в top-1, `any` не потерян (`experiments/xerj_2026_11/probe_scripts_demote.py`) |
 
 `EXPERIMENTS/**` исключён из индексации через `FileGuard` независимо от этих флагов:

docs/zh/CONFIGURATION.md

@@ -57,6 +57,7 @@
 | `MSCODEBASE_EVIDENCE_GATE` | `true` | 校验 `investigate` 的每一行（definition/ast/fresh/graph）并打印确定性结论；`0` 恢复门控前行为 |
 | `MSCODEBASE_ASK_REWRITE` | `smart` | `on` —— 任何没有代码形态标识符的西里尔文问题都要求给出标识符；`smart` —— 仅当问题还包含类专有名词的拉丁词（XERJ N03）；`off` —— 返回邻居并带结论。实测：`experiments/xerj_2026_11/evidence_gate/REPORT_REWRITE.md` |
 | `MSCODEBASE_FUSION` | `weights` | `investigate` 的合并公式：`weights` —— 通道得分 + 手工权重（默认发布）；`rrf` —— 对命中该行的通道求倒数秩之和；`norm` —— 两项都归一化到 [0, 1]。未知或空值回退到 `weights`。这是实验开关，不是调参旋钮：`experiments/xerj_2026_11/fusion/` |
+| `MSCODEBASE_FILE_DEDUP` | *（关）* | `investigate` 的作答顺序：`file` —— 每个文件先出一行，剩余行补到末尾；关 —— 发布顺序。未知或空值即为关。E72 实测：`sqlalchemy`（11/40）与 B1 dev（19/71）的 hit@5 均未变化，而“X 在哪里被使用”丢失了同一文件的行（20 个探针中保留 5 个）—— 这是实验开关，不是默认行为：`experiments/file_dedup_2026_11/` |
 | `MSCODEBASE_SCRIPTS_DEMOTE` | `true` | 对代码问题降低 `scripts/`、`examples/`、`benches/`、`fuzz/` 行的排名，与 tests、docs 相同。实测：XERJ P05 回到 top-1，`any` 无损失（`experiments/xerj_2026_11/probe_scripts_demote.py`） |
 
 无论这些开关如何设置，`EXPERIMENTS/**` 都会被 `FileGuard` 排除在索引之外：

experiments/file_dedup_2026_11/REPORT.md

@@ -0,0 +1,126 @@
+# E72 — file-level dedup: negative result
+
+Date: 2026-11-10/11. Frozen spec: `frozen/PLAN.md` (written before any code
+change). Switch: `MSCODEBASE_FILE_DEDUP=file`, **off by default**.
+
+## What was asked
+
+The tool answers with chunks; a "which file" question wants files. On the
+benchmark's own 40-commit task list, **70 of 216 rows (32%) repeat a file**,
+worst case 8 rows from 1 file. Hypothesis: repeated slots are why grep, which
+ranks whole files, wins on external repos.
+
+An earlier draft used 22-29% from a raw `git log` sample; that sample is not the
+task population (it includes docs-only commits), so the number here is the one
+measured with the benchmark's task filter applied.
+
+## What was measured
+
+Corpora: `sqlalchemy` at `8b62343` (668 files / 34 984 chunks, 40 commits) and
+B1 dev (71 commits, calibrated in E65/E67/E71).
+
+```
+sqlalchemy, 40 tasks          hit@1  hit@5
+  tool, flag off                 8     11
+  tool, flag on                  8     11     <- P1 (>=14) FAILED
+  neighbours                    11     19
+  search                        12     18
+  grep                           7     20
+```
+
+```
+B1 dev, 71 tasks              tool   neighbours  search  grep
+  flag off                    19/71      38        22      2
+  flag on                     19/71      38        22      2
+```
+
+**The flag changed no hit@5 number on either corpus**, and it did work: on the
+20-commit sample used to check its effect, the same answers carry **0 duplicate
+rows instead of 25**. So the repeats were real and were removed — the score
+simply did not move.
+
+`duplicate_positions.py`, the benchmark's 40 sqlalchemy commits, flag off:
+
+```
+rows=216  duplicate_rows=70 (0.32 of rows)  empty_answers=13
+duplicate row positions (1-based in the answer):
+  rank 2: 9   rank 3: 9   rank 4: 4   rank 5: 12
+  rank 6: 12  rank 7: 14  rank 8: 10
+duplicates at ranks 1-5: 34 of 70
+```
+
+So 34 of the 70 repeats sat inside the top-5 that hit@5 counts, and replacing
+them with distinct files still produced 11/40 — the swaps were neutral: the
+duplicate was not what kept the gold file out. What limits the tool is **which
+distinct files rank where**, not that one file took several slots.
+
+One more fact from the same run, unrelated to the flag but larger than it:
+**13 of the 40** solvable commits get an empty answer (0 rows) with and without
+the flag. E73 traced that to a false refusal, not to an absent subject.
+
+An earlier draft of this report said "6 of 20". That count came from
+`duplicate_positions.py`, which then sampled raw commits without the benchmark's
+task filter, so it included docs-only commits that have no `.py` gold and whose
+empty answer is correct. The script now applies the same filter and exits with
+code 2 instead of printing a share on a different denominator.
+
+## The harm the flag does (P3)
+
+20 probes built from the symbol index: functions with >= 2 distinct call lines
+inside one file, asked as `where is <name> used`.
+
+```
+probes kept (file still contributes >= 2 rows, before and after):  5 / 20
+```
+
+Examples:
+
+```
+select_from                      lib/sqlalchemy/dialects/mssql/base.py   before=3 after=1
+_join_fixture_compound_expression_1  test/orm/test_rel_fn.py              before=8 after=1
+_get_custom_operator_dispatch    lib/sqlalchemy/sql/compiler.py          before=7 after=1
+extend_string (kept)             lib/sqlalchemy/dialects/mysql/base.py   before=8 after=3
+```
+
+When the answer has >= 8 distinct files, the first pass fills all 8 slots and
+the same-file rows are pushed out entirely — which is exactly the wrong answer
+for a "where is X used" question. Two probes (`le_`, `to_instance`) never
+contained the file before either; they are not losses, they are probes whose
+answer was already wrong.
+
+## Gates
+
+| # | Gate | Result |
+|---|---|---|
+| P1 | sqlalchemy tool hit@5 >= 14/40 | **FAIL** (11, identical to flag off) |
+| P2 | B1 dev >= 19/71 | PASS (19, identical) |
+| P3 | >= 3 of 4 probe groups keep same-file rows | **FAIL** (5/20 = 25%; the frozen "3 of 4 groups" is 75%, i.e. 15 of 20 probes) |
+| P4 | flag off byte-identical on 10 tasks | PASS (`sha256` equal, sequential rerun) |
+
+P4 needed a second run: the first pair was measured while the sqlalchemy
+benchmark held the same SQLite index, one baseline answer carried
+`degraded: impact:InterfaceError`, and the two files differed. The difference
+was the harness, not the code — with both runs sequential the two files are
+`sha256`-identical.
+
+## Verdict
+
+The gap to grep is **not** a chunk-vs-file artifact. Dedup removes every
+duplicate row (25 -> 0 on the same 20 commits) and hit@5 stays at 11/40, so the
+repeated files were not what pushed the gold out of the top-5. The cost is real
+though: "where is X used" loses its same-file rows (5 of 20 probes kept), because
+once 8 distinct files are found the second pass never runs.
+
+Consequence for the product: keep the switch experimental and off. Do not claim,
+and do not ship, "dedup by file" as the explanation for the grep gap. The next
+honest question is the one this run points at: why 6 of 20 external commits get
+an empty answer at all, and why the distinct files rank below grep's.
+
+## Reproduce
+
+```
+python experiments/file_dedup_2026_11/p4_identity.py --src <tree> ...
+$env:MSCODEBASE_FILE_DEDUP="file"; python experiments/bench_2026_11/external.py ...
+python experiments/file_dedup_2026_11/usage_probe.py --corpus D:\Project\bench-ext\sa-B
+python experiments/file_dedup_2026_11/duplicate_positions.py --corpus ... --repo ...
+```

experiments/file_dedup_2026_11/duplicate_positions.py

@@ -0,0 +1,88 @@
+"""Where do the repeated files sit in the answer, ON THE TASK POPULATION?
+
+E72's motivation used a raw `git log` sample, which includes docs-only commits
+with no `.py` gold - a different population than the 40-task panel. This script
+applies the same task filter as the benchmark and asserts that the population
+matches, so the number cannot be quoted against the wrong denominator again.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import sys
+from collections import Counter
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+ROOT = HERE.parents[1]
+sys.path.insert(0, str(ROOT))
+sys.path.insert(0, str(ROOT / "experiments" / "bench_2026_11"))
+sys.stdout.reconfigure(encoding="utf-8")
+
+import re  # noqa: E402
+
+from external import _tasks  # noqa: E402
+from src.core.di_container import (  # noqa: E402
+    IndexerFactoryKey,
+    ProjectIndexerRegistry,
+    create_service_collection,
+)
+from src.mcp.tools.investigate_tool import InvestigateTool  # noqa: E402
+
+ROW_RE = re.compile(r"^\d+\. (.+?):(\d+) ", re.M)
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--corpus", required=True)
+    parser.add_argument("--repo", required=True)
+    parser.add_argument("--base", required=True)
+    parser.add_argument("--tasks", type=int, default=40)
+    args = parser.parse_args()
+
+    corpus = Path(args.corpus)
+    tasks = _tasks(Path(args.repo), args.base, args.tasks)
+    if len(tasks) != args.tasks:
+        print(f"population mismatch: asked {args.tasks}, benchmark rule kept "
+              f"{len(tasks)} - refusing to report a share on a different denominator")
+        return 2
+    print(f"corpus={corpus.name} tasks={len(tasks)} (benchmark filter applied)")
+
+    services = create_service_collection(corpus)
+    services.resolve(ProjectIndexerRegistry).get_indexer(
+        corpus, factory=services.resolve(IndexerFactoryKey))
+    tool = InvestigateTool(services)
+
+    positions: Counter = Counter()
+    duplicates = 0
+    rows_total = 0
+    empty = 0
+    for task in tasks:
+        text = asyncio.run(tool.execute(query=task["question"], limit=8,
+                                        project_root=str(corpus)))
+        files = [f for f, _ in ROW_RE.findall(text)]
+        if not files:
+            empty += 1
+        seen: set = set()
+        for index, file in enumerate(files, 1):
+            rows_total += 1
+            if file in seen:
+                duplicates += 1
+                positions[index] += 1
+            else:
+                seen.add(file)
+        print(f"  {task['id']} rows={len(files)} distinct={len(set(files))}", flush=True)
+
+    print(f"\nrows={rows_total} duplicate_rows={duplicates} "
+          f"({duplicates / max(rows_total, 1):.2f} of rows) empty_answers={empty}")
+    print("duplicate row positions (1-based):")
+    for pos in sorted(positions):
+        print(f"  rank {pos}: {positions[pos]}")
+    print(f"duplicates at ranks 1-5: "
+          f"{sum(positions[p] for p in positions if p <= 5)} of {duplicates}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())