test(experiment): generate a 281-task panel from the index

AGENT_DIARY.md

@@ -57,6 +57,13 @@
   Пункты «RU→EN словарь» и «concept→symbol словарь» сняты: их заменяет контракт.
 - **Урок:** не учить сервер языкам, которые знает хост; отдавать `hints` и честно
   просить их, когда в вопросе нет code-имени.
+- **E66 (панель из индекса):** 281 задача, сид = sha256(commit). name 154/160,
+  qualified 38/40, negative 40/40; якоря воспроизвели E65 **точно** (6/16 и 11/17) —
+  инструмент откалиброван. Это детектор регрессий, а не «поиск на 96%»: вопросы
+  содержат идентификатор, проза закрыта hints (E64).
+- **Найдено генератором:** в графе есть узлы из чужого проекта и фантомные
+  `Function` для вызовов (`callable` в `di_container.py:160`). Гейт не обманут
+  (нет `✓def`, вердикт `caveat 0.40`), но символьный канал такой узел поднимает.
 - **E65 (fusion A/B):** три руки в одном бинарнике (`MSCODEBASE_FUSION`). top1
   weights 21 / rrf 15 / norm 20 из 53; F1/F2/F3 провалены, шипим weights, ранжирование
   закрыто. Три попытки замера убиты до публикации: (1) кап демоушена + тай-брейк по

EXPERIMENTS_LOG.md

@@ -1,4 +1,37 @@
 # EXPERIMENTS_LOG.md — Audit Verification (2026-07-22)
+## [2026-11-10] — E66: generated 281-task panel — the instrument is calibrated
+
+**Design** (frozen before the run): tasks generated from the index, seed =
+`sha256(commit)`, so the panel is reproducible and changes only with the code.
+160 `name` + 40 `qualified` + 40 `negative` + the 41 existing anchors = 281.
+Symbols must have exactly one definition and a gold file that literally defines
+them (two index defects were found this way, see below).
+
+```
+bucket                    n   top1    any   chars/top1
+name                    160    154    157        2163
+qualified                40     38     40        2669
+negative                 40     40     40         332
+anchor:xerj/positive     16      6     10        5894
+anchor:mcp_audit/pos     17     11     14        3442
+```
+
+**Verdict:** S1-S4 all PASS. S3 is the claim that matters: the anchors
+reproduced E65's `weights` arm **exactly** (6/16 and 11/17), so the instrument
+is stable enough to compare experiments against.
+
+**What it does not say:** generated questions contain the identifier, so they
+measure the symbol channel, not prose. 0.963 is a regression detector ("exact
+names resolve"), not a quality headline; prose is covered by `hints` (E64).
+
+**Index defects found by the generator's own gold check:** the graph returns
+nodes whose file is not in the project (`src/sources/manifest/extract.py`), and
+it labels a call expression as a `Function` (`callable` in
+`di_container.py:160`). The evidence gate survives the second one — the row gets
+no `✓def` and the verdict is `caveat (0.40)` — but the symbol channel ranks a
+non-definition. `KNOWN_ISSUES.md` is at its 300-line limit, so both are recorded
+in the task state; the rotation is a separate task.
+
 ## [2026-11-10] — E65: fusion A/B — hand weights beat RRF by six tasks
 
 **Design** (frozen before the run, plus Amendment 1: latency is not a

experiments/scale_2026_11/REDTEAM.md

@@ -0,0 +1,96 @@
+# Scale panel — red team (after the first run)
+
+Attacks on the E66 result. Two of them are uncomfortable, and both change what
+the number may be used for.
+
+## A1. Triviality: does plain grep solve this panel?
+
+Control: for each positive task, find every `.py` file whose source literally
+defines the symbol (`^\s*(async def|def|class) <name>\b`) and take the first hit
+in path order. No index, no ranking.
+
+```
+bucket       grep top1   tool top1   grep any   tool any
+name         147/160     154/160     160/160    157/160
+qualified     23/40       38/40       33/40      40/40
+```
+
+**Verdict: the `name` bucket is a grep test.** A one-line `grep` with no
+retrieval at all reaches 91.9% top-1 and 100% recall, against the tool's 96.3%
+and 98.1%. The difference is real but small, and the tool *loses recall*: three
+tasks that grep finds are missed entirely.
+
+`qualified` is the opposite: grep 57.5% vs tool 95%, so that bucket does
+discriminate. Twelve points of the headline rate come from 40 tasks, not 160.
+
+**Consequence:** the 0.963 `name` rate must not be published as search quality.
+It is a regression detector, as the frozen plan said — but now with a number
+behind the caveat.
+
+## A2. A real bug the panel found: `test_impact.py` is treated as a test file
+
+The three `any` misses (tasks 1, 43, 127) are all definitions in
+`src/core/test_impact.py` — `risk_level:178`, `module_of:55`,
+`referenced_modules:63`. The tool returns `abstain` or never lists the file,
+while grep finds it at line level.
+
+Cause: `_bucket_penalty` classifies by path, and `_TEST_PATH_RE` matches any
+`test_*.py`, including a production module named for *test impact prediction*:
+
+```
+_bucket_penalty('src/core/test_impact.py', 'where is risk_level defined') -> -2.0
+_bucket_penalty('tests/test_x.py',        'where is X defined')            -> -2.0
+_bucket_penalty('src/core/artifact_paths.py', ...)                         ->  0.0
+```
+
+Blast radius, measured: 926 tracked `.py` files; 271 match the regex; 260 of
+them also contain `def test_` or import a test framework (real tests); **1
+production module is misclassified** (`src/core/test_impact.py`). The other
+matches are under `experiments/` and `scripts/`, which the panel does not sample.
+
+This is the most valuable output of the run: a defect the 41-task panel never
+touched, because none of its questions pointed at that file.
+
+## A3. Selection bias: who is in the panel
+
+```
+graph definitions                 5958 symbols
+with exactly one definition       5809
+after code-only filter            2584
+matching the `name` regex          791   -> 160 sampled (20%)
+matching `qualified` regex        1483   ->  40 sampled (3%)
+dropped by regex (short/dunder)    310
+```
+
+The panel samples distinctive names only: 5-40 characters, no dunders, no
+one-letter helpers. That is a defensible choice (a question about `x` is not a
+question), but it makes the panel easier than the corpus, and the rate inherits
+that.
+
+## A4. Determinism: one run is not enough
+
+24 positive tasks x 3 repeats: `unstable_metric=0`, `unstable_top3_rows=0`.
+But task 197 (`MultiProjectSearcher.cross_repo_search`) scored `top1=False` in
+the original run and `top1=True` in all three repeats afterwards, with the same
+code and the same panel. So the per-task result can move *between sessions*
+(index freshness), even though it is stable within one.
+
+**Consequence:** a single-run rate is a point with unmeasured session noise.
+Every published number from this panel must be a 3-repeat majority, as in E65.
+
+## A5. Leakage the panel cannot see
+
+Gold comes from the index, so by construction the gold file is indexed. The panel
+therefore cannot detect indexing gaps (a file the indexer skipped is never a
+gold). It measures ranking and name resolution, not coverage.
+
+## What changes because of this
+
+1. E66 stays as **instrument calibration** (the anchors reproduced E65 exactly),
+   not as a quality claim, and `REPORT.md` now says so with the grep control in
+   the same table.
+2. The `test_impact.py` misclassification becomes its own frozen experiment
+   (E67), not a silent edit: bar is `any@8` +3 on this panel with no loss on the
+   anchors, 3 repeats.
+3. Published rates from this panel require 3 repeats and the grep control next
+   to them.

experiments/scale_2026_11/REPORT.md

@@ -0,0 +1,110 @@
+# Scale panel — result (E66)
+
+Frozen before the run: `frozen/PLAN.md`. Panel generated from commit
+`e81e0b0` (`sha256(commit)` seed), 281 tasks, one arm (shipped defaults).
+
+## Result
+
+```
+bucket                    n   top1    any   chars/top1   95% CI top1
+name                    160    154    157        2163    [0.92, 0.98]
+qualified                40     38     40        2669    [0.83, 0.99]
+negative                 40     40     40         332    [0.91, 1.00]
+anchor:xerj/positive     16      6     10        5894    [0.18, 0.61]
+anchor:xerj/ambiguity     4      0      2         n/a    [0.00, 0.49]
+anchor:xerj/negative      4      4      4         359    [0.51, 1.00]
+anchor:mcp_audit/pos     17     11     14        3442    [0.41, 0.83]
+                                             invalid runs: 0
+```
+
+`chars/top1` is answer characters divided by the number of top-1 hits — an
+estimate of answer cost per correct answer, **not** a tokenizer measurement.
+
+## Frozen predictions
+
+| # | Prediction | Result |
+|---|---|---|
+| S1 | generated `name` top-1 >= 0.60 | **PASS** — 154/160 = 0.963 |
+| S2 | negatives yield no rows on >= 0.95 | **PASS** — 40/40 |
+| S3 | anchors reproduce E65 `weights` within +/- 2 | **PASS, exactly** — xerj 6/16 and mcp_audit 11/17, the same two numbers |
+| S4 | `qualified` top-1 <= `name` top-1 | **PASS** — 0.950 <= 0.963 |
+
+## What this does and does not mean
+
+**It means the instrument is calibrated.** The anchors did not merely stay within
+two tasks — they reproduced E65 exactly (6/16, 11/17). Two independent
+experiments, two different runners, same panel: the measurement is stable enough
+to compare against.
+
+**It does not mean "search is 96% accurate".** The generated questions contain
+the identifier, and the grep control below shows it: a plain `grep` reaches
+91.9% on the same `name` tasks. The number is a **regression detector**, not a
+quality claim — it says "exact names resolve, 154 of 160 times", and nothing
+about prose questions (`hints`, E64), ambiguous golds (XERJ A01-A04) or docs
+routing.
+
+The cost column shows the same thing from another side: a generated task costs
+2163 chars per correct answer, an XERJ prose task 5894. The difference is task
+difficulty, not an improvement in the tool.
+
+## Controls (red team, `REDTEAM.md`)
+
+**grep baseline** — every `.py` file that literally defines the symbol, first
+hit in path order:
+
+```
+bucket       grep top1   tool top1   grep any   tool any
+name         147/160     154/160     160/160    157/160
+qualified     23/40       38/40       33/40      40/40
+```
+
+So the `name` bucket is close to a grep test (91.9% vs 96.3%), while `qualified`
+discriminates strongly (57.5% vs 95%). Read the headline rate with that in mind.
+
+**Selection bias** — 5958 graph symbols -> 5809 unique -> 2584 code-only ->
+791 eligible `name` names -> 160 sampled (20%). The panel uses distinctive names
+(5-40 chars, no dunders), which makes it easier than the corpus.
+
+**Determinism** — 24 tasks x 3 repeats: metric stable. But one task
+(`MultiProjectSearcher.cross_repo_search`) differed between the original run and
+later repeats, so a single-run rate carries session noise: published numbers
+from this panel need 3 repeats, as in E65.
+
+## The 6 misses, honestly
+
+Three of the six are one bug, not three retrieval misses: `risk_level`,
+`module_of` and `referenced_modules` all live in `src/core/test_impact.py`, and
+the bucket classifier penalises that production module by `-2.0` because its
+name starts with `test_`. Measured blast radius: 926 `.py` files, 271 match the
+test regex, 1 production module is misclassified. That is E67, with its own
+frozen bar.
+
+The remaining three are ranking (`expect`, `CommitMemory.get_hotspots` found at
+row 2) and name resolution. The generator records ids, so the list is
+reproducible from the same commit.
+
+
+## Instrument findings (found while building the panel, not fixed here)
+
+1. **Foreign graph nodes.** `graph.find_nodes()` returned a node whose file
+   (`src/sources/manifest/extract.py`) does not exist in this project. The
+   generator now requires the file to exist; the tool still enumerates such
+   nodes.
+2. **Phantom `Function` nodes for calls.** `callable` in
+   `src/core/di_container.py:160` is `if callable(close_method):`, but the graph
+   labels it `Function`. The evidence gate is **not** fooled — `investigate`
+   returns that row without `✓def` and the verdict is `caveat (0.40) — no
+   verified declaration` — but the symbol channel still ranks a non-definition.
+   Follow-up: parser filter plus a regression test.
+
+Both would have silently depressed the generated rate (a task whose gold does
+not define the symbol can never be answered), which is why the generator now
+verifies gold at build time: `phantom_gold = 0` on the shipped panel.
+
+## Reproduce
+
+```
+python experiments/scale_2026_11/generate.py     # same commit -> same 281 tasks
+python experiments/scale_2026_11/run_scale.py    # ~15 min, 3 s per task
+python experiments/xerj_2026_11/wilson.py        # intervals
+```

experiments/scale_2026_11/controls.py

@@ -0,0 +1,121 @@
+"""Red-team controls for the E66 panel.
+
+Three questions, all adversarial:
+
+1. **Triviality** — would `grep "def <name>"` score the same? If yes, the panel
+   measures grep, not search, and the number must not be sold as quality.
+2. **Correctness of the misses** — for every top-1 miss, is the gold really a
+   definition, and what did the tool return instead?
+3. **Selection bias** — how many symbols did the generator drop, and why? A
+   panel built only from distinctive, unique names is easier than the corpus.
+
+Run: python experiments/scale_2026_11/controls.py
+"""
+
+from __future__ import annotations
+
+import json
+import re
+import subprocess
+import sys
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+ROOT = HERE.parents[1]
+sys.path.insert(0, str(ROOT))
+sys.stdout.reconfigure(encoding="utf-8")
+
+TASKS = HERE / "tasks.jsonl"
+RESULTS = HERE / "results.json"
+
+
+def _tasks() -> list[dict]:
+    return [json.loads(x) for x in TASKS.read_text(encoding="utf-8").splitlines() if x.strip()]
+
+
+def _py_files() -> list[str]:
+    out = subprocess.run(["git", "ls-files", "*.py"], cwd=ROOT,
+                         capture_output=True, text=True, encoding="utf-8")
+    return sorted(x for x in out.stdout.splitlines() if x.strip())
+
+
+def _symbol(question: str) -> str:
+    match = re.search(r"`([^`]+)`", question)
+    return match.group(1).split(".")[-1] if match else ""
+
+
+def main() -> int:
+    tasks = _tasks()
+    files = _py_files()
+    cache: dict[str, str] = {}
+
+    def text_of(path: str) -> str:
+        if path not in cache:
+            try:
+                cache[path] = (ROOT / path).read_text(encoding="utf-8", errors="ignore")
+            except OSError:
+                cache[path] = ""
+        return cache[path]
+
+    print("== 1. grep control (definition lines, path order) ==")
+    rows = []
+    for task in tasks:
+        if task["kind"] != "positive":
+            continue
+        name = _symbol(task["question"])
+        pattern = re.compile(
+            rf"^\s*(async\s+def|def|class)\s+{re.escape(name)}\b", re.M
+        )
+        hits = [path for path in files if pattern.search(text_of(path))]
+        gold = task["gold"][0]
+        top1 = bool(hits) and hits[0].endswith(gold)
+        any8 = any(h.endswith(gold) for h in hits[:8])
+        rows.append({"bucket": task["bucket"], "top1": top1, "any": any8})
+
+    for bucket in ("name", "qualified"):
+        subset = [r for r in rows if r["bucket"] == bucket]
+        if subset:
+            print(f"  grep {bucket:<10} top1={sum(r['top1'] for r in subset)}/{len(subset)} "
+                  f"any={sum(r['any'] for r in subset)}/{len(subset)}")
+
+    result_data = json.loads(RESULTS.read_text(encoding="utf-8"))
+    summary = result_data["summary"]
+    for bucket in ("name", "qualified"):
+        cell = summary.get(bucket)
+        if cell:
+            print(f"  tool {bucket:<10} top1={cell['top1']}/{cell['n']} "
+                  f"any={cell['any']}/{cell['n']}")
+
+    print("\n== 2. the misses, one by one ==")
+    # Per-task rows are not stored; re-derive the list from the run log if present.
+    log = Path(r"C:\Users\misha\AppData\Local\Temp\opencode\scale_run2.log")
+    misses = []
+    if log.is_file():
+        for line in log.read_text(encoding="utf-8", errors="ignore").splitlines():
+            match = re.search(r"^\s+\d+/\d+ (name|qualified).*top1=False", line)
+            if match:
+                misses.append(line.strip())
+    print(f"  misses found in the run log: {len(misses)}")
+    for line in misses[:10]:
+        print("   ", line[:120])
+
+    print("\n== 3. what the generator dropped ==")
+    from generate import _definitions, SKIP_FILES, NAME_RE, QUALIFIED_RE  # noqa: E402
+
+    defs = _definitions()
+    unique = {s: r for s, r in defs.items() if len(r) == 1}
+    code = {s: r for s, r in unique.items()
+            if not any(part in r[0][0] for part in SKIP_FILES)}
+    names = [s for s in code if NAME_RE.match(s)]
+    qualified = [s for s in code if QUALIFIED_RE.match(s)]
+    print(f"  graph definitions: {len(defs)} symbols, {len(unique)} with one definition")
+    print(f"  after code-only filter: {len(code)}")
+    print(f"  matching `name` regex: {len(names)} (panel uses 160)")
+    print(f"  matching `qualified` regex: {len(qualified)} (panel uses 40)")
+    print(f"  dropped by regex (short/dunder/generic names): "
+          f"{len(code) - len(set(names) | set(qualified))}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())