diff --git a/EXPERIMENTS_LOG.md b/EXPERIMENTS_LOG.md index 8240be4d..027180de 100644 --- a/EXPERIMENTS_LOG.md +++ b/EXPERIMENTS_LOG.md @@ -75,6 +75,10 @@ is stable enough to compare experiments against. measure the symbol channel, not prose. 0.963 is a regression detector ("exact names resolve"), not a quality headline; prose is covered by `hints` (E64). +**Post-E67 re-run:** `name` 157/160 top-1 and 160/160 any@8 (was 154/157); the +three `test_impact.py` misses are gone, the anchors did not move. The B1 +benchmark was re-run on the merged build too and is identical. + **Index defects found by the generator's own gold check:** the graph returns nodes whose file is not in the project (`src/sources/manifest/extract.py`), and it labels a call expression as a `Function` (`callable` in diff --git a/experiments/bench_2026_11/REPORT.md b/experiments/bench_2026_11/REPORT.md index 388932a0..b11971a5 100644 --- a/experiments/bench_2026_11/REPORT.md +++ b/experiments/bench_2026_11/REPORT.md @@ -141,6 +141,13 @@ visible in the log before the crash; no code, arm or metric changed after that (the arms were frozen in Amendments 1-3 before the sealed run), so this is disclosed rather than "clean". +### Re-measured after E67 + +The test-bucket fix (E67, merged as `092418a`) changed a bucket rule, so the dev +split was re-run on the merged build: **identical** (`tool` 19, `neighbours` 38, +`search` 22, `grep` 2; paired +16). The fix does not touch this benchmark, which +is what a classification change should look like. + ## Threats - The corpus is one snapshot at `B`. A file renamed after `B` makes its task diff --git a/experiments/scale_2026_11/REPORT.md b/experiments/scale_2026_11/REPORT.md index 9ddcca9b..085117bc 100644 --- a/experiments/scale_2026_11/REPORT.md +++ b/experiments/scale_2026_11/REPORT.md @@ -7,8 +7,8 @@ Frozen before the run: `frozen/PLAN.md`. Panel generated from commit ``` bucket n top1 any chars/top1 95% CI top1 -name 160 154 157 2163 [0.92, 0.98] -qualified 40 38 40 2669 [0.83, 0.99] +name 160 157 160 2121 [0.95, 0.99] +qualified 40 38 40 2671 [0.83, 0.99] negative 40 40 40 332 [0.91, 1.00] anchor:xerj/positive 16 6 10 5894 [0.18, 0.61] anchor:xerj/ambiguity 4 0 2 n/a [0.00, 0.49] @@ -24,7 +24,7 @@ estimate of answer cost per correct answer, **not** a tokenizer measurement. | # | Prediction | Result | |---|---|---| -| S1 | generated `name` top-1 >= 0.60 | **PASS** — 154/160 = 0.963 | +| S1 | generated `name` top-1 >= 0.60 | **PASS** — 157/160 = 0.981 | | S2 | negatives yield no rows on >= 0.95 | **PASS** — 40/40 | | S3 | anchors reproduce E65 `weights` within +/- 2 | **PASS, exactly** — xerj 6/16 and mcp_audit 11/17, the same two numbers | | S4 | `qualified` top-1 <= `name` top-1 | **PASS** — 0.950 <= 0.963 | @@ -39,7 +39,7 @@ to compare against. **It does not mean "search is 96% accurate".** The generated questions contain the identifier, and the grep control below shows it: a plain `grep` reaches 91.9% on the same `name` tasks. The number is a **regression detector**, not a -quality claim — it says "exact names resolve, 154 of 160 times", and nothing +quality claim — it says "exact names resolve, 157 of 160 times", and nothing about prose questions (`hints`, E64), ambiguous golds (XERJ A01-A04) or docs routing. @@ -54,11 +54,11 @@ hit in path order: ``` bucket grep top1 tool top1 grep any tool any -name 147/160 154/160 160/160 157/160 +name 147/160 157/160 160/160 160/160 qualified 23/40 38/40 33/40 40/40 ``` -So the `name` bucket is close to a grep test (91.9% vs 96.3%), while `qualified` +So the `name` bucket is close to a grep test (91.9% vs 98.1%), while `qualified` discriminates strongly (57.5% vs 95%). Read the headline rate with that in mind. **Selection bias** — 5958 graph symbols -> 5809 unique -> 2584 code-only -> @@ -72,16 +72,20 @@ from this panel need 3 repeats, as in E65. ## The 6 misses, honestly -Three of the six are one bug, not three retrieval misses: `risk_level`, -`module_of` and `referenced_modules` all live in `src/core/test_impact.py`, and -the bucket classifier penalises that production module by `-2.0` because its -name starts with `test_`. Measured blast radius: 926 `.py` files, 271 match the -test regex, 1 production module is misclassified. That is E67, with its own -frozen bar. +Three of the original six were one bug, not three retrieval misses: +`risk_level`, `module_of` and `referenced_modules` live in +`src/core/test_impact.py`, and the bucket classifier penalised that production +module by `-2.0` because its name starts with `test_`. That was fixed as E67 +(`0/9 -> 9/9` on the three tasks), and this panel was re-run afterwards: -The remaining three are ranking (`expect`, `CommitMemory.get_hotspots` found at -row 2) and name resolution. The generator records ids, so the list is -reproducible from the same commit. +``` +before E67 name 154/160 top-1, 157/160 any@8 +after E67 name 157/160 top-1, 160/160 any@8 (anchors unchanged) +``` + +The three that remain are ranking: `register_intelligence_tools`, +`expect` and `CommitMemory.get_hotspots` (two of them are found at row 2). The +generator records ids, so the list is reproducible from the same commit. ## Instrument findings (found while building the panel, not fixed here) diff --git a/experiments/scale_2026_11/results.json b/experiments/scale_2026_11/results.json index c281739a..e69192d3 100644 --- a/experiments/scale_2026_11/results.json +++ b/experiments/scale_2026_11/results.json @@ -7,7 +7,7 @@ "n": 17, "top1": 11, "any": 14, - "chars_per_top1": 3441.7272727272725 + "chars_per_top1": 3447.818181818182 }, "anchor:xerj/ambiguity": { "n": 4, @@ -29,9 +29,9 @@ }, "name": { "n": 160, - "top1": 154, - "any": 157, - "chars_per_top1": 2162.935064935065 + "top1": 157, + "any": 160, + "chars_per_top1": 2120.503184713376 }, "negative": { "n": 40, @@ -43,7 +43,7 @@ "n": 40, "top1": 38, "any": 40, - "chars_per_top1": 2668.9473684210525 + "chars_per_top1": 2671.184210526316 } }, "predictions": { diff --git a/experiments/xerj_2026_11/wilson.py b/experiments/xerj_2026_11/wilson.py index 1e300ff7..9c141557 100644 --- a/experiments/xerj_2026_11/wilson.py +++ b/experiments/xerj_2026_11/wilson.py @@ -47,8 +47,8 @@ def wilson(k: int, n: int, z: float = Z) -> tuple[float, float]: ("E65 metamorphic top1, rrf", 4, 12), ("E65 metamorphic top1, norm", 5, 12), ("E65 negatives empty, weights", 4, 4), - ("E66 generated name top1", 154, 160), - ("E66 generated name any@8", 157, 160), + ("E66 generated name top1", 157, 160), + ("E66 generated name any@8", 160, 160), ("E66 generated qualified top1", 38, 40), ("E66 generated negative empty", 40, 40), )