Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions EXPERIMENTS_LOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,10 @@ is stable enough to compare experiments against.
measure the symbol channel, not prose. 0.963 is a regression detector ("exact
names resolve"), not a quality headline; prose is covered by `hints` (E64).

**Post-E67 re-run:** `name` 157/160 top-1 and 160/160 any@8 (was 154/157); the
three `test_impact.py` misses are gone, the anchors did not move. The B1
benchmark was re-run on the merged build too and is identical.

**Index defects found by the generator's own gold check:** the graph returns
nodes whose file is not in the project (`src/sources/manifest/extract.py`), and
it labels a call expression as a `Function` (`callable` in
Expand Down
7 changes: 7 additions & 0 deletions experiments/bench_2026_11/REPORT.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,13 @@ visible in the log before the crash; no code, arm or metric changed after that
(the arms were frozen in Amendments 1-3 before the sealed run), so this is
disclosed rather than "clean".

### Re-measured after E67

The test-bucket fix (E67, merged as `092418a`) changed a bucket rule, so the dev
split was re-run on the merged build: **identical** (`tool` 19, `neighbours` 38,
`search` 22, `grep` 2; paired +16). The fix does not touch this benchmark, which
is what a classification change should look like.

## Threats

- The corpus is one snapshot at `B`. A file renamed after `B` makes its task
Expand Down
34 changes: 19 additions & 15 deletions experiments/scale_2026_11/REPORT.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ Frozen before the run: `frozen/PLAN.md`. Panel generated from commit

```
bucket n top1 any chars/top1 95% CI top1
name 160 154 157 2163 [0.92, 0.98]
qualified 40 38 40 2669 [0.83, 0.99]
name 160 157 160 2121 [0.95, 0.99]
qualified 40 38 40 2671 [0.83, 0.99]
negative 40 40 40 332 [0.91, 1.00]
anchor:xerj/positive 16 6 10 5894 [0.18, 0.61]
anchor:xerj/ambiguity 4 0 2 n/a [0.00, 0.49]
Expand All @@ -24,7 +24,7 @@ estimate of answer cost per correct answer, **not** a tokenizer measurement.

| # | Prediction | Result |
|---|---|---|
| S1 | generated `name` top-1 >= 0.60 | **PASS** — 154/160 = 0.963 |
| S1 | generated `name` top-1 >= 0.60 | **PASS** — 157/160 = 0.981 |
| S2 | negatives yield no rows on >= 0.95 | **PASS** — 40/40 |
| S3 | anchors reproduce E65 `weights` within +/- 2 | **PASS, exactly** — xerj 6/16 and mcp_audit 11/17, the same two numbers |
| S4 | `qualified` top-1 <= `name` top-1 | **PASS** — 0.950 <= 0.963 |
Expand All @@ -39,7 +39,7 @@ to compare against.
**It does not mean "search is 96% accurate".** The generated questions contain
the identifier, and the grep control below shows it: a plain `grep` reaches
91.9% on the same `name` tasks. The number is a **regression detector**, not a
quality claim — it says "exact names resolve, 154 of 160 times", and nothing
quality claim — it says "exact names resolve, 157 of 160 times", and nothing
about prose questions (`hints`, E64), ambiguous golds (XERJ A01-A04) or docs
routing.

Expand All @@ -54,11 +54,11 @@ hit in path order:

```
bucket grep top1 tool top1 grep any tool any
name 147/160 154/160 160/160 157/160
name 147/160 157/160 160/160 160/160
qualified 23/40 38/40 33/40 40/40
```

So the `name` bucket is close to a grep test (91.9% vs 96.3%), while `qualified`
So the `name` bucket is close to a grep test (91.9% vs 98.1%), while `qualified`
discriminates strongly (57.5% vs 95%). Read the headline rate with that in mind.

**Selection bias** — 5958 graph symbols -> 5809 unique -> 2584 code-only ->
Expand All @@ -72,16 +72,20 @@ from this panel need 3 repeats, as in E65.

## The 6 misses, honestly

Three of the six are one bug, not three retrieval misses: `risk_level`,
`module_of` and `referenced_modules` all live in `src/core/test_impact.py`, and
the bucket classifier penalises that production module by `-2.0` because its
name starts with `test_`. Measured blast radius: 926 `.py` files, 271 match the
test regex, 1 production module is misclassified. That is E67, with its own
frozen bar.
Three of the original six were one bug, not three retrieval misses:
`risk_level`, `module_of` and `referenced_modules` live in
`src/core/test_impact.py`, and the bucket classifier penalised that production
module by `-2.0` because its name starts with `test_`. That was fixed as E67
(`0/9 -> 9/9` on the three tasks), and this panel was re-run afterwards:

The remaining three are ranking (`expect`, `CommitMemory.get_hotspots` found at
row 2) and name resolution. The generator records ids, so the list is
reproducible from the same commit.
```
before E67 name 154/160 top-1, 157/160 any@8
after E67 name 157/160 top-1, 160/160 any@8 (anchors unchanged)
```

The three that remain are ranking: `register_intelligence_tools`,
`expect` and `CommitMemory.get_hotspots` (two of them are found at row 2). The
generator records ids, so the list is reproducible from the same commit.


## Instrument findings (found while building the panel, not fixed here)
Expand Down
10 changes: 5 additions & 5 deletions experiments/scale_2026_11/results.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
"n": 17,
"top1": 11,
"any": 14,
"chars_per_top1": 3441.7272727272725
"chars_per_top1": 3447.818181818182
},
"anchor:xerj/ambiguity": {
"n": 4,
Expand All @@ -29,9 +29,9 @@
},
"name": {
"n": 160,
"top1": 154,
"any": 157,
"chars_per_top1": 2162.935064935065
"top1": 157,
"any": 160,
"chars_per_top1": 2120.503184713376
},
"negative": {
"n": 40,
Expand All @@ -43,7 +43,7 @@
"n": 40,
"top1": 38,
"any": 40,
"chars_per_top1": 2668.9473684210525
"chars_per_top1": 2671.184210526316
}
},
"predictions": {
Expand Down
4 changes: 2 additions & 2 deletions experiments/xerj_2026_11/wilson.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,8 @@ def wilson(k: int, n: int, z: float = Z) -> tuple[float, float]:
("E65 metamorphic top1, rrf", 4, 12),
("E65 metamorphic top1, norm", 5, 12),
("E65 negatives empty, weights", 4, 4),
("E66 generated name top1", 154, 160),
("E66 generated name any@8", 157, 160),
("E66 generated name top1", 157, 160),
("E66 generated name any@8", 160, 160),
("E66 generated qualified top1", 38, 40),
("E66 generated negative empty", 40, 40),
)
Expand Down
Loading