Repository navigation
test(experiment): generate a 281-task panel from the index - #105
Merged
Merged
Conversation
…ate it
The 41-task panel makes one task worth 6 points, and every rate we publish is a
point estimate with a wide interval. This adds a panel built by a generator
instead of by hand: symbols are taken from the index, the sample is seeded by
`sha256(commit)`, and a symbol is used only if its gold file literally defines
it. 160 `name` + 40 `qualified` + 40 `negative` + the 41 existing anchors.
Frozen predictions all pass:
bucket n top1 any chars/top1
name 160 154 157 2163
qualified 40 38 40 2669
negative 40 40 40 332
anchor:xerj/positive 16 6 10 5894
anchor:mcp_audit/pos 17 11 14 3442
S3 is the point: the anchors reproduce E65's `weights` arm exactly (6/16,
11/17), so the instrument is stable enough to compare experiments against.
The generated rate is not a quality headline - the questions contain the
identifier, so they measure the symbol channel; prose is `hints` (E64).
Building the panel surfaced two index defects, both recorded rather than
silently absorbed into the rate: the graph returns nodes whose file is not in
this project (`src/sources/manifest/extract.py`), and it labels a call
expression as a `Function` (`callable` in `di_container.py:160`). The evidence
gate survives the second - the row gets no `✓def` and the verdict is
`caveat (0.40)` - but the symbol channel ranks a non-definition.
`phantom_gold = 0` on the shipped panel: every gold was checked against the
source at build time, so a miss is a retrieval miss, not a broken task.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
 |
Three controls on the E66 panel, and one of them changes what the number may be
used for.
Grep baseline (a file that literally defines the symbol, first in path order):
bucket grep top1 tool top1 grep any tool any
name 147/160 154/160 160/160 157/160
qualified 23/40 38/40 33/40 40/40
The `name` bucket is close to a grep test (91.9% vs 96.3%); `qualified` is where
the tool actually separates (57.5% vs 95%). So 0.963 is a regression detector,
not a quality claim, and the report now says that next to the number.
The three `any` misses are one bug, not three: `risk_level`, `module_of` and
`referenced_modules` live in `src/core/test_impact.py`, and the bucket
classifier penalises it by -2.0 because the file name starts with `test_`.
Blast radius: 926 tracked .py files, 271 match the test regex, 260 of them are
real tests, 1 production module is misclassified. `src/core/test_impact.py` is
exactly the file that computes test impact, so the penalty is backwards. Fixing
it is E67 with its own frozen bar, not a silent edit inside this PR.
Determinism: 24 tasks x 3 repeats are stable, but one task
(`MultiProjectSearcher.cross_repo_search`) scored False in the original run and
True in all three repeats afterwards. Single-run rates carry session noise, so
published numbers from this panel need 3 repeats, as in E65.
Also recorded: selection bias (791 eligible name symbols, 160 sampled, short and
dunder names dropped) and the leakage the panel cannot see (gold comes from the
index, so indexing gaps are invisible).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test(experiment): generate a 281-task panel from the index and calibrate it
The 41-task panel makes one task worth 6 points, and every rate we publish is a
point estimate with a wide interval. This adds a panel built by a generator
instead of by hand: symbols are taken from the index, the sample is seeded by
sha256(commit), and a symbol is used only if its gold file literally definesit. 160
name+ 40qualified+ 40negative+ the 41 existing anchors.Frozen predictions all pass:
S3 is the point: the anchors reproduce E65's
weightsarm exactly (6/16,11/17), so the instrument is stable enough to compare experiments against.
The generated rate is not a quality headline - the questions contain the
identifier, so they measure the symbol channel; prose is
hints(E64).Building the panel surfaced two index defects, both recorded rather than
silently absorbed into the rate: the graph returns nodes whose file is not in
this project (
src/sources/manifest/extract.py), and it labels a callexpression as a
Function(callableindi_container.py:160). The evidencegate survives the second - the row gets no
✓defand the verdict iscaveat (0.40)- but the symbol channel ranks a non-definition.phantom_gold = 0on the shipped panel: every gold was checked against thesource at build time, so a miss is a retrieval miss, not a broken task.