diff --git a/AGENT_DIARY.md b/AGENT_DIARY.md index b369887c..36ebb6cf 100644 --- a/AGENT_DIARY.md +++ b/AGENT_DIARY.md @@ -57,6 +57,10 @@ Пункты «RU→EN словарь» и «concept→symbol словарь» сняты: их заменяет контракт. - **Урок:** не учить сервер языкам, которые знает хост; отдавать `hints` и честно просить их, когда в вопросе нет code-имени. +- **E67 (бакет тестов):** `src/core/test_impact.py` штрафовался на −2.0 как тестовый + файл, потому что начинается с `test_`. Исправлено проверкой содержимого (тесты или + импорт фреймворка), дефолт переключён; откат `MSCODEBASE_TEST_BUCKET=path`. + Замер: 3/3 пострадавших, якоря 21/41 без изменений, sample 40/40; точечно 0/9 → 9/9. - **E68 (бенчмарк из истории git):** вопрос = сообщение коммита, gold = `git show`, корпус на коммите `B` старше задач. Dev/71: tool 19, neighbours 37, search 24, grep 2 (hit@5); sealed/37: 7 / 21 / 5 / 1. P1 и P2 провалены: дефолт **хуже** diff --git a/EXPERIMENTS_LOG.md b/EXPERIMENTS_LOG.md index 01f5175e..8240be4d 100644 --- a/EXPERIMENTS_LOG.md +++ b/EXPERIMENTS_LOG.md @@ -1,4 +1,23 @@ # EXPERIMENTS_LOG.md — Audit Verification (2026-07-22) +## [2026-11-10] — E67: a production module was penalised as a test file + +**Bug (found by the E66 generator):** `_bucket_penalty` classifies by path, and +`test_*.py` matched `src/core/test_impact.py` — a production module that +*predicts* test impact. Its definitions lost 2.0 rank on code questions, so three +E66 tasks failed there while a plain grep found them. Blast radius: 926 `.py` +files, 271 match the regex, 260 are real tests, **1 production module** is +misclassified. + +**Fix:** outside `tests/`, a `test_*.py` is a test only if it defines tests or +imports pytest/unittest; unreadable files keep the strict behaviour; +`MSCODEBASE_TEST_BUCKET=path` is the rollback. + +**Measurement** (3 affected + 41 anchors + 40 sampled positives, 3 repeats): +T1 3/3 fixed, T2 anchors 21/41 vs 21/41, T3 sample 40/40 vs 40/40, T4 identical +on unrelated tasks. Targeted re-check: `path` 0/9, `content` 9/9 top-1. + +**Shipped:** default flipped to `content`. + ## [2026-11-10] — E68: commit-localization benchmark — the default answers *less* than plain search **Why:** every earlier panel was authored by us, so "tuned to the test" was a diff --git a/experiments/test_bucket_2026_11/REPORT.md b/experiments/test_bucket_2026_11/REPORT.md new file mode 100644 index 00000000..28f93f3c --- /dev/null +++ b/experiments/test_bucket_2026_11/REPORT.md @@ -0,0 +1,50 @@ +# E67 — test-bucket classification + +Frozen: `frozen/PLAN.md`. Source: `experiments/scale_2026_11/REDTEAM.md` A2. + +## The bug + +`_bucket_penalty` treats any `test_*.py` as a test file and subtracts 2.0 from +its rows on a code question. `src/core/test_impact.py` is a production module (it +*predicts* test impact), so its definitions were pushed out of the answer: three +E66 tasks failed there and a plain grep found them. + +Blast radius, measured: 926 tracked `.py` files, 271 match the path regex, 260 of +them are real tests (they define `def test_*` or import a test framework), **1** +production module is misclassified. + +## Fix + +The path rule stays; a content check overrides it outside `tests/`: a `test_*.py` +counts as a test only if it defines tests or imports pytest/unittest. Unreadable +files keep the old (penalise) behaviour, so nothing becomes more permissive by +accident. `MSCODEBASE_TEST_BUCKET=path` restores the previous rule. + +## Measurement + +Arms: `path` vs `content`, one binary, interleaved. Panel: the 3 affected tasks + +41 anchors + 40 sampled positives, 3 repeats, per-task majority. + +``` +T1 affected fixed (any@8) 3/3 PASS +T2 anchors not worse 21/41 vs 21/41 PASS +T3 sampled positives not worse 40/40 vs 40/40 PASS +T4 identical on unrelated tasks PASS +``` + +Re-check of the load-bearing claim, 3 repeats per task per arm: + +``` +path risk_level / module_of / referenced_modules (valid, any, top1) x3 + all three: (True, False, False) x3 -> 0/9 +content all three: (True, True, True) x3 -> 9/9 +``` + +One of 504 calls in the full A/B returned an invalid answer (a degraded run); the +majority of 3 covered it, and the targeted re-check above has 18 valid calls. + +## Result + +The default is now `content`; `path` remains as a rollback via the env var. The +change is a classification fix, not ranking: T3/T4 confirm it does not reorder +anything that does not involve a `test_*.py` file. diff --git a/experiments/test_bucket_2026_11/frozen/PLAN.md b/experiments/test_bucket_2026_11/frozen/PLAN.md new file mode 100644 index 00000000..96261a9d --- /dev/null +++ b/experiments/test_bucket_2026_11/frozen/PLAN.md @@ -0,0 +1,57 @@ +# Test-bucket classification — frozen plan (before the code change) + +Date: 2026-11-10. Source: `experiments/scale_2026_11/REDTEAM.md` A2. + +**Bug.** `_bucket_penalty` treats any `test_*.py` as a test file and subtracts +2.0 from its rows on a code question. `src/core/test_impact.py` is a production +module (it *predicts* test impact), so all its definitions are pushed out of the +answer: three tasks of the E66 panel fail because of it, and grep finds them. + +**Change.** The path rule stays, and a content check overrides it: a `test_*.py` +outside `tests/` counts as a test only if it defines `def test_*` or imports +pytest/unittest. Unknown/unreadable files keep the old (penalise) behaviour, so +nothing becomes *more* permissive by accident. + +Measured: 926 tracked `.py` files; 271 match the path regex; 260 carry test +markers; **1** production module is misclassified. + +## Arms + +One binary, `MSCODEBASE_TEST_BUCKET`: + +- `path` — shipped behaviour (default; no change until the measurement says so); +- `content` — the override described above. + +## Panel + +The change can only affect rows whose file matches `test_*.py`, and exactly one +such file is outside `tests/`. So the panel is: + +- the 3 affected E66 tasks (`risk_level`, `module_of`, `referenced_modules`); +- the 41 anchors (regression); +- 40 random positives from the E66 panel (regression). + +84 tasks x 3 repeats, per-task majority, both arms interleaved. + +## Predictions (frozen) + +| # | Prediction | Falsifier | +|---|---|---| +| T1 | `content` fixes all 3 affected tasks (`any@8` true) | any of the 3 still misses | +| T2 | no anchor loses `top1` or `any` | any loss | +| T3 | the 40 sampled positives do not lose | any loss | +| T4 | per-task rows are identical between arms on tasks whose files do not match `test_*.py` | any difference | + +T4 is the guard against a scalar change leaking into unrelated ranking. + +## Kill + +T2 or T3 fails -> keep `path` (the shipped behaviour) and record the bug as a +KNOWN_ISSUE instead of a fix. T1 failing means the hypothesis is wrong about the +cause, not about the penalty. + +## Note on what this is not + +This is a **classification fix**, not ranking tuning. The bar is "the same +answers, plus the three that a production module was losing"; it is not allowed +to reorder the corpus. If it does, T4 trips. diff --git a/experiments/test_bucket_2026_11/results.json b/experiments/test_bucket_2026_11/results.json new file mode 100644 index 00000000..4e23f167 --- /dev/null +++ b/experiments/test_bucket_2026_11/results.json @@ -0,0 +1,687 @@ +{ + "tasks": 84, + "predictions": { + "T1": true, + "T2": true, + "T3": true, + "T4": true + }, + "per_arm": { + "path": { + "G-N-risk_level": { + "top1": false, + "any": false + }, + "G-N-module_of": { + "top1": false, + "any": false + }, + "G-N-referenced_modules": { + "top1": false, + "any": false + }, + "P01": { + "top1": true, + "any": true + }, + "P02": { + "top1": true, + "any": true + }, + "P03": { + "top1": false, + "any": false + }, + "P04": { + "top1": false, + "any": false + }, + "P05": { + "top1": true, + "any": true + }, + "P06": { + "top1": false, + "any": false + }, + "P07": { + "top1": false, + "any": true + }, + "P08": { + "top1": true, + "any": true + }, + "P09": { + "top1": false, + "any": true + }, + "P10": { + "top1": false, + "any": true + }, + "P11": { + "top1": true, + "any": true + }, + "P12": { + "top1": false, + "any": true + }, + "P13": { + "top1": false, + "any": false + }, + "P14": { + "top1": false, + "any": false + }, + "P15": { + "top1": false, + "any": false + }, + "P16": { + "top1": true, + "any": true + }, + "A01": { + "top1": false, + "any": true + }, + "A02": { + "top1": false, + "any": true + }, + "A03": { + "top1": false, + "any": false + }, + "A04": { + "top1": false, + "any": false + }, + "N01": { + "top1": true, + "any": true + }, + "N02": { + "top1": true, + "any": true + }, + "N03": { + "top1": true, + "any": true + }, + "N04": { + "top1": true, + "any": true + }, + "MSC01": { + "top1": true, + "any": true + }, + "MSC02": { + "top1": true, + "any": true + }, + "MSC03": { + "top1": true, + "any": true + }, + "MSC04": { + "top1": false, + "any": true + }, + "MSC05": { + "top1": true, + "any": true + }, + "MSC06": { + "top1": false, + "any": true + }, + "MSC07": { + "top1": false, + "any": false + }, + "MSC08": { + "top1": false, + "any": false + }, + "MSC09": { + "top1": false, + "any": false + }, + "BLA01": { + "top1": true, + "any": true + }, + "BLA02": { + "top1": true, + "any": true + }, + "BLA03": { + "top1": true, + "any": true + }, + "BLA04": { + "top1": true, + "any": true + }, + "HTT01": { + "top1": false, + "any": true + }, + "HTT02": { + "top1": true, + "any": true + }, + "HTT03": { + "top1": true, + "any": true + }, + "HTT04": { + "top1": true, + "any": true + }, + "G-N-_default_client_caps": { + "top1": true, + "any": true + }, + "G-N-get_hotspots": { + "top1": true, + "any": true + }, + "G-N-get_runtime_status": { + "top1": true, + "any": true + }, + "G-N-schema_check": { + "top1": true, + "any": true + }, + "G-N-apply_mcp_process_title": { + "top1": true, + "any": true + }, + "G-N-_enable_ansi": { + "top1": true, + "any": true + }, + "G-N-_cache_key": { + "top1": true, + "any": true + }, + "G-Q-CommitMemory.get_significant_commits": { + "top1": true, + "any": true + }, + "G-N-format_receipt": { + "top1": true, + "any": true + }, + "G-N-format_hotspots": { + "top1": true, + "any": true + }, + "G-N-_is_complex_query": { + "top1": true, + "any": true + }, + "G-N-_module_names": { + "top1": true, + "any": true + }, + "G-N-get_system_summary": { + "top1": true, + "any": true + }, + "G-N-_start_zed_parent_watchdog": { + "top1": true, + "any": true + }, + "G-Q-JobHistoryStore.load_history": { + "top1": true, + "any": true + }, + "G-N-register_all_tools": { + "top1": true, + "any": true + }, + "G-N-_ensure_lsp": { + "top1": true, + "any": true + }, + "G-Q-ConsistencyTracker.mark_stale": { + "top1": true, + "any": true + }, + "G-Q-GraphQueryTool._execute_query": { + "top1": true, + "any": true + }, + "G-N-user_brief": { + "top1": true, + "any": true + }, + "G-Q-PluginProcess.__exit__": { + "top1": true, + "any": true + }, + "G-N-_line_starts": { + "top1": true, + "any": true + }, + "G-N-_extract_go_mod": { + "top1": true, + "any": true + }, + "G-N-_proxy": { + "top1": true, + "any": true + }, + "G-N-_ensure_data_root": { + "top1": true, + "any": true + }, + "G-N-_build_vocabulary": { + "top1": true, + "any": true + }, + "G-N-has_instruction_risk": { + "top1": true, + "any": true + }, + "G-N-_run_trace": { + "top1": true, + "any": true + }, + "G-N-run_bootstrap_pipeline": { + "top1": true, + "any": true + }, + "G-N-iter_import_candidate_nodes": { + "top1": true, + "any": true + }, + "G-Q-SystemArtifacts.is_artifact": { + "top1": true, + "any": true + }, + "G-Q-_InterProcessLock.__exit__": { + "top1": true, + "any": true + }, + "G-Q-ProjectContext._capture_memory": { + "top1": true, + "any": true + }, + "G-N-_reject_self_index_target": { + "top1": true, + "any": true + }, + "G-Q-LiveBuffer.keys": { + "top1": true, + "any": true + }, + "G-N-_job_key": { + "top1": true, + "any": true + }, + "G-N-_get_max_chunk_preview_len": { + "top1": true, + "any": true + }, + "G-Q-CypherParser._parse_return_items": { + "top1": true, + "any": true + }, + "G-N-is_supported": { + "top1": true, + "any": true + }, + "G-N-_collect_tools": { + "top1": true, + "any": true + } + }, + "content": { + "G-N-risk_level": { + "top1": true, + "any": true + }, + "G-N-module_of": { + "top1": true, + "any": true + }, + "G-N-referenced_modules": { + "top1": true, + "any": true + }, + "P01": { + "top1": true, + "any": true + }, + "P02": { + "top1": true, + "any": true + }, + "P03": { + "top1": false, + "any": false + }, + "P04": { + "top1": false, + "any": false + }, + "P05": { + "top1": true, + "any": true + }, + "P06": { + "top1": false, + "any": false + }, + "P07": { + "top1": false, + "any": true + }, + "P08": { + "top1": true, + "any": true + }, + "P09": { + "top1": false, + "any": true + }, + "P10": { + "top1": false, + "any": true + }, + "P11": { + "top1": true, + "any": true + }, + "P12": { + "top1": false, + "any": true + }, + "P13": { + "top1": false, + "any": false + }, + "P14": { + "top1": false, + "any": false + }, + "P15": { + "top1": false, + "any": false + }, + "P16": { + "top1": true, + "any": true + }, + "A01": { + "top1": false, + "any": true + }, + "A02": { + "top1": false, + "any": true + }, + "A03": { + "top1": false, + "any": false + }, + "A04": { + "top1": false, + "any": false + }, + "N01": { + "top1": true, + "any": true + }, + "N02": { + "top1": true, + "any": true + }, + "N03": { + "top1": true, + "any": true + }, + "N04": { + "top1": true, + "any": true + }, + "MSC01": { + "top1": true, + "any": true + }, + "MSC02": { + "top1": true, + "any": true + }, + "MSC03": { + "top1": true, + "any": true + }, + "MSC04": { + "top1": false, + "any": true + }, + "MSC05": { + "top1": true, + "any": true + }, + "MSC06": { + "top1": false, + "any": true + }, + "MSC07": { + "top1": false, + "any": false + }, + "MSC08": { + "top1": false, + "any": false + }, + "MSC09": { + "top1": false, + "any": false + }, + "BLA01": { + "top1": true, + "any": true + }, + "BLA02": { + "top1": true, + "any": true + }, + "BLA03": { + "top1": true, + "any": true + }, + "BLA04": { + "top1": true, + "any": true + }, + "HTT01": { + "top1": false, + "any": true + }, + "HTT02": { + "top1": true, + "any": true + }, + "HTT03": { + "top1": true, + "any": true + }, + "HTT04": { + "top1": true, + "any": true + }, + "G-N-_default_client_caps": { + "top1": true, + "any": true + }, + "G-N-get_hotspots": { + "top1": true, + "any": true + }, + "G-N-get_runtime_status": { + "top1": true, + "any": true + }, + "G-N-schema_check": { + "top1": true, + "any": true + }, + "G-N-apply_mcp_process_title": { + "top1": true, + "any": true + }, + "G-N-_enable_ansi": { + "top1": true, + "any": true + }, + "G-N-_cache_key": { + "top1": true, + "any": true + }, + "G-Q-CommitMemory.get_significant_commits": { + "top1": true, + "any": true + }, + "G-N-format_receipt": { + "top1": true, + "any": true + }, + "G-N-format_hotspots": { + "top1": true, + "any": true + }, + "G-N-_is_complex_query": { + "top1": true, + "any": true + }, + "G-N-_module_names": { + "top1": true, + "any": true + }, + "G-N-get_system_summary": { + "top1": true, + "any": true + }, + "G-N-_start_zed_parent_watchdog": { + "top1": true, + "any": true + }, + "G-Q-JobHistoryStore.load_history": { + "top1": true, + "any": true + }, + "G-N-register_all_tools": { + "top1": true, + "any": true + }, + "G-N-_ensure_lsp": { + "top1": true, + "any": true + }, + "G-Q-ConsistencyTracker.mark_stale": { + "top1": true, + "any": true + }, + "G-Q-GraphQueryTool._execute_query": { + "top1": true, + "any": true + }, + "G-N-user_brief": { + "top1": true, + "any": true + }, + "G-Q-PluginProcess.__exit__": { + "top1": true, + "any": true + }, + "G-N-_line_starts": { + "top1": true, + "any": true + }, + "G-N-_extract_go_mod": { + "top1": true, + "any": true + }, + "G-N-_proxy": { + "top1": true, + "any": true + }, + "G-N-_ensure_data_root": { + "top1": true, + "any": true + }, + "G-N-_build_vocabulary": { + "top1": true, + "any": true + }, + "G-N-has_instruction_risk": { + "top1": true, + "any": true + }, + "G-N-_run_trace": { + "top1": true, + "any": true + }, + "G-N-run_bootstrap_pipeline": { + "top1": true, + "any": true + }, + "G-N-iter_import_candidate_nodes": { + "top1": true, + "any": true + }, + "G-Q-SystemArtifacts.is_artifact": { + "top1": true, + "any": true + }, + "G-Q-_InterProcessLock.__exit__": { + "top1": true, + "any": true + }, + "G-Q-ProjectContext._capture_memory": { + "top1": true, + "any": true + }, + "G-N-_reject_self_index_target": { + "top1": true, + "any": true + }, + "G-Q-LiveBuffer.keys": { + "top1": true, + "any": true + }, + "G-N-_job_key": { + "top1": true, + "any": true + }, + "G-N-_get_max_chunk_preview_len": { + "top1": true, + "any": true + }, + "G-Q-CypherParser._parse_return_items": { + "top1": true, + "any": true + }, + "G-N-is_supported": { + "top1": true, + "any": true + }, + "G-N-_collect_tools": { + "top1": true, + "any": true + } + } + } +} \ No newline at end of file diff --git a/experiments/test_bucket_2026_11/run_ab.py b/experiments/test_bucket_2026_11/run_ab.py new file mode 100644 index 00000000..b5b6dab5 --- /dev/null +++ b/experiments/test_bucket_2026_11/run_ab.py @@ -0,0 +1,130 @@ +"""E67: does content-aware test-bucket classification fix what it should? + +Frozen: `frozen/PLAN.md`. Arms: `path` (shipped) vs `content`, one binary via +`MSCODEBASE_TEST_BUCKET`. Panel: the 3 affected E66 tasks + 41 anchors + 40 +sampled positives, 3 repeats, per-task majority. + +Run: python experiments/test_bucket_2026_11/run_ab.py +""" + +from __future__ import annotations + +import asyncio +import json +import os +import random +import sys +from collections import defaultdict +from pathlib import Path + +HERE = Path(__file__).resolve().parent +EXP = HERE.parent +ROOT = HERE.parents[1] +SCALE = EXP / "scale_2026_11" +sys.path.insert(0, str(ROOT)) +sys.path.insert(0, str(EXP / "xerj_2026_11")) +sys.path.insert(0, str(EXP / "xerj_2026_11" / "evidence_gate")) +sys.stdout.reconfigure(encoding="utf-8") + +from run_baseline import score as canon_score # noqa: E402 +from run_gate import VERDICT_RE, _files, _invalid, _norm, _tools_for # noqa: E402 + +AFFECTED = {"risk_level", "module_of", "referenced_modules"} +ARMS = ("path", "content") + + +def _panel() -> list[dict]: + rows = [json.loads(x) for x in (SCALE / "tasks.jsonl").read_text(encoding="utf-8").splitlines() if x.strip()] + affected, anchors, positives = [], [], [] + for task in rows: + if task["anchor"]: + anchors.append(task) + elif task["id"].startswith("G-N-") and task["id"][4:] in AFFECTED: + affected.append(task) + elif task["kind"] == "positive" and not task["anchor"]: + positives.append(task) + rng = random.Random(20261110) + rng.shuffle(positives) + return affected + anchors + positives[:40] + + +def _run(tool, task: dict, arm: str) -> dict: + os.environ["MSCODEBASE_TEST_BUCKET"] = arm + text = asyncio.run(tool.execute(query=task["question"], limit=8, + project_root=str(task.get("root") or ROOT))) + files = _norm(_files(text), Path(task.get("root") or ROOT)) + valid = not _invalid(text) + verdict = VERDICT_RE.search(text) + return { + "arm": arm, + "id": task["id"], + "top1": None if not valid else canon_score(task, files[:1]), + "any": None if not valid else canon_score(task, files[:8]), + "valid": valid, + "verdict": verdict.group(1) if verdict else "", + } + + +def main() -> int: + tasks = _panel() + print(f"tasks={len(tasks)} (affected=3 anchors=41 sample=40)", flush=True) + tool = _tools_for({ROOT})[str(ROOT)] + results = {arm: [] for arm in ARMS} + for position, task in enumerate(tasks): + for run in range(3): + order = ARMS if (position + run) % 2 == 0 else ARMS[::-1] + for arm in order: + row = _run(tool, task, arm) + results[arm].append(row) + + def majority(rows): + bits = [r["top1"] for r in rows if r["top1"] is not None] + return bool(bits) and sum(bits) * 2 > len(bits) + + per_arm = {} + for arm in ARMS: + grouped = defaultdict(list) + for row in results[arm]: + grouped[row["id"]].append(row) + per_arm[arm] = {key: {"top1": majority(rows), + "any": any(r["any"] for r in rows)} + for key, rows in grouped.items()} + + affected = [t["id"] for t in tasks[:3]] + anchors = [t["id"] for t in tasks[3:44]] + sample = [t["id"] for t in tasks[44:]] + + def count(arm, key, ids, field="top1"): + return sum(per_arm[arm][i][field] for i in ids) + + print("\nfrozen predictions") + t1 = all(per_arm["content"][i]["any"] for i in affected) + print(f" T1 affected fixed (any@8): {count('content', '', affected, 'any')}/3 -> " + f"{'PASS' if t1 else 'FAIL'}") + t2 = all(per_arm["content"][i]["top1"] >= per_arm["path"][i]["top1"] and + per_arm["content"][i]["any"] >= per_arm["path"][i]["any"] + for i in anchors) + print(f" T2 anchors not worse: {'PASS' if t2 else 'FAIL'}") + t3 = all(per_arm["content"][i]["top1"] >= per_arm["path"][i]["top1"] and + per_arm["content"][i]["any"] >= per_arm["path"][i]["any"] + for i in sample) + print(f" T3 sampled positives not worse: {'PASS' if t3 else 'FAIL'}") + t4 = all(per_arm["content"][i] == per_arm["path"][i] for i in sample + anchors) + print(f" T4 identical on unrelated tasks: {'PASS' if t4 else 'FAIL'}") + + print(f"\n path anchors top1={count('path', '', anchors)}/41 " + f"sample top1={count('path', '', sample)}/40") + print(f" content anchors top1={count('content', '', anchors)}/41 " + f"sample top1={count('content', '', sample)}/40") + + invalid = sum(1 for a in ARMS for r in results[a] if not r["valid"]) + print(f" invalid runs: {invalid}") + (HERE / "results.json").write_text(json.dumps( + {"tasks": len(tasks), "predictions": {"T1": t1, "T2": t2, "T3": t3, "T4": t4}, + "per_arm": per_arm}, ensure_ascii=False, indent=1), + encoding="utf-8", newline="") + return 0 if all((t1, t2, t3)) else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/mcp/tools/investigate_tool.py b/src/mcp/tools/investigate_tool.py index 01d94a22..4a506c02 100644 --- a/src/mcp/tools/investigate_tool.py +++ b/src/mcp/tools/investigate_tool.py @@ -19,6 +19,7 @@ import os import re import threading +from functools import lru_cache from pathlib import Path from typing import Any, Dict, Iterable, List, Optional, Tuple @@ -255,6 +256,16 @@ def _asks_for_callers(query: str) -> bool: r"|(^|/)test_[^/]+$" r"|(^|/)conftest\.py$" ) +# A `test_*.py` name alone is not proof: `src/core/test_impact.py` *predicts* +# test impact and was penalised as a test file (E66 red team, A2). Under the +# `content` mode outside `tests/`, the file has to actually define tests or +# import a test framework before it counts. +_TEST_DEF_RE = re.compile(r"^\s*(async\s+)?def\s+test_[A-Za-z0-9_]*", re.M) +_TEST_FRAMEWORK_RE = re.compile( + r"^\s*(import\s+pytest|from\s+pytest|import\s+unittest|from\s+unittest)", + re.M, +) +_TEST_BUCKET_MODES = ("path", "content") _DOC_PATH_RE = re.compile(r"(^|/)docs?/") _HISTORY_PATH_RE = re.compile(r"(^|/)docs?/(archive|research|blog)/") _SCRIPT_PATH_RE = re.compile(r"(^|/)(scripts|examples|benches|fuzz)/") @@ -267,6 +278,40 @@ def _asks_for_tests(query: str) -> bool: )) +def _test_bucket_mode() -> str: + """`content` (shipped after E67) or `path` (the previous behaviour). + + Unknown or empty -> `content`: the fix is the default, and `path` exists as + a rollback, not as the normal state. + """ + mode = (os.environ.get("MSCODEBASE_TEST_BUCKET") or "").strip().lower() + return "path" if mode == "path" else "content" + + +@lru_cache(maxsize=1024) +def _content_says_test(path: str) -> bool: + """Does this file define test functions or import a test framework? + + Unreadable files answer `True`: an unknown file keeps the old, stricter + behaviour instead of silently becoming visible. + """ + try: + text = Path(path).read_text(encoding="utf-8", errors="ignore") + except OSError: + return True + return bool(_TEST_DEF_RE.search(text) or _TEST_FRAMEWORK_RE.search(text)) + + +def _test_penalty_applies(file: str, root: Any) -> bool: + if _test_bucket_mode() != "content": + return True + if file.replace("\\", "/").startswith(("tests/", "test/")): + return True # a tests/ path is a test; no I/O needed + base = Path(root) if root else None + absolute = str((base / file).resolve()) if base else file + return _content_says_test(absolute) + + def _asks_for_docs(query: str) -> bool: return bool(re.search( r"\b(docs?|documentation|readme|guide|faq|manual)\b" @@ -326,7 +371,7 @@ def _name_variants(identifiers: Iterable[str], max_variants: int = 12) -> List[s return variants[:max_variants] -def _bucket_penalty(file: str, query: str) -> float: +def _bucket_penalty(file: str, query: str, root: Any = None) -> float: """Rank penalty for a non-answer bucket (audit 2026-11-09). Measured on three corpora: `tests/**`, `tests/data/**` and `docs/archive/**` @@ -349,7 +394,8 @@ def _bucket_penalty(file: str, query: str) -> float: # ("on-demand rerank resolved" → EXPERIMENTS_LOG, audit follow-up). value -= 1.5 if _TEST_PATH_RE.search(file) and not _asks_for_tests(query): - value -= 2.0 + if _test_penalty_applies(file, root): + value -= 2.0 if ( _SCRIPT_PATH_RE.search(file) and not _asks_for_tests(query) @@ -524,7 +570,7 @@ def _merge( # numbers would be a hard ban rather than the documented "not a cap", so # there the bucket goes to the tail instead. for entry in merged.values(): - penalty = _bucket_penalty(entry["file"], query) + penalty = _bucket_penalty(entry["file"], query, root) if not penalty: continue if mode == "rrf": diff --git a/tests/test_test_bucket.py b/tests/test_test_bucket.py new file mode 100644 index 00000000..43e3b10d --- /dev/null +++ b/tests/test_test_bucket.py @@ -0,0 +1,74 @@ +"""Test-bucket classification (E67, `experiments/test_bucket_2026_11/`). + +After the measurement the shipped default is `content`; `path` is the rollback. +""" + +from __future__ import annotations + +from src.mcp.tools.investigate_tool import _bucket_penalty + + +def _project(tmp_path): + (tmp_path / "src" / "core").mkdir(parents=True) + (tmp_path / "src" / "core" / "test_impact.py").write_text( + "def risk_level(n):\n return 'high'\n", encoding="utf-8" + ) + (tmp_path / "src" / "tests").mkdir(parents=True) + (tmp_path / "src" / "tests" / "test_thing.py").write_text( + "from pytest import mark\n\ndef test_one():\n assert True\n", + encoding="utf-8", + ) + (tmp_path / "tests").mkdir() + (tmp_path / "tests" / "test_helper.py").write_text( + "def helper():\n return 1\n", encoding="utf-8" + ) + return tmp_path + + +def test_production_module_is_spared_by_default(tmp_path, monkeypatch): + root = _project(tmp_path) + monkeypatch.delenv("MSCODEBASE_TEST_BUCKET", raising=False) + + assert _bucket_penalty("src/core/test_impact.py", "where is X defined", root) == 0.0 + + +def test_path_mode_is_still_available_as_rollback(tmp_path, monkeypatch): + root = _project(tmp_path) + monkeypatch.setenv("MSCODEBASE_TEST_BUCKET", "path") + + assert _bucket_penalty("src/core/test_impact.py", "where is X defined", root) == -2.0 + + +def test_a_file_that_defines_tests_is_still_penalised(tmp_path, monkeypatch): + root = _project(tmp_path) + monkeypatch.delenv("MSCODEBASE_TEST_BUCKET", raising=False) + + assert _bucket_penalty("src/tests/test_thing.py", "where is X defined", root) == -2.0 + + +def test_no_io_needed_under_tests(tmp_path, monkeypatch): + root = _project(tmp_path) + monkeypatch.delenv("MSCODEBASE_TEST_BUCKET", raising=False) + + # The file has no test markers, but a tests/ path is a test by definition. + assert _bucket_penalty("tests/test_helper.py", "where is X defined", root) == -2.0 + + +def test_missing_file_fails_closed(tmp_path, monkeypatch): + monkeypatch.delenv("MSCODEBASE_TEST_BUCKET", raising=False) + + assert _bucket_penalty("src/ghost/test_missing.py", "where is X defined", tmp_path) == -2.0 + + +def test_unknown_mode_keeps_the_new_default(tmp_path, monkeypatch): + root = _project(tmp_path) + monkeypatch.setenv("MSCODEBASE_TEST_BUCKET", "nonsense") + + assert _bucket_penalty("src/core/test_impact.py", "where is X defined", root) == 0.0 + + +def test_a_question_about_tests_keeps_full_weight(tmp_path, monkeypatch): + root = _project(tmp_path) + monkeypatch.delenv("MSCODEBASE_TEST_BUCKET", raising=False) + + assert _bucket_penalty("src/tests/test_thing.py", "where are the tests for X", root) == 0.0