diff --git a/AGENT_DIARY.md b/AGENT_DIARY.md index caf59cb2..b369887c 100644 --- a/AGENT_DIARY.md +++ b/AGENT_DIARY.md @@ -57,6 +57,12 @@ Пункты «RU→EN словарь» и «concept→symbol словарь» сняты: их заменяет контракт. - **Урок:** не учить сервер языкам, которые знает хост; отдавать `hints` и честно просить их, когда в вопросе нет code-имени. +- **E68 (бенчмарк из истории git):** вопрос = сообщение коммита, gold = `git show`, + корпус на коммите `B` старше задач. Dev/71: tool 19, neighbours 37, search 24, + grep 2 (hit@5); sealed/37: 7 / 21 / 5 / 1. P1 и P2 провалены: дефолт **хуже** + обычного поиска. Причина — 36 из 52 промахов это abstain с нулём строк. +- **Урок:** «честный отказ» стоит recall, и это теперь измерено в задачах, а не + в мнении; решение (отказ или соседи с caveat) — за владельцем. - **E66 (панель из индекса):** 281 задача, сид = sha256(commit). name 154/160, qualified 38/40, negative 40/40; якоря воспроизвели E65 **точно** (6/16 и 11/17) — инструмент откалиброван. Это детектор регрессий, а не «поиск на 96%»: вопросы diff --git a/EXPERIMENTS_LOG.md b/EXPERIMENTS_LOG.md index ed3e4ba9..01f5175e 100644 --- a/EXPERIMENTS_LOG.md +++ b/EXPERIMENTS_LOG.md @@ -1,4 +1,36 @@ # EXPERIMENTS_LOG.md — Audit Verification (2026-07-22) +## [2026-11-10] — E68: commit-localization benchmark — the default answers *less* than plain search + +**Why:** every earlier panel was authored by us, so "tuned to the test" was a +fair objection. Here the question is a commit message we did not write, the gold +is `git show --name-only`, and the corpus is a worktree at `B = 7c2c7a3` +(2026-09-06) — the fixes are not in the index. + +**Panel:** 174 commits in `(B, HEAD]`; 66 reference files created after `B` and +are unsolvable by construction (dropped, counted); 108 solvable, split 71 dev / +37 sealed by date. Sealed ran once (`--sealed`). + +``` + hit@1 hit@5 any@8 dev (n=71) sealed (n=37) +tool (default) 12 19 23 0.268 0.189 +tool + neighbours 24 37 44 0.521 0.568 +search channel alone 10 24 29 0.338 0.135 +grep (floor) 0 2 6 0.028 0.027 +``` + +**Verdict:** P1 FAIL (default 0.268 < 0.35), P2 FAIL (default is **5 below** +plain search on dev), P3 PASS, P4 PASS (dev/sealed within 0.10 for both arms). + +**Cause, measured:** 36 of 52 default misses are `abstain` with zero rows — the +message names things the commit introduces, so the gate finds no occurrence and +withholds the eight neighbours it already computed. Turning them on recovers ++18 of 71 (dev) and +16 of 37 (sealed). This is a policy decision (honesty vs +recall), not a ranking defect. + +**Instrumentation:** the sealed process died twice (16 and 22 tasks) with no +traceback; the runner now resumes from `results_*.partial.jsonl`. Arms were +frozen before the run; no code or metric changed afterwards. + ## [2026-11-10] — E66: generated 281-task panel — the instrument is calibrated **Design** (frozen before the run): tasks generated from the index, seed = diff --git a/experiments/bench_2026_11/DESIGN.md b/experiments/bench_2026_11/DESIGN.md new file mode 100644 index 00000000..85f9d4ea --- /dev/null +++ b/experiments/bench_2026_11/DESIGN.md @@ -0,0 +1,103 @@ +# Benchmark design — a test we cannot tune to + +Date: 2026-11-10. Problem statement (owner): every panel so far was authored by +us, so "we tuned it ourselves" is a fair objection. This is the design that +removes the author from the loop. + +## The three ways a benchmark can be gamed, and the fix for each + +| Hole | How we exploited it (E66) | Fix | +|---|---|---| +| **We write the questions** | generated `name` tasks contain the identifier; grep solves 91.9% | take the query from **real text we did not write** (commit messages, issue bodies) | +| **We pick the gold** | gold = symbol definition, chosen by the generator's filters | gold is **computed by a rule** from git (`git show --name-only`), no human choice | +| **We can look and retune** | full panel visible while iterating | **temporal split + sealed eval**: tuning set and eval set are different commits, eval runs once and its hash is committed first | + +## B1 — commit localization (the main benchmark) + +For every non-merge commit `C` in a pinned range: + +- **query** = the commit message (subject + body), with the diff removed. The + text is written by whoever fixed the bug — not by us. +- **gold** = the set of files changed in `C`, minus merge commits, minus commits + touching more than 20 files, minus commits whose message is a changelog/typo + stub. The filter is one rule, applied to all commits, before any result is + seen. +- **corpus** = the repository at commit `B`, where `B` is *older* than every task + commit. One index build serves the whole task set, and the fix is not in the + corpus. This is the leak that makes naive "ask about HEAD" benchmarks + worthless: the fixed code is right there in the index. +- **metric** = file-level hit@1 / hit@5 / any@8 and MRR. Exact, no judge, no + rubric. Per task first, rates with Wilson intervals second. + +Why this resists tuning: the questions are the developers' own sentences +(including typos, "wip", "fix"), the gold is what git says changed, and the +corpus predates the fix. We do not get to pick a flattering subset without +saying so in the rule. + +### Split + +- Sort tasks by commit date. +- First 70% = **dev set** (visible, tuning allowed). +- Last 30% = **sealed set**. Its task list and gold are hashed and committed + *before* the first sealed run. The runner refuses to print per-task sealed + results unless `--sealed` is passed, and that flag is used once per release. +- Report dev and sealed separately. A change that helps dev and not sealed is + reported as such. + +### Baselines that must be printed next to us + +- `git grep` on the message's identifiers (the E66 control, measured: it wins on + easy tasks). +- BM25 over the same corpus with no graph, no symbols, no gate. + +If we do not beat both, the benchmark says so — that is the point. + +## B2 — external repositories (the strongest check) + +The same generator over 2-3 pinned external repos where we have never written a +line: the commit messages and the gold are entirely independent of us. Repos and +commits are pinned in a `frozen/PINS.md` with the sha, and the runner verifies +the sha before running. + +This is the only version where "they tuned to it" is impossible by construction; +its cost is cloning and indexing a few external trees. + +## B3 — definition lookup (what E66 was) + +Keep it, but label it **regression detector**, never a quality number: a plain +grep already reaches 91.9%, measured. + +## Determinism, the part the owner asked about explicitly + +"Same exact evaluation" means: same code + same commit + same task set + same +index state -> same numbers. Two things make that hard here: + +1. **Index freshness.** E66's task 197 differed between sessions while being + stable within a session. Fix: the runner builds the index once into a + dedicated data dir, records its `build_head` and file count, and refuses to + report if the index changed mid-run. +2. **Concurrency.** Channels run in parallel; equal scores are ordered by the + deterministic tie-break we added in E65, so repeated runs agree (measured + 24 tasks x 3 runs, 0 drift). + +Any published number must come from `bench/run.py` — not from a hand-written +table. The metric lives in code, the task set lives in a hashed file. + +## What this cannot fix + +- **We can still see the sealed set** and re-tune afterwards. The only real + defenses are the temporal split (future commits are unseen by construction) + and external repos (B2). +- **A benchmark is a sample.** n and intervals decide whether a delta is real; + the rule "one task is one task" (paired counts, not rates) stays. +- **Corpus drift**: the repo changes, so old tasks may become trivial (the fix + file gets renamed). Re-pinning the range is a new version of the benchmark; + versions are never edited in place. + +## Order of work + +1. `bench/commit_tasks.py` — build tasks from git, one rule, hashed manifest. +2. `bench/run.py` — index at `B`, run tasks, exact metrics, paired deltas. +3. `bench/baseline.py` — grep and BM25 rows in the same table. +4. Sealed split + manifest hash committed before the first sealed run. +5. B2 on one external repo, then a second. diff --git a/experiments/bench_2026_11/MANIFEST.sha256 b/experiments/bench_2026_11/MANIFEST.sha256 new file mode 100644 index 00000000..b0216eae --- /dev/null +++ b/experiments/bench_2026_11/MANIFEST.sha256 @@ -0,0 +1,2 @@ +71395d4533eb12aaad83a420308af87916d0a31f7c2ee460d0734441e579a205 tasks.jsonl +base=7c2c7a3c6417539bc08f807d6790fe6e722d3a84 diff --git a/experiments/bench_2026_11/REPORT.md b/experiments/bench_2026_11/REPORT.md new file mode 100644 index 00000000..388932a0 --- /dev/null +++ b/experiments/bench_2026_11/REPORT.md @@ -0,0 +1,177 @@ +# B1 — commit-localization benchmark: first result + +Frozen before the run: `frozen/PLAN.md` (design + Amendments 1-3). +Design rationale: `DESIGN.md`. Owner objection this answers: *"we wrote the test +and tuned to it"*. + +## What was measured, and why it cannot flatter us + +- **Corpus**: worktree at `B = 7c2c7a3` (2026-09-06). Index built once there: + 383 files, 6406 chunks, 0 errors, 218 s. Guarded: the runner refuses to report + if `build_head != B`. +- **Tasks**: 174 commits from `(B, HEAD]`. The question is the commit message + **verbatim**; the gold is `git show --name-only`, one rule for every commit. +- **Solvable subset**: 66 of 174 tasks reference files created after `B` and are + unsolvable by construction (Amendment 2); they are counted and printed, and + the metrics below are over the solvable subset. +- **Split**: 121 dev / 53 sealed by date; sealed needs `--sealed` and was run + once. + +Arms: `tool` (`investigate` default), `neighbours` (same with +`include_neighbours=true`, Amendment 3), `search` (the product's BM25/hybrid +channel alone — the fair baseline), `grep` (message words, floor). + +## Dev result (71 solvable tasks) + +``` +arm hit@1 hit@5 any@8 MRR hit@5 95% CI +tool 12 19 23 0.215 [0.179, 0.381] +neighbours 24 38 44 0.424 [0.420, 0.646] +search 11 22 29 0.239 [0.214, 0.425] +grep 0 2 6 0.051 [0.008, 0.097] +paired hit@5: neighbours-only 20, search-only 4 (delta +16) +``` + +Determinism guard: 10 tasks x 3 repeats, **0** unstable. + +## Frozen predictions + +| # | Prediction | Result | +|---|---|---| +| P1 | dev `hit@5` (tool) >= 0.35 | **FAIL** — 19/71 = 0.268 | +| P2 | tool >= search + 5 on `hit@5` | **FAIL** — 19 vs 22, i.e. **-3** | +| P3 | `any@8` >= `hit@5` everywhere | PASS | +| P4 | sealed within +/- 0.10 of dev | see below | + +The default `tool` arm is **worse than the plain search channel** on prose +queries. The `neighbours` arm is +13 over search. + +### Does the message give the answer away? + +Half of real commit messages name the file ("fix parser.py"). Split: + +``` + n tool neighbours search (hit@5, dev) +names file 37 11 22 11 +prose only 34 8 16 11 +``` + +The tool's deficit is in **both** halves (11 vs 11 named, 8 vs 11 prose), so the +loss is not an artifact of messages that leak the filename. `neighbours` wins in +both. + + +## Why the default loses (measured, not argued) + +52 tool misses at hit@5, classified: + +``` +36 abstain, zero rows -> identifiers in the message do not exist in the corpus +11 outside top-8 + 4 rank 6-8 + 1 outside top-8 (answer) +``` + +The messages describe changes, so they name things the commit *introduces* +(`iter_import_candidate_nodes`). The gate extracts those names, finds zero +occurrences, and — correctly by its contract — abstains and withholds the eight +neighbours it already computed. + +On exactly those 52 tasks, with `include_neighbours=true`: + +``` +default hit@1=0 hit@5=0 any@8=4 +neighbours hit@1=12 hit@5=18 any@8=25 +``` + +So the loss is a **policy** choice (`abstain` = "do not trust anything"), not a +ranking defect. This is the most useful thing the benchmark has produced so far, +and it is a product decision, not a bug fix: returning unverified neighbours +instead of nothing gains 18 of 71 tasks and risks an agent taking a neighbour for +an answer. + +## Sealed result (37 solvable tasks) + +``` +arm hit@1 hit@5 any@8 MRR hit@5 95% CI +tool 2 7 7 0.113 [0.095, 0.342] +neighbours 11 21 21 0.410 [0.409, 0.713] +search 0 5 7 0.062 [0.059, 0.280] +grep 0 1 1 0.036 [0.005, 0.138] +paired hit@5: neighbours-only 16, search-only 0 (delta +16) +``` + +P4: `tool` moves from 0.268 (dev) to 0.189 (sealed), a **0.079** drop — inside the ++/-0.10 band, so the overfit alarm does not fire. `neighbours` moves 0.521 -> +0.568 (+0.047). + +The sealed split is harder for **every** arm (`search` collapses from 24/71 to +5/37), which is consistent with it being the most recent commits: their files +changed after the corpus snapshot. That is a property of the benchmark, not of +the tool. + +Split by "does the message name the gold file" (sealed): + +``` + n tool neighbours search +names file 20 4 14 4 +prose only 17 3 7 1 +``` + +The policy gap reproduces out of sample: `neighbours` beats the default by 10/20 +on named tasks and 4/17 on prose. + +### Panel change after the first run (and why everything was re-run) + +The third-party data gate blocked the first commit: one commit message quoted a +contributor's email address. Messages are now redacted (``) **inside the +generator**, before the manifest is hashed, which changed the manifest from +`89940dd8...` to `71395d45...`. A changed panel cannot inherit a measurement, so +both splits were re-run from scratch on the redacted panel; the numbers above are +the re-run. The conclusions did not move (sealed is identical; dev `search` +shifted by two tasks because the email split into searchable terms). + +### Instrumentation note + +The sealed process died twice (after 16 and after 22 tasks) with no traceback and +no Python exception — a hard process exit, cause not identified. The runner now +appends every row to `results_sealed.partial.jsonl` and resumes by id; the table +above is the union of three chunks. Per-task booleans for the first 22 tasks were +visible in the log before the crash; no code, arm or metric changed after that +(the arms were frozen in Amendments 1-3 before the sealed run), so this is +disclosed rather than "clean". + +## Threats + +- The corpus is one snapshot at `B`. A file renamed after `B` makes its task + unsolvable; those tasks are not filtered (only *absent* files are), so a + rename shows up as a miss and is named in the report rather than hidden. +- Many commits in the range are ours (agents), so the messages are "natural" but + not independent authors. External repos (`DESIGN.md`, B2) are the fix. +- One run per split: dev was re-run after Amendment 3, sealed was not. +- `grep` is a weak floor by construction; `search` is the baseline that matters. + +## Decision this creates + +The tool currently optimises for "never show an unverified answer". This +benchmark prices that policy: **-33 tasks of 108 solvable** (dev 19 vs 38, +sealed 7 vs 21 for the same tool with neighbours shown). + +The options are: + +1. keep `abstain` (honest, loses the neighbours); +2. return neighbours as `caveat` when nothing verifies (gains them, risks a + neighbour being read as the answer); +3. keep `abstain` but make the retry obvious to the caller (it already prints + "Pass include_neighbours=true", the measurement says that hint is not acted + on by a single-shot caller). + +No option is implemented here — it changes the product's honesty contract. + +## What would make this benchmark stronger (next version, new pin) + +- External repos (B2): messages and gold written entirely outside this project. +- An issue-text arm: issue bodies are longer and further from the diff than + commit messages. +- The crash investigation: repeated `asyncio.run` + SQLite in one process dies + after ~20 tasks; a per-chunk subprocess would remove the resumption dance. diff --git a/experiments/bench_2026_11/commit_tasks.py b/experiments/bench_2026_11/commit_tasks.py new file mode 100644 index 00000000..0e60bc9f --- /dev/null +++ b/experiments/bench_2026_11/commit_tasks.py @@ -0,0 +1,111 @@ +"""Build commit-localization tasks from git history, by one fixed rule. + +Frozen spec: `frozen/PLAN.md`. The query is a commit message we did not write; +the gold is what git says the commit changed. Nothing here inspects the tool. + +Run: + python experiments/bench_2026_11/commit_tasks.py +""" + +from __future__ import annotations + +import hashlib +import json +import re +import subprocess +import sys +from pathlib import Path + +HERE = Path(__file__).resolve().parent +ROOT = HERE.parents[1] +sys.path.insert(0, str(ROOT)) +sys.stdout.reconfigure(encoding="utf-8") + +BASE = "7c2c7a3c6417539bc08f807d6790fe6e722d3a84" # session B, frozen in PLAN.md +CODE_DIRS = ("src/", "tests/", "scripts/", "tools/") +NEVER_GOLD = re.compile( + r"^(CHANGELOG\.md|.*\.lock|requirements.*\.txt|pyproject\.toml|experiments/)", + re.I, +) +MAX_FILES = 20 +MIN_MESSAGE = 20 +EMAIL_RE = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}") + + +def _git(*args: str) -> str: + out = subprocess.run(["git", *args], cwd=ROOT, capture_output=True, + text=True, encoding="utf-8", errors="replace") + if out.returncode != 0: + raise SystemExit(f"git {' '.join(args)} failed: {out.stderr[:200]}") + return out.stdout + + +def _commits() -> list[tuple[str, str, str]]: + """(sha, date, message) for non-merge commits in (BASE, HEAD].""" + raw = _git("log", f"{BASE}..HEAD", "--no-merges", + "--format=%H%x00%ad%x00%B%x00", "--date=short") + parts = raw.split("\x00") + out = [] + for sha, date, message in zip(parts[0::3], parts[1::3], parts[2::3]): + sha, date = sha.strip(), date.strip() + if sha and date: + out.append((sha, date, message.strip())) + return out + + +def _files(sha: str) -> list[str]: + return [line.strip() for line in + _git("show", "--name-only", "--format=", sha).splitlines() if line.strip()] + + +def main() -> int: + tasks = [] + for sha, date, message in _commits(): + files = _files(sha) + if not files or len(files) > MAX_FILES: + continue + if len(message) < MIN_MESSAGE: + continue + if not any(f.startswith(CODE_DIRS) for f in files): + continue + gold = sorted(f for f in files + if f.startswith(CODE_DIRS) and not NEVER_GOLD.match(f)) + if not gold: + continue + tasks.append({ + "id": sha[:12], + "sha": sha, + "date": date, + # The third-party data gate blocks personal emails (a commit message + # quoted a contributor address). Redaction is part of the rule, not a + # post-hoc edit: it is applied before the manifest is hashed. + "question": EMAIL_RE.sub("", message), + "gold": gold, + }) + + tasks.sort(key=lambda t: (t["date"], t["id"])) + cut = int(len(tasks) * 0.7) + for index, task in enumerate(tasks): + task["split"] = "dev" if index < cut else "sealed" + + out = HERE / "tasks.jsonl" + out.write_text("\n".join(json.dumps(t, ensure_ascii=False) for t in tasks) + "\n", + encoding="utf-8", newline="") + digest = hashlib.sha256(out.read_bytes()).hexdigest() + (HERE / "MANIFEST.sha256").write_text( + f"{digest} tasks.jsonl\nbase={BASE}\n", encoding="utf-8", newline="") + + print(f"base={BASE[:12]} tasks={len(tasks)} dev={cut} sealed={len(tasks) - cut}") + print(f"sha256={digest}") + for split in ("dev", "sealed"): + subset = [t for t in tasks if t["split"] == split] + files = [len(t["gold"]) for t in subset] + print(f" {split:<6} n={len(subset):<3} " + f"gold_files mean={sum(files) / max(len(files), 1):.1f} " + f"max={max(files) if files else 0} " + f"dates={subset[0]['date'] if subset else '-'}..{subset[-1]['date'] if subset else '-'}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/experiments/bench_2026_11/determinism.py b/experiments/bench_2026_11/determinism.py new file mode 100644 index 00000000..386db86d --- /dev/null +++ b/experiments/bench_2026_11/determinism.py @@ -0,0 +1,53 @@ +"""Determinism guard for the commit-localization benchmark (frozen PLAN). + +Ten dev tasks, three repeats each: a rate is only publishable if the per-task +result does not move between runs. + +Run: python experiments/bench_2026_11/determinism.py +""" + +from __future__ import annotations + +import asyncio +import json +import sys +from pathlib import Path + +HERE = Path(__file__).resolve().parent +ROOT = HERE.parents[1] +CORPUS = Path(r"D:\Project\bench-B") +sys.path.insert(0, str(ROOT)) +sys.path.insert(0, str(ROOT / "experiments" / "xerj_2026_11")) +sys.path.insert(0, str(ROOT / "experiments" / "xerj_2026_11" / "evidence_gate")) +sys.stdout.reconfigure(encoding="utf-8") + +from run_gate import _files, _norm, _tools_for # noqa: E402 + +from run import _hits # noqa: E402 + + +def main() -> int: + tasks = [json.loads(x) for x in (HERE / "tasks.jsonl").read_text(encoding="utf-8").splitlines() if x.strip()] + tasks = [t for t in tasks if t["split"] == "dev" and any((CORPUS / g).is_file() for g in t["gold"])][:10] + tool = _tools_for({CORPUS})[str(CORPUS)] + + seen: dict[str, list] = {} + for run in range(3): + for task in tasks: + text = asyncio.run(tool.execute(query=task["question"], limit=8, + project_root=str(CORPUS))) + files = _norm(_files(text), CORPUS) + seen.setdefault(task["id"], []).append( + (_hits(files, task["gold"], 1), tuple(files[:3])) + ) + print(f"run {run + 1}/3", flush=True) + + unstable = [k for k, v in seen.items() if len({x[0] for x in v}) > 1] + print(f"tasks={len(tasks)} unstable_top1={len(unstable)}") + for key in unstable: + print(" ", key, seen[key]) + return 1 if unstable else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/experiments/bench_2026_11/frozen/PLAN.md b/experiments/bench_2026_11/frozen/PLAN.md new file mode 100644 index 00000000..ba380de7 --- /dev/null +++ b/experiments/bench_2026_11/frozen/PLAN.md @@ -0,0 +1,127 @@ +# Commit-localization benchmark — frozen plan (before the first run) + +Date: 2026-11-10. Design rationale: `experiments/bench_2026_11/DESIGN.md`. +Owner objection this answers: *"we wrote the test and tuned to it"*. + +## Source (nobody authors it for us) + +- **Corpus commit** `B = 7c2c7a3c6417539bc08f807d6790fe6e722d3a84` (2026-09-06, + `HEAD~150` on this tree). One index is built at `B` and never at `HEAD`. +- **Tasks**: commits in `(B, HEAD]` — the fixes are *not* in the corpus, so a + search cannot find the answer by reading the fixed code. +- **Query** = the commit message (subject + body), verbatim. Typos, "wip", + Russian/English mix — all kept. We do not rewrite questions. +- **Gold** = the files that commit changed, by one rule applied to every commit + before any result is seen: + - merge commits: excluded; + - commits changing 0 or > 20 files: excluded; + - commits whose changed files are all outside `src/`, `tests/`, `scripts/`, + `tools/`: excluded (docs-only and experiment-only churn is not this task); + - `CHANGELOG.md`, `*.lock`, `requirements*.txt`, `pyproject.toml`, + `experiments/**`: never gold; + - remaining commits: gold = their changed code files. + +## Split + +Sorted by commit date: first 70% = **dev**, last 30% = **sealed**. The sealed +list and gold are hashed into `MANIFEST.sha256` and committed before the first +sealed run. The runner prints sealed numbers only with `--sealed`, used once per +release. + +## Metrics (computed, never typed) + +- file-level `hit@1`, `hit@5`, `any@8`, `MRR`; +- per task first, rates with Wilson 95% after; +- paired delta tool − grep on the same tasks (McNemar-style counts, printed). + +## Baselines in the same table + +- `grep`: every file whose source contains a content word from the message, + ranked by number of distinct matched terms, path order as the tie-break. +- The tool: `investigate` on the message, limit 8. + +If the tool does not beat grep, the report says so. + +## Amendment 1 — a fair baseline (recorded before the first tool run) + +While building the harness, the grep control scored 1/10 `hit@5` on the first ten +dev tasks. That is too weak to be an honest baseline: a weak baseline flatters +the tool, and this whole benchmark exists to avoid that. + +So a third arm is added, frozen here and computed from the same corpus: + +- `search`: the product's own search channel used **alone** — + `hybrid_search_async(query, limit=50, rerank=False)`, files from + `metadata.file`, order preserved, no merge, no graph, no evidence gate. + +`grep` stays as a floor, `search` becomes the baseline that matters: it is "what +this codebase does with a message and nothing else". P2 is read against +`search`. + +## Amendment 2 — unsolvable tasks (before the full measurement) + +The corpus at `B` does not contain files created after `B`. Measured on the +frozen panel: **66 of 174 tasks** (50 of 121 dev, 16 of 53 sealed) have **no** +gold file in the corpus. No tool can find a file that does not exist, and +counting those as misses understates every arm — and invites "fixing" it by +tuning. + +Rule, applied to every task before the numbers: + +- a task is **kept** if at least one of its gold files exists at `B`; +- a hit is any *kept* gold file in the top-k (so a task with a 3-file gold where + 1 file exists is still fair); +- dropped tasks are counted and printed with every rate, and the full panel + (including them) is reported in a second column. + +The 5-task smoke that ran before this rule was a harness check, not a +measurement; its numbers are discarded. + +## Amendment 3 — the `neighbours` arm (before the sealed run) + +Dev showed that 36 of 52 misses are `abstain` with **zero rows**: the gate +extracts identifiers from the message, they do not occur in the corpus (they are +names the commit introduces), and the tool withholds its neighbours. Re-running +those 52 tasks with `include_neighbours=true` recovered **18 at hit@5** and 25 at +any@8. + +So a fourth arm is frozen here, before the sealed run: + +- `neighbours` = the same tool with `include_neighbours=true`. + +Dev is re-run with all four arms so the table is consistent, and the sealed split +is run **once** with all four. The arm is a product variant, not a tuning of the +metric: it changes what the caller asks for, not how the score is computed. + + + +## Predictions (frozen) + +| # | Prediction | Falsifier | +|---|---|---| +| P1 | dev `hit@5` >= 0.35 | below 0.35 | +| P2 | tool `hit@5` >= grep `hit@5` + 5 tasks | gain < 5 | +| P3 | `any@8` >= `hit@5` on every bucket (by construction) | violated | +| P4 | sealed `hit@5` within +/- 0.10 of dev | outside the band | + +P4 is checkable only at the sealed run; it is the overfit alarm. + +## Guards + +- **Corpus pin**: the runner records `build_head` from the graph meta and refuses + to run if it is not `B`. +- **Leak check**: no task's gold file may be modified between `B` and the task's + own commit *before* that commit — that is what `B` being the parent guarantees + for the first commit of each file; later tasks may share files, and that is + reported as a known property, not hidden. +- **Determinism**: 10 tasks re-run 3 times; any change in `top1` is printed. +- **No silent denominator**: a task that fails to run aborts the report. + +## What this does not cover + +- Docs questions, ambiguity, prose-only retrieval: separate panels. +- The corpus at `B` is one snapshot; a file renamed after `B` makes its task + unsolvable by construction. Such tasks are counted as misses and reported with + the rename as the reason, not silently dropped. +- We can still look at the sealed set after the run. The only real defense is + time plus external repos (`DESIGN.md`, B2). diff --git a/experiments/bench_2026_11/results_dev.json b/experiments/bench_2026_11/results_dev.json new file mode 100644 index 00000000..50bdff19 --- /dev/null +++ b/experiments/bench_2026_11/results_dev.json @@ -0,0 +1,57 @@ +{ + "split": "dev", + "tasks": 71, + "dropped_unsolvable": 50, + "corpus_head": "7c2c7a3c6417539bc08f807d6790fe6e722d3a84", + "stats": { + "tool": { + "n": 71, + "top1": 12, + "top5": 19, + "any8": 23, + "mrr": 0.215, + "top5_ci": [ + 0.179, + 0.381 + ] + }, + "neighbours": { + "n": 71, + "top1": 24, + "top5": 38, + "any8": 44, + "mrr": 0.424, + "top5_ci": [ + 0.42, + 0.646 + ] + }, + "search": { + "n": 71, + "top1": 11, + "top5": 22, + "any8": 29, + "mrr": 0.239, + "top5_ci": [ + 0.214, + 0.425 + ] + }, + "grep": { + "n": 71, + "top1": 0, + "top5": 2, + "any8": 6, + "mrr": 0.051, + "top5_ci": [ + 0.008, + 0.097 + ] + } + }, + "paired_hit5": { + "tool_only": 20, + "search_only": 4 + }, + "invalid": 0 +} \ No newline at end of file diff --git a/experiments/bench_2026_11/results_dev.partial.jsonl b/experiments/bench_2026_11/results_dev.partial.jsonl new file mode 100644 index 00000000..2d87dd2e --- /dev/null +++ b/experiments/bench_2026_11/results_dev.partial.jsonl @@ -0,0 +1,71 @@ +{"id": "7ac30a45ca9c", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": true, "search_top5": true, "search_any8": true, "search_mrr": 1.0, "grep_top1": false, "grep_top5": true, "grep_any8": true, "grep_mrr": 0.2, "files": ["src/core/bug_correlation.py", "src/core/commit_memory.py", "src/core/commit_memory.py"]} +{"id": "8de862e6138a", "valid": true, "tool_top1": false, "tool_top5": true, "tool_any8": true, "tool_mrr": 0.25, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.25, "search_top1": false, "search_top5": true, "search_any8": true, "search_mrr": 0.5, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.05, "files": ["src/mcp/tools/git_tools.py", "src/mcp/tools/git_tools.py", "src/core/autonomous_fix.py"]} +{"id": "dbfe75970b9d", "valid": true, "tool_top1": false, "tool_top5": true, "tool_any8": true, "tool_mrr": 0.25, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.25, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.05, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.013157894736842105, "files": ["src/mcp/tools/codebase_tool.py", "src/mcp/tools/codebase_tool.py", "src/mcp/tools/codebase_tool.py"]} +{"id": "5ce444a59fae", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": true, "search_top5": true, "search_any8": true, "search_mrr": 1.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.05, "files": []} +{"id": "604324d00566", "valid": true, "tool_top1": true, "tool_top5": true, "tool_any8": true, "tool_mrr": 1.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": true, "search_any8": true, "search_mrr": 0.25, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.08333333333333333, "files": ["src/mcp/tools/write_tools.py", "src/core/search/cypher_sql.py", "src/core/search/cypher_sql.py"]} +{"id": "7cfc6d49ac83", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": true, "search_top5": true, "search_any8": true, "search_mrr": 1.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.047619047619047616, "files": []} +{"id": "d989b2dd1d2c", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": true, "search_any8": true, "search_mrr": 0.3333333333333333, "grep_top1": false, "grep_top5": false, "grep_any8": true, "grep_mrr": 0.16666666666666666, "files": []} +{"id": "f2e2517f34d8", "valid": true, "tool_top1": true, "tool_top5": true, "tool_any8": true, "tool_mrr": 1.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": true, "search_top5": true, "search_any8": true, "search_mrr": 1.0, "grep_top1": false, "grep_top5": false, "grep_any8": true, "grep_mrr": 0.125, "files": ["src/core/search/cypher_sql.py", "scripts/lock_guard.py", "src/core/graph.py"]} +{"id": "0d83bed81289", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.5, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.041666666666666664, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.02564102564102564, "files": []} +{"id": "144580e039a9", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.25, "search_top1": false, "search_top5": false, "search_any8": true, "search_mrr": 0.16666666666666666, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.1, "files": []} +{"id": "d96445ccca76", "valid": true, "tool_top1": true, "tool_top5": true, "tool_any8": true, "tool_mrr": 1.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.058823529411764705, "files": ["src/mcp/server_tools.py", "src/core/intelligence/layer.py", "docs/research/UNIVERSAL_ENGINE_PLAN.md"]} +{"id": "22a9ab7aec21", "valid": true, "tool_top1": true, "tool_top5": true, "tool_any8": true, "tool_mrr": 1.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": true, "search_any8": true, "search_mrr": 0.25, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.07692307692307693, "files": ["src/core/intelligence/verify_on_read.py", "src/core/intelligence/verify_on_read.py", "tools/stale_detector/stale_check.py"]} +{"id": "986ad6260465", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": true, "search_top5": true, "search_any8": true, "search_mrr": 1.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.030303030303030304, "files": []} +{"id": "8ab4d57357b0", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": true, "tool_mrr": 0.125, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": true, "neighbours_mrr": 0.125, "search_top1": false, "search_top5": false, "search_any8": true, "search_mrr": 0.16666666666666666, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.0019342359767891683, "files": ["extensions/vscode/mscodebase-sync/src/extension.ts", "src/core/indexing/resource_monitor.py", "src/core/search/agentic_search.py"]} +{"id": "f96470125845", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.019230769230769232, "files": []} +{"id": "131ac555a0ad", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.1, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.07692307692307693, "files": []} +{"id": "a4b4fd605685", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.05, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.010638297872340425, "files": ["docs/research/lsp-archive/lsp_main.py", "docs/research/lsp-archive/lsp_main.py", "extensions/vscode/mscodebase-sync/src/extension.ts"]} +{"id": "5d4b9352e1c8", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": true, "neighbours_mrr": 0.125, "search_top1": false, "search_top5": false, "search_any8": true, "search_mrr": 0.125, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.034482758620689655, "files": []} +{"id": "b865c9767713", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.25, "search_top1": false, "search_top5": true, "search_any8": true, "search_mrr": 0.2, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.017857142857142856, "files": []} +{"id": "ce3db26b9e63", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.05, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.05555555555555555, "files": []} +{"id": "457fd5b1d1e4", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.034482758620689655, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.043478260869565216, "files": ["AGENTS.md", "AGENTS.md", "AGENT_DIARY.md"]} +{"id": "6214889f08dc", "valid": true, "tool_top1": false, "tool_top5": true, "tool_any8": true, "tool_mrr": 0.2, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.2, "search_top1": true, "search_top5": true, "search_any8": true, "search_mrr": 1.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.041666666666666664, "files": ["extensions/vscode/mscodebase-sync/src/extension.ts", "scripts/check_tool_names.py", "scripts/revision_gate.py"]} +{"id": "bea7e62fa8ef", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.1, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.0, "files": ["extensions/vscode/mscodebase-sync/src/extension.ts", "scripts/check_tool_names.py", "src/core/bug_correlation.py"]} +{"id": "efc1df2e0f1b", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.003194888178913738, "files": []} +{"id": "1d6d75c8b4b7", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": true, "search_mrr": 0.125, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.03333333333333333, "files": ["install.py", "install.py", "install.py"]} +{"id": "3b28fe701256", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.04, "files": ["src/core/lsp_client.py", "src/core/modification_guard.py", "scripts/run_1L_live_arm.py"]} +{"id": "a11f062b55d1", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.5, "search_top1": false, "search_top5": false, "search_any8": true, "search_mrr": 0.125, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.0035714285714285713, "files": []} +{"id": "c1e04035ba4d", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.0036231884057971015, "files": []} +{"id": "0bd8526d9cf6", "valid": true, "tool_top1": true, "tool_top5": true, "tool_any8": true, "tool_mrr": 1.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": true, "search_top5": true, "search_any8": true, "search_mrr": 1.0, "grep_top1": false, "grep_top5": false, "grep_any8": true, "grep_mrr": 0.14285714285714285, "files": ["src/core/search/engine.py", "src/core/interfaces/searcher.py", "src/core/search/engine.py"]} +{"id": "7f6d5e71c9d0", "valid": true, "tool_top1": true, "tool_top5": true, "tool_any8": true, "tool_mrr": 1.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.03225806451612903, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.027777777777777776, "files": ["src/providers/reranker/llama_install.py", ".local/orphan_path_check.py", "src/core/artifact_paths.py"]} +{"id": "a189d13d8ae0", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.024390243902439025, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.005025125628140704, "files": []} +{"id": "fb7e3daadcf5", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.07692307692307693, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.02631578947368421, "files": ["src/core/intelligence/verify_on_read.py", "src/core/search/fts5_index.py", "src/core/intelligence/layer.py"]} +{"id": "0ac4da6863d5", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.03333333333333333, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.015625, "files": ["src/mcp/tools/duplication_tool.py", "src/core/auto_doc_updater.py", "src/core/indexing/parser.py"]} +{"id": "323dbd875c15", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.023255813953488372, "files": []} +{"id": "87ae60337dfd", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": true, "neighbours_mrr": 0.14285714285714285, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.01818181818181818, "files": []} +{"id": "8c0ac276d1de", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.3333333333333333, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.02702702702702703, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.02040816326530612, "files": []} +{"id": "8c7f4479eb5c", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.25, "search_top1": false, "search_top5": false, "search_any8": true, "search_mrr": 0.14285714285714285, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.00641025641025641, "files": []} +{"id": "aad01ec7879d", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": true, "search_top5": true, "search_any8": true, "search_mrr": 1.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.047619047619047616, "files": ["install.py", "src/config/settings.py", "src/config/settings.py"]} +{"id": "ff18a8a16fb2", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.00390625, "files": []} +{"id": "3faae1f54867", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": true, "tool_mrr": 0.14285714285714285, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": true, "neighbours_mrr": 0.14285714285714285, "search_top1": false, "search_top5": true, "search_any8": true, "search_mrr": 0.5, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.02702702702702703, "files": ["src/core/auto_doc_updater.py", "src/core/auto_doc_updater.py", "install.py"]} +{"id": "8855220bfe29", "valid": true, "tool_top1": true, "tool_top5": true, "tool_any8": true, "tool_mrr": 1.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": true, "search_any8": true, "search_mrr": 0.2, "grep_top1": false, "grep_top5": false, "grep_any8": true, "grep_mrr": 0.125, "files": ["scripts/verify_diary.py", "src/core/eta_predictor.py", "scripts/console_flash_monitor.py"]} +{"id": "940c3491eb63", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.043478260869565216, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.005917159763313609, "files": ["src/core/auto_doc_updater.py", "src/core/autonomous_fix.py", "src/core/autonomous_fix.py"]} +{"id": "98716d36cce2", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.005235602094240838, "files": []} +{"id": "d1e77fad76cf", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": true, "tool_mrr": 0.14285714285714285, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": true, "neighbours_mrr": 0.14285714285714285, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.02702702702702703, "files": ["src/core/lsp_client.py", "scripts/negative_controls_runner.py", "src/core/lsp_client.py"]} +{"id": "d49b44bc24cd", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": false, "search_any8": true, "search_mrr": 0.16666666666666666, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.041666666666666664, "files": []} +{"id": "e0579136e8ac", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.006666666666666667, "files": []} +{"id": "1c7abbcf8472", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.006535947712418301, "files": []} +{"id": "7f04f5db9da4", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.058823529411764705, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.1, "files": []} +{"id": "896d8d9f9936", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.5, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.038461538461538464, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.02702702702702703, "files": []} +{"id": "12e64a940db9", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.05555555555555555, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.04, "files": []} +{"id": "39b6e9c997fe", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.05555555555555555, "files": []} +{"id": "47f85ee08ff3", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": true, "search_top5": true, "search_any8": true, "search_mrr": 1.0, "grep_top1": false, "grep_top5": true, "grep_any8": true, "grep_mrr": 0.5, "files": []} +{"id": "67f13d02e6ec", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": true, "search_any8": true, "search_mrr": 0.5, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.1, "files": []} +{"id": "a178e87ac8c2", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.08333333333333333, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.022222222222222223, "files": []} +{"id": "5b6e34560233", "valid": true, "tool_top1": true, "tool_top5": true, "tool_any8": true, "tool_mrr": 1.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": true, "search_top5": true, "search_any8": true, "search_mrr": 1.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.034482758620689655, "files": ["src/core/indexing/index_project_runner.py", "src/core/indexing/index_project_runner.py", "src/core/indexing/index_project_runner.py"]} +{"id": "5c432409ab56", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.009345794392523364, "files": []} +{"id": "9e44e4b5c5f9", "valid": true, "tool_top1": true, "tool_top5": true, "tool_any8": true, "tool_mrr": 1.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.037037037037037035, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.03571428571428571, "files": ["src/core/indexing/index_project_runner.py", "src/core/indexing/index_project_runner.py", "src/core/lsp_client.py"]} +{"id": "f982a15d45fb", "valid": true, "tool_top1": false, "tool_top5": true, "tool_any8": true, "tool_mrr": 0.5, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.5, "search_top1": false, "search_top5": true, "search_any8": true, "search_mrr": 0.5, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.045454545454545456, "files": ["scripts/stale_detector.py", "scripts/verify_diary.py", "scripts/verify_diary.py"]} +{"id": "13e235f681b0", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.024390243902439025, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.1, "files": []} +{"id": "189fac8fb277", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.02127659574468085, "files": ["src/core/di_container.py", "src/core/language_pack.py", "src/core/language_pack.py"]} +{"id": "7178e3425aaa", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.014492753623188406, "files": []} +{"id": "c1fb3a9736dd", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.00847457627118644, "files": []} +{"id": "042762f11857", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.05555555555555555, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.09090909090909091, "files": ["scripts/console_flash_monitor.py", "scripts/diag_quality_hang.py", "src/core/indexing/index_project_runner.py"]} +{"id": "08f4854311ef", "valid": true, "tool_top1": false, "tool_top5": true, "tool_any8": true, "tool_mrr": 0.5, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.5, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.008064516129032258, "files": ["install.py", "src/core/structural_search.py", "src/core/structural_search.py"]} +{"id": "0a95f68d884f", "valid": true, "tool_top1": true, "tool_top5": true, "tool_any8": true, "tool_mrr": 1.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.02702702702702703, "files": ["src/core/indexing/freshness.py", "scripts/verify_diary.py", "scripts/verify_diary.py"]} +{"id": "144ded4c60ee", "valid": true, "tool_top1": true, "tool_top5": true, "tool_any8": true, "tool_mrr": 1.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": true, "search_any8": true, "search_mrr": 0.2, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.07692307692307693, "files": ["scripts/verify_diary.py", "scripts/verify_diary.py", "scripts/verify_diary.py"]} +{"id": "2724dfd6a38a", "valid": true, "tool_top1": false, "tool_top5": true, "tool_any8": true, "tool_mrr": 0.5, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.5, "search_top1": false, "search_top5": true, "search_any8": true, "search_mrr": 0.5, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.09090909090909091, "files": ["src/core/indexing/index_project_runner.py", "src/core/indexing/indexer.py", "src/core/search/utils.py"]} +{"id": "35157951fbf2", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.005263157894736842, "files": []} +{"id": "389913427c89", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": true, "tool_mrr": 0.14285714285714285, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": true, "neighbours_mrr": 0.14285714285714285, "search_top1": true, "search_top5": true, "search_any8": true, "search_mrr": 1.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.08333333333333333, "files": ["install.py", "src/core/eta_predictor.py", "src/core/indexing/resource_monitor.py"]} +{"id": "85538026288b", "valid": true, "tool_top1": true, "tool_top5": true, "tool_any8": true, "tool_mrr": 1.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.0625, "files": ["scripts/verify_diary.py", "src/core/lsp_client.py", "src/core/lsp_client.py"]} +{"id": "97a32f126df6", "valid": true, "tool_top1": false, "tool_top5": true, "tool_any8": true, "tool_mrr": 0.5, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.5, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.03125, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.041666666666666664, "files": ["src/mcp/tools/base.py", "src/core/indexing/index_status.py", "src/core/indexing/indexer.py"]} diff --git a/experiments/bench_2026_11/results_sealed.json b/experiments/bench_2026_11/results_sealed.json new file mode 100644 index 00000000..fceb5553 --- /dev/null +++ b/experiments/bench_2026_11/results_sealed.json @@ -0,0 +1,57 @@ +{ + "split": "sealed", + "tasks": 37, + "dropped_unsolvable": 16, + "corpus_head": "7c2c7a3c6417539bc08f807d6790fe6e722d3a84", + "stats": { + "tool": { + "n": 37, + "top1": 2, + "top5": 7, + "any8": 7, + "mrr": 0.113, + "top5_ci": [ + 0.095, + 0.342 + ] + }, + "neighbours": { + "n": 37, + "top1": 11, + "top5": 21, + "any8": 21, + "mrr": 0.41, + "top5_ci": [ + 0.409, + 0.713 + ] + }, + "search": { + "n": 37, + "top1": 0, + "top5": 5, + "any8": 7, + "mrr": 0.062, + "top5_ci": [ + 0.059, + 0.28 + ] + }, + "grep": { + "n": 37, + "top1": 0, + "top5": 1, + "any8": 1, + "mrr": 0.036, + "top5_ci": [ + 0.005, + 0.138 + ] + } + }, + "paired_hit5": { + "tool_only": 16, + "search_only": 0 + }, + "invalid": 0 +} \ No newline at end of file diff --git a/experiments/bench_2026_11/results_sealed.partial.jsonl b/experiments/bench_2026_11/results_sealed.partial.jsonl new file mode 100644 index 00000000..b29d48fc --- /dev/null +++ b/experiments/bench_2026_11/results_sealed.partial.jsonl @@ -0,0 +1,37 @@ +{"id": "b07317cacffa", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.25, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.0625, "files": []} +{"id": "f33234753006", "valid": true, "tool_top1": true, "tool_top5": true, "tool_any8": true, "tool_mrr": 1.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.02702702702702703, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.043478260869565216, "files": ["src/core/indexing/resource_monitor.py", "src/config/settings.py", "src/config/settings.py"]} +{"id": "fca8916a79b2", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.034482758620689655, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.05, "files": []} +{"id": "0846cbda34e2", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.020833333333333332, "files": []} +{"id": "0b81a3707cb0", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": true, "search_any8": true, "search_mrr": 0.5, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.029411764705882353, "files": []} +{"id": "16d3a11fe706", "valid": true, "tool_top1": false, "tool_top5": true, "tool_any8": true, "tool_mrr": 0.5, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.5, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.037037037037037035, "files": ["src/core/intelligence/layer.py", "src/mcp/server_tools.py", "src/mcp/server_tools.py"]} +{"id": "1f13d8db2922", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.0, "files": []} +{"id": "2974d3af594b", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.0, "files": ["scripts/run_experiment_pagerank.py", "src/config/settings.py", "src/core/embedder/onnx_client.py"]} +{"id": "2af9d314126e", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.01694915254237288, "files": []} +{"id": "2c069a3b9302", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.025, "files": []} +{"id": "4323fa32f834", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.029411764705882353, "files": ["scripts/stale_detector.py", "tools/stale_detector/stale_check.py", "tools/stale_detector/stale_check.py"]} +{"id": "624933ad7b39", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.041666666666666664, "files": []} +{"id": "6e908b9fd99f", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.5, "search_top1": false, "search_top5": true, "search_any8": true, "search_mrr": 0.5, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.05555555555555555, "files": []} +{"id": "716405a4430f", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.25, "search_top1": false, "search_top5": true, "search_any8": true, "search_mrr": 0.25, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.025, "files": []} +{"id": "76046254d5c9", "valid": true, "tool_top1": false, "tool_top5": true, "tool_any8": true, "tool_mrr": 0.5, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.5, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.022727272727272728, "files": ["src/core/intelligence/layer.py", "src/utils/ui_formatter.py", "src/utils/ui_formatter.py"]} +{"id": "78e69add6259", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": false, "search_any8": true, "search_mrr": 0.16666666666666666, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.02631578947368421, "files": []} +{"id": "7bab133be02a", "valid": true, "tool_top1": false, "tool_top5": true, "tool_any8": true, "tool_mrr": 0.3333333333333333, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.3333333333333333, "search_top1": false, "search_top5": true, "search_any8": true, "search_mrr": 0.25, "grep_top1": false, "grep_top5": true, "grep_any8": true, "grep_mrr": 0.2, "files": ["src/core/lsp_client.py", "src/mcp/tools/base.py", "src/mcp/tools/write_tools.py"]} +{"id": "80a69faea54e", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.02857142857142857, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.047619047619047616, "files": []} +{"id": "85ab4aecd7e7", "valid": true, "tool_top1": false, "tool_top5": true, "tool_any8": true, "tool_mrr": 0.3333333333333333, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.3333333333333333, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.08333333333333333, "files": ["src/core/intelligence/layer.py", "src/core/intelligence/layer.py", "src/mcp/server_tools.py"]} +{"id": "93b2e9134e44", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.017857142857142856, "files": ["docs/research/UNIVERSAL_ENGINE_PLAN.md", "docs/research/UNIVERSAL_ENGINE_PLAN.md", "docs/archive/DEV_DIARY_2026_07.md"]} +{"id": "974bb1c2332c", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.005714285714285714, "files": []} +{"id": "aa96f96ed315", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.038461538461538464, "files": []} +{"id": "bc9ececf6780", "valid": true, "tool_top1": false, "tool_top5": true, "tool_any8": true, "tool_mrr": 0.5, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.5, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.03125, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.0625, "files": ["scripts/live_search_audit.py", "src/core/indexing/indexer.py", "scripts/backfill_chunk_hash.py"]} +{"id": "d30b3a2c8487", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.015384615384615385, "files": ["scripts/download_model.py", "scripts/download_model.py", "scripts/download_model.py"]} +{"id": "d9e52f0d4998", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.005291005291005291, "files": ["src/core/version_manager.py", "src/utils/i18n.py", "src/core/lsp_client.py"]} +{"id": "db49d9ae1539", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.008849557522123894, "files": []} +{"id": "e827ab45b067", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.00847457627118644, "files": []} +{"id": "f6c234cc2434", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.04, "files": []} +{"id": "1090188f3490", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.008, "files": ["src/core/project_resolution.py", "src/mcp/server_tools.py", "src/mcp/server_tools.py"]} +{"id": "4e8adb9bfbe1", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.05555555555555555, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.0625, "files": []} +{"id": "6bbed1843196", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.015151515151515152, "files": ["scripts/verify_diary.py", "scripts/verify_diary.py", "src/core/eta_predictor.py"]} +{"id": "8845633115d7", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.047619047619047616, "files": []} +{"id": "8d5136de3fd0", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.5, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.03571428571428571, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.03125, "files": []} +{"id": "985c80f463b7", "valid": true, "tool_top1": true, "tool_top5": true, "tool_any8": true, "tool_mrr": 1.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": false, "search_any8": true, "search_mrr": 0.16666666666666666, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.043478260869565216, "files": ["src/mcp/server_tools.py", "scripts/benchmark_search_stages.py", "scripts/bump_version.py"]} +{"id": "dc9fa9d49651", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 0.5, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.047619047619047616, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.038461538461538464, "files": []} +{"id": "8dec11e490d4", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": false, "neighbours_top5": false, "neighbours_any8": false, "neighbours_mrr": 0.0, "search_top1": false, "search_top5": false, "search_any8": false, "search_mrr": 0.0, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.0016051364365971107, "files": []} +{"id": "f69052743638", "valid": true, "tool_top1": false, "tool_top5": false, "tool_any8": false, "tool_mrr": 0.0, "neighbours_top1": true, "neighbours_top5": true, "neighbours_any8": true, "neighbours_mrr": 1.0, "search_top1": false, "search_top5": true, "search_any8": true, "search_mrr": 0.2, "grep_top1": false, "grep_top5": false, "grep_any8": false, "grep_mrr": 0.047619047619047616, "files": []} diff --git a/experiments/bench_2026_11/run.py b/experiments/bench_2026_11/run.py new file mode 100644 index 00000000..da5bd9ba --- /dev/null +++ b/experiments/bench_2026_11/run.py @@ -0,0 +1,265 @@ +"""Run the commit-localization benchmark (frozen: `frozen/PLAN.md`). + +Corpus is a worktree at the frozen base commit `B`; the tasks are commits after +`B`, so the fix is not in the index. The query is the commit message verbatim. + +Run: + python experiments/bench_2026_11/run.py --split dev + python experiments/bench_2026_11/run.py --split sealed --sealed + +The sealed split refuses to run without `--sealed`, which is passed once per +release. +""" + +from __future__ import annotations + +import argparse +import asyncio +import json +import re +import subprocess +import sys +from pathlib import Path + +HERE = Path(__file__).resolve().parent +ROOT = HERE.parents[1] +sys.path.insert(0, str(ROOT)) +sys.path.insert(0, str(ROOT / "experiments" / "xerj_2026_11")) +sys.path.insert(0, str(ROOT / "experiments" / "xerj_2026_11" / "evidence_gate")) +sys.stdout.reconfigure(encoding="utf-8") + +from run_gate import _files, _invalid, _norm, _tools_for # noqa: E402 +from wilson import wilson # noqa: E402 + +BASE = "7c2c7a3c6417539bc08f807d6790fe6e722d3a84" +CORPUS = Path(r"D:\Project\bench-B") +STOP = { + "the", "and", "for", "with", "that", "this", "from", "into", "when", "what", + "where", "which", "have", "has", "not", "are", "was", "were", "will", "can", + "all", "one", "two", "fix", "add", "update", "test", "tests", "code", + "правка", "тесты", "код", "для", "что", "как", "или", "при", +} + + +def _tasks(split: str) -> list[dict]: + rows = [json.loads(x) for x in (HERE / "tasks.jsonl").read_text(encoding="utf-8").splitlines() if x.strip()] + return [r for r in rows if split == "all" or r["split"] == split] + + +def _solvable(task: dict) -> bool: + """At least one gold file must exist in the corpus (PLAN Amendment 2).""" + return any((CORPUS / g).is_file() for g in task["gold"]) + + +def _kept_gold(task: dict) -> list[str]: + return [g for g in task["gold"] if (CORPUS / g).is_file()] + + +def _corpus_files() -> list[str]: + out = subprocess.run(["git", "ls-files", "*.py", "*.md", "*.toml", "*.json"], + cwd=CORPUS, capture_output=True, text=True, encoding="utf-8") + return sorted(x for x in out.stdout.splitlines() if x.strip()) + + +class GrepBaseline: + """Files ranked by how many distinct message terms they contain.""" + + def __init__(self, files: list[str]): + self.files = files + self.texts = {} + for path in files: + try: + self.texts[path] = (CORPUS / path).read_text( + encoding="utf-8", errors="ignore").lower() + except OSError: + self.texts[path] = "" + + def rank(self, message: str) -> list[str]: + terms = {w for w in re.findall(r"[a-zA-Zа-яА-Я_]{4,}", message.lower()) + if w not in STOP} + if not terms: + return [] + scored = [] + for path, text in self.texts.items(): + hits = sum(1 for term in terms if term in text) + if hits: + scored.append((-hits, path)) + scored.sort() + return [path for _, path in scored] + + +def _hits(files: list[str], gold: list[str], k: int) -> bool: + return any(any(g in f for f in files[:k]) for g in gold) + + +def _mrr(files: list[str], gold: list[str]) -> float: + for rank, path in enumerate(files, 1): + if any(g in path for g in gold): + return 1.0 / rank + return 0.0 + + +def _stats(rows: list[dict], key: str) -> dict: + top1 = sum(r[f"{key}_top1"] for r in rows) + top5 = sum(r[f"{key}_top5"] for r in rows) + any8 = sum(r[f"{key}_any8"] for r in rows) + mrr = sum(r[f"{key}_mrr"] for r in rows) / len(rows) + return {"n": len(rows), "top1": top1, "top5": top5, "any8": any8, + "mrr": round(mrr, 3), + "top5_ci": [round(x, 3) for x in wilson(top5, len(rows))]} + + +def _files_from_rows(rows: list[dict]) -> list[str]: + out: list[str] = [] + for row in rows or []: + meta = row.get("metadata") or {} + file = str(meta.get("file") or row.get("file") or "") + if file: + file = file.replace("\\", "/") + base = str(CORPUS).replace("\\", "/").rstrip("/") + if file.startswith(base): + file = file[len(base):].lstrip("/") + out.append(file) + return out + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--split", default="dev", choices=("dev", "sealed", "all")) + parser.add_argument("--sealed", action="store_true") + parser.add_argument("--limit", type=int, default=0) + args = parser.parse_args() + + if args.split == "sealed" and not args.sealed: + print("refused: sealed split needs --sealed (one run per release)") + return 2 + if args.split == "all": + print("refused: report dev and sealed separately") + return 2 + + tasks_all = _tasks(args.split) + if args.limit: + tasks_all = tasks_all[:args.limit] + tasks = [] + for task in tasks_all: + gold = _kept_gold(task) + if gold: + tasks.append({**task, "gold": gold}) + dropped = len(tasks_all) - len(tasks) + print(f"split={args.split} tasks={len(tasks)} dropped_unsolvable={dropped} " + f"corpus={CORPUS}", flush=True) + + tools = _tools_for({CORPUS}) + tool = tools[str(CORPUS)] + + # Guard: the corpus must be the frozen base, not a moving HEAD. + from src.core.di_container import ( # noqa: E402 + IndexerFactoryKey, + ProjectIndexerRegistry, + create_service_collection, + ) + + services = create_service_collection(CORPUS) + indexer = services.resolve(ProjectIndexerRegistry).get_indexer( + CORPUS, factory=services.resolve(IndexerFactoryKey) + ) + head = indexer._symbol_index.graph.get_meta("build_head") or "" + print(f"corpus build_head={head[:12]} expected={BASE[:12]}") + baseline = GrepBaseline(_corpus_files()) + + rows = [] + partial = HERE / f"results_{args.split}.partial.jsonl" + done = set() + if partial.is_file(): + for line in partial.read_text(encoding="utf-8").splitlines(): + if line.strip(): + old = json.loads(line) + rows.append(old) + done.add(old["id"]) + print(f"resuming: {len(done)} tasks already captured", flush=True) + + searcher = tool.resolve_searcher(explicit_project_root=str(CORPUS)) + for index, task in enumerate(tasks, 1): + if task["id"] in done: + continue + text = asyncio.run(tool.execute(query=task["question"], limit=8, + project_root=str(CORPUS))) + files = _norm(_files(text), CORPUS) + nb_text = asyncio.run(tool.execute(query=task["question"], limit=8, + include_neighbours=True, + project_root=str(CORPUS))) + nb_files = _norm(_files(nb_text), CORPUS) + grep = baseline.rank(task["question"]) + raw = asyncio.run( + searcher.hybrid_search_async(task["question"], limit=50, rerank=False) + ) + search_files = _files_from_rows(raw) + row = { + "id": task["id"], + "valid": not _invalid(text), + "tool_top1": _hits(files, task["gold"], 1), + "tool_top5": _hits(files, task["gold"], 5), + "tool_any8": _hits(files, task["gold"], 8), + "tool_mrr": _mrr(files, task["gold"]), + "neighbours_top1": _hits(nb_files, task["gold"], 1), + "neighbours_top5": _hits(nb_files, task["gold"], 5), + "neighbours_any8": _hits(nb_files, task["gold"], 8), + "neighbours_mrr": _mrr(nb_files, task["gold"]), + "search_top1": _hits(search_files, task["gold"], 1), + "search_top5": _hits(search_files, task["gold"], 5), + "search_any8": _hits(search_files, task["gold"], 8), + "search_mrr": _mrr(search_files, task["gold"]), + "grep_top1": _hits(grep, task["gold"], 1), + "grep_top5": _hits(grep, task["gold"], 5), + "grep_any8": _hits(grep, task["gold"], 8), + "grep_mrr": _mrr(grep, task["gold"]), + "files": files[:3], + } + rows.append(row) + with partial.open("a", encoding="utf-8") as handle: + handle.write(json.dumps(row, ensure_ascii=False) + "\n") + print(f" {index:>3}/{len(tasks)} {task['id']} " + f"tool@5={row['tool_top5']} nb@5={row['neighbours_top5']} " + f"search@5={row['search_top5']}", flush=True) + + invalid = sum(1 for r in rows if not r["valid"]) + stats = {arm: _stats(rows, arm) for arm in ("tool", "neighbours", "search", "grep")} + only_tool = sum(1 for r in rows if r["neighbours_top5"] and not r["search_top5"]) + only_search = sum(1 for r in rows if r["search_top5"] and not r["neighbours_top5"]) + + print(f"\n{'arm':<6} {'n':>3} {'hit@1':>6} {'hit@5':>6} {'any@8':>6} " + f"{'MRR':>5} hit@5 95% CI") + for arm in ("tool", "neighbours", "search", "grep"): + cell = stats[arm] + print(f"{arm:<6} {cell['n']:>3} {cell['top1']:>6} {cell['top5']:>6} " + f"{cell['any8']:>6} {cell['mrr']:>5} {cell['top5_ci']}") + print(f"\npaired hit@5: neighbours-only={only_tool} search-only={only_search} " + f"delta={only_tool - only_search:+d}") + print(f"invalid runs: {invalid}") + if dropped: + total = len(tasks) + dropped + print(f"\nfull panel (dropped {dropped} unsolvable tasks counted as misses):") + for arm in ("tool", "neighbours", "search", "grep"): + cell = stats[arm] + print(f" {arm:<6} hit@1={cell['top1']}/{total} " + f"hit@5={cell['top5']}/{total} any@8={cell['any8']}/{total}") + + out = HERE / f"results_{args.split}.json" + out.write_text(json.dumps({ + "split": args.split, "tasks": len(tasks), "dropped_unsolvable": dropped, + "corpus_head": head, "stats": stats, + "paired_hit5": {"tool_only": only_tool, "search_only": only_search}, + "invalid": invalid, + }, ensure_ascii=False, indent=1), encoding="utf-8", newline="") + + if invalid: + print("ABORT: a task did not run; the denominator is not the panel.") + return 3 + if head[:12] != BASE[:12]: + print("ABORT: corpus is not the frozen base commit.") + return 4 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/experiments/bench_2026_11/tasks.jsonl b/experiments/bench_2026_11/tasks.jsonl new file mode 100644 index 00000000..4b3ed0d0 --- /dev/null +++ b/experiments/bench_2026_11/tasks.jsonl @@ -0,0 +1,174 @@ +{"id": "7ac30a45ca9c", "sha": "7ac30a45ca9cc9049a654c9a90534504389d4835", "date": "2026-09-06", "question": "fix(lock_guard): raise _run timeout 60s to 900s — pre-commit hook needs up to 10min", "gold": ["scripts/lock_guard.py"], "split": "dev"} +{"id": "8de862e6138a", "sha": "8de862e6138ae07330dc428ca0c15a7e5c18022f", "date": "2026-09-06", "question": "fix(subprocess): add CREATE_NO_WINDOW to powershell/nvidia-smi spawns (P-001 reoccurrence); real guard test", "gold": ["src/core/indexing/resource_monitor.py", "src/providers/reranker/llama_runner.py", "tests/test_subprocess_windows.py"], "split": "dev"} +{"id": "dbfe75970b9d", "sha": "dbfe75970b9dc21b9a407799031664acff22e404", "date": "2026-09-06", "question": "fix(tools): wrap sync subprocess in asyncio.to_thread for async MCP tools (context_tool, system_tools)", "gold": ["src/mcp/tools/context_tool.py", "src/mcp/tools/system_tools.py"], "split": "dev"} +{"id": "5ce444a59fae", "sha": "5ce444a59faee6fa7198dbc1d1d113c326ba2903", "date": "2026-09-08", "question": "feat(indexing): derive import maps from parser and gate fallback behind language pack flag\n\nRoute import extraction through a single source of truth: LANGUAGE_IMPORT_NODES is now a derived view of CodeParser.IMPORT_NODE_MAP instead of a separate literal that had drifted for kt/dart/php.\n\n- parser._walk_file: run fallback (mode 2) only in the else-branch when ext has no map, behind a double gate (else-branch + language_pack.is_enabled inside iter_import_candidate_nodes); main grammar path stays priority\n- extract_imports_from_file reads the flag via language_pack.is_enabled (removes env re-read)\n- dedup moved out to a shared path used by both precise and fallback modes\n- LANGUAGE_IMPORT_NODES: lazy derivation via module __getattr__ (PEP 562) so parser is never imported at module load; static parser<->language_imports cycle registered as deliberate tech debt in _ALLOWED_CORE_CYCLES (KNOWN_ISSUES 2026-09-08)\n- tests: TestMapConsistency (card drift), TestFallbackImports (4 integration), fallback-gate negative controls; full suite green", "gold": ["scripts/architecture_linter.py", "src/core/indexing/parser.py", "src/core/language_imports.py", "tests/test_language_imports.py", "tests/test_symbol_index_call_graph.py"], "split": "dev"} +{"id": "604324d00566", "sha": "604324d00566c065761231483fafa3f3d85c1ae5", "date": "2026-09-08", "question": "fix(write): record ActionReceipt in all write paths and fix Cypher anonymous nodes\n\nWriteTool._contract_record now emits an ActionReceipt alongside ChangeIntent via new _contract_receipt(); _contract_record itself was added to replace/insert/rename-LSP/move (incl. refs) paths that previously wrote files without any contract artifact.\n\nCypher engine (cypher_sql.py): resolve from_node_alias for anonymous nodes (n{path_idx*2}), register edge variables ([e:]) into edge_vars/edge_prop_map so e.type/source_id/target_id and count(e) translate to SQL.", "gold": ["src/mcp/tools/write_tools.py", "tests/test_write_tools.py"], "split": "dev"} +{"id": "7cfc6d49ac83", "sha": "7cfc6d49ac834d5d9ab677099b7a9c112cb9e51f", "date": "2026-09-08", "question": "fix(cypher): translate collect() to json_group_array with typed decode (#29)\n\nCOLLECT(expr) now lowers to json_group_array(expr) FILTER (WHERE expr IS NOT NULL): Neo4j semantics (null-ignored, empty match -> []). collect(*) and nested collect raise explicit ValueError; DISTINCT is rejected by the parser layer. Translator marks collect columns (collect_cols); cypher_executor decodes json.loads only for marked string columns (collision-safe, warning-only on decode failure).\n\n13 new SQL/E2E/error tests incl. decode-collision guard; full pytest 1663 passed / 6 skipped / 91 deselected (verified 5x incl. sanitized-env gate-zero replica); ruff clean; verify_diary 16/0; negative controls: stale_detector PROVEN. Red Team 5/5. Experiments G1/G2 raw-verified on sqlite 3.50.4.\n\nRefs KNOWN_ISSUES 2026-09-07 (collect open -> fixed).\n\nCo-authored-by: MSCodeBase Agent <>", "gold": ["scripts/verify_diary.py", "src/core/search/cypher_executor.py", "src/core/search/cypher_sql.py", "tests/test_cypher_engine.py"], "split": "dev"} +{"id": "d989b2dd1d2c", "sha": "d989b2dd1d2c9aa4d8fdd9290740756ade0c5ccd", "date": "2026-09-08", "question": "feat(indexing): per-language grammar maps for imports, calls, assigns, conditions\n\nWire CALL_NODES/IMPORT_NODE_MAP/ASSIGNMENT_NODE_MAP/CONDITIONAL_NODE_MAP from universal-engine study into CodeParser walk, verified against live tree-sitter grammars for 11 languages.\n\n- calls: simple_identifier (Swift/Kotlin/Dart), method_invocation name field (Java), PHP *_call_expression last-name rule, Ruby method target\n- imports: C/C++ preproc_include regex, Kotlin keyword-import filter, Dart dedup, Ruby call require/include filter\n- assigns: variable_name left (PHP), directly_assignable_expression + simple_identifier (Swift), name-node RHS\n- conditional: per-language CONDITIONAL_NODE_MAP replaces shared set\n- live tests: TestMultiLangCalls (7) + TestMultiLangImports (9) + php assign", "gold": ["src/core/indexing/parser.py", "tests/test_assignments.py", "tests/test_symbol_index_call_graph.py"], "split": "dev"} +{"id": "f2e2517f34d8", "sha": "f2e2517f34d8df5d3abd83e451181085aee34478", "date": "2026-09-08", "question": "fix(cypher): support anonymous nodes and edge variables in SQL generator\n\nRoot cause: from_node_alias defaulted to n1 while joins used n{idx*2} for anonymous left nodes (no such column: n0.id); edge variable [e:] was dropped (alias hardcoded e{idx}), so e.type resolved to json_extract on a missing table. Fix: resolve from_node_alias to the generated alias; register edge vars in edge_vars and map e.type/source_id/target_id to columns; support count(e). 10 regression tests (SQL+E2E) + 5 Red Team attacks covered.", "gold": ["src/core/search/cypher_sql.py", "tests/test_cypher_engine.py"], "split": "dev"} +{"id": "0d83bed81289", "sha": "0d83bed812897951ad955450aa45ffa29839dd26", "date": "2026-09-09", "question": "feat(mcp): graph-backed sections in get_context with structured handoff and VOR filter (#30)\n\n* feat(mcp): graph-backed sections in get_context with structured handoff and vor filter\n\nNew deterministic sections assembled from the PropertyGraph (no LLM): dataflow (ASSIGNED_FROM/TO chains with condition_path for local variables), writes (WRITES edges), receipts (ActionReceipts filtered by target file), tests (affected tests from impact affected_files). Wired into intent profiles: modify/debug/prepare_change/verify_change/test; budgets and priorities extended.\n\nStructured handoff: _section_symbols now carries meta (file_path/line/symbol/affected_files) consumed by source/git and the new sections; regex text-parsing demoted to fallback. Memory section applies VOR filtering: only VERIFIED/ACTIVE (or status-less) nodes are shown. Lazy SymbolIndexAdapter init; every new section degrades to None when the graph is unavailable.\n\n12 new tests (test_context_sections.py): 14 passed with existing context_tool tests; full suite 1659 passed / 9 skipped / 91 deselected; ruff clean.\n\nRefs: agent-research on Anthropic code-execution-with-mcp + writing-tools-for-agents (tool consolidation, meaningful context, token efficiency).\n\n* ci: raise full-suite timeout from 10 to 20 minutes\n\nThe test job started exceeding the 10-minute job timeout once get_context v2 (+433 lines, 12 tests) and the collect() work landed: locally the suite takes ~15.6 min with coverage. Observed: job 102074901674/102093822441 timed out at 10 min mid-suite (last file test_hybrid_cache.py, all tests green up to that point). clean-state (same suite, filtered) stays green.\n\n---------\n\nCo-authored-by: MSCodeBase Agent <>", "gold": ["src/mcp/tools/context_tool.py", "tests/test_context_sections.py"], "split": "dev"} +{"id": "144580e039a9", "sha": "144580e039a98abac701e6cc51f83b12a365807b", "date": "2026-09-10", "question": "fix(search): get_symbol_info AMBIGUOUS + docs + README badge (#33)\n\n* docs(readme): bump tests badge to 1773 and tools count to 64\n\n* docs(experiments): record VOR catch-up (exp1) and HEAD-polling (exp3) results\n\n* fix(search): get_symbol_info returns AMBIGUOUS with candidates when len(defs) > 1\n\nCloses issue #21 read-path fork. When build_call_graph returns multiple\ndefinitions, the tool now surfaces all candidates (up to 10) instead of\nsilently picking defs[0]. Single-definition path is unchanged.\n\nVerified: 3 new tests + 38 related tests passed, 1704 total, ruff clean.\n\n* feat(memory): fail-closed freshness gate (read+write) and dirty-tree interval fix\n\nResearch (Exp 2 chain): inform-the-agent is unreliable (agent can ignore STALE\nalerts, PlanFence 30/30 failures); server-side blocking gives 0% stale. Add a\nfail-closed gate so memory reads/writes no longer silently use stale data.\n\n- config: MemoryConfig.freshness_gate (off|read|write|both, default both)\n- read gate: intel_get_project_memory marks unverified nodes stale_unverified\n on incomplete VOR pass, marks domain CONSISTENT on full pass, surfaces\n freshness_gate status in stats and ui_formatter\n- write gate: intel_add_memory_node refuses while memory is STALE with pointer\n to intel_get_project_memory\n- dirty tree: _is_dirty() git-status probe with TTL cache; dirty fingerprints\n rebuilt every pass (never persisted) and dirty verdicts bypass the cache\n entirely (dirty-bit in cache key), closing the notify->commit toxic interval\n- background verify marks memory CONSISTENT after a full pass\n- tests: conftest autouse ConsistencyTracker reset (singleton leak)\n\n---------\n\nCo-authored-by: MSCodeBase Agent <>", "gold": ["src/config/settings.py", "src/core/intelligence/layer.py", "src/core/intelligence/verify_on_read.py", "src/mcp/tools/search_tools.py", "src/utils/ui_formatter.py", "tests/conftest.py", "tests/test_freshness_gate.py", "tests/test_get_symbol_info_ambiguous.py"], "split": "dev"} +{"id": "d96445ccca76", "sha": "d96445ccca76ea592951545a353c29ecce4b3f15", "date": "2026-09-10", "question": "chore(experiments): E-S1/E-S2 live-repo results + Exhibit #23 audit docs (#32)\n\n* fix(cypher): translate collect() to json_group_array with typed decode\n\nCOLLECT(expr) now lowers to json_group_array(expr) FILTER (WHERE expr IS NOT NULL): Neo4j semantics (null-ignored, empty match -> []). collect(*) and nested collect raise explicit ValueError; DISTINCT is rejected by the parser layer. Translator marks collect columns (collect_cols); cypher_executor decodes json.loads only for marked string columns (collision-safe, warning-only on decode failure).\n\n13 new SQL/E2E/error tests incl. decode-collision guard; full pytest 1663 passed / 6 skipped / 91 deselected (verified 5x incl. sanitized-env gate-zero replica); ruff clean; verify_diary 16/0; negative controls: stale_detector PROVEN. Red Team 5/5. Experiments G1/G2 raw-verified on sqlite 3.50.4.\n\nRefs KNOWN_ISSUES 2026-09-07 (collect open -> fixed).\n\n* chore(issues): dedup entries, add temperatures and deadlines\n\nRemove 3 duplicate issue blocks (full version kept), fix false entry counter (11 -> 14) after dedup, set yellow temperature + deadline + owner on open items (VOR lazy-only by 2026-09-15, B4 refactor by 2026-10-01).\n\n* docs(logs): E-S1/E-S2 live-repo results; add .h issue; fix tests badge\n\n- EXPERIMENTS_LOG: E-S1/E-S2 on 6 real top-repos (imports edge-rate\n java 0.867 / php 0.797 / c 0.680 / kotlin 0.853 / dart 0.940 /\n ruby 0.618; calls php 0.813 / ruby 0.562 / c 0.250 / dart 0.080 -\n synthetic \"PHP/Ruby/C/Dart calls = 0\" defect withdrawn; skeleton\n ASSIGNED_FROM+condition_path thousands of rows outside Python)\n- KNOWN_ISSUES: new Open entry - .h headers not indexed\n (SUPPORTED_EXTENSIONS lacks .h; curl 65/300 map_lies mostly .h)\n- README: tests badge -> 1663 (real full-suite count: 1663 passed /\n 6 skipped / 91 deselected, verified twice on this HEAD)\n- telemetry: runtime rotation from MCP tool runs\n\nRefs: docs/research/universal-engine-study/05 sec.3, 08 sec.3,\nKNOWN_ISSUES #13 (lazy reindex: symbols +28, graph not rebuilt)\n\n* docs(logs): record Exhibit #23 audit (absent tool call) for continuous verification\n\n* feat(intel): idle background VOR pass via IdleScheduler hook\n\nVerify memory without an agent call: _check_index_health now invokes a\nregistered hook (set_idle_vor_callback) that runs VerifyOnRead in the\nbackground with budget=250ms and locked()-guard. Uses the shared\n_write_lock and get_verifier registry so idle and agent passes\nserialize without a second lock. Extract _build_symbol_resolver for\nDRY. Closes KNOWN_ISSUES 2026-09-07 (deadline 2026-09-15).\n\n* docs(issues): mark second lazy-only record as Fixed (sync with H1)\n\n* feat(parse): index .h C headers as parseable C files\n\nPARSE_EXTENSIONS now includes .h; the C parser handles .h, env and\nimport node maps map .h->c/preproc_include. Header files previously\nproduced no AST imports/calls even though they were vector-indexed\n(dart-http curl 65/300 chunks were .h). Closes the E-S1 map_lies gap.\n\n* docs(issues): mark .h header indexing as Fixed (H2, commit 0301fa93)\n\n* chore(cleanup): remove stray test capture file\n\n* chore(cleanup): remove stray commit capture file\n\n* feat(alerts): system_alerts chain stale->starved with one-shot delivery\n\nAdd AlertStore (per-project JSON, thread-safe, dedup by kind+payload,\natomic collect_and_clear limit=5). Generate memory_stale on first\nSTALE transition in notify_change and memory_starved from idle VOR\nand intel_get_project_memory. Deliver via format_system_alerts into\nintel_get_project_memory and intel_explain_project_state.\n\nTests: 14 new (AlertStore + formatter); full pytest 1689 passed.\n\n* docs(log): log system_alerts chain probe (exp 2026-09-10)\n\n---------\n\nCo-authored-by: MSCodeBase Agent <>", "gold": ["src/core/extensions.py", "src/core/indexing/parser.py", "src/core/intelligence/alert_store.py", "src/core/intelligence/layer.py", "src/core/intelligence/tools_reg.py", "src/core/task_queue.py", "src/mcp/server_tools.py", "src/mcp/tools/indexing_tools.py", "src/utils/ui_formatter.py", "tests/test_alert_store.py", "tests/test_symbol_index_call_graph.py", "tests/test_task_queue.py", "tests/test_ui_formatter_memory.py", "tests/test_verify_on_read.py"], "split": "dev"} +{"id": "22a9ab7aec21", "sha": "22a9ab7aec216d361a36652b04d3e5b2eedbfe17", "date": "2026-09-11", "question": "fix(intelligence): VOR read-path skips prose scan when explicit anchors exist (#34)\n\n* fix(intelligence): VOR read-path skips prose scan when explicit anchors exist\n\nRead-path re-scanned prose via _PATH_RE even when data.anchors were already\ncaptured at write-time. ADR body prose contains historical \"X -> Y\" paths,\nso a rename-sweep resurrected the old path and falsely REFUTED live nodes\n(ADR-d35fd1e7fee5: src/utils/paths.py -> adapters/local_fs/windows.py).\n\nextract_anchors now accepts read_path=True; VerifyOnRead.run() passes it.\nWhen the node has explicit anchors the prose is not scanned, so historical\npaths never become checkable anchors. Legacy nodes without explicit anchors\nkeep prose scanning (backward compat).\n\nAdd regression tests: explicit anchors trusted on read-path, historical\nprose path not refuting live node, legacy node still refutes on real drift.\n\nVerified: 53 tests in test_verify_on_read.py + 28 in test_memory_retraction.py.\n\n* docs: record VOR read-path PR #34 and test-overhead owner decision\n\n- AGENT_DIARY: session entry for PR #34 (extract_anchors read_path=True,\n prose skip when explicit anchors exist) + decision that 178s gate-zero\n is normal and pytest stays single-threaded.\n- WISDOM: \"Commit overhead — НЕ баг\" — 8-min commit = llama-server\n contention, not slow tests; basetemp race is known, do not fix.\n\n---------\n\nCo-authored-by: MSCodeBase Agent <>", "gold": ["src/core/intelligence/verify_on_read.py", "tests/test_verify_on_read.py"], "split": "dev"} +{"id": "986ad6260465", "sha": "986ad62604654eb2f60d76792514c63143cf431f", "date": "2026-09-12", "question": "feat(intelligence): H3 TTL-rotation for verify-on-read (last_checked + stale_ttl)", "gold": ["src/core/intelligence/layer.py", "src/core/intelligence/verify_on_read.py", "src/utils/ui_formatter.py", "tests/test_verify_on_read_ttl.py"], "split": "dev"} +{"id": "8ab4d57357b0", "sha": "8ab4d57357b02d061babe9d7dad90307da25e175", "date": "2026-09-15", "question": "fix(ui): resolve actual TTL days in stale_ttl label (#35)\n\nCo-authored-by: MSCodeBase Agent <>", "gold": ["src/utils/ui_formatter.py"], "split": "dev"} +{"id": "a1991ef60523", "sha": "a1991ef605238f8ff4fbb5df0be41a1fbd6d2935", "date": "2026-09-16", "question": "feat(core): Bootstrap A2 - TESTS edges from dynamic trace into PropertyGraph\n\nNew src/core/bootstrap_tests.py builds Test nodes (label=Test, existing\nFunction nodes reused via get_node to avoid ON CONFLICT label overwrite)\nand links each test to executed src functions via EdgeType.TESTS.\n\nLive run on real graph DB: 1595 Test nodes created, 132 reused, 14985/15669\nsrc functions matched (95.7%), 16172 unique TESTS edges (idempotent UPSERT).\nUnmatched (684) are only / with no symbol nodes.\nRed Team 5/5: empty trace, name duplicates, label overwrite, re-runs, params.\n\nDocs: KNOWN_ISSUES #FEATURE progress, six a2_*.py analysis scripts.", "gold": ["src/core/bootstrap_tests.py", "tests/test_bootstrap_tests.py"], "split": "dev"} +{"id": "8fcf06adb7fe", "sha": "8fcf06adb7fea459c8bf68b36ef2979555b00cf4", "date": "2026-09-17", "question": "feat(bootstrap): add entity detector for data structures (Шаг B1)\n\nAST-based detector of clean data structures for new-project bootstrap:\n@dataclass (Name/Call/Attribute forms) and NamedTuple bases, without the\n`Table(` regex noise (db.open_table calls are LanceDB, not SQL schemas).\n\nLive check vs graph DB: 46/46 graph dataclass classes covered, zero losses;\n+3 are the new bootstrap classes themselves (not yet reindexed); NamedTuple=0\nand open_table=12 calls confirmed non-entities. 8 unit tests. Deterministic,\nread-only, idempotent, concurrency-safe.", "gold": ["src/core/bootstrap_entities.py", "tests/test_bootstrap_entities.py"], "split": "dev"} +{"id": "e0c303b8146e", "sha": "e0c303b8146e4d6471e46fb3bbda8b4f16834b3c", "date": "2026-09-17", "question": "feat(bootstrap): auto-detect source root instead of hardcoded src/", "gold": ["src/core/bootstrap_entities.py", "tests/test_bootstrap_entities.py"], "split": "dev"} +{"id": "f96470125845", "sha": "f9647012584500ae068b31ebabd9e84788000b88", "date": "2026-09-18", "question": "feat(bootstrap): add bootstrap_pipeline command (step 3) for MCP+CLI\n\nMove dynamic_trace_plugin.py to src/core as bootstrap_trace_plugin.py and\norchestrate detect_entities -> pytest trace -> TESTS edges in PropertyGraph.\n\n- bootstrap_pipeline.py: run_bootstrap_pipeline() + BootstrapPipelineStats\n- bootstrap_tool.py: MCP tool bootstrap_pipeline (asyncio.to_thread)\n- cli.py invites allowlist; server_tools.py registers the tool\n- static Functional index (index_src_functions) so a fresh graph matches\n- PYTHONPATH is only appended when the plugin is not importable from the\n tracked project's cwd (avoids src-package namespace shadowing)\n- read .py as utf-8-sig (BOM from Windows editors)\n\nTests: tests/test_bootstrap_pipeline.py (5 integration) + 3 unit for\nindex_src_functions. Full suite: 1753 passed.", "gold": ["src/cli.py", "src/core/bootstrap_entities.py", "src/core/bootstrap_pipeline.py", "src/core/bootstrap_tests.py", "src/core/bootstrap_trace_plugin.py", "src/mcp/server_tools.py", "src/mcp/tools/bootstrap_tool.py", "tests/test_bootstrap_pipeline.py", "tests/test_bootstrap_tests.py"], "split": "dev"} +{"id": "131ac555a0ad", "sha": "131ac555a0adb7a54b317594c08b571d22a2d261", "date": "2026-09-19", "question": "feat(indexing): incremental hot-reload for index freshness\n\nRestore and rewire FreshnessChecker as a stat-first incremental\nsweep that runs before each search: mtime+size match skips the\ncontent hash, changed files are re-indexed on the fly, and NEW\nfiles on disk are picked up without notify_change (closes KI-109).\nAdd schema columns file_mtime_ns/file_size with add_columns\nmigration, close FTS5/PropertyGraph holes in _index_single_file,\nand wire a _maybe_hot_reload hook into search_code with debounce\n(FRESHNESS_INTERVAL_SEC) + Lock + is_reindexing gate. Full reindex\nremains the fallback (empty index only). 7 fresh tests incl.\nconcurrency stress (N=16, correctness of hash not just 0 errors).\nE7 benchmark: stat sweep 17.8ms vs sha256 214.8ms over 1100 files.", "gold": ["src/config/settings.py", "src/core/indexing/db_manager.py", "src/core/indexing/db_writer.py", "src/core/indexing/freshness.py", "src/core/indexing/index_pipeline.py", "src/core/indexing/indexer.py", "src/core/indexing/indexer_table.py", "src/mcp/tools/search_tools.py", "tests/test_freshness_checker.py"], "split": "dev"} +{"id": "a4b4fd605685", "sha": "a4b4fd605685d438350d71d13da2a2276278b6a4", "date": "2026-09-19", "question": "fix(indexing): lancedb migration guards + E10 negative result (#36)\n\n* fix(indexing): guard lancedb migration + recreate against silent schema loss\n\n* refactor(settings): restore MAX_RERANKER_INPUT default 30 behind env toggle\n\n* chore(experiments): keep E10 search-quality harness and e2e quality tool\n\n* docs: record lancedb migration incident and E10 negative result\n\n* fix(ci): bump tool-count contract to 65 (bootstrap_pipeline 65th tool)\n\n---------\n\nCo-authored-by: MSCodeBase Agent <>", "gold": ["scripts/e2e_quality_search.py", "src/config/settings.py", "src/core/auto_doc_updater.py", "src/core/indexing/db_manager.py", "src/core/indexing/db_writer.py", "tests/test_auto_doc_updater.py", "tests/test_lancedb_recreate.py"], "split": "dev"} +{"id": "5d4b9352e1c8", "sha": "5d4b9352e1c892a1a9f089954e6afc96197b5893", "date": "2026-09-20", "question": "fix: resume-incremental write + role-specific ubatch (INC-6C62) (#37)\n\n* fix(indexing): resume-incremental write + table integrity (INC-6C62)\n\nIndexProjectRunner.run() previously accumulated all embeddings in\nmemory and wrote them in a single bulk_write at the end. A crash\non a large project (330K chunks, ~9h) lost all work: empty table\n-> empty known_hashes -> full re-embed on restart.\n\nFix:\n- WRITE_FLUSH_FILES=32: completed files flushed incrementally via\n _flush_pending(), RAM freed (embeddings zeroed + gc.collect)\n- _verify_and_repair_table_integrity() run once at start of run():\n detects dead fragment refs from drop+create inheritance, recreates\n table physically before indexing starts\n- Resume: on restart, known_hashes (table) skip already-written files;\n only the unfinished tail is re-processed\n- Double-write guard: test_no_double_write_of_completed_files\n- Crash preservation: test_crash_preserves_written_files_in_db\n- RAM invariant: test_ram_is_freed_after_flush\n\nRed Team 5/5: concurrency/idempotent, BATCH_SIZE boundary, TOCTOU\ncrash during bulk_write, runner restart, integrity-recreate vs skip.\n\n* perf(rerank): role-specific ubatch + query truncate (A/B 2026-09-20)\n\n- resolve_ubatch(role): embed->512, rerank->1024 (ceil to 128)\n env LLAMA_UBATCH_SIZE is hard override for both roles\n- _ubatch_arg(model_key): selects ubatch by role in llama_runner\n- _truncate_rerank_pair(query, passages): heuristic trim so\n query+max_passage <= ubatch tokens (1 token ~ 2 chars)\n- remote_embedder uses LLAMA_EMBED_MAX_TOKENS instead of hardcoded 480\n\nA/B on 2227 real chunks, port 8082:\n ubatch 512 -> 596 MB / 18.9 ch/s\n ubatch 2048 -> 1686 MB / 16.1 ch/s\n\nTests: test_ubatch_roles.py (6 cases), ruff clean.\n\n* docs: research notes KI-R1..R11 + diary + experiments log\n\n- ISSUE.md: 11 research tasks (KI-R1..R11) for search quality / E13\n- AGENT_DIARY.md: [2026-09-20] entry with priority order\n- EXPERIMENTS_LOG.md: E13 context + plan\n- KNOWN_ISSUES.md: auto-sync (entry added)\n- README.md, .env.example: minor updates\n\nNot committed: experiments/ search_quality artifacts (scratch).\n\n---------\n\nCo-authored-by: MSCodeBase Agent <>", "gold": ["src/core/indexing/index_project_runner.py", "src/providers/embedder/remote_embedder.py", "src/providers/reranker/llama_install.py", "src/providers/reranker/llama_runner.py", "src/providers/reranker/multi_provider.py", "tests/test_index_resume_incremental.py", "tests/test_ubatch_roles.py"], "split": "dev"} +{"id": "aeae56094a52", "sha": "aeae56094a521117106b66729ca12cefe91cfdbf", "date": "2026-09-22", "question": "docs: KNOWN_ISSUES dedup (379->240) + Bootstrap external review + E11/E14 harness (#38)\n\n* docs: KNOWN_ISSUES dedup (16 dups, 379->240 lines) + diary resume entry\n\n* docs(issues): add Bootstrap Pipeline external review (2026-09-20) + fix KI-R11 indent\n\n* chore(experiments): commit E11/E14 harness, embed bench and text chunk eval\n\n---------\n\nCo-authored-by: MSCodeBase Agent <>", "gold": ["scripts/eval_text_chunks.py"], "split": "dev"} +{"id": "b865c9767713", "sha": "b865c976771331f5c79a969f9d0b2fcee7a0da46", "date": "2026-09-22", "question": "fix(search): normalize file_path in get_tests_for_symbol\n\nBug: get_tests_for_symbol returned 0 tests because file_path format\nmismatch (relative vs absolute). Graph stores absolute POSIX paths\n(D:/Project/MSCodeBase/...) but callers passed relative paths\n(src/core/...).\n\nFix: normalize file_path to absolute POSIX before find_nodes lookup.", "gold": ["src/core/search/graph_adapter.py"], "split": "dev"} +{"id": "ce3db26b9e63", "sha": "ce3db26b9e63328ffb99f2836d72a70ebb8d8141", "date": "2026-09-22", "question": "feat(search): add opt-in TESTS-signal to graph-stage (E17) (#39)\n\nAppend covering tests to symbol-search results behind MSCODEBASE_TESTS_SIGNAL (off by default, production unchanged).\n\n- SearchConfig.tests_signal (env flag, default false)\n\n- SymbolIndexAdapter.get_tests_for_symbol: incoming TESTS edges of a function\n\n- Searcher._append_tests_signal: up to 3 tests per function as separate rank (graph_score 0.4 vs 1.0), sentinel chunk_index -(20_000_000+line)\n\n- 7 unit tests in test_graph_stage_e4.py\n\n- Reproducible harnesses: e17_ab_tests_signal.py, e17_wide_panel.py (35 queries + language coverage), e17_redteam.py (5 attacks)\n\nFindings: hit@1 unchanged (94.3% both arms) - signal adds LLM context only; TESTS coverage Python 34.0%, Other/TypeScript 0%; red team 5/5 repelled.\n\nArticle draft: docs/blog/bootstrap-pipeline.md\n\nCo-authored-by: MSCodeBase Agent <>", "gold": ["src/config/settings.py", "src/core/search/engine.py", "src/core/search/graph_adapter.py", "tests/test_graph_stage_e4.py"], "split": "dev"} +{"id": "1746bbf89878", "sha": "1746bbf89878f72f15a951231428e0e8cad10c98", "date": "2026-09-26", "question": "feat(experiments): pin reasoning variant; harden blind-runner (F4 rerun)\n\nReplace ad-hoc PowerShell+Tee with scripts/f4_blind_run.py (subprocess argv-list, UTF-8, ANSI-strip); --variant is required so an unpinned run fails loudly. Rerun with --variant high: valid 10/11 (longcat 4/5, qwen 3/3, deepseek 3/3), controls 6/6 in all valid, #16 -> NONE everywhere. The earlier 'deepseek 0/3' was an unpinned-budget confound, not a model property. Pitfall #19 added to the registry.", "gold": ["scripts/f4_blind_run.py"], "split": "dev"} +{"id": "17db525bdc55", "sha": "17db525bdc55dc05e311d8bf4e7dee2936bd8ad3", "date": "2026-09-26", "question": "fix(experiments): gate counts only first contiguous numbered block\n\nThe G6 overlap checker counted a trailing numbered 'next steps' section as symptoms (f4b reported 11 items instead of 8). It now stops at the first non-item line after the block starts. Verified: f4b=8, recovered handout=16, both PASS.", "gold": ["scripts/frozen_overlap_check.py"], "split": "dev"} +{"id": "30ff79d042b0", "sha": "30ff79d042b00494297da2239818823b069d0681", "date": "2026-09-26", "question": "test(experiments): guard frozen inputs tracked + recover E7 handout\n\nRoot cause of the 2026-09-26 loss: the E7/E11 frozen symptom list lived in %TEMP% and was never committed, so verbatim regression became impossible. Recovery: the handout content was found in opencode.db (write rid=75331, 2026-09-22; cross-checked against read rid=75864) and committed to experiments/4A_unit_of_return/frozen/ (sha256 a6f719df100ec0e3256a91e1d78469a1dc0ba3a9595aa266046199fe9f68de0a). New rule: frozen inputs live only in the repo; tests/test_frozen_inputs_tracked.py fails if any experiments/**/frozen file is untracked.", "gold": ["tests/test_frozen_inputs_tracked.py"], "split": "dev"} +{"id": "343288d20f59", "sha": "343288d20f59982fb0ce2e4637179980c22541ee", "date": "2026-09-26", "question": "feat(experiments): operationalize G6 overlap gate and judge-confound controls\n\nClose two gaps from review. (1) Judge-confound architecture: judge sees only the arm output bundle, identical reference across arms, opaque arm tokens, no self-grading, plus a leak control (same artifact under two labels must yield the same verdict). (2) G6 becomes executable: scripts/frozen_overlap_check.py fails on exact duplicates or >=3 shared content tokens vs any used list (stopwords filtered); verified with a negative control. Recorded in 4A README and in the global AGENTS rules (mini/compressed/full).", "gold": ["scripts/frozen_overlap_check.py"], "split": "dev"} +{"id": "457fd5b1d1e4", "sha": "457fd5b1d1e4eaa8d3e405d0c7c3872d9129b98f", "date": "2026-09-26", "question": "fix(docs): keep README tool count env-independent (#44)\n\n_count_tools added +1 for ExecuteScriptTool when\nMSCODEBASE_EXECUTE_SCRIPT_ENABLED=true, so a reindex on a machine with\nthe flag on rewrote the README heading to \"MCP Tools (66 total)\" while\nthe note and AGENTS.md still say 65 — and re-dirtied README on every\nreindex. _update_readme now uses the deterministic base count via\ninclude_env_optional=False; _count_tools keeps the runtime count for\ncallers that need it.\n\nCo-authored-by: MSCodeBase Agent <>", "gold": ["src/core/auto_doc_updater.py", "tests/test_auto_doc_updater.py"], "split": "dev"} +{"id": "5a11cbeee35b", "sha": "5a11cbeee35b58a50e2762ef59c3822dd5ca24f5", "date": "2026-09-26", "question": "feat(experiments): freeze F4b handouts (symptom+arrival) and manifest\n\nTwo conditions over the same 14 items (8 fresh held-out symptoms + 6 controls). Handout sha256 and catalogue sha256 recorded in manifest.json. Overlap gate scoped to previously-used lists (sibling condition excluded), both PASS.", "gold": ["scripts/frozen_overlap_check.py"], "split": "dev"} +{"id": "6214889f08dc", "sha": "6214889f08dcbec91f427497990b47f4f20f6910", "date": "2026-09-26", "question": "perf(dev): run the gate-zero suite in parallel in verify_diary (#47)\n\nverify_diary runs the full suite on every commit (pre-commit) and it was\nserial (~200-360s locally). Gate-zero now adds `-n auto` when pytest-xdist\nis importable, falling back to serial otherwise, so the gate keeps working\nin a minimal environment. The selection is unchanged (pyproject addopts\nkeep -m 'not slow and not benchmark'); measured 197s -> 71s locally.\n\nCo-authored-by: MSCodeBase Agent <>", "gold": ["scripts/verify_diary.py"], "split": "dev"} +{"id": "72c171457f6b", "sha": "72c171457f6b50605628bcd674d42576c6b4b7a2", "date": "2026-09-26", "question": "feat(experiments): F5 4-arm retrieval harness + in-sample pilot\n\nscripts/f5_retrieve_arms.py serves arms A (top-k chunks), B (whole\ntop-1 doc), C (oracle file), D (closed book) over the live index\nread-only, and emits objective metrics + context bundles for the\njudged reader arm. Population split is data set before results.\n\nPilot on 12 in-sample benchmark2 tasks (calibration, NOT an F5\nresult): A hit@1 8.3%, hit@3 41.7%; B top1-is-gold 8.3%; B context\n27x larger than A. Phase Zero, gates and the fresh-set requirement\nare recorded in f5/README.md.", "gold": ["scripts/f5_retrieve_arms.py"], "split": "dev"} +{"id": "8975b6a9d4ce", "sha": "8975b6a9d4ce42177afe28d55db2c4df5d99f9eb", "date": "2026-09-26", "question": "fix(experiments): raise blind-run timeout to 900s; add F4b orchestrator\n\nOne opencode run with --variant high on the 14-item handout takes ~5 min, so the 300s cap killed calls (TimeoutExpired). Timeout is now configurable (default 900) and a timed-out run is recorded instead of raising. scripts/f4b_run_all.py runs both conditions x 3 models sequentially, launched detached with a log.", "gold": ["scripts/f4_blind_run.py", "scripts/f4b_run_all.py"], "split": "dev"} +{"id": "8def35df211c", "sha": "8def35df211c9abe03332b71f270c8406317d2b7", "date": "2026-09-26", "question": "fix(experiments): opaque filenames in F5 judge harness (defense-in-depth)\n\nopencode reveals the --file name to the model. Arm/query no longer\nappear in ctx/cand filenames, so neither reader nor judge can see them.\nVerified benign (arm labels are opaque to the models) but hardened anyway.", "gold": ["scripts/f5_judged_run.py"], "split": "dev"} +{"id": "9266613b9f2a", "sha": "9266613b9f2a93a09a6b3b7b639cfafb239eb1f9", "date": "2026-09-26", "question": "fix(experiments): make the G6 novelty gate catch paraphrase twins\n\nF4b exposed that frozen_overlap_check.py v1 was blind to probes\nthat paraphrase a catalogue arrival phrase (item #3: 'timing out /\nretry' vs 'times out / try'): it only saw numbered lines and did no\nmorphology. v2 compares numbered probes of previous lists AND the\ncatalogue arrival register, with light stemming; item #3 is now\nflagged with zero false positives.\n\nControls: --selftest (negative twin flagged + positive clean) and\ntests/test_frozen_overlap_check.py pin the behaviour. Docs, diary and\nKNOWN_ISSUES updated (G6 gap -> Fixed).", "gold": ["scripts/frozen_overlap_check.py", "tests/test_frozen_overlap_check.py"], "split": "dev"} +{"id": "9b0417c2e385", "sha": "9b0417c2e385296f158c47ebe2c9456df9bf166e", "date": "2026-09-26", "question": "fix(experiments): reject silent opencode model fallback in F5 judge run\n\nA reader/judge call that silently falls back to another model (observed\n'google/gemma-4-e2b' on a real call) would corrupt the experiment\n(channel #1: model identity). _run now marks such outputs with\n[MODEL-MISMATCH:...] / [ERROR] / [TIMEOUT] and the run records them as\n'invalid' instead of grading them. Dry-run and ruff clean.", "gold": ["scripts/f5_judged_run.py"], "split": "dev"} +{"id": "9be20efc6964", "sha": "9be20efc6964aef108658a40c18e05fb648fdf91", "date": "2026-09-26", "question": "feat(experiments): F5 judged-arm harness + measured compute blocker\n\nscripts/f5_judged_run.py implements the blind judged reader arm\n(reader context per arm, different judge model, opaque tokens,\nmajority, leak-check). Dry-run and direct calls validated.\n\nMeasured blocker: a realistic reader call via the opencode CLI is\n~200s (trivial 8.4s), so the full design (n=16 x 4 arms x 10-20\ntrials) is hours-to-days and cannot run as designed. Options (raw API\n= HALT, reduced pilot, or defer to F0c) are recorded in\nf5/JUDGED_PLAN.md. AGENT_DIARY notes the incident: a broad\n'opencode' process filter killed the host session.", "gold": ["scripts/f5_judged_run.py"], "split": "dev"} +{"id": "bea7e62fa8ef", "sha": "bea7e62fa8efa1edb685c8ac7853bac5418cc50e", "date": "2026-09-26", "question": "perf(ci): parallelize the clean-state suite too (#46)\n\nclean-state was the last long CI job (~13m) after the test job dropped to\n~2m with pytest-xdist. Its test selection is identical to the test job:\nno file list, and pyproject addopts keep `-m 'not slow and not benchmark'`\n(98 deselected). The same parallel run already passed on ubuntu+windows,\nso parallel safety is proven; locally it is 197s -> 71s (12 cores).\n\nThe fresh venv, lockfile-drift gate (+ its negative control), the\nnegative-controls inventory and the revision gate are all unchanged.\n\nCo-authored-by: MSCodeBase Agent <>", "gold": ["scripts/verify_clean_state.sh"], "split": "dev"} +{"id": "efc1df2e0f1b", "sha": "efc1df2e0f1b0936618aa4b2023e1a567bdf0af3", "date": "2026-09-26", "question": "perf(ci): run tests in parallel via pytest-xdist (#45)\n\nThe test job ran serially (13m44s on ubuntu). pytest-xdist -n auto cuts\nthe local suite from 197s to 71s (x2.76 on 12 cores); CI runners have 4\ncores, so the job time drops proportionally. Both OSes keep the exact\nsame test command, so coverage of ubuntu+windows is unchanged.\n\nCoverage moves to .coveragerc.ci (COVERAGE_RCFILE): the local\n.coveragerc sets dynamic_context=test_function, which pytest-cov rejects\nunder xdist (DistCovError, pytest-cov#604) and which costs ~+13.6%\n(sys.settrace, AGENT_DIARY 2026-09-15). CI only needs the % gate, so it\nuses the context-free config; local and bootstrap runs keep contexts.\n\nAlso cache the temporal facts generator per module (36.6s -> 20.8s): the\nthree tests share one deterministic dataset but each re-ran generate(),\nwhich spawns dozens of git subprocesses.\n\nMeasured locally (12 cores): serial 197.1s -> xdist 71.4s (no cov) ->\nxdist + ci-cov 86.4s; 1865 passed, coverage 54.58% (> 38% gate).\n\nCo-authored-by: MSCodeBase Agent <>", "gold": ["tests/test_temporal_facts_generator.py"], "split": "dev"} +{"id": "fa616acf2a31", "sha": "fa616acf2a316061937a9d1454d21c346afaa17a", "date": "2026-09-26", "question": "fix(experiments): real failure markers + guard the judge call\n\nThe reader guard was startswith(\"[\"), which flagged a valid answer as\ninvalid whenever the agent led with a bracketed heading (observed live:\nthe agent's own \"[ИТОГ]\" summary). Observed once in 24 units.\n\nThe judge had no guard at all: a failed call fell through to the keyword\nfallback in _parse_verdict and was recorded as a verdict instead of\nbeing rejected. Both now check FAIL_MARKERS explicitly.\n\nAlso adds arm S: the retrieved chunk plus the file's imports and the\nsignatures of functions it calls. Callee-to-signature resolution was\nmeasured at 11.3% (274/2432) with a repo-wide index, 1.0% file-local,\nso S is imports + a small number of local callee signatures.\n\nCo-Authored-By: Claude Opus 4.8 (1M context) <>", "gold": ["scripts/f5_judged_run.py"], "split": "dev"} +{"id": "1d6d75c8b4b7", "sha": "1d6d75c8b4b74b8365997cd391fe3ef7f6f30f2a", "date": "2026-09-27", "question": "fix(windows): run gate-zero pytest under pythonw to stop console popups\n\npytest-xdist workers spawned from python.exe open a black CMD console per worker on Windows. Resolving the gate-zero pytest interpreter to the pythonw.exe sibling of sys.executable (win32 only, fallback to sys.executable) keeps pipes/timeouts/serial-fallback unchanged while stopping the popups.", "gold": ["scripts/verify_diary.py"], "split": "dev"} +{"id": "3b28fe701256", "sha": "3b28fe701256e5e013d1c744a014e1e30c7ac50e", "date": "2026-09-27", "question": "fix(repo): normalize 17 production path leaks after closure-walk", "gold": ["scripts/architecture_linter.py", "scripts/backfill_chunk_hash.py", "scripts/db_health.py", "scripts/diag_quality_hang.py", "scripts/full_reindex.py", "scripts/patch_parser.py", "scripts/sync_src.py", "src/cli.py", "src/core/bootstrap_tests.py", "src/core/indexing/symbol_index.py", "src/core/search/graph_adapter.py", "src/mcp/tools/graph_tools.py", "src/utils/ui_formatter.py"], "split": "dev"} +{"id": "46f7220e2973", "sha": "46f7220e2973bfa92d7c54a9236acc5b2272f60f", "date": "2026-09-27", "question": "feat(scripts): redact.py — scrub personal paths with planted-key test (5/5 pass)", "gold": ["scripts/redact.py", "tests/test_redact.py"], "split": "dev"} +{"id": "4e57021da777", "sha": "4e57021da777cad75de01fb7ce35dfb3132fb685", "date": "2026-09-27", "question": "docs(f5): D-ablation floor test, exp-54 plan, P-004, grader principle + README refresh\n\nF5 D-arm no-abstention ablation (floor test): 16 frozen queries x 3 trials = 48 reader (longcat-2.0, instr frozen/f5/reader_D_ablation.txt sha 91b16b8b) + 48 judge (qwen3.7-plus, unchanged). D 2/48 (4.2%), code 0/24, prose 2/24 (F5S-10, F5S-13 single middle trials), majority 0/16, 0 invalid/uncertain. Verdict: reader priors at noise level - Tom's mechanism confirmed, scale not (his 6/14 vs our 2/48); prose B 18.8% vs ~4% floor, margin survives. Files: judged_raw_D_ablation.json + judged_aggregate_D_ablation.json. Also: exp-54 CoT plan (PLANNED), P-004 letter-vs-spirit, judge-verdict-contract issue, deterministic-verdict-by-code principle (AGENTS.md), f5_judged_run.py --reader-instr-file, README Recent results block + badge 1889->1965 (pre-existing). token_reduction_v3_lancedb relocated to TEMP backup for this commit (frozen-inputs gate), restore after.", "gold": ["scripts/f5_judged_run.py"], "split": "dev"} +{"id": "6b50d7f28e48", "sha": "6b50d7f28e4833d67ff9c1773fe0c66dfd0f0afa", "date": "2026-09-27", "question": "feat(tests): planted-break gate — 6 controls (3 guards x pos/neg)", "gold": ["tests/test_planted_break_gate.py"], "split": "dev"} +{"id": "a11f062b55d1", "sha": "a11f062b55d1207860414d01961158cb46033a21", "date": "2026-09-27", "question": "style: fix 4 ruff errors failing CI lint\n\nF821 cli.py unreachable allowlist line; BLE001 staleness.py narrowed except; F541 store.py plain string; F401 test_redact.py dropped import", "gold": ["src/cli.py", "src/core/intelligence/staleness.py", "src/core/intelligence/store.py", "tests/test_redact.py"], "split": "dev"} +{"id": "bc6cfea480da", "sha": "bc6cfea480da8524a3660f2a374b6a878b6d6c6f", "date": "2026-09-27", "question": "fix(f5-judge): explicit-final verdict contract plus CoT backfill\n\nParser took first verdict mention, inverting self-correcting judges;\nnow last JSON match else last word wins (53/61 flips vs 34/61).\nBackfill 1014 judge sessions from opencode.db to judged_cot_backfill.json.\nFuture runs persist judge_texts per trial. Guard: test_f5_judged_verdict.", "gold": ["scripts/f5_judged_run.py", "scripts/reconstruct_judge_cot.py", "tests/test_f5_judged_verdict.py"], "split": "dev"} +{"id": "c1e04035ba4d", "sha": "c1e04035ba4da6228073bafea0c4c85691957b99", "date": "2026-09-27", "question": "feat(memory): stale_after + discriminator for memory notes — 17/17 tests pass", "gold": ["src/cli.py", "src/core/intelligence/staleness.py", "src/core/intelligence/store.py", "tests/test_stale_after.py"], "split": "dev"} +{"id": "d72fa3230ba8", "sha": "d72fa3230ba83bff54703a965944f37ff87f1828", "date": "2026-09-27", "question": "fix(tests): script import no longer pollutes xdist worker environ\n\nf5_judged_run set PYTHONPATH at import time; importing it in\ntest_f5_judged_verdict broke plugin subprocess tests under -n auto\n(ModuleNotFoundError: No module named src). Side effects moved to\n_ensure_importable(), called only from __main__.", "gold": ["scripts/f5_judged_run.py"], "split": "dev"} +{"id": "0561e331f633", "sha": "0561e331f6330ad76ea30878d68aac7c7fa3765d", "date": "2026-09-28", "question": "fix(harness): single event loop for o1 holdout gate", "gold": ["scripts/o1_holdout_gate.py", "tests/test_holdout_harness_loop.py"], "split": "dev"} +{"id": "0bd8526d9cf6", "sha": "0bd8526d9cf63b843bfd4b9a336339018342ca60", "date": "2026-09-28", "question": "feat(search): O1 rare-identifier x100 boost at pre-rerank pool", "gold": ["scripts/o1_holdout_gate.py", "src/core/search/engine.py", "tests/test_o1_identifier_boost.py"], "split": "dev"} +{"id": "7f6d5e71c9d0", "sha": "7f6d5e71c9d0a74b5a62c8fadecc5e6a8107a78b", "date": "2026-09-28", "question": "fix(search): cold FTS build out of 2s budget plus ext-dir off-by-one\n\nCold to_pandas build measured 2.47s > 2.0s wait_for: first search in a\nfresh process silently dropped the FTS tier (flaky A/B 18/20 vs 8/20).\nBuild hoisted out of timeout (idempotent, locked); 2s kept for search.\n_get_ext_dir took 3 parents instead of 4: dev-mode branch dead, models\nresolved to empty data_root. Embedder GGUF copied to git-ignored models/.", "gold": ["src/core/search/engine.py", "src/providers/reranker/llama_install.py"], "split": "dev"} +{"id": "a189d13d8ae0", "sha": "a189d13d8ae0aa9c2f58fd8a317414071ce3520d", "date": "2026-09-28", "question": "feat(index): exclude experiment outputs from index, keep git tracking\n\n3127/15426 chunks (20.3 pct) were experiments slash results|work dumps\n(ctx files up to 329 chunks each) choking lexical search (P2) and cold\nFTS build (2.47s). New SystemArtifacts.is_experiment_output layer wired\ninto FileGuard; frozen inputs and experiment sources stay indexed.\nTakes effect on full reindex. Guard: test_experiment_output_guard.", "gold": ["src/core/indexing/file_guard.py", "src/core/system_artifacts.py", "tests/test_experiment_output_guard.py"], "split": "dev"} +{"id": "fb7e3daadcf5", "sha": "fb7e3daadcf58672c14b3b3c9bb0de92dc2024a2", "date": "2026-09-28", "question": "feat(search): anchor rare-identifier symbols into pre-rerank pool (P2 root cause)\n\nP2 gold (engine.py) sat at BM25#126/FTS#74 vs [:limit] cut; O1 boosted the\nwrong file (rarest token != gold symbol). Anchors resolve exact-symbol\nchunks via single-token FTS (def-first, df<=120, docs/data excluded,\ncaps 2+3+MAX) — P2 rank 1 live, holdout gate 15/15 PASS.", "gold": ["scripts/p2_holdout_gate.py", "src/core/search/engine.py", "tests/test_p2_pool_anchors.py"], "split": "dev"} +{"id": "fede348468a6", "sha": "fede348468a61ffc4e0afbf5a1b47ee2749c23f3", "date": "2026-09-28", "question": "fix(redteam): fail-closed hook, STARTUPINFO guards, holdout timeouts", "gold": ["scripts/o1_holdout_gate.py", "scripts/p2_holdout_gate.py", "src/core/silent_subprocess.py", "tests/test_holdout_harness_timeout.py", "tests/test_hook_root.py", "tests/test_silent_subprocess.py"], "split": "dev"} +{"id": "0ac4da6863d5", "sha": "0ac4da6863d5e4954ba9c50addf796612fbb2a68", "date": "2026-09-29", "question": "fix(reranker): normalize llama.cpp raw logits via sigmoid (scale contract)\n\nllama.cpp returns logits ~[-11,+11] vs threshold calibrated for [0,1]; filter ate 70-97%. Threshold 0.3 untouched. Does NOT fix P3 ranking — see KNOWN_ISSUES P3 entry.", "gold": ["src/providers/reranker/multi_provider.py"], "split": "dev"} +{"id": "132d80299389", "sha": "132d80299389e5335ee65ebdd608dbbab3561f66", "date": "2026-09-29", "question": "feat(hygiene): add disk_guard with TTL caps and disk-space CI check", "gold": ["scripts/disk_guard.py", "tests/test_disk_guard.py"], "split": "dev"} +{"id": "323dbd875c15", "sha": "323dbd875c15abc59820525da1e7d489c251629c", "date": "2026-09-29", "question": "fix(chunker): emit module docstring as chunk 0, live-RED (see reranker_p3/RESULT-chunker.md)", "gold": ["src/core/indexing/parser.py", "tests/test_chunker_module_doc.py"], "split": "dev"} +{"id": "49b60c1ac3ae", "sha": "49b60c1ac3aee82ddd64be20803077777b661cba", "date": "2026-09-29", "question": "fix(tests): remove unused pytest import in test_disk_guard", "gold": ["tests/test_disk_guard.py"], "split": "dev"} +{"id": "87ae60337dfd", "sha": "87ae60337dfd25f3a12c8bb276194305a2e2eb2e", "date": "2026-09-29", "question": "feat(reranker): salvage PR52 scale floor plus holdout calibration\n\nFrom conflicting PR52 (supserseded by merged P54 for P2): sigmoid\nnormalization of llama.cpp raw logits to [0,1] (else MIN_RERANK_SCORE\ncuts 70-97pct incl. targets), top-N recall floor reranker_topn_keep\n(default 0 = off), holdout threshold calibration with eval sources\nrejected by code. Tier-winner anchors deliberately skipped: P2 covered\nby merged P54 at the same pool site. Guards: 4 sigmoid + 13 floor tests.", "gold": ["src/config/settings.py", "src/providers/reranker/multi_provider.py", "src/providers/reranker/threshold_calibration.py", "tests/test_reranker.py", "tests/test_reranker_pool_and_threshold.py"], "split": "dev"} +{"id": "8c0ac276d1de", "sha": "8c0ac276d1de2e5f647a69b0c00bd39f4f1dd9e2", "date": "2026-09-29", "question": "feat(search): P3 module-head pool anchor, live-FALSIFIED (see reranker_p3/RESULT.md)", "gold": ["scripts/p3_holdout_gate.py", "src/core/search/engine.py", "tests/test_p3_module_head_anchor.py"], "split": "dev"} +{"id": "8c7f4479eb5c", "sha": "8c7f4479eb5c17701b8a3a74f95ac3f6bf80ca83", "date": "2026-09-29", "question": "fix(index): exclude measurement artifacts from index (Layer 5)\n\nProbe/result JSON dumps and experiments results/work files contain\nverbatim query terms and dominate BM25, invalidating gates. Canonical\nexclusion via SystemArtifacts.is_measurement_artifact, wired into\nFileGuard.should_skip_file used by all indexer walks.", "gold": ["src/core/indexing/file_guard.py", "src/core/system_artifacts.py", "tests/test_measurement_hygiene.py"], "split": "dev"} +{"id": "aaa64d2a5fec", "sha": "aaa64d2a5fec517068819ca10f1b6242f57d1955", "date": "2026-09-29", "question": "feat(hygiene): disk_guard also sweeps stale pytest worker dirs", "gold": ["scripts/disk_guard.py", "tests/test_disk_guard.py"], "split": "dev"} +{"id": "aad01ec7879d", "sha": "aad01ec7879dcec4ef7c82e0e38ac5cdf5b10d8c", "date": "2026-09-29", "question": "feat(process): distinguishable MCP and llama process names\n\nMCP sets Windows console title mscodebase-mcp:- at\nstartup (win32-guarded, logged). Llama embed/rerank spawn via NTFS\nhardlinks llama-embed.exe/llama-rerank.exe (os.link, same inode,\nfallback to plain spawn). PID-reuse and port guards accept new names.", "gold": ["src/core/process_titles.py", "src/main.py", "src/providers/reranker/llama_runner.py", "tests/test_process_titles.py"], "split": "dev"} +{"id": "d70b6be0600a", "sha": "d70b6be0600a7bc6fd309f697fe1136ee75fbb7d", "date": "2026-09-29", "question": "fix(hygiene): tolerate missing disk anchors in disk_guard", "gold": ["scripts/disk_guard.py", "tests/test_disk_guard.py"], "split": "dev"} +{"id": "dbab8c9ebee7", "sha": "dbab8c9ebee78e051b9cce11e64c85819f5357b4", "date": "2026-09-29", "question": "feat(process): mcp exe hardlink fallback for windowless pythonw", "gold": ["src/core/process_titles.py", "tests/test_process_titles.py"], "split": "dev"} +{"id": "fc88ee73c98c", "sha": "fc88ee73c98ccebebad4a427b169c18c55cb0230", "date": "2026-09-29", "question": "tool(index): one-time purge of experiment outputs from shared DB\n\n772 files / 2152 chunks under experiments slash results|work removed via\ndelete_file plus graph cleanup plus compaction (prune safety-guard at\n50pct would refuse: garbage was 52.4pct of files). Verified 0 remaining.\nDry-run by default.", "gold": ["scripts/purge_experiment_outputs.py"], "split": "dev"} +{"id": "ff18a8a16fb2", "sha": "ff18a8a16fb22d0efa3e727dcb2c6d7517aa1dc7", "date": "2026-09-29", "question": "fix(redteam): wire silent_subprocess into entry point, close red-team tails\n\n- src/main.py imports silent_subprocess + apply() at startup (module was INERT since PR #56)\n- silent_subprocess.apply(): TypeError guard for non-class Popen (test shims)\n- tests/test_silent_subprocess_wired.py: fresh-process liveness (Popen=_SilentPopen on win32)\n- KNOWN_ISSUES.md: 3x 2026-09-28 red-team entries (Open) -> Fixed with SHAs (PR #56 merged cca01095, CI green)", "gold": ["src/core/silent_subprocess.py", "src/main.py", "tests/test_silent_subprocess_wired.py"], "split": "dev"} +{"id": "18955c9bf9c5", "sha": "18955c9bf9c5bc59b06b7b9e0cb8e048af25afc1", "date": "2026-10-03", "question": "docs(ledger): propose KNOWN_ISSUES entries for the owning session\n\nKNOWN_ISSUES.md is uncommitted in the parallel session's tree, and\ncheck_parallel_sessions blocked the commit - correctly. So this branch leaves\nthat file alone and records both intended entries here instead: the correction\nto the judge-verdict entry, and the three measured reasons a commit could not\nbe made.\n\nAlso notes that the ruff gate was the mirror image of the UNSEEN proposal:\nit had seen its condition and could not say so.", "gold": ["tools/knowledge/PENDING_LEDGER.md"], "split": "dev"} +{"id": "29ce3b42d633", "sha": "29ce3b42d63365268981c8266ea9bd52b420328d", "date": "2026-10-03", "question": "chore(tests): drop the duplicated ruff-gate contract test\n\ntests/test_ruff_gate.py already covered the gate. The duplicate was\ncreated without checking, broke four CI jobs, and is replaced there by the\ntwo cases it actually added.", "gold": ["tests/test_ruff_gate_contract.py"], "split": "dev"} +{"id": "3c68f5182fd0", "sha": "3c68f5182fd0c985e832cb22a8d1732c68ec534d", "date": "2026-10-03", "question": "docs: bind published counts to the command that re-derives them (#66)\n\nBenchmark decay: the badge read 1965 while the suite collects 2007, the\narchitecture table read ~1889, and WISDOM's census read intel_*=14 / tests=1180\nagainst a live 20 / 2007. Every one of those is a public number a reader could cite.\n\nA manual edit fixes the symptom once. The guard is the fix: each claim now\ncarries the command that re-derives it, and disagreement exits rc=3.\n\nThe comparison is symmetric on purpose. The first cut used `live <= stated`,\nwhich only catches understatement — its own selftest caught an inflated badge\nsliding through. Two synthetic cases must be rejected, and both directions are\nnow covered.\n\nThe dated 2026-08-12 snapshot (intel_*=14, tests=1180) was correct on the day it\nwas written; it is marked as a snapshot and left intact. The live census is a\nseparate section, so published history is superseded by a new record rather than\nsilently rewritten.\n\nAlso records the measured false-positive share of the protocol guards (62.5%,\n5 of 8) in the suite output that previously still said UNMEASURED — quoting the\nraw count overstated defects by 2.7x.\n\nVerified on this tree:\n pytest tests/ 2001 passed, 6 skipped, exit 0\n tools/verification/run_all.py 14/14 provability steps OK, exit 0\n verify_public_claims --selftest 4/4, incl. 2 that must be rejected\n verify_public_claims 4/4 published numbers reproduce\n\nSources for naming the class: GTM-Bench \"Keeping a Benchmark Honest\" (2026-09-06)\n— benchmark decay, \"a number without a version is uninterpretable\"; READU\n(arXiv 2607.15780) — an alert judge to remove false positives is part of the\nconstruction, not an option; driftmd — \"badge versions\" as its own check.\n\nCo-authored-by: MSCodeBase Agent <>", "gold": ["tools/knowledge/PATTERNS.md", "tools/knowledge/RESEARCH-benchmark-decay.md", "tools/verification/denominator_manifest.json", "tools/verification/run_all.py", "tools/verification/verify_public_claims.py"], "split": "dev"} +{"id": "3faae1f54867", "sha": "3faae1f54867fa61c20be91c60d02ae42bf7761a", "date": "2026-10-03", "question": "fix(negative-controls): resolve GitBash outside PATH\n\nThe runner resolved bash via shutil.which only. From an agent shell Git's bin dir is often absent from PATH, so the WSL shim System32\\\\bash.exe won the lookup and the resolver returned None, reporting drift_gate as BROKEN even though GitBash was installed. That made the pre-commit chain unusable outside the author's own shell.\n\nNow also probes the standard Git install roots and the root implied by the git on PATH. drift_gate goes BROKEN -> PROVEN, and the whole hook passes: ALL PROVEN (3).", "gold": ["scripts/negative_controls_runner.py"], "split": "dev"} +{"id": "8653200353f5", "sha": "8653200353f50bc66710a521ab2268ed37cd43c9", "date": "2026-10-03", "question": "feat(guards): add third-party data and cross-session pre-commit gates\n\ncheck_third_party_data: R1 personal email, R2 exported-profile signature, R3 bulk\ndump (advisory). Vendored dependency metadata is allowlisted as legitimate\nattribution. --selftest proves positive 2/2 and negative 2/2 so a matcher that\ncannot fail cannot pass; --all over 1897 tracked files reports blocking=0.\n\ncheck_parallel_sessions: lists foreign worktrees with branches and uncommitted\nfiles, blocks when a staged file is also modified in another tree, advisory\notherwise. Wired into .githooks/pre-commit; AGENTS.md documents both.\n\nThree defects were found while verifying them. Each is reproduced by a test.\n\n1. GIT_DIR leak. git exports GIT_DIR into the hook environment and _git()\ninherited it, so 'git -C status' kept reading the CURRENT\nrepository. The gate inspected the wrong tree: with GIT_DIR set it reported\nthree pre-commit-stage files as modified in D:/Project/wt-pr-v3 and exited 1\nwhile that worktree was clean. Bidirectional - it also let real cross-tree\nconflicts through. Fix drops GIT_DIR/GIT_WORK_TREE/GIT_INDEX_FILE/\nGIT_COMMON_DIR from the child environment.\n\n2. The third-party gate could not be tested at all. Its R1 fixture held a real\nperson's address (), so committing the detector would\nhave published a third party's email - exactly what the rule forbids. Test\nfixtures used gmail.com addresses and tripped R1 on synthetic data. Both now\nuse the RFC 2606 reserved domain personal.invalid, which can never be\nregistered, and personal_domains is injectable so R1 is still proven to fire.\n\n3. R2 counted 11 profile markers in the detector itself, because PROFILE_MARKERS\nlives in that file. R2 is skipped for exactly two paths; R1 still applies to\nthem, so a dump cannot be hidden there.", "gold": ["scripts/check_parallel_sessions.py", "scripts/check_third_party_data.py", "tests/test_check_parallel_sessions.py", "tests/test_check_third_party_data.py"], "split": "dev"} +{"id": "87fbe9d953cb", "sha": "87fbe9d953cb6519459b128a4af3ed0707590784", "date": "2026-10-03", "question": "fix(judge): one verdict parser, last-match in both copies\n\nKNOWN_ISSUES recorded this class as fixed on 2026-09-27, but only for\nf5_judged_run.py. reconstruct_judge_cot.py kept a first-match substring\nparser, so a self-correcting judge was recorded INVERTED and silently, and the\nscript had no test at all. Two of two locations carried the logic; one was\nbuggy.\n\nBoth now import scripts/judge_verdict.py (last-match contract, validated on\n1014 judge sessions), so a third copy cannot appear.\n\nTests: 11 for the previously untested script, with a control that re-runs the\nold parser and requires it to disagree. Verified: 7 fail before the fix.", "gold": ["scripts/f5_judged_run.py", "scripts/judge_verdict.py", "scripts/reconstruct_judge_cot.py", "tests/test_reconstruct_judge_verdict.py"], "split": "dev"} +{"id": "8855220bfe29", "sha": "8855220bfe292c2da3f95c6b5dd30365c8d3538b", "date": "2026-10-03", "question": "fix(gate-zero): run pytest under python.exe, not pythonw\n\ngate_zero_full_suite() picked pythonw.exe on Windows to avoid a console flash.\npythonw has no console, so every test that spawns a subprocess (git, lock\nhelpers, discriminators) dies inside subprocess with OSError [WinError 50].\n\nMeasured on the same tree and commit: 'python -m pytest tests/' gives\n2006 passed and 0 failed, 'pythonw.exe -m pytest tests/' gives 72 failed and\n6 errors, every failure WinError 50 from subprocess.\n\nCREATE_NO_WINDOW already suppresses the console window, so pythonw bought\nnothing and cost the entire gate.", "gold": ["scripts/verify_diary.py"], "split": "dev"} +{"id": "89b03c7ea234", "sha": "89b03c7ea234e25ad75af0fdb20b1c28f559c940", "date": "2026-10-03", "question": "docs(claims): mark A10 measurement SUPERSEDED (closes T-17)\n\norphan30s->120ms was published in three places with a confirmed label\nwhile claims_audit/RESULTS.md A10 records it as NOT REPRODUCED: the ORPHAN\npath was removed by design (R3TF). Relabel with measured-on sha 867dbe56\nand the R3TF reference; the raw run output is left untouched.\n\nAlso registers P-18 (absence claimed from a single referent, seen twice in\none session) and opens tools/knowledge/PENDING_LEDGER.md for the session\nthat owns KNOWN_ISSUES.md.\n\nCommitted with --no-verify; reason in AGENT_DIARY.md, 2026-10-03.", "gold": ["tools/knowledge/PATTERNS.md", "tools/knowledge/PENDING_LEDGER.md"], "split": "dev"} +{"id": "940c3491eb63", "sha": "940c3491eb63420ff91b2be92a72856bc1aab86f", "date": "2026-10-03", "question": "fix(ruff-gate): resolve a ruff that can actually run\n\nThe gate invoked \\python -m ruff\\, but the ruff installed in this venv\n(0.15.22) ships no __main__: the call printed nothing and exited 1. The gate\nwas therefore permanently red and mute, which reads as broken lint rather than\nas a broken runner - and a permanently red gate is a gate people disable.\n\nruff_cmd() now resolves a live tool and proves it with --version before use.\nA non-zero exit with EMPTY output is reported as a broken runner, not as a lint\nverdict. stderr is no longer discarded.\n\nTests: 6, incl. the gate failing on a real diagnostic and the module form\nbeing rejected when it is dead.", "gold": ["scripts/ruff_gate.py", "tests/test_gates_kwarg_contract.py", "tests/test_ruff_gate_contract.py"], "split": "dev"} +{"id": "98716d36cce2", "sha": "98716d36cce21bf4d4c2b840275d752a7bf4f035", "date": "2026-10-03", "question": "fix(tests): repair what CI caught in the gate work\n\nCI ran the suite and found four failures, all from this branch:\n- tests/test_ruff_gate.py broke because ruff_cmd() probed with\n subprocess.run, and the existing tests stub subprocess.Popen globally;\n availability is now decided by importability first, with no subprocess.\n- tests/test_ruff_gate_contract.py duplicated tests/test_ruff_gate.py, which\n already existed. Deleted; the two genuinely new cases were added there.\n- the hook's negative control read the old hook back with 'git show HEAD', so\n it stopped being a control the moment the fix was committed; and it wrote a\n temp file under ROOT, tripping test_no_tracked_file_mutation. The old\n behaviour is now reproduced inline, with no filesystem and no git.\n- verify_diary flagged the diary reference to the deleted file.", "gold": ["scripts/ruff_gate.py", "tests/test_precommit_hook_contract.py", "tests/test_ruff_gate.py", "tools/knowledge/PENDING_LEDGER.md"], "split": "dev"} +{"id": "d1e77fad76cf", "sha": "d1e77fad76cfc4175f6be9f4fd4de61cc069511b", "date": "2026-10-03", "question": "fix(tests): remove the cross-test race that made CI classify a healthy guard as UNPROVEN\n\nCI reported `NEGATIVE CONTROLS: FAILED (broken=0, unproven=1)` with\n`dead_guard_classifier` marked UNPROVEN — green locally, red on every runner.\n\nRoot cause: tests/test_negative_controls_runner.py proved the digest pin by\nediting the REAL fixture\nscripts/negative_controls/fixtures/dead_guard.py and restoring it in `finally`.\nUnder `pytest -n auto` another worker read that fixture's digest inside the\nwindow between the write and the restore, computed a different digest, and\nclassified a healthy guard as UNPROVEN. The digest guard was never wrong; the\nrace was between two tests, and it presented as a guard defect.\n\nEvery digest matched locally and in a clean checkout, which is why this survived\nlocal verification — the trigger is worker count, not content.\n\nThe test now copies the fixture into a scratch directory it creates and removes,\nand never touches the tracked file. It asserts a PROVEN control before mutating,\nso the property under test is still \"a byte change flips PROVEN to UNPROVEN\".\n\nAdds tests/test_no_tracked_file_mutation.py so the class cannot come back. It is\na test rather than a script because a script nobody runs is not a gate, and its\nfirst version collected zero tests under pytest while reading as coverage.\n\nThat guard's selftest earned its place immediately: it caught a real second\ninstance in tests/test_planted_break_gate.py, which wrote\nexperiments/planted_break/results.json from two xdist workers with no atomicity.\nThat write is now temp-file + os.replace, the artifact is gitignored, and the\none exemption is recorded with its reason.\n\nVerified: 1991 passed, 6 skipped, two consecutive runs of the exact CI command\nwith -n auto.", "gold": ["tests/test_negative_controls_runner.py", "tests/test_no_tracked_file_mutation.py", "tests/test_planted_break_gate.py"], "split": "dev"} +{"id": "d49b44bc24cd", "sha": "d49b44bc24cd532cd68ab7319761274138fcda07", "date": "2026-10-03", "question": "fix(hooks): make a slow gate return a verdict instead of a traceback\n\nThe hook printed a gate's result only after it finished, and an\nunhandled TimeoutExpired killed the whole hook with no verdict. A slow gate\ntherefore looked like a broken hook, which is how commits ended up bypassing\nthe gates with --no-verify.\n\nNow each gate announces itself before it runs, a timeout is a verdict naming\nthe gate, and both budgets are configurable.\n\nMeasured: gate-zero runs the full pytest tests/ and reached 36% in 600s here,\nso the 900s cap could not pass ANY commit. Fast mode is explicit\n(MSCB_PRECOMMIT_FAST=1), prints a warning, and leaves the full run to CI.\n\nTests: 7, incl. a negative control that loads the pre-fix hook from git and\nrequires it to be unable to answer a timeout.\n\nCommitted with MSCB_PRECOMMIT_FAST=1 (see AGENT_DIARY.md, 2026-10-03).", "gold": ["scripts/verify_diary.py", "tests/test_precommit_hook_contract.py"], "split": "dev"} +{"id": "e0579136e8ac", "sha": "e0579136e8ac3edf71199a043d64a38a00bab669", "date": "2026-10-03", "question": "fix(deps): bump urllib3 and PyJWT to clear 16 known CVEs (#65)\n\n* chore(docs): archive auto-synced diary copies out of the issue board\n\nKNOWN_ISSUES.md held 448 lines against a 300-line limit (rule R1), which blocked\nevery commit on main. 224 of those lines were not board entries: 29 blocks\ninjected verbatim from AGENT_DIARY.md by AutoDocUpdater during a full reindex,\neach carrying \"Источник: AGENT_DIARY.md\" and \"Статус: автоматически\nсинхронизировано\".\n\nA board that mirrors the diary holds every issue twice and cannot be trimmed\nwithout losing information. The diary stays the source; the board keeps the 24\nhand-authored sections. Live board is now 224 lines.\n\nscripts/archive_autosync_entries.py does this by marker rather than by date, so\nthe classification does not rot as the file ages, and a second run finds nothing\nto move and changes nothing. Nothing is deleted — the blocks are appended to\ndocs/archive/KNOWN_ISSUES_2026_10_AUTO_SYNC.md, per §4.8 R4.\n\n* fix(deps): bump urllib3 and PyJWT to clear 16 known CVEs\n\npip-audit failed on both test jobs with vulnerabilities that appeared after the\nlast green main run on 2026-09-29, so this is a time-dependent failure rather\nthan a regression from any recent change:\n\n urllib3 2.7.0 PYSEC-2026-4175 / -4176 / -4177 -> 2.8.0\n PyJWT 2.13.0 13 advisories, highest fix 2.15.0 -> 2.15.0\n\nBoth are transitive: neither is imported as an API. PyJWT appears in the codebase\nonly as a token pattern in src/core/redact.py, and the line above the pin claimed\na verified RS256 roundtrip against pyjwt 2.13.0 — that claim is re-verified here\nrather than assumed, and the comment is updated to the version actually tested.\n\nVerified after the bump, not before:\n - pip-audit -r requirements-lock.txt --no-deps --disable-pip -> \"No known\n vulnerabilities found\", rc=0\n - RS256 roundtrip with cryptography 50.0.0, and `import mcp` -> ok\n - pytest tests/ -> 1989 passed, 6 skipped, 98 deselected\n - scripts/smoke_e2e.py -> SMOKE E2E: PASSED, which exercises the real embed\n (llama.cpp), the real rerank (BGE-M3) and a real index search, i.e. the\n network stack urllib3 actually backs\n\n* fix(tests): remove the cross-test race that made CI classify a healthy guard as UNPROVEN\n\nCI reported `NEGATIVE CONTROLS: FAILED (broken=0, unproven=1)` with\n`dead_guard_classifier` marked UNPROVEN — green locally, red on every runner.\n\nRoot cause: tests/test_negative_controls_runner.py proved the digest pin by\nediting the REAL fixture\nscripts/negative_controls/fixtures/dead_guard.py and restoring it in `finally`.\nUnder `pytest -n auto` another worker read that fixture's digest inside the\nwindow between the write and the restore, computed a different digest, and\nclassified a healthy guard as UNPROVEN. The digest guard was never wrong; the\nrace was between two tests, and it presented as a guard defect.\n\nEvery digest matched locally and in a clean checkout, which is why this survived\nlocal verification — the trigger is worker count, not content.\n\nThe test now copies the fixture into a scratch directory it creates and removes,\nand never touches the tracked file. It asserts a PROVEN control before mutating,\nso the property under test is still \"a byte change flips PROVEN to UNPROVEN\".\n\nAdds tests/test_no_tracked_file_mutation.py so the class cannot come back. It is\na test rather than a script because a script nobody runs is not a gate, and its\nfirst version collected zero tests under pytest while reading as coverage.\n\nThat guard's selftest earned its place immediately: it caught a real second\ninstance in tests/test_planted_break_gate.py, which wrote\nexperiments/planted_break/results.json from two xdist workers with no atomicity.\nThat write is now temp-file + os.replace, the artifact is gitignored, and the\none exemption is recorded with its reason.\n\nVerified: 1991 passed, 6 skipped, two consecutive runs of the exact CI command\nwith -n auto.\n\n---------\n\nCo-authored-by: MSCodeBase Agent <>", "gold": ["scripts/archive_autosync_entries.py", "tests/test_negative_controls_runner.py", "tests/test_no_tracked_file_mutation.py", "tests/test_planted_break_gate.py"], "split": "dev"} +{"id": "e40338b311d5", "sha": "e40338b311d566c390829566b826d05f885ed638", "date": "2026-10-03", "question": "chore(docs): archive auto-synced diary copies out of the issue board\n\nKNOWN_ISSUES.md held 448 lines against a 300-line limit (rule R1), which blocked\nevery commit on main. 224 of those lines were not board entries: 29 blocks\ninjected verbatim from AGENT_DIARY.md by AutoDocUpdater during a full reindex,\neach carrying \"Источник: AGENT_DIARY.md\" and \"Статус: автоматически\nсинхронизировано\".\n\nA board that mirrors the diary holds every issue twice and cannot be trimmed\nwithout losing information. The diary stays the source; the board keeps the 24\nhand-authored sections. Live board is now 224 lines.\n\nscripts/archive_autosync_entries.py does this by marker rather than by date, so\nthe classification does not rot as the file ages, and a second run finds nothing\nto move and changes nothing. Nothing is deleted — the blocks are appended to\ndocs/archive/KNOWN_ISSUES_2026_10_AUTO_SYNC.md, per §4.8 R4.", "gold": ["scripts/archive_autosync_entries.py"], "split": "dev"} +{"id": "f3837127c187", "sha": "f3837127c187b090762b2b9af80a235bbd664809", "date": "2026-10-03", "question": "fix(gates): select gate kwargs and report unusable input\n\nrun() called fn(**kw) without selecting the gate's own parameters.\nThe one-tool-four-gates action sends a superset of kwargs, so every call\nraised TypeError, which the top-level handler reported as exit 2 plus a\ntraceback - indistinguishable from an unavailable gate.\n\nNow: parameters are selected via inspect.signature, dropped keys come back\nas ignored_kwargs, and a missing required kwarg answers UNKNOWN/rc=2 with\nrequired_kwargs instead of raising.\n\nNegative control: 4 of the 10 new tests fail on the pre-fix code.\nCommitted with --no-verify: the pre-commit hook produces no output and does\nnot finish on this machine (reason in AGENT_DIARY.md, 2026-10-03).", "gold": ["tests/test_gates_kwarg_contract.py", "tools/verification/gates.py"], "split": "dev"} +{"id": "fbfb474c36ba", "sha": "fbfb474c36baae66528460526b77ab1bff1acb04", "date": "2026-10-03", "question": "fix(hooks): keep the spawn stub local to the hook module\n\nThe contract tests stubbed subprocess.Popen on the shared subprocess\nmodule, so the stub leaked into every other test running in the same worker:\nCI reported 14 unrelated failures naming test_precommit_hook_contract's fake.\n\nThe hook now resolves its spawn through a module-level Popen, so a test can\nreplace this module's reference without touching the global one.", "gold": ["tests/test_precommit_hook_contract.py"], "split": "dev"} +{"id": "1c7abbcf8472", "sha": "1c7abbcf8472b133a3ec49f5cfd903323f42d81c", "date": "2026-10-08", "question": "feat(index): E26 file-level chunk augmentation behind flag (default OFF)\n\nPhase 1 of the approved stack migration. New pure helper chunk_augment.py (FILE/SYMBOLS/DOC = frozen E26 template, parity 27/27); single wiring point in index_parser.parse_one_file covering AST+fallback paths; MSCODEBASE_AUGMENT_CHUNKS default off (byte-identical output). Flip changes chunk_hash=sha256(text) so vectors recompute instead of going stale. Tests: test_chunk_augment.py 10/10 (template pin, flag on/off, wiring both modes); adjacent suites 47/47. No prod reindex triggered.", "gold": ["src/core/indexing/chunk_augment.py", "src/core/indexing/index_parser.py", "tests/test_chunk_augment.py"], "split": "dev"} +{"id": "7f04f5db9da4", "sha": "7f04f5db9da41d2833cfca845b9a457100b635ab", "date": "2026-10-08", "question": "feat(search): on-demand cosine rerank path behind flag (default OFF)\n\nPhase 2 of the approved stack migration (E25). New pure ondemand_rerank.py (cosine, stable ties, skips text-less); MSCODEBASE_ONDEMAND_RERANK (+TOP_N) config mirroring late_enrichment; hybrid hook reranks top-N fused pool with live embeddings INSTEAD of BGE only when flagged. Default OFF: prod path byte-identical. Tests: test_ondemand_rerank.py 6/6 exact math on fake vectors; adjacent 36/36. Live proof with flag: quality h1 3/10 h5 4/10 = baseline, rerank_timing.mode=ondemand_cosine in 1041ms; fast untouched.", "gold": ["src/config/settings.py", "src/core/search/engine.py", "src/core/search/ondemand_rerank.py", "tests/test_ondemand_rerank.py"], "split": "dev"} +{"id": "896d8d9f9936", "sha": "896d8d9f9936af60673a418f05d84a7b2278551c", "date": "2026-10-10", "question": "feat(search): MSCODEBASE_DENSE_OFF kill-switch + EXP-28 dense-off A/B\n\nPhase 3 as a measurement, not a deletion. Flag skips the whole dense stage (query embedding + LanceDB vector search); default OFF so the prod path is byte-identical.\n\nEXP-28: frozen 33-case panel (sha256 f31e594e, union of three pre-existing frozen sets, frozen before the run), 2 runs per arm, same session and index.\nA dense ON: hit1 11/33, hit5 15/33, 0 errors, mean 3359/3534ms.\nB dense OFF: hit1 12/33, hit5 16/33, 0 errors, mean 3163/2492ms.\nPer case 2 better / 1 worse / 30 same; speedup mean-of-means 18% but per-run spread 5.8-29.5%.\nVerdict: H1/H2/H6 refuted as expected, H4 confirmed (0 errors), H3 PARTIAL, H5 untested (no symbol/NL labelling in the panel). Harness proven by a _vector_search_async call counter 1 -> 0 with the embedder alive.\n\nKey finding: of 52 table-API call sites in 13 src files only 5 are vector search; bm25.py:93 builds BM25 from table.to_pandas(), so LanceDB is the chunk primary store, not a vector index. Dense removal NOT recommended; the flag stays as an operational kill-switch.\n\nKNOWN_ISSUES untouched (foreign session edits it) - proposals in experiments/e28_denseoff/PENDING_LEDGER.md. Tests: 3 flag tests + adjacent suites 29/29, ruff clean.", "gold": ["src/config/settings.py", "src/core/search/engine.py", "tests/test_dense_off_flag.py"], "split": "dev"} +{"id": "0a3d92d6196c", "sha": "0a3d92d6196c151b53ef5b2c72895f273f2fbbee", "date": "2026-10-22", "question": "fix(tools): repair the pipeline paths after the move and ignore its run dir\n\nThe move into tools/microtask_pipeline/ broke two relative paths, and the\ncross-project check caught them: run artifacts landed in tools/runs instead of\nthe pipeline's own runs dir, and live_smoke pointed at tools/harness. Both are\nfixed and the run directory is ignored.\n\nProven after the fix: the server produces a machine-checked receipt for a\nproject in a temp directory that is not a git repository, so the pipeline is\nnot tied to this repository.", "gold": ["tools/microtask_pipeline/.gitignore", "tools/microtask_pipeline/live_smoke.py", "tools/microtask_pipeline/mcp_server.py"], "split": "dev"} +{"id": "12e64a940db9", "sha": "12e64a940db94271c476d15e8975c3684264d9f3", "date": "2026-10-22", "question": "fix(docs): auto-sync KNOWN_ISSUES enforces R1 300-line limit + archives\n\n_sync_known_issues now checks KNOWN_ISSUES.md size before appending:\nif >300 lines, entries older than the current calendar month are\narchived to docs/archive/KNOWN_ISSUES_YYYY_MM.md (append-only, per\n§4.8 R4). This prevents auto-sync from becoming the driver of\noverflow (the very issue that broke commits on 2026-09-30 with +267\nlines).\n\nIncludes 5 new tests covering archivation of stale sections,\nretention of current-month entries, idempotency, and the >300-line\ntrigger path in _sync_known_issues.", "gold": ["src/core/auto_doc_updater.py", "tests/test_auto_doc_updater.py"], "split": "dev"} +{"id": "1e8ffadd090d", "sha": "1e8ffadd090d5a5a53e40dd66c9f45eda743c17e", "date": "2026-10-22", "question": "fix(tools): connect gates.py verification gates to pre-commit hook\n\nAdd scripts/gate_cli.py as the 11th pre-commit check, exposing G1-G4\nverification gates (population, referent, generalization, control) from\ntools/verification/gates.py. rc=1 (BLOCK) blocks the commit; rc=2\n(UNKNOWN/not enough input) is advisory and never blocks per §19.6.\nDefault behaviour runs --selftest to prove each gate blocks for its own\nreason.\n\nIncludes tests/test_gate_cli.py covering rc passthrough, env-var and\nCLI-flag configuration, selftest, and bad-input handling.\n\nAlso fixes pre-existing repo issues that blocked all commits:\n- tests/test_microtask_pipeline_red_team.py: unused 'import pytest' (ruff F401)\n- AGENT_DIARY.md: 2 entries referenced non-existent commit 0301fa93\n- KNOWN_ISSUES.md: 532 lines > 300 R1 limit — archived 420 lines of\n pre-October-2026 entries to docs/archive/KNOWN_ISSUES_2026_09.md\n\nAll pre-commit checks pass.", "gold": ["scripts/gate_cli.py", "tests/test_gate_cli.py", "tests/test_microtask_pipeline_red_team.py"], "split": "dev"} +{"id": "39b6e9c997fe", "sha": "39b6e9c997fefec15fb3f4181b856591c85f39e0", "date": "2026-10-22", "question": "fix(core): embedder lease ref-count prevents shared-port kill\n\nembedder_lease now tracks per-PID ref-count instead of a single timestamp.\nThe watchdog skips embedder unload while any process holds a non-zero count,\npreventing one workspace's idle watchdog from killing the shared :8080/:8081\nembedder while another workspace is indexing.\n\n- src/core/embedder_lease.py: acquire/release/is_active with per-PID ref-count\n- src/core/intelligence/layer.py: acquire on reindex start/progress, release in finally\n- Backward-compatible touch() wrapper preserved\n- tests/test_embedder_lease.py: 5 tests (ref-count, multi-pid, stale expiry, touch, holder)", "gold": ["src/core/embedder_lease.py", "src/core/intelligence/layer.py", "tests/test_embedder_lease.py"], "split": "dev"} +{"id": "47f85ee08ff3", "sha": "47f85ee08ff390e07ae2c21f3602d42d37911cd5", "date": "2026-10-22", "question": "fix(mcp): ETA per-phase instead of embed-only\n\nJobHistoryStore теперь хранит за фазу (parsing/scanning/embedding/finalizing/writing)\nвместо только общей длительности. _enrich_job_response считает каждую фазу\nотдельно по истории, затем суммирует — вместо ложного «~8с» от линейной\nэкстраполяции первых секунд. Без данных фазы — честный None.\n\n- src/core/intelligence/store.py: append_record(phase=...),\n get_estimated_phase_duration(), get_estimated_duration() delegating\n- src/core/intelligence/layer.py: phase tracking in _index_progress_callback,\n per-phase ETA in _enrich_job_response\n- tests/test_job_history.py: 2 new ETA tests, updated mock\n\nAll 108 related tests pass.", "gold": ["src/core/intelligence/layer.py", "src/core/intelligence/store.py", "tests/test_job_history.py"], "split": "dev"} +{"id": "67f13d02e6ec", "sha": "67f13d02e6ec32baa8703100fe28812d11179661", "date": "2026-10-22", "question": "fix(mcp): counters show 'нет данных' instead of fake % at zero calls\n\nReplace max(calls, 1) division guard with explicit zero-check in both\nruntime counter outputs. When calls == 0, show 'нет данных' instead of a\nplausible-looking percentage (100% blocked / 0% OK) that masks 'no data'\nas a measurement — the silent zero anti-pattern (§19.6).\n\n- src/mcp/tools/meta_tools.py:332 — blocked_pct\n- src/mcp/server_tools.py:657 — ok_rate health bar\n\nAll 55 existing tests in test_reranker.py + test_quiet_break_gate.py pass.", "gold": ["src/mcp/server_tools.py", "src/mcp/tools/meta_tools.py"], "split": "dev"} +{"id": "6e6bddad32ce", "sha": "6e6bddad32ce9c222e855b899ae8232a779dcbcb", "date": "2026-10-22", "question": "feat(tools): serve the pipeline as an OpenCode tool, and research the built-in shape\n\nThe complaint was fair: a tool the agent must be told to call is a script.\nOpenCode has the native points to make it a mechanism — a custom tool that\ntakes precedence over a built-in one, and tool.execute.before hooks.\n\n- tools/microtask_pipeline/run_step.py: the sidecar for one edit. Two modes,\n the architect/editor split: 'apply' (the parent already specified the edit,\n so the harness applies it deterministically and verifies — no model call) and\n 'solve' (the local model writes the diff). Both return a receipt.\n- tools/microtask_pipeline/opencode/microtask_edit.ts: the tool definition.\n Put it in ~/.config/opencode/tools/edit.ts and it replaces the built-in edit.\n- tools/microtask_pipeline/INTEGRATION.md: the research — the native extension\n points, and that aider's architect/editor split is the same contract\n (85% vs 79.7% solo on their benchmark; the same failure mode we measured:\n a strong model reasons, a weak one cannot format the edit).\n- 23 tests (5 new for the sidecar).\n\nShadowing the built-in tools is deliberately NOT enabled yet: the pipeline's\nsuccess rate is not established (1 of 2 live audits), the executor is slow on\nour server, and a failed pipeline would leave the parent stuck. The order is\ntool -> plugin in observe-only mode -> shadowing.\n\nTwo pre-commit gates still fail and are NOT ours: verify_diary wants a commit\n0301fa93 that does not exist in this repository, and check_known_issues reports\nKNOWN_ISSUES.md at 532 lines while a parallel session rotates that file.", "gold": ["tests/test_microtask_run_step.py", "tools/microtask_pipeline/INTEGRATION.md", "tools/microtask_pipeline/opencode/microtask_edit.ts", "tools/microtask_pipeline/run_step.py"], "split": "dev"} +{"id": "a178e87ac8c2", "sha": "a178e87ac8c237035d2d43c41580408e9dcc7543", "date": "2026-10-22", "question": "fix(gate): judge only tracked and staged docs in the stale detector\n\nThe pre-commit hook used to scan the whole working tree, so a parallel\nsession's untracked drafts could block an unrelated commit, and a doc that\nquotes a third-party version (OpenCode 1.18.23) or a private host looked like\na project-version claim.\n\n- stale_check.py: --staged scans only the files in the pending commit; the\n full-tree walk now skips files git does not track.\n- pre-commit: the stale gate runs with --staged.\n- stale_config.json: exclude our report by basename, exclude the experiment\n directory.\n- negative controls re-proven after the guard changed (pin + reason).\n\nTwo pre-commit gates still fail and are NOT ours: verify_diary wants a commit\n0301fa93 that does not exist in this repository (pre-existing entry), and\ncheck_known_issues reports KNOWN_ISSUES.md at 532 lines while a parallel\nsession is rotating that file right now.", "gold": ["scripts/negative_controls/manifest.json", "scripts/negative_controls/pin_log.json", "tools/stale_detector/stale_check.py", "tools/stale_detector/stale_config.json"], "split": "dev"} +{"id": "ac858785cd1e", "sha": "ac858785cd1edb761ff85418ebac035f91a7b4a2", "date": "2026-10-22", "question": "fix(tools): take the brief's code slice from the AST, and add the audit tool\n\nThe live audit found a real defect in the pipeline: the slice of the code\nunder test was cut at the closing line of a multi-line signature, so the model\nreceived a function signature and no body, and said so. The slice now comes\nfrom the AST (exact node source), with the line-based guess as a fallback for\nnon-Python files.\n\n- tools/microtask_pipeline/audit.py: the audit as a repeatable procedure\n (real module, real test, seeded defect, live pipeline, machine verdict).\n- regression tests for the multi-line signature and the body content\n (18 passed).\n\nLive audit after the fix, on a copy of this repository with one seeded defect\n(run_bounded swallowed the worker's exception): 2 model calls, 231 s,\nacceptance exit 0, receipt with the artifact sha256, verdict PASS.\n\nTwo pre-commit gates still fail and are NOT ours: verify_diary wants a commit\n0301fa93 that does not exist in this repository, and check_known_issues reports\nKNOWN_ISSUES.md at 532 lines while a parallel session rotates that file.", "gold": ["tests/test_microtask_pipeline.py", "tools/microtask_pipeline/audit.py", "tools/microtask_pipeline/engine.py"], "split": "dev"} +{"id": "ffa2f8330996", "sha": "ffa2f8330996247fee40d1312c179f7e2a5872b8", "date": "2026-10-22", "question": "feat(tools): move the microtask pipeline into its own folder and route work through it\n\nThe pipeline is a tool of this repository now, not a file inside an\nexperiment: tools/microtask_pipeline/ (engine, MCP server, red team, live\nsmoke) with its own README, and its tests in the repo suite so the guards\nstay proven on every run.\n\n- tools/microtask_pipeline/engine.py: the diff applier kept the hunks but\n dropped the line terminators (found by the new unit test), and\n restore_tests() restored a tampered test without recording the violation.\n- tools/microtask_pipeline/red_team.py: A6 was asserting on a violation left\n by an earlier attack, so it could pass without its own guard working.\n- tests/test_microtask_pipeline.py (14 tests) and\n tests/test_microtask_pipeline_red_team.py (2) — 16 passed.\n- the MCP server in the global OpenCode config now points at the new folder,\n and the pipeline-orchestrator agent has edit/shell/task denied, so the\n parent cannot do the work itself: it delegates and reads the receipt.\n\nTwo pre-commit gates still fail and are NOT ours: verify_diary wants a\ncommit 0301fa93 that does not exist in this repository, and\ncheck_known_issues reports KNOWN_ISSUES.md at 532 lines while a parallel\nsession rotates that file.", "gold": ["tests/test_microtask_pipeline.py", "tests/test_microtask_pipeline_red_team.py", "tools/microtask_pipeline/README.md", "tools/microtask_pipeline/engine.py", "tools/microtask_pipeline/live_smoke.py", "tools/microtask_pipeline/mcp_server.py", "tools/microtask_pipeline/red_team.py"], "split": "dev"} +{"id": "5b6e34560233", "sha": "5b6e34560233a574869f09c5097f68f5cc0b99e4", "date": "2026-10-23", "question": "fix(indexing): IVF auto-disable после N abandonóв optimize/create_index\n\nНативный LanceDB optimize() нельзя отменить (Future.result(timeout) не\nстопит поток, shutdown(wait=True) джойнит зависший воркер). run_bounded\n'abandonит' поток, но нативный вызов продолжает висеть в процессе.\nПосле _IVF_MAX_ABANDONS__':\n abandon'ов IVF отключается полностью (оптимизация пропускается,\nиндекс без IVF-файла остаётся рабочим).\n\n- src/core/indexing/index_project_runner.py: _ivf_abandon_count,\n _ivf_disabled flag, auto-disable в _call_with_timeout\n- MSCODEBASE_IVF_MAX_ABANDONS env (default 3)\n- MSCODEBASE_LINK_TIMEOUT_SEC env уже используется как таймаут по умолчанию\n- tests/test_index_runner_deadlock.py: 3 новых теста\n\nВсе 20 тестов в test_index_runner_deadlock + test_lancedb_recreate +\ntest_lancedb_race проходят.", "gold": ["src/core/indexing/index_project_runner.py", "tests/test_index_runner_deadlock.py"], "split": "dev"} +{"id": "5c432409ab56", "sha": "5c432409ab562526e5f9fe2ae3f501b6be5d277a", "date": "2026-10-24", "question": "fix(site): resync published counts, deref stale refs, sanitize paths\n\nREADME/WISDOM test counts and both G2 claim entries now match the live\ncensus (2142 collected / 2131 passed, full run 2026-10-24). PATTERNS P-18\nno longer cites a rotated-away line or a guard script that was never\nwritten. Author-absolute paths removed from the research doc and from the\nfrozen-input selftest; repoint_stale_refs derives the repo root from\n__file__. Track the frozen CORPUS input the guard requires. Re-pin the\nstale_detector negative control after fixture byte re-normalization.", "gold": ["scripts/check_parallel_sessions.py", "scripts/negative_controls/manifest.json", "scripts/negative_controls/pin_log.json", "tools/knowledge/PATTERNS.md", "tools/knowledge/repoint_stale_refs.py", "tools/verification/denominator_manifest.json", "tools/verification/verify_public_claims.py"], "split": "dev"} +{"id": "7e2b668e3c4a", "sha": "7e2b668e3c4a31a1ebc39ed0b7b7623e8dc870c3", "date": "2026-10-24", "question": "chore(verification): refresh denominator manifest after audit entry\n\nG5 derived counts for KNOWN_ISSUES.md only (n_sig1 0->3, n_sig2 185->221);\nno new unregistered numbers.", "gold": ["tools/verification/denominator_manifest.json"], "split": "dev"} +{"id": "9e44e4b5c5f9", "sha": "9e44e4b5c5f983baf5852a37263ad7c4e14a9cb1", "date": "2026-10-24", "question": "fix(indexing): resolve IVF timeout at call time\n\n_safe_ivf_index/_run_ivf_index_body bound _LINK_TIMEOUT_SEC as a default\nargument, freezing it at import time. Any test module that imported the\nrunner first (default 300s) made test_index_runner_deadlock wait out the\nreal timeout and fail on the full suite while passing in isolation.\n\nRead the constant inside the body instead, and scope the short timeout in\nthe test with an autouse monkeypatch fixture so it cannot leak into other\nmodules (P-011 class: hidden global binding).", "gold": ["src/core/indexing/index_project_runner.py", "tests/test_index_runner_deadlock.py"], "split": "dev"} +{"id": "f982a15d45fb", "sha": "f982a15d45fb89e3d36759e62322e912dd9c7741", "date": "2026-10-24", "question": "fix(guards): stop task-id hex and hook env from breaking gates\n\nverify_diary read task ids like _health as commit refs; after the\nhistory rewrite those prefixes no longer resolve and blocked the ledger.\n\nstale_detector's negative-control fixture inherited GIT_DIR/GIT_INDEX_FILE\nfrom the invoking hook, so the fixture project queried the real repo index\nand scanned nothing (control reported BROKEN). Both git calls are now\nisolated from GIT_* env. Also exclude foreign/frozen-transcript audit dirs\nfrom our-version drift; re-pin the control.", "gold": ["scripts/negative_controls/manifest.json", "scripts/negative_controls/pin_log.json", "scripts/verify_diary.py", "tools/stale_detector/stale_check.py", "tools/verification/denominator_manifest.json"], "split": "dev"} +{"id": "fba0be9a36c9", "sha": "fba0be9a36c909d144542d815d9dfcac5e46c43a", "date": "2026-10-24", "question": "feat(guard): block verbatim third-party thread dumps (R4)\n\nThe 55 KB dev.to thread that shipped on 2026-10-24 had only 3 external\ndomains, so R3 (>=5 domains, advisory) never fired. R4 matches the shape a\nfrozen thread dump actually has: the frozen-corpus header, declared\ncomment counts, the author tally, or a devto-thread/raw_fetch filename.\nBlocking, with selftest positive+negative cases.\n\n.gitignore now also refuses the devto dump patterns. Verified on the real\npurged blob: R4 fires on it; --all on this repo reports blocking=0.", "gold": ["scripts/check_third_party_data.py"], "split": "dev"} +{"id": "13e235f681b0", "sha": "13e235f681b0d47210e994ceb67ae47c337702b5", "date": "2026-10-25", "question": "fix(search): tolerate fake searchers without a chunk store\n\nP3 anchor degradation must not assume every Searcher instance has _chunk_store. The unit-test doubles exercise the degraded path directly; use getattr and keep the deterministic fallback.", "gold": ["src/core/search/engine.py"], "split": "dev"} +{"id": "189fac8fb277", "sha": "189fac8fb277f628d6afadd8dd4b2a50f54b5be6", "date": "2026-10-25", "question": "docs(tools): report 66 registered and 67 optional tools\n\nThe registered set now has 33 core classes + 16 intel + 13 inline + 4 dev = 66; execute_script is the 67th optional tool. Update the EN/RU/ZH contract and the real-project guard instead of publishing 64/65. The exact default-visible set is described as a curated allowlist rather than a stale hardcoded count.", "gold": ["tests/test_auto_doc_updater.py", "tests/test_factory.py"], "split": "dev"} +{"id": "3824f031e519", "sha": "3824f031e5194c0d39911b8a8027523a6fc6f6a3", "date": "2026-10-25", "question": "chore(test): drop test_factory.py from this PR\n\nThe file imports tools/factory/ops.py, which is not tracked in this branch. Keep the PR focused on the Phase 3 search and docs changes; the factory work needs its own commit with its implementation.", "gold": ["tests/test_factory.py"], "split": "dev"} +{"id": "4861055aaba2", "sha": "4861055aaba2391e4c86e4ee67860ba61826c6f5", "date": "2026-10-25", "question": "test(microtask): use a direct import probe as acceptance\n\npytest in the clean-state job resolves the installed src package before the temporary fixture. The acceptance command now imports the edited module directly from the project root, which is what the test is about.", "gold": ["tests/test_microtask_run_step.py"], "split": "dev"} +{"id": "7178e3425aaa", "sha": "7178e3425aaa2b53fee3fa8e8510ff1a060d2111", "date": "2026-10-25", "question": "fix(search): require a real boolean from the docs auto-router\n\nMocked searchers return a truthy MagicMock from prefers_docs, which routed every test and every non-Searcher double into the docs channel. Require is True so only the real BM25-margin decision can switch channels.", "gold": ["src/mcp/tools/search_tools.py"], "split": "dev"} +{"id": "7507b903d7d8", "sha": "7507b903d7d891eaf960d71bc95603bc74186a0a", "date": "2026-10-25", "question": "test(microtask): make the acceptance fixture importable\n\nThe temporary project used a namespace-package src/ that can be shadowed by the installed package on CI. Add an explicit src/__init__.py so the acceptance command tests the edit, not an import accident.", "gold": ["tests/test_microtask_run_step.py"], "split": "dev"} +{"id": "a0ba261be440", "sha": "a0ba261be4405545d0cd8bbc5e2b7691107f96bf", "date": "2026-10-25", "question": "fix(microtask): pin acceptance cwd on PYTHONPATH\n\nThe acceptance command runs inside a temporary project whose src/ may be shadowed by the installed package. Put the project directory first on PYTHONPATH so the edited file is the one under test.", "gold": ["tools/microtask_pipeline/run_step.py"], "split": "dev"} +{"id": "a9c0764f5ed6", "sha": "a9c0764f5ed6051684dc013e04f323899d0fbfe2", "date": "2026-10-25", "question": "test(microtask): pin the temporary project on sys.path\n\nA conftest in the fixture keeps the local src package ahead of the installed one on CI, so the acceptance command validates the edited file rather than an import accident.", "gold": ["tests/test_microtask_run_step.py"], "split": "dev"} +{"id": "c1fb3a9736dd", "sha": "c1fb3a9736dd14fb5f772a38225eba32a760b046", "date": "2026-10-25", "question": "test(search): align dense and cache tests with Phase 3 defaults\n\nThe default runtime is text-first: dense_off and chunk augmentation are on, and on-demand rerank defaults on. Tests that exercise dense/cache paths now opt back into them explicitly instead of asserting the old defaults.", "gold": ["tests/test_chunk_augment.py", "tests/test_chunk_cache.py", "tests/test_dense_off_flag.py", "tests/test_hybrid_cache.py", "tests/test_ondemand_rerank.py"], "split": "dev"} +{"id": "042762f11857", "sha": "042762f11857c3c396c978c66dd0e2ff07c6a333", "date": "2026-11-07", "question": "fix(health): probe git with Popen+communicate, not check_output\n\nThe execution-contract check ran subprocess.check_output inside a\nThreadPoolExecutor thread while the MCP process had stdout redirected to\nthe JSON-RPC pipe. On Windows the child's pipe is never drained, so the\nprobe deadlocked and every health report carried \"Git операция не удалась:\nФункция превысила таймаут 15 секунд\" — while `git log -1` by hand took\n40 ms. The inner and outer timeouts were also both 15 s, a race by\nconstruction.\n\nThe probe now spawns Popen(stdout=PIPE, stderr=DEVNULL), reads it with\ncommunicate(timeout=10) and kills the child on timeout (AGENTS.md §5.16).\nTwo tests pin the API and the timeout path.", "gold": ["src/core/intelligence/health.py", "tests/test_health_git_probe.py"], "split": "dev"} +{"id": "08f4854311ef", "sha": "08f4854311ef4c04c325557ea6dd76004f4af362", "date": "2026-11-07", "question": "fix(tools): stop three tools from hanging on their own budgets\n\nAudit of the request-to-answer chain on 2026-11-07 found three tools that\nnever returned:\n\n* `structural_search` collected files with `project_path.rglob()` over\n venv/, models/ and experiments/ (~2 GB) and filtered afterwards, so a\n built-in pattern hit the 15 s budget. The walk now prunes heavy\n directories; the same pattern scans 167 files in 2.7 s.\n* `predict_change` waited on read-only git: `_git` used DEFAULT_TIMEOUT\n (300 s) while the tool's budget is 120 s, and neither\n GIT_OPTIONAL_LOCKS=0 nor a closed stdin was set. Read-only git now gets\n 20 s; worktree add/remove keep 120 s.\n* the six LSP tools used a 15 s budget, but basedpyright's first start on\n this repo is slower than that, so every call timed out. Budget is 60 s.\n\n`stale_detector` keeps a 90 s budget and now logs its resolved root and\nscan duration, because the same scan takes 7 s outside the MCP and timed\nout inside it — a hypothesis to confirm from the log, not to guess.\n\nTests: pruning (heavy dirs excluded, scan < 5 s), git env/timeouts.", "gold": ["src/core/change_preview.py", "src/core/structural_search.py", "src/mcp/tools/doc_tools.py", "src/mcp/tools/lsp_tools.py", "tests/test_predict_git_timeout.py", "tests/test_structural_search_prune.py"], "split": "dev"} +{"id": "0a95f68d884f", "sha": "0a95f68d884f6de4eb1e9df58aa3f94fa712f902", "date": "2026-11-07", "question": "fix(index): read hot-reload state from the SQLite store\n\nPhase 3 moved the index to SQLite/FTS5 but FreshnessChecker still read\n`table.to_lance()` (the empty LanceDB table). With an empty state it\nreturned early by design, so no file was ever reindexed after the initial\nbuild: a file changed while the server was down stayed invisible.\n\nObserved 2026-11-07: `_read_head_sha`, added to scripts/verify_diary.py in\ncommit 8553802 before that restart, could not be found by `get_symbol_info`\nor `search_code` — the index still described the pre-commit tree.\n\nThe store now exposes `indexed_state()` (file hash, mtime, size per file)\nand the checker prefers it whenever the store is populated, falling back to\nLanceDB otherwise. Tests cover unchanged / modified / new files against a\nfake store, plus the real store's state payload.", "gold": ["src/core/indexing/chunk_store.py", "src/core/indexing/freshness.py", "src/core/indexing/indexer.py", "tests/test_freshness_sqlite.py"], "split": "dev"} +{"id": "144ded4c60ee", "sha": "144ded4c60ee7b85dd205397acc3efb848e21bc0", "date": "2026-11-07", "question": "fix(gate): run gate-zero serially to stop flaky commit blocks\n\n`verify_diary.py` ran the full suite with `-n auto`. Different runs failed\nwith 4, 31, 5 and 1 tests while the same suite is green serially; the\nfailing tests spawn git subprocesses (temporal_facts_generator alone runs\n\"dozens of git subprocesses\" per call), and 16 workers turn process\ncontention into red gates on healthy code. The result was that a commit\ncould not pass without a retry.\n\nThe suite now runs serially by default (~250 s at a 900 s budget). Parallel\nexecution stays available explicitly with MSCB_GATE_PARALLEL=1, and the\nrecorded lesson \"pytest stays single-thread, do not hthreadinate\" now holds\nfor the gate too.", "gold": ["scripts/verify_diary.py", "tests/test_verify_diary_commit_guard.py"], "split": "dev"} +{"id": "2724dfd6a38a", "sha": "2724dfd6a38ab4e55c9d64c49956d1113acdbc5d", "date": "2026-11-07", "question": "fix(index): report text-index progress to the reindex job\n\nA full text reindex held the job at 10% for its whole run: index_project\nreceived progress_callback/phase_callback and dropped them, so the UI could\nnot tell a working index from a hung one (verified by watching CPU, RSS and\nindexed_at advance while the job stayed at 10%).\n\nThe text path now collects the file list first, reports (current, done,\ntotal, \"parsing\") per file and sends a watchdog heartbeat every 10 files. A\ncallback that raises is logged and ignored: progress must never abort\nindexing. Tests cover the final (done, total) pair and the raising callback.", "gold": ["src/core/indexing/indexer.py", "tests/test_text_index_progress.py"], "split": "dev"} +{"id": "35157951fbf2", "sha": "35157951fbf2f58a1083178cab2f15eadbf22dfe", "date": "2026-11-07", "question": "fix(tests): recognise the FAIL column in frozen manifests\n\nTwo gates failed on the real working tree and blocked every commit.\n\ncheck_191 (§19.1) looked only for the phrase \"ПРОВАЛ\" / \"expected to fail\",\nbut the E30-E57 manifests encode expected failures as a bold **FAIL** cell\nin the Expectation column, so 31 compliant manifests were flagged once they\nwere on disk. The check now counts that encoding and requires two, as the\nrule says. Six frozen manifests carry fewer; they are listed with their\nreal counts and a closed allowlist, because editing a frozen input\nretroactively would destroy the record (§17).\n\nThe auto-doc archive tests hard-coded September/October 2026 as\nold/current. In November both became old and the tests failed without any\ncode change, so a commit could not pass the full-suite gate. The dates are\nnow derived from today.", "gold": ["scripts/audit_protocol_guards.py", "tests/test_auto_doc_updater.py"], "split": "dev"} +{"id": "389913427c89", "sha": "389913427c8930cba0a277be041b3d30483509ed", "date": "2026-11-07", "question": "perf(ledger): verify commits in one git process instead of N\n\nThe Contradiction Ledger called `git cat-file -t` once per commit hash found\nin the diary. In this repo that is 27 spawns per check; on Windows each spawn\ncosts seconds under Defender and they compete with everything else — a health\ncheck's `git log` timed out while those processes were alive, and the diary's\nown note (2026-09-05) records cat-file missing a 5 s budget.\n\nAll hashes now go through a single `git cat-file --batch-check` process.\nNumeric tokens (run-IDs, byte counts, dates) are never cached: the digit guard\nmust still decide them, otherwise two diary entries fail falsely. Measured:\n27 commit tokens resolved by 1 process, 0 issues, ~50 ms of git.", "gold": ["scripts/verify_diary.py", "tests/test_verify_diary_commit_guard.py"], "split": "dev"} +{"id": "85538026288b", "sha": "85538026288b2735c1c08acd299625d2c4651ee1", "date": "2026-11-07", "question": "fix(health): stop spawning git in the startup diagnostics\n\nAfter the previous fix the ledger still never finished on a restart: no\n\"result received\" line appeared at all. Two causes, both in the cache path:\n\n* `_ledger_cache_key` called `git rev-parse` through `subprocess.run(\n capture_output=True)` — the §5.16 pattern — so the cache lookup itself\n could hang, and a failed lookup fell through to the full check.\n* When the batch `cat-file` failed, `check_commit_exists` fell back to one\n spawn per hash with a 30 s timeout and a retry, turning one slow git into\n a multi-minute spawn storm.\n\nThe cache key now reads HEAD from .git/HEAD and the ref file — no process.\nIf git is unavailable the ledger marks `git_unavailable` and skips commit\nverification instead of reporting commits as missing, so a slow sandbox can\nno longer produce false \"commit not found\" entries.\n\nThe health probe no longer runs `git log` (it used to wait 15 s for the\n\"commit line\"); it asks `git rev-parse --short HEAD` with a 5 s budget, and\na timeout is recorded as `metrics[\"git_probe\"] = \"timeout\"` instead of a\nwarning in every report. Measured outside the MCP: 40 ms; inside: the probe\nis the only thing that cannot be made fast, so it stops being an alarm.", "gold": ["scripts/verify_diary.py", "src/core/intelligence/health.py", "tests/test_health_git_probe.py", "tests/test_verify_diary_commit_guard.py"], "split": "dev"} +{"id": "97a32f126df6", "sha": "97a32f126df604980df3b501ab55e943ed588a92", "date": "2026-11-07", "question": "fix(index): report the active SQLite store in index status\n\nPhase 3 (dense_off=true) keeps chunks in /.mscodebase/chunks.db\nwhile the LanceDB table stays empty. IndexStatusReporter read only\ntable.count_rows(), so get_status() returned total_chunks=0 and\nrequire_ready_project blocked search_code, get_symbol_info, all LSP tools\nand the write hub actions. Auto-index then called index_project(), which\nreturned 0 immediately under the same flag, so the state could never heal.\n\nStatus now takes the larger of LanceDB and the SQLite store, names the\nactive store, and skips the LanceDB stale scan when SQLite serves. Under\ndense_off, index_project runs a text-only full index (chunks + graph, no\nembeddings) instead of returning 0.\n\nVerified live: the reporter returns 7314 chunks / 487 files for this\nproject, where it returned 0 before.", "gold": ["src/core/indexing/index_status.py", "src/core/indexing/indexer.py", "tests/test_status_sqlite_store.py"], "split": "dev"} +{"id": "b07317cacffa", "sha": "b07317cacffa3248fbd81224f38a860138998168", "date": "2026-11-07", "question": "fix(autodoc): stop rewriting the agent-owned registries\n\nA full reindex runs AutoDocUpdater, which synced diary entries into the\ntracked KNOWN_ISSUES.md. That happened three times — 2026-09-30 twice (+267\nand +241 lines) and 2026-11-07 (+242 lines, 197 -> 439) — and each time the\nresult was discovered by eye, not by a gate; the last one blocked commits via\ncheck_known_issues.\n\nKNOWN_ISSUES.md and AGENT_DIARY.md are protocol registries maintained by the\nagent, not generated documentation. Auto-sync is now opt-in behind\nMSCODEBASE_AUTODOC_LEDGERS=1 and the skip is reported in the update summary.\nA test pins both sides: default run leaves the file byte-identical, explicit\nflag still syncs.", "gold": ["src/core/auto_doc_updater.py", "tests/test_auto_doc_updater.py"], "split": "sealed"} +{"id": "f33234753006", "sha": "f3323475300615f1cd70556f1474c9a2bad739b4", "date": "2026-11-07", "question": "fix(health): back off provider scans and require real RAM growth\n\nTwo chronic warnings in every health report, both false signals on a healthy\nserver.\n\nProvider scanner: a fixed 13 s interval ran up to 20 iterations even when\nllama.cpp was serving, so a running-but-empty LM Studio produced ~20\nidentical \"empty response\" warnings and a steady stream of probe processes.\nThe interval now doubles up to 5 minutes and the budget is 8 iterations\n(~22 min of recovery time, as before, with far less churn).\n\nRAM trend: `is_growing` was true after three samples, so the startup warm-up\n(embedding model + index load) always looked like a leak. It now needs at\nleast five samples and more than 50 MB of growth inside the window. Tests\ncover warm-up, flat, noisy and genuinely growing histories.", "gold": ["src/core/indexing/resource_monitor.py", "src/providers/embedder/remote_embedder.py", "tests/test_provider_scan_backoff.py", "tests/test_ram_trend.py"], "split": "sealed"} +{"id": "fca8916a79b2", "sha": "fca8916a79b247bcfe150c090a76f16dc82bd8ad", "date": "2026-11-07", "question": "perf(ledger): cache the startup check until the diary or HEAD moves\n\nThe Contradiction Ledger ran the full check on every server start: 34\nentries, 30 commit hashes, and an rglob over tests/ for every symbol of\nevery entry. Measured 2026-11-07: 18:06:56 -> 18:09:29, i.e. 2.5 minutes,\nwith the disk saturated enough that the health probe's `git log` timed out\n(10 s) and the report carried \"Превышен внутренний таймаут Git команды\".\n\nThe result is now cached in /.mscodebase/contradiction_ledger.json\nfor up to six hours and reused only while the diary (mtime, size, sha256)\nand HEAD are unchanged. Any edit to the diary or a new commit triggers a\nfull re-check, so the cache cannot certify a diary it never read.\n\nTests: second run served from cache; a diary edit invalidates it.", "gold": ["scripts/verify_diary.py", "tests/test_ledger_cache.py"], "split": "sealed"} +{"id": "0846cbda34e2", "sha": "0846cbda34e2a5f90f0768c3b91ce894e2a26fa2", "date": "2026-11-08", "question": "fix(tools): bound the stale scan inside the MCP and expose its root\n\n`stale_detector` timed out at 60 s and then 90 s inside the MCP, while the\nsame scan completes in 0.9 s in-process (77 docs) and in 10 s even against\nthe extension root. It also logged nothing, because this module's logger\n(`src.mcp.tools.doc_tools`) is not a child of the configured\n`mscodebase_server` logger, so its lines never reached the file.\n\nThe scan still runs off the event loop, but the tool now enforces its own\n15 s limit and returns a diagnostic with the resolved root and cwd instead\nof hanging until the decorator gives up. Logging goes through\n`mscodebase_server.doc_tools`, so the next run leaves a trace.\n\nThe hypothesis under test is that `asyncio.to_thread` is starved inside the\nMCP (the same process where git subprocesses hang); the diagnostic will\nconfirm it instead of leaving it as a guess.", "gold": ["src/mcp/tools/doc_tools.py"], "split": "sealed"} +{"id": "0b81a3707cb0", "sha": "0b81a3707cb057245784e33df2dbdff24d1c09b5", "date": "2026-11-08", "question": "fix(parser): align the language-pack contract with 1.25.0\n\n`test_language_pack.py` asserted `tags_queries >= 50`, a number taken from\nlanguage-pack 1.14.3. On 1.25.0 the pack still returns parsers for all 53\nlanguages in `LANG_EXT_MAP`, but ships tags.scm for 49: the queries for\n`cuda`, `sflog`, `soql` and `sosl` are gone. That made the clean-state CI job\nfail with `assert 49 >= 50` even though the layer itself works.\n\n* The guard now asserts `languages >= 50` and `tags_queries >= 45`, with a\n comment naming the four languages that lost their tags query and the\n measured 53/49 split, so the threshold catches a broken registration\n without pinning an upstream package's exact count.\n* `language_pack.py` docstring said \"+56 языков\" and cited 1.14.3 as the last\n verification; it now states the measured 53 parsers / 49 tags split for\n 1.25.0.\n* `pyproject.toml` and `.env.example` carried the same stale \"+56 языков\"\n line; both now say \"до 53 языков, 49 с tags.scm\".\n\nVerified: `pytest tests/test_language_pack.py tests/test_language_imports.py`\n23 passed, ruff clean, lockfile in sync.", "gold": ["src/core/language_pack.py", "tests/test_language_pack.py"], "split": "sealed"} +{"id": "16d3a11fe706", "sha": "16d3a11fe706bf53ebd043997009068cd390260b", "date": "2026-11-08", "question": "feat(investigate): one query, parallel channels, one merged answer\n\nThe agent should formulate a question, not pick among tools. `investigate`\nruns the three retrieval channels concurrently and merges them:\n\n* search — hybrid FTS5/BM25 over code and docs;\n* symbols — exact/prefix symbol matches with file:line;\n* grep — regex over the query's identifiers, AST-scoped (comments and\n strings skipped).\n\nEach channel has an 8 s timeout. A slow or failing channel degrades the\nanswer (`degraded: grep:timeout`) instead of failing the call, so partial\nevidence is returned with its provenance. Items are deduplicated by\n(file, line) and ranked by the number of channels that found them, then by\nscore; every item lists its channels (`[search+symbol]`). No matches returns\nan explicit \"nothing found\" rather than a plausible answer.\n\nThe default profile is now 13 entries: investigate, codebase (hub), the\nthree precise lookups for follow-ups, graph_query, read_live_file,\nnotify_change, get_health_report, intel_get_project_memory, predict_change,\ntool_help.\n\nTests: identifier extraction (CamelCase/snake_case first), merge/dedup with\nchannel union, honest empty answer, and an execute path with fake channels.", "gold": ["src/mcp/server_tools.py", "src/mcp/tools/investigate_tool.py", "tests/test_investigate.py"], "split": "sealed"} +{"id": "1f13d8db2922", "sha": "1f13d8db29221c958885b1f2d6e0adf981d41f58", "date": "2026-11-08", "question": "ci: print the slowest tests instead of throwing them away\n\n`--durations=15` was added yesterday but the suite output is captured into a\nshell variable and only `tail -5` is printed, so the numbers were computed and\ndiscarded. Three runs have now taken ~13 minutes (13m29s, 13m31s, 13m14s) while\nthe rest take ~2 minutes, and nothing in the log said where the time went.\n\nThis run's captured output shows a maximum test duration of 2.16 s, so the time\nis not in test bodies - it is in the process around them (install/collection).\nPrinting the durations block makes the next occurrence answer that question\ninstead of producing another unexplained number.", "gold": ["scripts/verify_clean_state.sh"], "split": "sealed"} +{"id": "2890374e7b5e", "sha": "2890374e7b5e7900ee9940ada48be5c95bad825e", "date": "2026-11-08", "question": "chore(tools): land the pending write-integrity guard and its registry entries\n\nThese files were sitting uncommitted in the working tree since 2026-11-07 and\nmade every later session believe another agent was still working (the same\nconfusion cost this session a wrong claim about a \"busy\" diary file). They are\nfinished and self-verifying, so they are landed here rather than deleted.\n\n- `scripts/check_write_bytes.py` — post-write byte check (UTF-8 strict, no BOM,\n homogeneous EOL, binary files skipped). Empty population exits 2, not \"0\n violations\" (§19.6). `--selftest` verified in this session: dirty (BOM) → rc=1,\n clean → rc=0, empty → rc=2.\n- `scripts/recurrence.py` — counts `BREACH:` tags per class and, separately,\n how many happened *after* the guard existed. Run now: 3 tags, 1 class\n (MSC-P-19), after_guard=0.\n- `tools/knowledge/PATTERNS.md` — the P-19 row these two tools implement.\n- `.gitignore` — `.factory/` runtime directory.\n- `WISDOM.md` — removes a leftover `` marker (a hook test\n wrote it and it stayed).\n\nAuthored by the 2026-11-07 session; this commit only verifies and lands it.", "gold": ["scripts/check_write_bytes.py", "scripts/recurrence.py", "tools/knowledge/PATTERNS.md"], "split": "sealed"} +{"id": "2974d3af594b", "sha": "2974d3af594bbacc66cb812b9d56ad7d4a2f86b5", "date": "2026-11-08", "question": "ci: keep clean-state from timing out on a slow test run\n\nTwo pushes took this job 13m29s and 13m31s while the others take 2m10s, and the\njob timeout was 15 minutes. The 13m31s log has a 743.7 s silent gap inside\n\"Running full test suite (no filters, parallel)\": the suite finished and passed,\nso nothing failed — but a slightly slower runner would have been killed for it.\n\n- `--durations=15` in verify_clean_state.sh names the slow tests on the next\n occurrence; today's log cannot say which one they were.\n- `timeout-minutes: 15 -> 25` so a slow runner cannot turn main red without a\n failing test.\n\nKNOWN_ISSUES.md records both measurements and that the cause is not identified.", "gold": ["scripts/verify_clean_state.sh"], "split": "sealed"} +{"id": "2af9d314126e", "sha": "2af9d314126e816b6e056274a1fac4093f5d41c0", "date": "2026-11-08", "question": "fix(search): cap query length; make the text reindex runnable as documented\n\nRed-team findings from the live audit (2026-11-08):\n\n* `search_code` with a 2000-character query took 47.5 s and returned zero\n results — the FTS/embedding path was fed a wall of noise. Queries longer\n than 512 characters are now refused with the length in the message.\n* `python scripts/reindex_text_only.py --project .` failed with\n `ModuleNotFoundError: No module named 'src'`, because `python script.py`\n puts `scripts/` on sys.path, not the repository root. The script now\n inserts its parent directory before importing the package, so the\n documented command works.\n\nVerified: the canonical reindex then ran to completion — 498 files,\n7375 chunks, 13106 graph nodes, 49306 edges, 0 errors, 262 s.", "gold": ["scripts/reindex_text_only.py", "src/mcp/tools/search_tools.py"], "split": "sealed"} +{"id": "2c069a3b9302", "sha": "2c069a3b930236f5e548c5390cf70682a88651cb", "date": "2026-11-08", "question": "feat(mcp): report the JSON error envelope as isError=true\n\nFollow-up to `716405a`: the envelope was machine-readable but the protocol flag\nsaid `isError=false`, so an agent branching on the flag still saw a failed call\nas success. This closes that gap at the boundary.\n\n`src/mcp/error_results.py` wraps the low-level CallTool handler once (installed\nby `create_mcp_server`, right after `register_all_tools`) and flips `isError`\nwhen the result text is exactly our envelope. The text is not modified, so\n`json.loads(content[0].text)` still works.\n\nWhy not raise: FastMCP's `Tool.run` catches every exception and re-raises\n`ToolError(\"Error executing tool : ...\")`, so the JSON would arrive with a\nprefix and stop being parseable.\n\nCost and how it is paid for: `mcp._mcp_server.request_handlers` is a private\nsurface, pinned by `mcp==1.30.0`. `tests/test_mcp_is_error.py` (19) asserts that\nthe install returns True, and covers negative controls (plain text, `ok:true`\nJSON, dict results stay successful; raising tools still report errors) plus an\nend-to-end call through the real `error_boundary`. If a future `mcp` changes the\nsurface, the install returns False with a warning and the flag degrades to\n`false` — the payload is never mangled.\n\nDocs: `docs/{en,ru,zh}/GRACEFUL_DEGRADATION.md` (the \"known residual\" paragraph\nis replaced by the implemented behaviour), `CHANGELOG.md`,\n`docs/research/ideal_system_patterns_2026-09-25.md` (S2 residual closed).", "gold": ["src/mcp/error_results.py", "src/mcp/server_factory.py", "tests/test_mcp_is_error.py"], "split": "sealed"} +{"id": "3fe46aa803da", "sha": "3fe46aa803daf4b978a08160974df2be3e8f8404", "date": "2026-11-08", "question": "feat(investigate): say when a result is a neighbour, not a match\n\n`investigate` merged seven channels into one ranked list. The semantic and\nsymbol channels always answer with neighbours, so \"where is implemented\" produced eight confident `file:line` rows and no\nwarning — the same shape XERJ ships as a defect (#3021 readiness, #3015\n\"partner_agreement: true\" on a wrong count).\n\nThe header now carries the per-identifier exact count from the grep channel,\nand when the first identifier occurs nowhere it says so:\n\n ⚠️ exact: missing_widget=0, implemented=5 — the first identifier does not\n occur in the indexed code; the results below are neighbours, not matches.\n\nCounts are kept per identifier on purpose: \"where is X implemented\" carries the\nnoise word \"implemented\", and a single sum would hide that X itself is absent.\nWhen the grep channel failed, no count is printed at all — silence is not a\nmeasurement.\n\nGuard: tests/test_no_silent_success.py (8 tests). Five of them fail on the\nprevious implementation (verified by running them against the stashed file),\nincluding the rule that two non-exact channels naming the same place is\ncorrelation, not proof. The contract is documented in\ndocs/{en,ru,zh}/GRACEFUL_DEGRADATION.md.", "gold": ["src/mcp/tools/investigate_tool.py", "tests/test_no_silent_success.py"], "split": "sealed"} +{"id": "4323fa32f834", "sha": "4323fa32f834a71e6753cd1a419f9758fb422322", "date": "2026-11-08", "question": "feat(grep): AST-scoped regex search adapted from coderlm\n\nMSCodeBase had no regex grep at all: agents fell back to `shell grep`, which\nis noisy (matches inside comments and strings) and bypasses the project\nguard. coderlm (JaredStewart/coderlm, MIT) solves this with\n`grep?scope=code`; its implementation decides per line and inspects only the\nfirst match on it, so a line with both code and a comment is dropped whole.\n\nThis version judges every match by its own byte offset, keeping the code\nmatch on mixed lines. It reuses FileGuard (same excludes as the index),\nrefuses paths outside the project, skips files >1 MB and binaries, caps the\npattern at 512 chars, and reports files scanned plus elapsed time.\n\nMeasured on this repo (509 files): `capture_output` 94 -> 81 matches,\n`TODO` 25 -> 17, `stale_detector` 145 -> 103 (14-32% of hits were in\ncomments/strings), at ~1.2 s vs ~2.2 s. The pre-registered guess of a 40%\nreduction is not reproduced — recorded as measured, not as the hypothesis.\n\nTests: scope=code vs all, negative control, path traversal, invalid regex,\nempty pattern. Attribution is in the module docstring (MIT).", "gold": ["src/mcp/server_tools.py", "src/mcp/tools/grep_tool.py", "tests/test_grep_code.py"], "split": "sealed"} +{"id": "624933ad7b39", "sha": "624933ad7b39151a0a3b34cc4d14d0b40f39d0c2", "date": "2026-11-08", "question": "fix(health): keep the commit subject in last_commit and repair stale slow tests\n\n`_check_execution_contract` was switched to `git rev-parse --short HEAD` when\nthe deadlock fix replaced `check_output`. That returned only the hash, while\nthe metric contract (and `test_health_git_probe`) expects \"hash subject\". The\nprobe now runs `git log -1 --format=%h %s` — one spawn, same budget, and the\nreport is readable again.\n\nThe mocked probe test could not catch this: it pins the payload shape, not the\ncommand. A new `test_real_probe_returns_hash_and_subject` runs the probe\nagainst a real temporary repository and asserts two space-separated fields, so\na hash-only command fails.\n\nTwo other `slow` tests were red on `main` and in CI-ignored state:\n\n* `test_logs_with_errors` wrote to the legacy `/.codebase_indices/logs`\n with dates from 2026-06-28. Logs live in the central data root\n (`artifact_paths`, task 4/5) and `_count_log_levels` only counts the last\n 24 hours — the metric was honestly 0. The test now patches `get_log_dir` and\n writes today's `[ERROR]`/`[WARNING]` lines, asserting exact counts.\n* `test_search_quality_passed` / `_degraded` mocked `searcher.search`, but the\n check calls `searcher.hybrid_search()` and requires `file_path` + text on\n every item. Both tests now mock `hybrid_search` with the real result shape.\n\nVerified: `pytest tests/test_health_report.py tests/test_health_git_probe.py`\n25 passed; full suite 2313 passed, 10 skipped.", "gold": ["src/core/intelligence/health.py", "tests/test_health_git_probe.py", "tests/test_health_report.py"], "split": "sealed"} +{"id": "686b9165a9bf", "sha": "686b9165a9bf1070ca6b5355e4a22827aee4d571", "date": "2026-11-08", "question": "feat(investigate): second pass — impact, callers, tests, project memory\n\nPass 1 (search, symbols, grep) now feeds a second pass, because these\nchannels need a symbol or a file to ask about:\n\n* impact — blast radius of the top identifiers (risk level, affected files,\n definition site);\n* graph — direct callers of the top identifiers;\n* tests — which test files cover the top files (test_impact predictor);\n* memory — project-memory nodes (ADR/incident) sharing a meaningful token\n with the question.\n\nBoth passes run under the same 8 s per-channel timeout, so a slow channel\nstill degrades instead of failing. Items from both passes are merged by\n(file, line): a caller found by `graph` and a hit found by `search` become\none row with `[search+graph]`. Memory rows are printed as `[memory] title`.\n\nTests: memory matches on title/body, short tokens are ignored, merge still\nunions channels.", "gold": ["src/mcp/tools/investigate_tool.py", "tests/test_investigate.py"], "split": "sealed"} +{"id": "6e908b9fd99f", "sha": "6e908b9fd99f8750be4ba382d47b3bc449c9cb95", "date": "2026-11-08", "question": "feat(indexer): publish wait_reason so \"running\" is distinguishable from \"hung\"\n\nPattern P-B from the 2026-09-25 research: every native link of the indexing chain\nis bounded to 300 s, but while one runs the job is silent — CPU idle, status\n\"running\", nothing says what it waits on. Diagnostics, not correctness, but it is\nthe difference between \"prune is slow\" and \"the process is stuck\".\n\n`src/core/indexing/wait_state.py` keeps a small stack of wait frames (nested-safe,\ncleared in `finally`, so an exception cannot leak a stale reason). It is set by:\n\n- `IndexProjectRunner._bounded_link` -> `native: