Team-convention queries return no ranked exemplar files (0/6 in niche benchmark) #157

Summary

On the frozen niche benchmark from PR #156 (results/niche-benchmark-evidence.json, 2026-10-30), Codebase Context is 0/6 top-3 file hits on team-convention tasks. The same slice is 2/6 (33%) for our colleague Aider repo map (Paul Gauthier, aider-chat). jCodeMunch, Repomix, and ripgrep are also 0/6 there. Overall Codebase Context is still 9/18 (50%) because edit-preflight (4/6) and symbol retrieval (5/6) do return ranked paths.

Team-convention answers match pattern signals (dependencyInjection, stateManagement → usefulness 1.0 on as-team-01, as-team-02, ex-team-01) while topFiles stays []. Three of the six tasks (ex-team-02, fa-team-01, fa-team-02) return a 51-byte payload: empty patterns and empty goldenFiles.

Root cause

The patterns command dispatches get_team_patterns (src/cli.ts). That tool (src/tools/get-team-patterns.ts) returns a category map plus an optional side list. It never returns a ranked path list.

1. Index stores exemplars off to the side, and truncates the path. src/core/indexer.ts sets relPath to the last three path segments (file.split(/[\\/]/).slice(-3)) and passes that into UsageTracker.track / trackGoldenFile. A golden file is recorded only when one file hits two or more pattern categories. getConsensus (src/utils/usage-tracker.ts) keeps one canonicalExample for the primary pattern (prefer core/ or shared/, then a shorter snippet). alsoDetected entries get a name and a frequency and no file.

2. get_team_patterns emits signals, not a ranking. The JSON is { patterns: { dependencyInjection: { primary, alsoDetected } }, goldenFiles }. patterns is a record, not results[]. Category filters keep golden files whose flags start with dependencyInjection: or stateManagement:. Files that only demonstrate that one pattern never qualify as golden. FastAPI in this run has no detected DI/router patterns, so both FastAPI tasks (and Excalidraw DI) are the empty 51-byte object. There is nothing to rank.

3. Search ranking will not fill the gap. Team-convention tasks call patterns, not search. On the search path, hasGroundedCanonicalExample (src/tools/search-codebase.ts) attaches example only when the stored path equals exactly one hit path and that hit's snippet contains the stored snippet. getBestExample only echoes a golden file that is already in the top hits. A last-three-segment path never equals a repo-relative hit path, so the example is dropped. Preferred-pattern examples are also limited to edit/refactor/migrate preflight.

4. The niche lane only scores results. extractSearchFilesFromJson (scripts/lib/niche-benchmark/lanes.mjs) reads results[].file|path, and patterns[].bestExample only when patterns is an array. The real payload is an object, so topFiles is [] even if a nested canonicalExample.file or goldenFiles[].file is present. Signal matching still scores usefulness. That is the 0/6.

How colleagues surface a file

• Aider repo map (scripts/lib/niche-benchmark/aider-repomap-query.py): query tokens become mentioned identifiers; RepoMap.get_ranked_tags runs personalized PageRank over tree-sitter tags and returns rel_fname paths. That is why it can name fastapi/routing.py on fa-team-02 (expected substring routing) without knowing the word dependencyInjection. The other hit, ex-team-02, matches because the frozen expected patterns include the loose substring src (rank 3). Same lane is 0/6 on edit-preflight. Credit: Paul Gauthier's Aider.
• jCodeMunch search_symbols does return file paths, but these questions are not symbol names, so the team-convention slice is 0/6. Credit: jCodeMunch (jcodemunch-mcp).
• Repomix ranks pack sections by query-token overlap and also scores 0/6 on this slice. Credit: Repomix.
• ripgrep returns lexical match paths and scores 0/6 on this slice. Credit: BurntSushi ripgrep.

Candidate fixes (low effort first)

Do not edit tests/fixtures/niche-benchmark-manifest.json after seeing output. Do not add task-id or query-string special cases whose only job is to pass these six tasks.

1. Low — publish the files we already store, and let the scorer read them

• In get_team_patterns, add a capped rankedExemplars list (file, score, category): category-filtered goldenFiles (already sorted by score) plus primary.canonicalExample, deduped.
• When tracking examples and golden files, store the full repo-relative path (normalizedRelPath already computed beside relPath), not the last three segments.
• In extractSearchFilesFromJson, also collect rankedExemplars[].file, goldenFiles[].file, and patterns[category].primary.canonicalExample.file / examples[].

This wires existing data into the metric. It can move the Angular tasks whose golden or canonical paths contain service, inject, store, or state. It does not help the 51-byte FastAPI/Excalidraw-DI tasks until a pattern exists.

2. Medium — top-k exemplars per pattern, usable without a prior search hit

• The tracker keeps dates per pattern, not the file list. Record full paths per (category, pattern) and return the top three by in-file count, then the existing core/shared preference.
• Return that list from get_team_patterns and from search preflight. Stop requiring the exemplar to already be a search hit. Match on path suffix, and still require the stored snippet to occur in that file's indexed chunk.
• Select the category from the query the way testing already does in shouldIncludePatternConflictCategory (src/preflight/query-scope.ts): general DI/state terms, not the six benchmark strings.

This should rank a DI or state file for the TypeScript convention questions. FastAPI stays empty until Depends / APIRouter are detected as patterns.

3. Higher — rank convention files the way Aider ranks a repo map

• Take pattern-bearing files as nodes on the import graph we already store (internalFileGraph) and personalize the rank with query tokens plus the matched pattern category. Return the top three full paths next to the frequency signal. Same shape as Aider's mentioned-identifier ranking, using our graph instead of a second tag index.
• When the pattern catalog is empty (FastAPI in this run), fall back to graph hubs whose path or exported names overlap the query tokens (routing.py, depends), so the tool still returns files.
• Gate it on the full 18-task run so edit-preflight and symbol retrieval do not regress.

How to verify each fix

Frozen fixtures stay frozen. From a clean build:

pnpm install && pnpm run build
npm run benchmark:niche

Optional narrower lanes: node scripts/run-niche-benchmark.mjs --lanes codebase-context,aider-repo-map.

Read results/niche-benchmark-evidence.json → comparator codebase-context:

• byNiche.team-convention.top3HitRate is above 0 (today 0/6).
• Each *-team-* row has a non-empty topFiles, and firstRelevantHit is 1–3 when an expected substring is actually present.
• matchedSignals stay where they already match (dependencyInjection, stateManagement). Usefulness must not be traded away for a file list.
• byNiche.edit-preflight.top3HitRate stays at least 4/6 and symbol-retrieval at least 5/6.
• Report the new fraction honestly, including tasks that stay empty (likely FastAPI until fix 3). Compare the team-convention slice with Aider's 2/6 on the same evidence file. Aider's ex-team-02 hit is the loose src substring; do not treat 33% as a ceiling to game.

Agent: cursor-cloud/cbc-convention-gap