diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 4b114b2..0319cd0 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -515,3 +515,15 @@ Promotion rule: 5. commit the manifest, scorer, limits, failure policy, and report schema 6. fingerprint the frozen set 7. only after G1 PASS may any claim-bearing treatment run begin + +--- + +# Next + +Remaining gaps from the backend and mixed-repo operational smoke in `docs/benchmark.md`. These are not Phase 1 work. + +- Python and Go file-import graphs are empty, so convention maps and edit-preflight impact cannot name callers outside TypeScript and JavaScript. On the FastAPI full-stack template, impact named generated TypeScript client files for a Python route change. +- The keyword index keeps the first 5000 chunks. Dex analyzed 13,940 chunks and healthchecks 5,812; later chunks are absent from search. Status now reports that omission. Choosing which chunks to keep is still open. +- Style-guide lookup reads a fixed filename list and only returns sections that match the query. NestJS `authentication` and Dex `connector` returned no content even though `CONTRIBUTING.md` was present. +- Team-pattern detection is still oriented around JavaScript and TypeScript libraries. After removing the false RxJS match on JWT and email `Subject`, Python and Go maps have no language-native patterns. +- Edit preflight stayed low-confidence for prose change requests on the Python, Go, and mixed repos. Symbol search for a concrete name succeeded on all four. Routing a prose edit to that symbol is still open. diff --git a/docs/benchmark.md b/docs/benchmark.md index 73d073b..fb2e353 100644 --- a/docs/benchmark.md +++ b/docs/benchmark.md @@ -78,3 +78,39 @@ The benchmark does not measure patch correctness, end-to-end task completion, or Historical note: the [earlier public report archive](./benchmark-prior-public-report.md) contains a one-task official pilot and a separate 24-task discovery report. Those records and their comparator statuses remain separate, are not folded into this current four-family summary, and retain their claim gates. See the [comparison table](./comparison-table.md) for the retrieval family in a compact form. The repository walkthrough is in the [demo](./demo.md). + +## Repo-kind operational smoke + +This section records whether `init`, `reindex`, `status`, convention lookup (`map`, plus `style-guide`), edit preflight (`search --intent edit`), and symbol search (`refs`) finish with a usable payload. It is not a retrieval-quality score and it does not change the claim gates above. Wall times are one local run on Linux x64, Node.js 22.23.4, model-free indexing (embeddings skipped). Clones were `--depth 40`. A command is a pass only when it exits 0 and the payload is usable for that step. + +Checked revisions: + +| Kind | Repository | Commit | +| --- | --- | --- | +| NestJS backend | [brocoders/nestjs-boilerplate](https://github.com/brocoders/nestjs-boilerplate) | `9620f15` | +| Django app | [healthchecks/healthchecks](https://github.com/healthchecks/healthchecks) | `f6c46b5` | +| Go service | [dexidp/dex](https://github.com/dexidp/dex) | `b02ead0` | +| TypeScript frontend + Python FastAPI | [fastapi/full-stack-fastapi-template](https://github.com/fastapi/full-stack-fastapi-template) | `f4bcfab` | + +`init --yes` detected Cursor from the home directory and wrote `.cursor/mcp.json` in each repo. + +### Backend and mixed repositories + +| Repo | init | reindex | status | conventions | edit preflight | symbol search | +| --- | --- | --- | --- | --- | --- | --- | +| NestJS boilerplate | Pass 0.68s | Pass 3.20s. 226 files, 694 chunks, indexer 2.60s | Fail 0.57s. Ready, but a new process omitted file and chunk counts | Pass 0.64s. Nest patterns and 428 import edges. `style-guide --query authentication` found no section | Pass 0.99s. Top hit `MailService`; impact includes `auth.service.ts`. `ready` is false | Pass 1.34s. `MailService`, 6 syntactic usages, complete | +| healthchecks (Django) | Pass 0.70s | Pass 4.09s. 1,018 files, 5,812 chunks analyzed, indexer 3.47s. Keyword index kept 5,000 | Fail 0.62s. Ready, counts and the 812-chunk omission were hidden | Fail 0.63s. Only pattern was RxJS at 100% adoption. Import edges 0. `style-guide --query contributing` matched `CONTRIBUTING.md` | Fail 1.58s. Low confidence (0.07). Abstain told the caller to refresh a current index. Top hits were not `format_duration` | Pass 0.71s. `format_duration`, 17 syntactic usages. `isComplete` false because the limit was 8 | +| dex (Go) | Pass 0.66s | Pass 3.88s. 485 files, 13,940 chunks analyzed, indexer 3.29s. Keyword index kept 5,000 | Fail 0.64s. Ready, counts and the 8,940-chunk omission were hidden | Fail 0.65s. Only pattern was RxJS at 100% adoption. Import edges 0. `style-guide --query connector` found no section | Fail 1.29s. Rank 2 was `retryOnConflict`, then the card said to refresh a current index. Low confidence 0.17 | Pass 0.70s. `retryOnConflict`, 8 syntactic usages, at the limit | +| full-stack FastAPI template | Pass 0.65s | Pass 3.21s. 180 files, 943 chunks, indexer 2.57s | Fail 0.59s. Ready, but a new process omitted file and chunk counts | Fail 0.70s for the Python API. Map patterns are React-only. Graph is 33 files and 91 edges, and the examples are TSX. `style-guide --query development` matched `CONTRIBUTING.md` | Fail 0.81s. Results include `delete_item` in `backend/app/api/routes/items.py`. Impact names generated TS client files. Low confidence 0.09, `ready` false | Pass 0.73s. `get_current_user`, 2 syntactic usages, both in `backend/app/api/deps.py`, which matches the source | + +Edit requests were: NestJS password-reset mail via `MailService`; healthchecks downtime duration formatting; Dex Kubernetes conflict retry; full-stack superuser check before item delete. + +The false RxJS pattern came from bare `Subject` in JWT claims and email headers. The refresh instruction came from the search candidate budget, not from stale files (`skippedFiles` was 0). + +### After those three fixes + +Re-checked on the same clones. This does not turn the failed convention or preflight cells into passes. + +- `status` on a new process now reads persisted stats. healthchecks reports 5,000 searchable chunks of 5,812 (812 omitted). Dex reports 5,000 of 13,940 (8,940 omitted). NestJS (694) and the full-stack template (943) are under the cap, so they show counts only. +- After reindex, healthchecks (wall 2.22s, indexer 1.73s) and Dex (wall 2.18s, indexer 1.70s) no longer report RxJS. Their maps have no active patterns, no best examples, and 0 import edges. +- Edit preflight for the same Dex and healthchecks queries now says the candidate budget was exhausted and to narrow the query. Both stay `ready: false` with low confidence. Dex still ranks `retryOnConflict` second. healthchecks still does not surface `format_duration` in the top 5. diff --git a/src/cli-formatters.ts b/src/cli-formatters.ts index c89295f..678d4ae 100644 --- a/src/cli-formatters.ts +++ b/src/cli-formatters.ts @@ -723,6 +723,8 @@ type IndexingStatusPayload = { totalFiles?: number; indexedFiles?: number; totalChunks?: number; + searchableChunks?: number; + omittedChunks?: number; duration?: string; incremental?: boolean; }; @@ -761,6 +763,16 @@ export function formatStatus(data: IndexingStatusPayload, rootPath: string): voi if (s.duration) parts.push(s.duration); if (typeof s.incremental === 'boolean') parts.push(s.incremental ? 'incremental' : 'full'); if (parts.length > 0) lines.push(`Stats: ${parts.join(` ${g.dot} `)}`); + if ( + s.omittedChunks && + s.omittedChunks > 0 && + s.searchableChunks != null && + s.totalChunks != null + ) { + lines.push( + `Searchable: ${s.searchableChunks} of ${s.totalChunks} chunks (keyword index cap)` + ); + } } if (data.progress) { diff --git a/src/core/source-coverage-action.ts b/src/core/source-coverage-action.ts new file mode 100644 index 0000000..7dc7144 --- /dev/null +++ b/src/core/source-coverage-action.ts @@ -0,0 +1,33 @@ +export interface SourceCoverageActionInput { + skippedFiles: number; + candidateBudgetExhausted?: boolean; + candidateRetrievalFailed?: boolean; + refreshAction?: string; +} + +const SOURCE_REFRESH_ACTION = + 'Run refresh_index with incrementalOnly=true to refresh changed or unavailable sources, then retry.'; +const CANDIDATE_BUDGET_ACTION = + 'Search candidate budget was exhausted. Narrow the query to a concrete symbol or path and retry.'; + +/** Stale or unreadable sources need a refresh. An exhausted candidate budget does not. */ +export function sourceCoverageRefreshAction(input: { + skippedFiles: number; + candidateBudgetExhausted: boolean; + candidateRetrievalFailed: boolean; +}): string | undefined { + if (input.skippedFiles > 0 || input.candidateRetrievalFailed) return SOURCE_REFRESH_ACTION; + if (input.candidateBudgetExhausted) return CANDIDATE_BUDGET_ACTION; + return undefined; +} + +export function presentSourceCoverage(coverage: T): T { + if (coverage.skippedFiles > 0 || coverage.candidateRetrievalFailed === true) return coverage; + const refreshAction = sourceCoverageRefreshAction({ + skippedFiles: 0, + candidateBudgetExhausted: coverage.candidateBudgetExhausted === true, + candidateRetrievalFailed: false + }); + if (!refreshAction || refreshAction === coverage.refreshAction) return coverage; + return { ...coverage, refreshAction }; +} diff --git a/src/tools/get-indexing-status.ts b/src/tools/get-indexing-status.ts index 1dc5e59..b7b3a7b 100644 --- a/src/tools/get-indexing-status.ts +++ b/src/tools/get-indexing-status.ts @@ -1,7 +1,61 @@ +import { promises as fs } from 'node:fs'; +import path from 'node:path'; import { fileURLToPath } from 'node:url'; import type { Tool } from '@modelcontextprotocol/sdk/types.js'; +import { INDEXING_STATS_FILENAME, KEYWORD_INDEX_FILENAME } from '../constants/codebase-context.js'; +import { resolveIndexArtifact } from '../core/index-generation.js'; import type { ToolContext, ToolResponse } from './types.js'; +interface PersistedIndexingStats { + indexedFiles: number; + totalChunks: number; + totalFiles: number; + generatedAt: string; +} + +interface DiskIndexSummary { + stats: PersistedIndexingStats; + searchableChunks: number; +} + +function isPersistedIndexingStats(value: unknown): value is PersistedIndexingStats { + if (typeof value !== 'object' || value === null) return false; + const record = value as Record; + return ( + typeof record.indexedFiles === 'number' && + Number.isFinite(record.indexedFiles) && + typeof record.totalChunks === 'number' && + Number.isFinite(record.totalChunks) && + typeof record.totalFiles === 'number' && + Number.isFinite(record.totalFiles) && + typeof record.generatedAt === 'string' + ); +} + +function searchableChunkCount(value: unknown): number | null { + if (typeof value !== 'object' || value === null || !('chunks' in value)) return null; + const chunks = (value as { chunks?: unknown }).chunks; + return Array.isArray(chunks) ? chunks.length : null; +} + +async function readDiskIndexSummary(rootPath: string): Promise { + try { + const keywordPath = await resolveIndexArtifact(rootPath, KEYWORD_INDEX_FILENAME); + const statsPath = path.join(path.dirname(keywordPath), INDEXING_STATS_FILENAME); + const [statsRaw, indexRaw] = await Promise.all([ + fs.readFile(statsPath, 'utf-8'), + fs.readFile(keywordPath, 'utf-8') + ]); + const statsParsed: unknown = JSON.parse(statsRaw); + const indexParsed: unknown = JSON.parse(indexRaw); + const searchableChunks = searchableChunkCount(indexParsed); + if (!isPersistedIndexingStats(statsParsed) || searchableChunks === null) return null; + return { stats: statsParsed, searchableChunks }; + } catch { + return null; + } +} + const loadedStatusModulePath = fileURLToPath(import.meta.url); export const definition: Tool = { @@ -20,6 +74,15 @@ export async function handle( ctx: ToolContext ): Promise { const progress = ctx.indexState.indexer?.getProgress(); + const disk = await readDiskIndexSummary(ctx.rootPath); + const memory = ctx.indexState.stats; + const omittedChunks = + disk && disk.stats.totalChunks > disk.searchableChunks + ? disk.stats.totalChunks - disk.searchableChunks + : undefined; + const capFields = omittedChunks + ? { searchableChunks: disk?.searchableChunks, omittedChunks } + : {}; return { content: [ @@ -30,16 +93,24 @@ export async function handle( status: ctx.indexState.status, rootPath: ctx.rootPath, loadedStatusModulePath, - lastIndexed: ctx.indexState.lastIndexed?.toISOString(), - stats: ctx.indexState.stats + lastIndexed: ctx.indexState.lastIndexed?.toISOString() ?? disk?.stats.generatedAt, + stats: memory ? { - totalFiles: ctx.indexState.stats.totalFiles, - indexedFiles: ctx.indexState.stats.indexedFiles, - totalChunks: ctx.indexState.stats.totalChunks, - duration: `${(ctx.indexState.stats.duration / 1000).toFixed(2)}s`, - incremental: ctx.indexState.stats.incremental + totalFiles: memory.totalFiles, + indexedFiles: memory.indexedFiles, + totalChunks: memory.totalChunks, + duration: `${(memory.duration / 1000).toFixed(2)}s`, + incremental: memory.incremental, + ...capFields } - : undefined, + : disk + ? { + totalFiles: disk.stats.totalFiles, + indexedFiles: disk.stats.indexedFiles, + totalChunks: disk.stats.totalChunks, + ...capFields + } + : undefined, progress: progress ? { phase: progress.phase, diff --git a/src/tools/search-codebase.ts b/src/tools/search-codebase.ts index d423ce2..4182734 100644 --- a/src/tools/search-codebase.ts +++ b/src/tools/search-codebase.ts @@ -4,6 +4,7 @@ import type { Tool } from '@modelcontextprotocol/sdk/types.js'; import { promises as fs } from 'fs'; import path from 'path'; import type { ToolContext, ToolResponse, DecisionCard } from './types.js'; +import { presentSourceCoverage } from '../core/source-coverage-action.js'; import { CodebaseSearcher } from '../core/search.js'; import type { SearchIntentProfile, SearchSourceCoverage } from '../core/search.js'; import type { @@ -232,7 +233,7 @@ async function handlePinned( profile: searchProfile }); results = outcome.results; - sourceCoverage = outcome.sourceCoverage; + sourceCoverage = presentSourceCoverage(outcome.sourceCoverage); } catch (error) { if (error instanceof IndexCorruptedError) { console.error('[Auto-Heal] Index corrupted. Requesting background re-index...'); diff --git a/src/utils/usage-tracker.ts b/src/utils/usage-tracker.ts index 7bd9d9f..461f709 100644 --- a/src/utils/usage-tracker.ts +++ b/src/utils/usage-tracker.ts @@ -587,8 +587,13 @@ export class PatternDetector { } } - // Generic state patterns (framework-agnostic) - if (/BehaviorSubject|ReplaySubject|Subject|Observable/.test(content)) { + // RxJS only. Bare "Subject" and "Observable" also appear in JWT claims and + // email headers, so they are not evidence on their own. + const importsRxjs = + /from\s+['"]rxjs(?:\/[^'"]*)?['"]/.test(content) || + /require\(\s*['"]rxjs(?:\/[^'"]*)?['"]\s*\)/.test(content); + const usesRxjsSubject = /\b(?:BehaviorSubject|ReplaySubject|AsyncSubject)\b/.test(content); + if (importsRxjs || usesRxjsSubject) { this.track('stateManagement', 'RxJS'); } if (content.includes('createStore') || content.includes('configureStore')) { diff --git a/tests/indexing-status-disk-stats.test.ts b/tests/indexing-status-disk-stats.test.ts new file mode 100644 index 0000000..eba1bce --- /dev/null +++ b/tests/indexing-status-disk-stats.test.ts @@ -0,0 +1,60 @@ +import { mkdtemp, mkdir, rm, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import { afterEach, describe, expect, it } from 'vitest'; +import { makePaths } from '../src/project-state.js'; +import { handle } from '../src/tools/get-indexing-status.js'; +import type { ToolContext } from '../src/tools/types.js'; + +const roots: string[] = []; + +afterEach(async () => { + await Promise.all(roots.splice(0).map((root) => rm(root, { recursive: true, force: true }))); +}); + +describe('indexing status disk stats', () => { + it('reports the keyword cap from persisted stats when the process has no in-memory stats', async () => { + const root = await mkdtemp(path.join(tmpdir(), 'cbc-status-cap-')); + roots.push(root); + const dir = path.join(root, '.codebase-context'); + await mkdir(dir); + await writeFile( + path.join(dir, 'indexing-stats.json'), + JSON.stringify({ + indexedFiles: 4, + totalChunks: 12, + totalFiles: 4, + generatedAt: '2026-10-30T22:40:09.166Z' + }) + ); + await writeFile( + path.join(dir, 'index.json'), + JSON.stringify({ chunks: [{ id: 'a' }, { id: 'b' }] }) + ); + + const ctx: ToolContext = { + indexState: { status: 'ready' }, + paths: makePaths(root), + rootPath: root, + performIndexing: async () => undefined + }; + const response = await handle({}, ctx); + const payload = JSON.parse(response.content?.[0]?.text ?? '{}') as { + lastIndexed?: string; + stats?: { + indexedFiles?: number; + totalChunks?: number; + searchableChunks?: number; + omittedChunks?: number; + }; + }; + + expect(payload.lastIndexed).toBe('2026-10-30T22:40:09.166Z'); + expect(payload.stats).toMatchObject({ + indexedFiles: 4, + totalChunks: 12, + searchableChunks: 2, + omittedChunks: 10 + }); + }); +}); diff --git a/tests/pattern-detector.test.ts b/tests/pattern-detector.test.ts index c5d668d..7b50744 100644 --- a/tests/pattern-detector.test.ts +++ b/tests/pattern-detector.test.ts @@ -27,4 +27,33 @@ describe('PatternDetector testing categories', () => { expect(patterns.unitTestFramework?.primary.name).toBe('Jest'); expect(patterns.testingFramework?.primary.name).toBe('Jest'); }); + + it('does not treat JWT or email Subject as RxJS', () => { + const detector = new PatternDetector(); + detector.detectFromCode( + 'type Claims struct {\n\tSubject string `json:"sub"`\n}\n', + 'server/tokens/claims.go' + ); + detector.detectFromCode( + 'body = b"""Subject: Undelivered Mail Returned to Sender\\n"""\n', + 'hc/api/tests/test_smtpd.py' + ); + expect(detector.getAllPatterns().stateManagement).toBeUndefined(); + }); + + it('tracks an rxjs import and BehaviorSubject as RxJS', () => { + const imported = new PatternDetector(); + imported.detectFromCode( + "import { Subject } from 'rxjs';\nconst events = new Subject();\n", + 'src/events.ts' + ); + expect(imported.getAllPatterns().stateManagement?.primary.name).toBe('RxJS'); + + const typed = new PatternDetector(); + typed.detectFromCode( + 'export class AuthService { state = new BehaviorSubject(false); }\n', + 'src/auth.service.ts' + ); + expect(typed.getAllPatterns().stateManagement?.primary.name).toBe('RxJS'); + }); }); diff --git a/tests/source-coverage-refresh-action.test.ts b/tests/source-coverage-refresh-action.test.ts new file mode 100644 index 0000000..765ebee --- /dev/null +++ b/tests/source-coverage-refresh-action.test.ts @@ -0,0 +1,37 @@ +import { describe, expect, it } from 'vitest'; +import { + presentSourceCoverage, + sourceCoverageRefreshAction +} from '../src/core/source-coverage-action.js'; + +describe('sourceCoverageRefreshAction', () => { + it('asks for a narrower query when only the candidate budget is exhausted', () => { + const action = sourceCoverageRefreshAction({ + skippedFiles: 0, + candidateBudgetExhausted: true, + candidateRetrievalFailed: false + }); + expect(action).toMatch(/candidate budget/i); + expect(action).not.toMatch(/refresh_index/); + }); + + it('keeps refresh guidance when nominated sources were skipped', () => { + const action = sourceCoverageRefreshAction({ + skippedFiles: 2, + candidateBudgetExhausted: true, + candidateRetrievalFailed: false + }); + expect(action).toMatch(/refresh_index/); + }); + + it('replaces a refresh instruction when the only gap is the candidate budget', () => { + const presented = presentSourceCoverage({ + skippedFiles: 0, + candidateBudgetExhausted: true, + refreshAction: + 'Run refresh_index with incrementalOnly=true to refresh changed or unavailable sources, then retry.' + }); + expect(presented.refreshAction).toMatch(/candidate budget/i); + expect(presented.refreshAction).not.toMatch(/refresh_index/); + }); +});