From e702be0b2ac187bbb0629013663a4d8769cf9862 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Thu, 29 Oct 2026 08:02:23 +0000 Subject: [PATCH 01/15] fix: set up detected tools from one command Init configures Claude Code, Codex, Cursor, and OpenCode together and saves the Node binary plus the package file from the npx cache. map indexes before it prints, --version prints the package version, and a low-confidence search tells the agent a weak result is not proof the code is missing. --- README.md | 6 +- docs/cli.md | 11 +- docs/client-setup.md | 25 +- src/cli-init.ts | 637 ++++++++++++++++++-------------- src/cli-map.ts | 29 +- src/cli.ts | 12 +- src/core/codebase-map.ts | 14 +- src/core/search-quality.ts | 7 +- src/index.ts | 12 +- tests/cli-init-windows.test.ts | 29 ++ tests/cli-init.test.ts | 655 ++++++++++++++++++++++----------- tests/cli-map-export.test.ts | 18 +- tests/cli-version.test.ts | 26 ++ tests/codebase-map.test.ts | 37 +- tests/search-quality.test.ts | 2 +- 15 files changed, 984 insertions(+), 536 deletions(-) create mode 100644 tests/cli-init-windows.test.ts create mode 100644 tests/cli-version.test.ts diff --git a/README.md b/README.md index aa7f416..c08aa40 100644 --- a/README.md +++ b/README.md @@ -12,13 +12,17 @@ Use Node.js 22 or newer. These commands use the current npm release of Codebase # Claude Code claude mcp add --scope user --transport stdio codebase-context -- npx -y codebase-context -# Codex CLI +# Codex CLI (this is the Codex setup line) codex mcp add codebase-context -- npx -y codebase-context # OpenCode 1.x (keep the quoted separator on Windows) opencode mcp add codebase-context '--' npx -y codebase-context ``` +`npx -y codebase-context --version` prints the installed package version. `npx -y codebase-context map` builds a file index when the project does not have one, then prints the map. A low-confidence search includes a hint: the matches are weak, and that is not a finding that the code is absent. + +This repository's `init` command detects Claude Code, Codex, Cursor, and OpenCode and writes a project config for each selected tool. The saved launch is the Node binary and the package file from the npx cache. That command is in this source and is not part of published 2.2.0. Until it is published, use the copy-paste lines above. The [client setup guide](./docs/client-setup.md) marks the unpublished command. + Start a new agent session in your project, then ask: > Use Codebase Context to find [feature] in this repository. Pass this repository's absolute path as project, wait for indexing if needed, and open the relevant source files. diff --git a/docs/cli.md b/docs/cli.md index c561ab9..e23c347 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -36,7 +36,7 @@ CODEBASE_CONTEXT_ASCII=1 npx -y codebase-context patterns ## Commands -- `map` — conventions map: architecture layers, patterns, golden files +- `map` — conventions map: architecture layers, patterns, golden files. Builds a file index first when the project does not have one, so a fresh checkout does not report 0 files - `metadata` — tech stack overview - `patterns` — team conventions + adoption/trends - `search --query ` — ranked results; add `--intent edit` for a preflight card @@ -46,13 +46,14 @@ CODEBASE_CONTEXT_ASCII=1 npx -y codebase-context patterns - `reindex` — rebuild index (full or incremental) - `style-guide` — find style guide sections in docs - `memory list|add|remove` — manage team memory (stored in `.codebase-context/memory.json`) -- MCP client registration — use the recipes in the [client setup guide](./client-setup.md); `init` is an interactive HTTP setup wizard with the limitations described there +- MCP client registration — `init` detects Claude Code, Codex, Cursor, and OpenCode and configures the selected tools together. See the [client setup guide](./client-setup.md). Codex setup line: `codex mcp add codebase-context -- npx -y codebase-context` +- `--version` — print the installed package version --- ## First use -Run these CLI commands from the project root. Build or refresh the local index first; the first run downloads a local model and indexes the repository; time depends on repository size, CPU and model. Then inspect the conventions map and search for the code you need: +Run these CLI commands from the project root. `map` indexes the project files when no index exists yet. `reindex` builds or refreshes the full index, including local embeddings; the first embedding run downloads a local model. Time depends on repository size, CPU and model. ```bash npx -y codebase-context reindex @@ -69,7 +70,7 @@ npx -y codebase-context reindex npx -y codebase-context map ``` -The conventions map - run this first on an unfamiliar repo. It shows architecture layers, active patterns with adoption rates and trend direction, and the golden files the team treats as the strongest examples. This is also what the MCP server delivers to AI agents via the `codebase://context` resource on first call, before search narrows to a specific local example. +The conventions map - run this first on an unfamiliar repo. If the project has no index, `map` builds the file index before it prints, so the file count is not stuck at 0. It shows architecture layers, active patterns with adoption rates and trend direction, and the golden files the team treats as the strongest examples. This is also what the MCP server delivers to AI agents via the `codebase://context` resource on first call, before search narrows to a specific local example. That automatic index skips embeddings; run `reindex` when you want semantic search. Example output (truncated): @@ -148,6 +149,8 @@ npx -y codebase-context reindex npx -y codebase-context search --query "file watcher" --intent edit --limit 3 ``` +A low-confidence result includes a hint. The hint means the matches are weak. It is not a finding that the code is absent. + Example output (truncated): ```text diff --git a/docs/client-setup.md b/docs/client-setup.md index 4ad2c3e..29498b9 100644 --- a/docs/client-setup.md +++ b/docs/client-setup.md @@ -103,23 +103,26 @@ On October 6 the exact published `2.2.0` `npx` entrypoint was tested without a f ## Interactive setup wizard -> **Requires 2.4.0 (not yet published)**: The non-interactive `--client` and `--yes` flags are not available in published 2.2.0 (which runs an interactive HTTP wizard). These commands require the unpublished 2.4.0 release and will not work with the current npm release. +> **Requires 2.4.0 (not yet published)**: One `init` command detects Claude Code, Codex, Cursor, and OpenCode and configures them together. Published 2.2.0 still runs an older interactive wizard. These commands require the unpublished 2.4.0 release and will not work with the current npm release. -Run setup from the project you want the client to search: +Run setup from the project you want the clients to search: ```bash -# Claude Code (requires 2.4.0, not yet published) -npx -y codebase-context@2.4.0 init --client claude-code --yes - -# Codex (requires 2.4.0, not yet published) -npx -y codebase-context@2.4.0 init --client codex --yes +# requires the unpublished 2.4.0 +npx -y codebase-context@2.4.0 init --yes ``` -Claude Code writes `.mcp.json`; Codex writes `.codex/config.toml`. Both configs start the installed server with the selected project. Setup uses the current folder unless `--root` or `CODEBASE_ROOT` selects another one; `--root` takes priority. +That writes `.mcp.json` for Claude Code, `.codex/config.toml` for Codex, `.cursor/mcp.json` for Cursor, and `opencode.json` for OpenCode. Each file saves the Node binary that ran `init` and the package file from the npx cache, plus the project folder. The client starts that file directly and does not look up `npx` again. + +Codex setup line, including for the published package: + +```bash +codex mcp add codebase-context -- npx -y codebase-context +``` -`--yes` writes the config without prompts. In 2.2.0, running `init` without flags starts the older interactive wizard. Cursor and OpenCode still use HTTP and need a separately running server. +Repeat `--client` to choose tools, for example `--client claude-code --client codex`. `--root` selects another folder and takes priority over `CODEBASE_ROOT`. With a terminal and without `--yes`, detected tools start checked. -Existing settings are kept. If a different Codebase Context entry is already present, setup leaves the file unchanged and reports the conflict. Restart the client after setup and approve the server if asked. The first use builds the code index. Rerun setup if the Codebase Context installation moves, because the config uses its installed location. +`--yes` writes the config without prompts. Existing settings are kept. A recognized generated launch is updated when you rerun setup, including when the npx cache path moves. A different Codebase Context entry is left unchanged and reported. Restart the client after setup and approve the server if asked. `map` builds the file index when one is missing. ## Optional fixed-folder setup and CLI preparation @@ -131,7 +134,7 @@ To prepare the index separately, run from the project folder: npx -y codebase-context reindex ``` -The interactive wizard limitations described above are still present in the current source. A successful wizard exit alone does not establish working client registration. +A successful setup exit alone does not establish working client registration. ## Gemini CLI diff --git a/src/cli-init.ts b/src/cli-init.ts index bf126cc..0fa873f 100644 --- a/src/cli-init.ts +++ b/src/cli-init.ts @@ -1,6 +1,6 @@ /** * Interactive setup wizard for codebase-context. - * Handles `codebase-context init` — generates MCP config and instruction block + * Handles `codebase-context init` — detects clients and merges project MCP configs * for the four first-wave AI clients: Claude Code, Cursor, Codex, OpenCode. * * Depends only on Node built-ins and @inquirer/prompts. @@ -10,96 +10,149 @@ import path from 'path'; import * as fs from 'node:fs/promises'; -import { select, confirm } from '@inquirer/prompts'; +import { checkbox } from '@inquirer/prompts'; +import { homedir } from 'node:os'; +import { constants } from 'node:fs'; -export type Client = 'claude-code' | 'cursor' | 'codex' | 'opencode'; +export type Client = (typeof CLIENTS)[number]['id']; export type McpConfigResult = { kind: 'file'; path: string; content: string }; export type McpConfigOptions = { rootPath?: string; + /** Node binary written into the client config. Defaults to the current process. */ nodeExecutable?: string; + /** + * Package entrypoint written into the client config. + * When init runs through npx this is the file inside the npx cache. + */ serverEntrypoint?: string; }; type JsonObject = Record; -function quoteTomlString(value: string): string { - return JSON.stringify(value); +export const CLIENTS = [ + { id: 'claude-code', name: 'Claude Code', executable: 'claude', directory: '.claude' }, + { id: 'codex', name: 'Codex', executable: 'codex', directory: '.codex' }, + { id: 'cursor', name: 'Cursor', executable: 'cursor', directory: '.cursor' }, + { id: 'opencode', name: 'OpenCode', executable: 'opencode', directory: '.opencode' } +] as const; + +export async function detectClients( + rootPath: string, + env: NodeJS.ProcessEnv = process.env, + platform: NodeJS.Platform = process.platform +): Promise { + const paths = platform === 'win32' ? path.win32 : path.posix; + const home = env.HOME || env.USERPROFILE || homedir(); + const configHome = env.XDG_CONFIG_HOME || paths.join(home, '.config'); + const searchPath = (env.PATH || env.Path || '').split(platform === 'win32' ? ';' : ':'); + const extensions = platform === 'win32' ? ['', '.exe', '.cmd', '.bat', '.ps1'] : ['']; + const detected: Client[] = []; + for (const client of CLIENTS) { + const userDirectory = + client.id === 'codex' + ? env.CODEX_HOME || paths.join(home, '.codex') + : client.id === 'claude-code' + ? env.CLAUDE_CONFIG_DIR || paths.join(home, '.claude') + : client.id === 'opencode' + ? paths.join(configHome, 'opencode') + : paths.join(home, client.directory); + const directories = [paths.join(rootPath, client.directory), userDirectory]; + const executables = searchPath + .filter(Boolean) + .flatMap((directory) => + extensions.map((extension) => paths.join(directory, client.executable + extension)) + ); + const candidates = [ + ...directories.map((candidate) => ({ path: candidate, directory: true })), + ...executables.map((candidate) => ({ path: candidate, directory: false })) + ]; + for (const candidate of candidates) { + try { + await fs.access( + candidate.path, + candidate.directory || platform === 'win32' ? constants.F_OK : constants.X_OK + ); + const info = await fs.stat(candidate.path); + if (candidate.directory ? !info.isDirectory() : !info.isFile()) continue; + detected.push(client.id); + break; + } catch (error) { + if ( + !isFileNotFound(error) && + !( + typeof error === 'object' && + error !== null && + 'code' in error && + (error.code === 'EACCES' || error.code === 'ENOTDIR') + ) + ) + throw error; + } + } + } + return detected; } +function resolveServerEntrypoint(explicit?: string): string { + if (explicit) return path.resolve(explicit); + const entrypoint = process.argv[1]; + if (!entrypoint) { + throw new Error('Cannot locate the codebase-context CLI entrypoint.'); + } + return path.resolve(entrypoint); +} + +/** + * Launch command saved for every client: the Node binary that ran init, the + * package entrypoint (the npx cache path when init itself was started with npx), + * and the project folder. Clients then start that file directly. + */ export function generateMcpConfig(client: Client, options: McpConfigOptions = {}): McpConfigResult { const rootPath = path.resolve(options.rootPath ?? process.cwd()); - const nodeExecutable = options.nodeExecutable ?? process.execPath; - const serverEntrypoint = path.resolve( - options.serverEntrypoint ?? process.argv[1] ?? path.join(rootPath, 'dist', 'index.js') - ); - const stdioArgs = [serverEntrypoint, rootPath]; - - switch (client) { - case 'claude-code': - return { - kind: 'file', - path: path.join(rootPath, '.mcp.json'), - content: JSON.stringify( - { - mcpServers: { - 'codebase-context': { - command: nodeExecutable, - args: stdioArgs - } - } - }, - null, - 2 - ) - }; - case 'cursor': - return { - kind: 'file', - path: path.join(rootPath, '.cursor', 'mcp.json'), - content: JSON.stringify( - { - mcpServers: { - 'codebase-context': { - type: 'http', - url: 'http://127.0.0.1:3100/mcp' - } - } - }, - null, - 2 - ) - }; - case 'codex': - return { - kind: 'file', - path: path.join(rootPath, '.codex', 'config.toml'), - content: - `[mcp_servers.codebase-context]\n` + - `command = ${quoteTomlString(nodeExecutable)}\n` + - `args = [${stdioArgs.map(quoteTomlString).join(', ')}]\n` + - 'startup_timeout_sec = 120\n' - }; - case 'opencode': - return { - kind: 'file', - path: path.join(rootPath, 'opencode.json'), - content: JSON.stringify( - { + const nodeExecutable = path.resolve(options.nodeExecutable ?? process.execPath); + const serverEntrypoint = resolveServerEntrypoint(options.serverEntrypoint); + const args = [serverEntrypoint, rootPath]; + if (client === 'codex') { + return { + kind: 'file', + path: path.join(rootPath, '.codex', 'config.toml'), + content: + `[mcp_servers.codebase-context]\ncommand = ${JSON.stringify(nodeExecutable)}\n` + + `args = [${args.map((value) => JSON.stringify(value)).join(', ')}]\nstartup_timeout_sec = 120\n` + }; + } + const configPath = + client === 'claude-code' + ? '.mcp.json' + : client === 'cursor' + ? '.cursor/mcp.json' + : 'opencode.json'; + return { + kind: 'file', + path: path.join(rootPath, configPath), + content: JSON.stringify( + client === 'opencode' + ? { $schema: 'https://opencode.ai/config.json', mcp: { + 'codebase-context': { type: 'local', command: [nodeExecutable, ...args] } + } + } + : { + mcpServers: { 'codebase-context': { - type: 'remote', - url: 'http://127.0.0.1:3100/mcp' + ...(client === 'cursor' ? { type: 'stdio' } : {}), + command: nodeExecutable, + args } } }, - null, - 2 - ) - }; - } + null, + 2 + ) + }; } export function resolveInitRoot( @@ -130,58 +183,6 @@ export function resolveInitRoot( return path.resolve(explicitRoot ?? (envRoot ? envRoot : cwd)); } -export function generateInstructionBlock(): string { - return ` -## Codebase Context (MCP) - -**Start of every task:** Call \`get_memory\` to load team conventions before writing any code. - -**Before editing existing code:** Call \`search_codebase\` with \`intent: "edit"\`. If the preflight card says \`ready: false\`, read the listed files before touching anything. - -**Before writing new code:** Call \`get_team_patterns\` to check how the team handles DI, state, testing, and library wrappers — don't introduce a new pattern if one already exists. - -**When asked to "remember" or "record" something:** Call \`remember\` immediately, before doing anything else. - -**When adding imports that cross module boundaries:** Call \`detect_circular_dependencies\` with the relevant scope after adding the import. - -`; -} - -export function resolveInstructionFilePath(client: Client, cwd: string): string | null { - switch (client) { - case 'claude-code': - return path.join(cwd, 'CLAUDE.md'); - case 'cursor': - return path.join(cwd, '.cursorrules'); - case 'codex': - return path.join(cwd, 'AGENTS.md'); - case 'opencode': - return null; - } -} - -/** - * Appends the instruction block to an existing file, or skips if already present. - * Exported with leading underscore to signal internal/test-only use. - */ -export async function _appendInstructionBlock(filePath: string): Promise<'written' | 'skipped'> { - let existing = ''; - try { - existing = await fs.readFile(filePath, 'utf8'); - } catch { - // file does not exist — write fresh - await fs.writeFile(filePath, generateInstructionBlock(), 'utf8'); - return 'written'; - } - - if (existing.includes('')) { - return 'skipped'; - } - - await fs.writeFile(filePath, existing + '\n' + generateInstructionBlock(), 'utf8'); - return 'written'; -} - function isJsonObject(value: unknown): value is JsonObject { return typeof value === 'object' && value !== null && !Array.isArray(value); } @@ -217,9 +218,82 @@ async function readExistingConfig(filePath: string): Promise { } } +function isAbsolutePath(value: string): boolean { + return path.posix.isAbsolute(value) || path.win32.isAbsolute(value); +} + +function samePath(left: string, right: string): boolean { + const normalize = (value: string) => value.replace(/\\/g, '/').replace(/\/+$/, '').toLowerCase(); + return normalize(left) === normalize(right); +} + +function isNpxCacheEntrypoint(value: string): boolean { + return /[\\/]node_modules[\\/]codebase-context[\\/]dist[\\/]index\.js$/.test(value); +} + +/** + * Recognize only CBC-specific launch shapes written by init. + * A node launch counts when it points at the npx cache copy of this package, + * or at the entrypoint this run is about to save. + */ +function isGeneratedLaunch(command: unknown, args: unknown, entrypoint?: string): boolean { + if ( + typeof command !== 'string' || + !Array.isArray(args) || + !args.every((arg) => typeof arg === 'string') + ) + return false; + const executable = path.basename(command.replace(/\\/g, '/')).toLowerCase(); + if (executable === 'node' || executable === 'node.exe') { + const script = args[0] ?? ''; + return ( + args.length === 2 && + isAbsolutePath(script) && + isAbsolutePath(args[1] ?? '') && + (isNpxCacheEntrypoint(script) || (entrypoint !== undefined && samePath(script, entrypoint))) + ); + } + const launch = command === 'npx' || command === 'npx.cmd' ? args : []; + return ( + launch.length === 3 && + (launch[0] === '--yes' || launch[0] === '-y') && + /^codebase-context@\d+\.\d+\.\d+(?:-[\w.-]+)?$/.test(launch[1] ?? '') && + isAbsolutePath(launch[2] ?? '') + ); +} + +function generatedEntrypoint(generated: unknown, client: Client): string | undefined { + if (!isJsonObject(generated)) return undefined; + if (client === 'opencode' && Array.isArray(generated.command)) { + return typeof generated.command[1] === 'string' ? generated.command[1] : undefined; + } + return Array.isArray(generated.args) && typeof generated.args[0] === 'string' + ? generated.args[0] + : undefined; +} + +function mergeGeneratedServer(existing: unknown, generated: unknown, client: Client): unknown { + if (jsonValuesEqual(existing, generated)) return existing; + if (!isJsonObject(existing) || !isJsonObject(generated) || 'url' in existing) return null; + const local = client === 'opencode'; + const command = local && Array.isArray(existing.command) ? existing.command[0] : existing.command; + const args = local && Array.isArray(existing.command) ? existing.command.slice(1) : existing.args; + if ( + (existing.type !== undefined && existing.type !== (local ? 'local' : 'stdio')) || + !isGeneratedLaunch(command, args, generatedEntrypoint(generated, client)) + ) + return null; + return { + ...existing, + command: generated.command, + ...(!local ? { args: generated.args } : {}), + ...(generated.type ? { type: generated.type } : {}) + }; +} + /** * Merge generated MCP config into an existing JSON config file when possible. - * Preserves unrelated keys and refuses to replace a different codebase-context entry. + * Preserves unrelated keys and updates only recognized generated launches. */ export async function _buildMergedMcpContent( filePath: string, @@ -262,28 +336,21 @@ export async function _buildMergedMcpContent( 'codebase-context' ); const generatedCodebaseContext = generatedServersValue['codebase-context']; - if ( - hasExistingCodebaseContext && - !jsonValuesEqual(existingServers['codebase-context'], generatedCodebaseContext) - ) { + const mergedServer = hasExistingCodebaseContext + ? mergeGeneratedServer(existingServers['codebase-context'], generatedCodebaseContext, client) + : generatedCodebaseContext; + if (mergedServer === null) { throw new Error( `Cannot merge ${client} config at ${filePath}: it already contains a different codebase-context configuration.` ); } - + const merged = { + ...generated, + ...existing, + [serverKey]: { ...existingServers, 'codebase-context': mergedServer } + }; return { - content: JSON.stringify( - { - ...existing, - ...generated, - [serverKey]: { - ...existingServers, - ...generatedServersValue - } - }, - null, - 2 - ), + content: jsonValuesEqual(existing, merged) ? existingText : JSON.stringify(merged, null, 2), mergedFromExisting: true }; } @@ -297,8 +364,7 @@ function tomlKeyPattern(key: string): string { } /** - * Appends the project MCP table to Codex TOML, preserving other settings and - * refusing to replace an existing codebase-context table. + * Merges the generated Codex launch, preserving unrelated settings and custom entries. */ export async function _buildMergedCodexContent( filePath: string, @@ -333,13 +399,69 @@ export async function _buildMergedCodexContent( if (comparableSection === normalizedGenerated) { return { content: existing, mergedFromExisting: true }; } + const commandLine = /^([ \t]*command[ \t]*=[ \t]*)("(?:[^"\\]|\\.)*")([ \t]*(?:#.*)?)$/m; + const argsLine = /^([ \t]*args[ \t]*=[ \t]*)(\[[^\r\n]*\])([ \t]*(?:#.*)?)$/m; + const oldCommand = commandLine.exec(section); + const oldArgs = argsLine.exec(section); + const newCommand = commandLine.exec(normalizedGenerated); + const newArgs = argsLine.exec(normalizedGenerated); + if ( + exactEntryHeader.test(section) && + oldCommand && + oldArgs && + newCommand && + newArgs && + !/^[ \t]*url[ \t]*=/m.test(section) && + section.match(/^[ \t]*command[ \t]*=/gm)?.length === 1 && + section.match(/^[ \t]*args[ \t]*=/gm)?.length === 1 + ) { + try { + const generatedArgs: unknown = JSON.parse(newArgs[2]); + const generatedEntrypoint = + Array.isArray(generatedArgs) && typeof generatedArgs[0] === 'string' + ? generatedArgs[0] + : undefined; + if ( + isGeneratedLaunch(JSON.parse(oldCommand[2]), JSON.parse(oldArgs[2]), generatedEntrypoint) + ) { + if ( + jsonValuesEqual(JSON.parse(oldCommand[2]), JSON.parse(newCommand[2])) && + jsonValuesEqual(JSON.parse(oldArgs[2]), JSON.parse(newArgs[2])) + ) { + return { content: existing, mergedFromExisting: true }; + } + const updated = normalizedExisting + .slice(existingEntry.index, sectionEnd) + .replace( + commandLine, + (_line, prefix: string, _value: string, comment: string) => + prefix + newCommand[2] + comment + ) + .replace( + argsLine, + (_line, prefix: string, _value: string, comment: string) => + prefix + newArgs[2] + comment + ); + const content = + normalizedExisting.slice(0, existingEntry.index) + + updated + + normalizedExisting.slice(sectionEnd); + return { + content: existing.includes('\r\n') ? content.replace(/\n/g, '\r\n') : content, + mergedFromExisting: true + }; + } + } catch { + // Non-generated TOML values remain untouched and are reported as conflicts. + } + } throw new Error( `Cannot merge Codex config at ${filePath}: it already contains a different codebase-context configuration.` ); } const dottedEntry = new RegExp( - `^\\s*${mcpServersKey}\\s*\\.\\s*${codebaseContextKey}\\s*\\.`, + `^\\s*${mcpServersKey}\\s*\\.\\s*${codebaseContextKey}\\s*(?:=|\\.)`, 'm' ); if (dottedEntry.test(normalizedExisting)) { @@ -384,51 +506,40 @@ export async function _buildMergedCodexContent( } type InitOptions = { - client?: Client; + clients: Client[]; rootPath: string; yes: boolean; help: boolean; }; function parseClient(value: string): Client { - if (value === 'claude-code' || value === 'cursor' || value === 'codex' || value === 'opencode') { - return value; - } - throw new Error(`Unknown client "${value}". Choose claude-code, cursor, codex, or opencode.`); + const client = CLIENTS.find((client) => client.id === value); + if (client) return client.id; + throw new Error( + `Unknown client "${value}". Choose ${CLIENTS.map((client) => client.id).join(', ')}.` + ); } function parseInitOptions(args: string[]): InitOptions { - if (args.includes('--help') || args.includes('-h')) { - return { rootPath: process.cwd(), yes: false, help: true }; - } - - let client: Client | undefined; + if (args.includes('--help') || args.includes('-h')) + return { clients: [], rootPath: process.cwd(), yes: false, help: true }; + const clients: Client[] = []; let yes = false; for (let index = 0; index < args.length; index++) { const arg = args[index]; - if (arg === '--yes' || arg === '-y') { - yes = true; - } else if (arg === '--client') { - const value = args[index + 1]; - if (!value || value.startsWith('--')) { - throw new Error('Expected a client after --client.'); - } - client = parseClient(value); - index++; - } else if (arg?.startsWith('--client=')) { - client = parseClient(arg.slice('--client='.length)); - } else if (arg === '--root') { - index++; - } else if (arg?.startsWith('--root=')) { - // Root values are validated once by resolveInitRoot below. - } else { - throw new Error(`Unknown init option "${arg}".`); - } + if (arg === '--yes' || arg === '-y') yes = true; + else if (arg === '--client') { + const value = args[++index]; + if (!value || value.startsWith('--')) throw new Error('Expected a client after --client.'); + clients.push(parseClient(value)); + } else if (arg?.startsWith('--client=')) + clients.push(parseClient(arg.slice('--client='.length))); + else if (arg === '--root') index++; + else if (!arg?.startsWith('--root=')) throw new Error(`Unknown init option "${arg}".`); } - return { - ...(client ? { client } : {}), - rootPath: resolveInitRoot(args, process.env.CODEBASE_ROOT, process.cwd()), + clients: [...new Set(clients)], + rootPath: resolveInitRoot(args, process.env.CODEBASE_ROOT), yes, help: false }; @@ -436,131 +547,87 @@ function parseInitOptions(args: string[]): InitOptions { function printInitUsage(): void { console.log('codebase-context init [options]'); - console.log(''); - console.log('Set up MCP config for an AI client.'); - console.log(''); - console.log('Options:'); - console.log(' --client claude-code, cursor, codex, or opencode'); + console.log( + 'Detect and configure coding tools together. With a terminal, Space toggles and Enter applies.' + ); + console.log( + 'Each config saves this Node binary and the package file from the npx cache, plus the project folder.' + ); + console.log('Codex setup line: codex mcp add codebase-context -- npx -y codebase-context'); + console.log( + ' --client Choose a tool (repeatable): ' + + CLIENTS.map((client) => client.id).join(', ') + ); console.log(' --root Project root (default: CODEBASE_ROOT or cwd)'); - console.log(' --yes, -y Write the MCP config without prompts'); + console.log(' --yes, -y Configure every detected tool without prompts'); console.log(' --help, -h Show this help'); } -function getCurrentCliEntrypoint(): string { - const entrypoint = process.argv[1]; - if (!entrypoint) { - throw new Error('Cannot locate the codebase-context CLI entrypoint.'); - } - return path.resolve(entrypoint); -} - export async function handleInitCli(args: string[]): Promise { - let options: InitOptions; try { - options = parseInitOptions(args); - } catch (error) { - console.error(error instanceof Error ? error.message : String(error)); - printInitUsage(); - process.exitCode = 1; - return; - } - - if (options.help) { - printInitUsage(); - return; - } - if (options.yes && !options.client) { - console.error('Error: --yes requires --client.'); - printInitUsage(); - process.exitCode = 1; - return; - } - - console.log('\nSet up codebase-context for your AI client\n'); - - const client = - options.client ?? - (await select({ - message: 'Which client?', - choices: [ - { name: 'Claude Code', value: 'claude-code' }, - { name: 'Cursor', value: 'cursor' }, - { name: 'Codex', value: 'codex' }, - { name: 'OpenCode', value: 'opencode' } - ] - })); - - const mcpResult = generateMcpConfig(client, { - rootPath: options.rootPath, - nodeExecutable: process.execPath, - serverEntrypoint: getCurrentCliEntrypoint() - }); - - console.log('\n--- MCP Config Preview ---'); - console.log( - 'Existing unrelated settings will be preserved; a different codebase-context entry will stop setup.' - ); - console.log(`File: ${mcpResult.path}\n${mcpResult.content}`); - - const applyMcp = options.yes - ? true - : await confirm({ - message: 'Apply MCP config? [y/N]', - default: false - }); - - const instructionPath = options.yes ? null : resolveInstructionFilePath(client, options.rootPath); - - let applyInstruction = false; - if (instructionPath !== null) { - console.log(`\n--- Instruction Block Preview (will append to ${instructionPath}) ---`); - console.log(generateInstructionBlock()); - - applyInstruction = await confirm({ - message: 'Write instruction block? [y/N]', - default: false - }); - } + const options = parseInitOptions(args); + if (options.help) { + printInitUsage(); + return; + } + if (!(await fs.stat(options.rootPath)).isDirectory()) + throw new Error(`Project root is not a directory: ${options.rootPath}`); + console.log(`Set up Codebase Context in ${options.rootPath}`); + let clients = options.clients; + if (!clients.length) { + const detected = await detectClients(options.rootPath); + clients = + !options.yes && process.stdin.isTTY && process.stdout.isTTY + ? await checkbox({ + message: 'Choose tools (Space toggles, Enter sets up):', + choices: CLIENTS.map((client) => ({ + name: client.name + (detected.includes(client.id) ? ' (detected)' : ''), + value: client.id, + checked: detected.includes(client.id) + })) + }) + : detected; + } + if (!clients.length) + throw new Error( + 'No coding tools selected or detected. Choose with --client claude-code, --client codex, --client cursor, or --client opencode.' + ); - try { - if (applyMcp) { - try { - await fs.access(options.rootPath); - } catch { - throw new Error(`Project root does not exist or is not accessible: ${options.rootPath}`); - } - const dir = path.dirname(mcpResult.path); - await fs.mkdir(dir, { recursive: true }); - const mergedConfig = + const prepared = []; + for (const client of clients) { + const config = generateMcpConfig(client, { rootPath: options.rootPath }); + const merged = client === 'codex' - ? await _buildMergedCodexContent(mcpResult.path, mcpResult.content) - : await _buildMergedMcpContent(mcpResult.path, mcpResult.content, client); - await fs.writeFile(mcpResult.path, mergedConfig.content, 'utf8'); - if (mergedConfig.mergedFromExisting) { - console.log(`Merged: ${mcpResult.path} (existing entries preserved)`); - } - console.log(`Written: ${mcpResult.path}`); + ? await _buildMergedCodexContent(config.path, config.content) + : await _buildMergedMcpContent(config.path, config.content, client); + prepared.push({ + ...config, + content: merged.content, + unchanged: (await readExistingConfig(config.path)) === merged.content + }); } - - if (applyInstruction && instructionPath !== null) { - const status = await _appendInstructionBlock(instructionPath); - if (status === 'skipped') { - console.log(`Instruction block already present in ${instructionPath}, skipping.`); - } else { - console.log(`Written: ${instructionPath}`); + for (const config of prepared) { + if (!config.unchanged) { + await fs.mkdir(path.dirname(config.path), { recursive: true }); + await fs.writeFile(config.path, config.content, 'utf8'); } + console.log( + `${config.unchanged ? 'Unchanged' : 'Configured'}: ${path.relative(options.rootPath, config.path)}` + ); } + console.log('Reopen your tools; approve the server or project trust if asked.'); + console.log("The first search builds this repository's index."); } catch (error) { + if ( + error instanceof Error && + (error.name === 'ExitPromptError' || + error.name === 'AbortPromptError' || + error.name === 'CancelPromptError') + ) { + console.log('Setup cancelled; no files written.'); + return; + } console.error(`Init failed: ${error instanceof Error ? error.message : String(error)}`); process.exitCode = 1; - return; } - - const nextSteps = - client === 'claude-code' || client === 'codex' - ? ' Restart or reopen the client in this project to load the MCP server.\n' + - ' Run `npx codebase-context map` to see your codebase conventions\n' - : ' Run `npx codebase-context map` to see your codebase conventions\n' + - ' Start the HTTP server: npx codebase-context --http\n'; - console.log(`\nNext steps:\n${nextSteps}`); } diff --git a/src/cli-map.ts b/src/cli-map.ts index 4479afe..8dac640 100644 --- a/src/cli-map.ts +++ b/src/cli-map.ts @@ -19,7 +19,9 @@ import { MEMORY_FILENAME } from './constants/codebase-context.js'; import { createProjectState } from './project-state.js'; +import { CodebaseIndexer } from './core/indexer.js'; import { buildCodebaseMap, renderMapMarkdown, renderMapPretty } from './core/codebase-map.js'; +import { loadProjectConfig } from './server/config.js'; import type { IndexState } from './tools/types.js'; function resolveMapRootPath(): string { @@ -41,6 +43,29 @@ function printMapUsage(): void { console.log('Default output is markdown (pipeable).'); } +async function ensureProjectIndex(rootPath: string, keywordIndex: string): Promise { + try { + await fs.access(keywordIndex); + return; + } catch { + // A missing index is why `map` used to report 0 files on a fresh project. + } + + console.error(`Indexing ${rootPath} before building the map.`); + const projectConfig = await loadProjectConfig(rootPath); + const indexer = new CodebaseIndexer({ + rootPath, + config: { + skipEmbedding: true, + ...(projectConfig?.parsing ? { parsing: projectConfig.parsing } : {}) + } + }); + const stats = await indexer.index(); + console.error( + `Indexed ${stats.indexedFiles} files for the map. Run \`reindex\` to add local embeddings.` + ); +} + export async function handleMapCli(args: string[]): Promise { const useJson = args.includes('--json'); const usePretty = args.includes('--pretty'); @@ -65,12 +90,14 @@ export async function handleMapCli(args: string[]): Promise { vectorDb: path.join(baseDir, VECTOR_DB_DIRNAME) }; + await ensureProjectIndex(rootPath, paths.keywordIndex); + let indexExists = false; try { await fs.access(paths.keywordIndex); indexExists = true; } catch { - // no index yet + // Indexing did not produce a keyword index. The map degrades to an empty project. } const indexState: IndexState = { status: indexExists ? 'ready' : 'idle' }; diff --git a/src/cli.ts b/src/cli.ts index 2e5f17e..4596cf7 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -87,12 +87,16 @@ function printUsage(): void { console.log(' patterns [--category all|di|state|testing|libraries] Team patterns'); console.log(' refs --symbol [--limit ] Symbol references'); console.log(' cycles [--scope ] Circular dependency detection'); - console.log(' init Interactive setup wizard for AI clients'); - console.log(' map [--json] [--pretty] Conventions map'); + console.log(' init Set up detected AI clients for this project'); + console.log(' map [--json] [--pretty] Conventions map (indexes first if needed)'); console.log(''); console.log('Global flags:'); - console.log(' --json Output raw JSON (default: human-readable)'); - console.log(' --help Show this help'); + console.log(' --json Output raw JSON (default: human-readable)'); + console.log(' --version Print the package version'); + console.log(' --help Show this help'); + console.log(''); + console.log('Codex setup line:'); + console.log(' codex mcp add codebase-context -- npx -y codebase-context'); console.log(''); console.log('Environment:'); console.log(' CODEBASE_ROOT Project root path (default: cwd for CLI only)'); diff --git a/src/core/codebase-map.ts b/src/core/codebase-map.ts index 61a4495..e9754cb 100644 --- a/src/core/codebase-map.ts +++ b/src/core/codebase-map.ts @@ -234,8 +234,20 @@ export async function buildCodebaseMap( })); // --- Graph stats --- + // Prefer recorded stats. When they are missing or say 0 files but the index + // already has files, count those files so the map does not report an empty project. + const countedFiles = new Set([ + ...Object.keys(graph.imports ?? {}), + ...Object.keys(graph.importedBy ?? {}), + ...chunks.flatMap((chunk) => + typeof chunk.relativePath === 'string' && chunk.relativePath.length > 0 + ? [chunk.relativePath] + : [] + ) + ]).size; + const recordedFiles = statsSource.files ?? 0; const graphStats = { - files: statsSource.files ?? 0, + files: recordedFiles > 0 ? recordedFiles : countedFiles, edges: statsSource.edges ?? 0, avgDependencies: statsSource.avgDependencies ?? 0 }; diff --git a/src/core/search-quality.ts b/src/core/search-quality.ts index 49c1938..f7e9e80 100644 --- a/src/core/search-quality.ts +++ b/src/core/search-quality.ts @@ -18,6 +18,9 @@ export function isTestArtifactPath(filePath: string): boolean { ); } +const LOW_CONFIDENCE_HINT = + 'Low confidence: this search is not evidence that the code is missing. Try one concrete symbol, file, or route, or call get_symbol_references.'; + export function assessSearchQuality( query: string, results: SearchResult[] @@ -28,7 +31,7 @@ export function assessSearchQuality( confidence: 0, signals: ['no results returned'], nextSteps: [ - 'Try a narrower query with one concrete symbol, route, or file hint.', + LOW_CONFIDENCE_HINT, 'Apply search filters (framework/language/componentType/layer).', 'Use get_symbol_references to find where a specific symbol is used across the codebase.' ] @@ -74,7 +77,7 @@ export function assessSearchQuality( signals, ...(lowConfidence && { nextSteps: [ - 'Add one or two concrete symbols, routes, or file hints to the query.', + LOW_CONFIDENCE_HINT, 'Apply filters (framework/language/componentType/layer) to narrow candidates.', 'Use get_symbol_references when the question is about where a symbol or function is used.' ] diff --git a/src/index.ts b/src/index.ts index 3e7a989..038f8f6 100644 --- a/src/index.ts +++ b/src/index.ts @@ -81,7 +81,7 @@ let mcpRuntimeReady = false; let mcpRuntimePromise: Promise | undefined; // Flags that are NOT project paths — skip them when resolving the bootstrap root. -const KNOWN_FLAGS = new Set(['--http', '--port', '--help']); +const KNOWN_FLAGS = new Set(['--http', '--port', '--help', '--version', '-v']); const CLI_SUBCOMMANDS = [ 'memory', 'search', @@ -105,7 +105,10 @@ const directSubcommand = process.argv[2]; const isDirectCliSubcommand = isDirectRun && typeof directSubcommand === 'string' && - (CLI_SUBCOMMANDS.includes(directSubcommand) || directSubcommand === '--help'); + (CLI_SUBCOMMANDS.includes(directSubcommand) || + directSubcommand === '--help' || + directSubcommand === '--version' || + directSubcommand === '-v'); // Resolve optional bootstrap root with validation handled later in main(). function resolveRootPath(): string | undefined { @@ -1985,7 +1988,10 @@ if (!isDirectCliSubcommand) { if (isDirectRun) { const subcommand = process.argv[2]; - if (CLI_SUBCOMMANDS.includes(subcommand) || subcommand === '--help') { + if (subcommand === '--version' || subcommand === '-v') { + console.log(PKG_VERSION); + process.exit(0); + } else if (CLI_SUBCOMMANDS.includes(subcommand) || subcommand === '--help') { handleCliCommand(process.argv.slice(2)).catch((error) => { console.error('Error:', error instanceof Error ? error.message : String(error)); process.exit(1); diff --git a/tests/cli-init-windows.test.ts b/tests/cli-init-windows.test.ts new file mode 100644 index 0000000..548e27f --- /dev/null +++ b/tests/cli-init-windows.test.ts @@ -0,0 +1,29 @@ +import path from 'node:path'; +import { describe, expect, it } from 'vitest'; +import { generateMcpConfig } from '../src/cli-init.js'; + +describe('saved launch arguments', () => { + it.each(['An App', 'App&MissingCbcLaunchCommand', 'App%cbc_literal%End'])( + 'keeps the repository path %s as one argument beside the npx cache path', + (name) => { + const root = path.resolve('C:/projects', name); + const nodeExecutable = path.resolve('C:/Program Files/nodejs/node.exe'); + const cacheEntrypoint = path.resolve( + 'C:/Users/dev/AppData/Local/npm-cache/_npx/hash/node_modules/codebase-context/dist/index.js' + ); + const generated = generateMcpConfig('claude-code', { + rootPath: root, + nodeExecutable, + serverEntrypoint: cacheEntrypoint + }); + if (generated.kind !== 'file') throw new Error('expected file config'); + const server = JSON.parse(generated.content).mcpServers['codebase-context'] as { + command: string; + args: string[]; + }; + + expect(server.command).toBe(nodeExecutable); + expect(server.args).toEqual([cacheEntrypoint, root]); + } + ); +}); diff --git a/tests/cli-init.test.ts b/tests/cli-init.test.ts index 8d139a2..aeb20eb 100644 --- a/tests/cli-init.test.ts +++ b/tests/cli-init.test.ts @@ -1,4 +1,4 @@ -import { describe, it, expect, vi, beforeEach } from 'vitest'; +import { describe, it, expect, vi, beforeEach, afterAll } from 'vitest'; import path from 'path'; // Mock node:fs/promises at the top level so Vitest can hoist it correctly. @@ -7,6 +7,7 @@ vi.mock('node:fs/promises', async (importOriginal) => { return { ...actual, access: vi.fn(), + stat: vi.fn(), readFile: vi.fn(), writeFile: vi.fn().mockResolvedValue(undefined), mkdir: vi.fn().mockResolvedValue(undefined) @@ -14,18 +15,16 @@ vi.mock('node:fs/promises', async (importOriginal) => { }); vi.mock('@inquirer/prompts', () => ({ - select: vi.fn(), - confirm: vi.fn() + checkbox: vi.fn() })); import * as fsMod from 'node:fs/promises'; -import { confirm, select } from '@inquirer/prompts'; +import { checkbox } from '@inquirer/prompts'; import { + CLIENTS, generateMcpConfig, - generateInstructionBlock, - resolveInstructionFilePath, + detectClients, resolveInitRoot, - _appendInstructionBlock, _buildMergedMcpContent, _buildMergedCodexContent, handleInitCli @@ -33,17 +32,24 @@ import { // --- generateMcpConfig --- +const nodeExecutable = path.resolve('C:/Program Files/nodejs/node.exe'); +const cacheEntrypoint = path.resolve( + 'C:/npm-cache/_npx/hash/node_modules/codebase-context/dist/index.js' +); + +function savedLaunch(rootPath: string) { + return { + rootPath, + nodeExecutable, + serverEntrypoint: cacheEntrypoint + }; +} + describe('generateMcpConfig', () => { const projectRoot = path.resolve('C:/projects/Example Project'); - const nodeExecutable = path.resolve('C:/Program Files/nodejs/node.exe'); - const serverEntrypoint = path.resolve('C:/Program Files/codebase-context/dist/index.js'); - - it('Claude Code gets a project stdio config that passes the project root', () => { - const result = generateMcpConfig('claude-code', { - rootPath: projectRoot, - nodeExecutable, - serverEntrypoint - }); + + it('saves the npx cache path for Claude Code', () => { + const result = generateMcpConfig('claude-code', savedLaunch(projectRoot)); expect(result.kind).toBe('file'); if (result.kind !== 'file') return; @@ -52,47 +58,62 @@ describe('generateMcpConfig', () => { mcpServers: { 'codebase-context': { command: nodeExecutable, - args: [serverEntrypoint, projectRoot] + args: [cacheEntrypoint, projectRoot] } } }); }); - it('cursor returns a file at .cursor/mcp.json with http type', () => { - const result = generateMcpConfig('cursor'); + it('Cursor gets a project stdio config that saves the npx cache path', () => { + const result = generateMcpConfig('cursor', savedLaunch(projectRoot)); expect(result.kind).toBe('file'); if (result.kind !== 'file') return; - expect(result.path).toBe(path.resolve('.cursor/mcp.json')); + expect(result.path).toBe(path.join(projectRoot, '.cursor', 'mcp.json')); + const parsed = JSON.parse(result.content) as { + mcpServers: { 'codebase-context': { type: string; command: string; args: string[] } }; + }; + expect(parsed.mcpServers['codebase-context']).toEqual({ + type: 'stdio', + command: nodeExecutable, + args: [cacheEntrypoint, projectRoot] + }); + }); + + it('defaults to the current Node binary and the running package file', () => { + const result = generateMcpConfig('claude-code', { rootPath: projectRoot }); + if (result.kind !== 'file') throw new Error('expected file config'); const parsed = JSON.parse(result.content) as { - mcpServers: { 'codebase-context': { type: string } }; + mcpServers: { 'codebase-context': { command: string; args: string[] } }; }; - expect(parsed.mcpServers['codebase-context'].type).toBe('http'); + const server = parsed.mcpServers['codebase-context']; + expect(server.command).toBe(path.resolve(process.execPath)); + expect(server.args?.[0]).toBe(path.resolve(process.argv[1] ?? '')); + expect(server.args?.[1]).toBe(projectRoot); }); - it('Codex gets a project TOML config that passes paths containing spaces safely', () => { - const result = generateMcpConfig('codex', { - rootPath: projectRoot, - nodeExecutable, - serverEntrypoint - }); + it('Codex saves the node binary and the npx cache path', () => { + const result = generateMcpConfig('codex', savedLaunch(projectRoot)); expect(result.kind).toBe('file'); if (result.kind !== 'file') return; expect(result.path).toBe(path.join(projectRoot, '.codex', 'config.toml')); expect(result.content).toBe( - `[mcp_servers.codebase-context]\ncommand = ${JSON.stringify(nodeExecutable)}\nargs = [${JSON.stringify(serverEntrypoint)}, ${JSON.stringify(projectRoot)}]\nstartup_timeout_sec = 120\n` + `[mcp_servers.codebase-context]\ncommand = ${JSON.stringify(nodeExecutable)}\nargs = [${JSON.stringify(cacheEntrypoint)}, ${JSON.stringify(projectRoot)}]\nstartup_timeout_sec = 120\n` ); }); - it('opencode returns a file at opencode.json with remote type', () => { - const result = generateMcpConfig('opencode'); + it('OpenCode uses a local command that saves the npx cache path', () => { + const result = generateMcpConfig('opencode', savedLaunch(projectRoot)); expect(result.kind).toBe('file'); if (result.kind !== 'file') return; - expect(result.path).toBe(path.resolve('opencode.json')); + expect(result.path).toBe(path.join(projectRoot, 'opencode.json')); const parsed = JSON.parse(result.content) as { - mcp: { 'codebase-context': { type: string } }; + mcp: { 'codebase-context': { type: string; command: string[] } }; }; - expect(parsed.mcp['codebase-context'].type).toBe('remote'); + expect(parsed.mcp['codebase-context']).toEqual({ + type: 'local', + command: [nodeExecutable, cacheEntrypoint, projectRoot] + }); }); }); @@ -122,90 +143,6 @@ describe('resolveInitRoot', () => { }); }); -// --- generateInstructionBlock --- - -describe('generateInstructionBlock', () => { - it('contains opening and closing delimiters', () => { - const block = generateInstructionBlock(); - expect(block).toContain(''); - expect(block).toContain(''); - }); - - it('contains all five tool call rules', () => { - const block = generateInstructionBlock(); - expect(block).toContain('get_memory'); - expect(block).toContain('search_codebase'); - expect(block).toContain('get_team_patterns'); - expect(block).toContain('remember'); - expect(block).toContain('detect_circular_dependencies'); - }); -}); - -// --- resolveInstructionFilePath --- - -describe('resolveInstructionFilePath', () => { - const cwd = '/test-cwd'; - - it('claude-code resolves to CLAUDE.md', () => { - const result = resolveInstructionFilePath('claude-code', cwd); - expect(result).toBe(path.join(cwd, 'CLAUDE.md')); - }); - - it('cursor resolves to .cursorrules', () => { - const result = resolveInstructionFilePath('cursor', cwd); - expect(result).toBe(path.join(cwd, '.cursorrules')); - }); - - it('codex resolves to AGENTS.md', () => { - const result = resolveInstructionFilePath('codex', cwd); - expect(result).toBe(path.join(cwd, 'AGENTS.md')); - }); - - it('opencode returns null', () => { - const result = resolveInstructionFilePath('opencode', cwd); - expect(result).toBeNull(); - }); -}); - -// --- _appendInstructionBlock file-write behavior --- - -describe('_appendInstructionBlock', () => { - const accessMock = vi.mocked(fsMod.access); - const readFileMock = vi.mocked(fsMod.readFile); - const writeFileMock = vi.mocked(fsMod.writeFile); - - beforeEach(() => { - vi.resetAllMocks(); - writeFileMock.mockResolvedValue(undefined); - }); - - it('appends block when file exists and block not yet present', async () => { - // Simulate file exists (access resolves) and has content without the delimiter - accessMock.mockResolvedValue(undefined); - readFileMock.mockResolvedValue('# existing content' as unknown as Buffer); - - await _appendInstructionBlock('/test/CLAUDE.md'); - - expect(writeFileMock).toHaveBeenCalledOnce(); - const callArgs = writeFileMock.mock.calls[0] as [string, string, ...unknown[]]; - const content = callArgs[1]; - expect(content).toContain('# existing content'); - expect(content).toContain(''); - }); - - it('skips when block already present', async () => { - accessMock.mockResolvedValue(undefined); - readFileMock.mockResolvedValue( - '# existing content\n\nsome block\n' as unknown as Buffer - ); - - const result = await _appendInstructionBlock('/test/CLAUDE.md'); - - expect(result).toBe('skipped'); - expect(writeFileMock).not.toHaveBeenCalled(); - }); -}); - describe('_buildMergedMcpContent', () => { const readFileMock = vi.mocked(fsMod.readFile); @@ -223,7 +160,7 @@ describe('_buildMergedMcpContent', () => { }) as unknown as Buffer ); - const generated = generateMcpConfig('cursor'); + const generated = generateMcpConfig('cursor', { platform: 'linux' }); if (generated.kind !== 'file') throw new Error('expected file config'); const merged = await _buildMergedMcpContent( @@ -232,7 +169,7 @@ describe('_buildMergedMcpContent', () => { 'cursor' ); const parsed = JSON.parse(merged.content) as { - mcpServers: Record; + mcpServers: Record; someOtherKey: boolean; }; @@ -242,9 +179,9 @@ describe('_buildMergedMcpContent', () => { type: 'http', url: 'http://127.0.0.1:4000/mcp' }); - expect(parsed.mcpServers['codebase-context']).toEqual({ - type: 'http', - url: 'http://127.0.0.1:3100/mcp' + expect(parsed.mcpServers['codebase-context']).toMatchObject({ + type: 'stdio', + command: path.resolve(process.execPath) }); }); @@ -258,11 +195,10 @@ describe('_buildMergedMcpContent', () => { }) as unknown as Buffer ); - const generated = generateMcpConfig('claude-code', { - rootPath: path.resolve('C:/projects/app'), - nodeExecutable: 'C:/node.exe', - serverEntrypoint: path.resolve('C:/cbc/index.js') - }); + const generated = generateMcpConfig( + 'claude-code', + savedLaunch(path.resolve('C:/projects/app')) + ); if (generated.kind !== 'file') throw new Error('expected file config'); const merged = await _buildMergedMcpContent( @@ -283,8 +219,8 @@ describe('_buildMergedMcpContent', () => { args: ['server.js'] }); expect(parsed.mcpServers['codebase-context']).toEqual({ - command: 'C:/node.exe', - args: [path.resolve('C:/cbc/index.js'), path.resolve('C:/projects/app')] + command: nodeExecutable, + args: [cacheEntrypoint, path.resolve('C:/projects/app')] }); }); @@ -303,8 +239,7 @@ describe('_buildMergedMcpContent', () => { ); const generated = generateMcpConfig('claude-code', { rootPath: path.resolve('C:/projects/app'), - nodeExecutable: 'C:/node.exe', - serverEntrypoint: path.resolve('C:/cbc/index.js') + platform: 'win32' }); if (generated.kind !== 'file') throw new Error('expected file config'); @@ -332,6 +267,156 @@ describe('_buildMergedMcpContent', () => { _buildMergedMcpContent('/test/opencode.json', generated.content, 'opencode') ).rejects.toThrow(/already contains a different codebase-context configuration/i); }); + + it('treats a custom Cursor HTTP entry as a conflict', async () => { + readFileMock.mockResolvedValue( + JSON.stringify({ + mcpServers: { 'codebase-context': { type: 'http', url: 'https://custom/mcp' } } + }) as unknown as Buffer + ); + const generated = generateMcpConfig('cursor', { rootPath: '/test', platform: 'linux' }); + if (generated.kind !== 'file') throw new Error('expected file config'); + + await expect( + _buildMergedMcpContent('/test/.cursor/mcp.json', generated.content, 'cursor') + ).rejects.toThrow(/already contains a different codebase-context configuration/i); + }); + + it('updates a generated JSON launch when the project root moves and preserves entry extras', async () => { + const oldRoot = path.resolve('C:/projects/Old App'); + const newRoot = path.resolve('C:/projects/New App'); + readFileMock.mockResolvedValue( + JSON.stringify({ + customSetting: true, + mcpServers: { + 'codebase-context': { + command: 'C:/node.exe', + args: ['C:/cache/node_modules/codebase-context/dist/index.js', oldRoot], + customTimeout: 8 + } + } + }) as unknown as Buffer + ); + const generated = generateMcpConfig('claude-code', savedLaunch(newRoot)); + if (generated.kind !== 'file') throw new Error('expected file config'); + + const merged = await _buildMergedMcpContent( + '/test/.mcp.json', + generated.content, + 'claude-code' + ); + const parsed = JSON.parse(merged.content) as { + customSetting: boolean; + mcpServers: { 'codebase-context': Record }; + }; + expect(parsed.customSetting).toBe(true); + expect(parsed.mcpServers['codebase-context']).toEqual({ + command: nodeExecutable, + args: [cacheEntrypoint, newRoot], + customTimeout: 8 + }); + }); + + it('updates a previously generated pinned npx command when the project root moves', async () => { + const oldRoot = path.resolve('C:/projects/Old Pinned App'); + const newRoot = path.resolve('C:/projects/New Pinned App'); + readFileMock.mockResolvedValue( + JSON.stringify({ + mcpServers: { + 'codebase-context': { + command: 'npx', + args: ['--yes', 'codebase-context@2.3.0', oldRoot], + customLabel: 'preserve' + } + } + }) as unknown as Buffer + ); + const generated = generateMcpConfig('claude-code', savedLaunch(newRoot)); + if (generated.kind !== 'file') throw new Error('expected file config'); + + const merged = await _buildMergedMcpContent( + '/test/.mcp.json', + generated.content, + 'claude-code' + ); + const parsed = JSON.parse(merged.content) as { + mcpServers: { 'codebase-context': { command: string; args: string[]; customLabel: string } }; + }; + expect(parsed.mcpServers['codebase-context']).toEqual({ + command: nodeExecutable, + args: [cacheEntrypoint, newRoot], + customLabel: 'preserve' + }); + }); +}); + +describe('detectClients', () => { + const accessMock = vi.mocked(fsMod.access); + const missing = () => Object.assign(new Error('missing'), { code: 'ENOENT' }); + + beforeEach(() => { + vi.resetAllMocks(); + accessMock.mockRejectedValue(missing()); + vi.mocked(fsMod.stat).mockImplementation(async (candidate) => { + const executable = String(candidate) + .replace(/\\/g, '/') + .toLowerCase() + .startsWith('c:/tools/'); + return { isFile: () => executable, isDirectory: () => !executable } as Awaited< + ReturnType + >; + }); + }); + + it('uses one four-client table and detects installed tools through PATH candidates', async () => { + accessMock.mockImplementation(async (candidate) => { + const pathValue = String(candidate).toLowerCase(); + if (pathValue.startsWith('c:\\tools\\') && pathValue.endsWith('codex.exe')) return undefined; + throw missing(); + }); + + const clients = await detectClients('/project', { PATH: 'C:/tools' }, 'win32'); + + expect(CLIENTS.map(({ id }) => id)).toEqual(['claude-code', 'codex', 'cursor', 'opencode']); + expect(clients).toContain('codex'); + expect(clients).not.toContain('claude-code'); + expect( + accessMock.mock.calls.some(([candidate]) => String(candidate).includes('C:\\tools')) + ).toBe(true); + }); + + it.each([ + { + name: 'CODEX_HOME', + env: { PATH: '', CODEX_HOME: 'C:/custom-codex-home' }, + marker: 'custom-codex-home', + client: 'codex' + }, + { + name: 'CLAUDE_CONFIG_DIR', + env: { PATH: '', CLAUDE_CONFIG_DIR: 'C:/custom-claude-config' }, + marker: 'custom-claude-config', + client: 'claude-code' + }, + { + name: 'XDG_CONFIG_HOME', + env: { PATH: '', XDG_CONFIG_HOME: 'C:/custom-xdg-config' }, + marker: 'custom-xdg-config', + client: 'opencode' + } + ])('checks $name when detecting installed clients', async ({ env, marker, client }) => { + accessMock.mockImplementation(async (candidate) => { + if (String(candidate).toLowerCase().includes(marker)) return undefined; + throw missing(); + }); + + const clients = await detectClients('/project', env, 'win32'); + + expect(clients).toContain(client); + expect(accessMock.mock.calls.some(([candidate]) => String(candidate).includes(marker))).toBe( + true + ); + }); }); describe('handleInitCli', () => { @@ -339,118 +424,239 @@ describe('handleInitCli', () => { const writeFileMock = vi.mocked(fsMod.writeFile); const mkdirMock = vi.mocked(fsMod.mkdir); const accessMock = vi.mocked(fsMod.access); + const statMock = vi.mocked(fsMod.stat); + const previousExitCode = process.exitCode; + const previousIsTTY = process.stdin.isTTY; + const previousStdoutIsTTY = process.stdout.isTTY; + + const missing = () => Object.assign(new Error('missing'), { code: 'ENOENT' }); + const detectDirectories = (...names: string[]) => { + accessMock.mockImplementation(async (candidate) => { + const basename = String(candidate).toLowerCase().split(/[\\/]/).at(-1); + if (names.some((name) => basename === (name === 'opencode' ? 'opencode' : `.${name}`))) + return undefined; + throw missing(); + }); + }; beforeEach(() => { vi.resetAllMocks(); writeFileMock.mockResolvedValue(undefined); mkdirMock.mockResolvedValue(undefined); - readFileMock.mockRejectedValue(Object.assign(new Error('missing'), { code: 'ENOENT' })); + readFileMock.mockRejectedValue(missing()); + accessMock.mockRejectedValue(missing()); + statMock.mockResolvedValue({ isDirectory: () => true } as Awaited< + ReturnType + >); + process.exitCode = undefined; + Object.defineProperty(process.stdin, 'isTTY', { configurable: true, value: false }); + Object.defineProperty(process.stdout, 'isTTY', { configurable: true, value: false }); + }); + + afterAll(() => { + process.exitCode = previousExitCode; + Object.defineProperty(process.stdin, 'isTTY', { configurable: true, value: previousIsTTY }); + Object.defineProperty(process.stdout, 'isTTY', { + configurable: true, + value: previousStdoutIsTTY + }); }); - it('writes only the selected project config in non-interactive mode', async () => { + it('configures all detected clients with --yes', async () => { const projectRoot = path.resolve('C:/projects/Configured App'); + detectDirectories('claude', 'codex'); - await handleInitCli(['--client', 'claude-code', '--yes', '--root', projectRoot]); + await handleInitCli(['--yes', '--root', projectRoot]); - expect(select).not.toHaveBeenCalled(); - expect(confirm).not.toHaveBeenCalled(); - expect(writeFileMock).toHaveBeenCalledOnce(); - const [writtenPath, writtenContent] = writeFileMock.mock.calls[0] as [ - string, - string, - ...unknown[] - ]; - expect(writtenPath).toBe(path.join(projectRoot, '.mcp.json')); - const parsed = JSON.parse(writtenContent) as { - mcpServers: { 'codebase-context': { args: string[] } }; - }; - expect(parsed.mcpServers['codebase-context'].args.at(-1)).toBe(projectRoot); - expect(mkdirMock).toHaveBeenCalledWith(projectRoot, { recursive: true }); + expect(checkbox).not.toHaveBeenCalled(); + expect(writeFileMock.mock.calls.map(([filePath]) => filePath)).toEqual( + expect.arrayContaining([ + path.join(projectRoot, '.mcp.json'), + path.join(projectRoot, '.codex', 'config.toml') + ]) + ); + expect(writeFileMock).toHaveBeenCalledTimes(2); }); - it('writes a project Codex TOML file when Codex is selected non-interactively', async () => { - const projectRoot = path.resolve('C:/projects/Codex App'); + it('configures all detected clients without a terminal even without --yes', async () => { + const projectRoot = path.resolve('C:/projects/No Terminal'); + detectDirectories('cursor', 'opencode'); - await handleInitCli(['--client', 'codex', '--yes', '--root', projectRoot]); + await handleInitCli(['--root', projectRoot]); + expect(checkbox).not.toHaveBeenCalled(); + expect(writeFileMock.mock.calls.map(([filePath]) => filePath)).toEqual( + expect.arrayContaining([ + path.join(projectRoot, '.cursor', 'mcp.json'), + path.join(projectRoot, 'opencode.json') + ]) + ); + expect(writeFileMock).toHaveBeenCalledTimes(2); + }); + + it('lets repeated --client options override detection', async () => { + const projectRoot = path.resolve('C:/projects/Explicit Clients'); + + await handleInitCli([ + '--client', + 'claude-code', + '--client', + 'codex', + '--yes', + '--root', + projectRoot + ]); + + expect(accessMock).not.toHaveBeenCalled(); + expect(writeFileMock.mock.calls.map(([filePath]) => filePath)).toEqual( + expect.arrayContaining([ + path.join(projectRoot, '.mcp.json'), + path.join(projectRoot, '.codex', 'config.toml') + ]) + ); + expect(writeFileMock).toHaveBeenCalledTimes(2); + }); + + it('reports no detected clients without writing when there is no terminal', async () => { + await handleInitCli(['--yes', '--root', path.resolve('C:/projects/Empty')]); + + expect(writeFileMock).not.toHaveBeenCalled(); + expect(mkdirMock).not.toHaveBeenCalled(); + expect(process.exitCode).toBe(1); + }); + + it('shows detected clients checked in the terminal multi-select', async () => { + detectDirectories('claude', 'codex'); + vi.mocked(checkbox).mockResolvedValue(['claude-code']); + Object.defineProperty(process.stdin, 'isTTY', { configurable: true, value: true }); + Object.defineProperty(process.stdout, 'isTTY', { configurable: true, value: true }); + + await handleInitCli(['--root', path.resolve('C:/projects/Interactive')]); + + const choices = vi.mocked(checkbox).mock.calls[0]?.[0].choices; + expect(choices).toEqual( + expect.arrayContaining([ + expect.objectContaining({ value: 'claude-code', checked: true }), + expect.objectContaining({ value: 'codex', checked: true }), + expect.objectContaining({ value: 'cursor', checked: false }), + expect.objectContaining({ value: 'opencode', checked: false }) + ]) + ); expect(writeFileMock).toHaveBeenCalledOnce(); - const [writtenPath, writtenContent] = writeFileMock.mock.calls[0] as [ - string, - string, - ...unknown[] - ]; - expect(writtenPath).toBe(path.join(projectRoot, '.codex', 'config.toml')); - expect(writtenContent).toContain('[mcp_servers.codebase-context]'); - expect(writtenContent).toContain(`args = [`); - expect(writtenContent).toContain(JSON.stringify(projectRoot)); - expect(mkdirMock).toHaveBeenCalledWith(path.join(projectRoot, '.codex'), { recursive: true }); - }); - - it('uses cwd as the project when init is run without a root override', async () => { - const previousEnvRoot = process.env.CODEBASE_ROOT; - delete process.env.CODEBASE_ROOT; + }); - try { - await handleInitCli(['--client', 'codex', '--yes']); - } finally { - if (previousEnvRoot === undefined) delete process.env.CODEBASE_ROOT; - else process.env.CODEBASE_ROOT = previousEnvRoot; - } + it('leaves all no-detection terminal choices unchecked and writes nothing on empty selection', async () => { + vi.mocked(checkbox).mockResolvedValue([]); + Object.defineProperty(process.stdin, 'isTTY', { configurable: true, value: true }); + Object.defineProperty(process.stdout, 'isTTY', { configurable: true, value: true }); + + await handleInitCli(['--root', path.resolve('C:/projects/Cancelled')]); - const [writtenPath, writtenContent] = writeFileMock.mock.calls[0] as [ - string, - string, - ...unknown[] - ]; - expect(writtenPath).toBe(path.join(process.cwd(), '.codex', 'config.toml')); - expect(writtenContent).toContain(JSON.stringify(path.resolve(process.cwd()))); + const choices = vi.mocked(checkbox).mock.calls[0]?.[0].choices; + expect(choices).toEqual( + expect.arrayContaining([ + expect.objectContaining({ value: 'claude-code', checked: false }), + expect.objectContaining({ value: 'codex', checked: false }), + expect.objectContaining({ value: 'cursor', checked: false }), + expect.objectContaining({ value: 'opencode', checked: false }) + ]) + ); + expect(writeFileMock).not.toHaveBeenCalled(); + expect(mkdirMock).not.toHaveBeenCalled(); + expect(process.exitCode).toBe(1); }); - it('does not overwrite a conflicting project MCP entry', async () => { - const projectRoot = path.resolve('C:/projects/Configured App'); - readFileMock.mockResolvedValue( - JSON.stringify({ - mcpServers: { - 'codebase-context': { disabled: true, command: 'old-server' } - } - }) as unknown as Buffer + it('does not write when the terminal prompt throws its cancellation error', async () => { + detectDirectories('claude'); + vi.mocked(checkbox).mockRejectedValue( + Object.assign(new Error('User force closed the prompt'), { name: 'ExitPromptError' }) ); - const previousExitCode = process.exitCode; + Object.defineProperty(process.stdin, 'isTTY', { configurable: true, value: true }); + Object.defineProperty(process.stdout, 'isTTY', { configurable: true, value: true }); - await handleInitCli(['--client', 'claude-code', '--yes', '--root', projectRoot]); + await handleInitCli(['--root', path.resolve('C:/projects/Prompt Cancelled')]); + expect(writeFileMock).not.toHaveBeenCalled(); + expect(mkdirMock).not.toHaveBeenCalled(); + }); + + it('requires an existing directory for --root', async () => { + statMock.mockResolvedValue({ isDirectory: () => false } as Awaited< + ReturnType + >); + + await handleInitCli(['--client', 'codex', '--yes', '--root', path.resolve('C:/projects/File')]); + + expect(mkdirMock).not.toHaveBeenCalled(); expect(writeFileMock).not.toHaveBeenCalled(); expect(process.exitCode).toBe(1); - process.exitCode = previousExitCode; }); - it('does not create a project config when the selected root is missing', async () => { - const projectRoot = path.resolve('C:/projects/Missing App'); - accessMock.mockRejectedValue(Object.assign(new Error('missing'), { code: 'ENOENT' })); - const previousExitCode = process.exitCode; + it('does not create configs when --root does not exist', async () => { + statMock.mockRejectedValue(missing()); - await handleInitCli(['--client', 'codex', '--yes', '--root', projectRoot]); + await handleInitCli([ + '--client', + 'codex', + '--yes', + '--root', + path.resolve('C:/projects/Missing Directory') + ]); expect(mkdirMock).not.toHaveBeenCalled(); expect(writeFileMock).not.toHaveBeenCalled(); expect(process.exitCode).toBe(1); - process.exitCode = previousExitCode; }); - it('shows init help without prompting or writing files', async () => { - await handleInitCli(['--help']); + it('leaves an identical generated config untouched', async () => { + const projectRoot = path.resolve('C:/projects/Already Configured'); + const generated = generateMcpConfig('claude-code', { rootPath: projectRoot }); + readFileMock.mockResolvedValue(generated.content as unknown as Buffer); + + await handleInitCli(['--client', 'claude-code', '--yes', '--root', projectRoot]); - expect(select).not.toHaveBeenCalled(); - expect(confirm).not.toHaveBeenCalled(); expect(writeFileMock).not.toHaveBeenCalled(); }); - it('keeps the interactive preview read-only when the user declines', async () => { - vi.mocked(select).mockResolvedValue('claude-code'); - vi.mocked(confirm).mockResolvedValue(false); + it('prepares every selected config before writing when a later client conflicts', async () => { + const projectRoot = path.resolve('C:/projects/Batch Conflict'); + const opencodePath = path.join(projectRoot, 'opencode.json'); + readFileMock.mockImplementation(async (filePath) => { + if (String(filePath) === opencodePath) { + return JSON.stringify({ + mcp: { 'codebase-context': { type: 'remote', url: 'https://custom' } } + }) as unknown as Buffer; + } + throw missing(); + }); + + await handleInitCli([ + '--client', + 'claude-code', + '--client', + 'opencode', + '--yes', + '--root', + projectRoot + ]); + + expect(process.exitCode).toBe(1); + expect(mkdirMock).not.toHaveBeenCalled(); + expect(writeFileMock).not.toHaveBeenCalled(); + }); - await handleInitCli([]); + it('shows help without prompting or writing files', async () => { + const log = vi.spyOn(console, 'log').mockImplementation(() => undefined); + try { + await handleInitCli(['--help']); + expect(log.mock.calls.flat().join('\n')).toContain( + 'codex mcp add codebase-context -- npx -y codebase-context' + ); + } finally { + log.mockRestore(); + } + expect(checkbox).not.toHaveBeenCalled(); expect(writeFileMock).not.toHaveBeenCalled(); }); }); @@ -459,11 +665,15 @@ describe('_buildMergedCodexContent', () => { const readFileMock = vi.mocked(fsMod.readFile); const writeFileMock = vi.mocked(fsMod.writeFile); const mkdirMock = vi.mocked(fsMod.mkdir); + const statMock = vi.mocked(fsMod.stat); beforeEach(() => { vi.resetAllMocks(); writeFileMock.mockResolvedValue(undefined); mkdirMock.mockResolvedValue(undefined); + statMock.mockResolvedValue({ isDirectory: () => true } as Awaited< + ReturnType + >); readFileMock.mockRejectedValue(Object.assign(new Error('missing'), { code: 'ENOENT' })); }); @@ -492,8 +702,7 @@ describe('_buildMergedCodexContent', () => { ); const generated = generateMcpConfig('codex', { rootPath: 'C:/projects/app', - nodeExecutable: 'C:/node.exe', - serverEntrypoint: 'C:/cbc/index.js' + platform: 'win32' }); if (generated.kind !== 'file') throw new Error('expected file config'); @@ -508,8 +717,7 @@ describe('_buildMergedCodexContent', () => { it('leaves the same Codex codebase-context entry unchanged on a repeated setup', async () => { const generated = generateMcpConfig('codex', { rootPath: path.resolve('C:/projects/app'), - nodeExecutable: 'C:/node.exe', - serverEntrypoint: path.resolve('C:/cbc/index.js') + platform: 'win32' }); if (generated.kind !== 'file') throw new Error('expected file config'); readFileMock.mockResolvedValue(generated.content as unknown as Buffer); @@ -522,8 +730,7 @@ describe('_buildMergedCodexContent', () => { it('treats quoted table keys as the same idempotent Codex entry', async () => { const generated = generateMcpConfig('codex', { rootPath: path.resolve('C:/projects/app'), - nodeExecutable: 'C:/node.exe', - serverEntrypoint: path.resolve('C:/cbc/index.js') + platform: 'win32' }); if (generated.kind !== 'file') throw new Error('expected file config'); const existing = generated.content.replace( @@ -543,8 +750,7 @@ describe('_buildMergedCodexContent', () => { ); const generated = generateMcpConfig('codex', { rootPath: 'C:/projects/app', - nodeExecutable: 'C:/node.exe', - serverEntrypoint: 'C:/cbc/index.js' + platform: 'win32' }); if (generated.kind !== 'file') throw new Error('expected file config'); @@ -553,6 +759,29 @@ describe('_buildMergedCodexContent', () => { ); }); + it('updates a generated Codex launch for a moved root while preserving custom settings', async () => { + const oldRoot = path.resolve('C:/projects/Old Codex App'); + const newRoot = path.resolve('C:/projects/New Codex App'); + const existing = + '[mcp_servers.codebase-context]\n' + + 'command = "node"\n' + + `args = ["C:/cache/node_modules/codebase-context/dist/index.js", ${JSON.stringify(oldRoot)}]\n` + + 'startup_timeout_sec = 75\n' + + 'custom_label = "keep me"\n'; + readFileMock.mockResolvedValue(existing as unknown as Buffer); + const generated = generateMcpConfig('codex', savedLaunch(newRoot)); + if (generated.kind !== 'file') throw new Error('expected file config'); + + const merged = await _buildMergedCodexContent(generated.path, generated.content); + + expect(merged.content).toContain(`command = ${JSON.stringify(nodeExecutable)}`); + expect(merged.content).toContain( + `args = [${JSON.stringify(cacheEntrypoint)}, ${JSON.stringify(newRoot)}]` + ); + expect(merged.content).toContain('startup_timeout_sec = 75'); + expect(merged.content).toContain('custom_label = "keep me"'); + }); + it.each([ { name: 'double-quoted parent and child keys', diff --git a/tests/cli-map-export.test.ts b/tests/cli-map-export.test.ts index 78dd9cc..54b8e88 100644 --- a/tests/cli-map-export.test.ts +++ b/tests/cli-map-export.test.ts @@ -1,5 +1,5 @@ import { describe, it, expect, afterEach } from 'vitest'; -import { mkdtemp, rm, access, readFile } from 'fs/promises'; +import { mkdtemp, rm, access, readFile, writeFile } from 'fs/promises'; import os from 'os'; import path from 'path'; import { handleMapCli } from '../src/cli-map.js'; @@ -60,4 +60,20 @@ describe('handleMapCli --export', () => { else process.env.CODEBASE_ROOT = original; } }); + + it('indexes a project that has no index before printing the map', async () => { + tmpDir = await mkdtemp(path.join(os.tmpdir(), 'cli-map-index-')); + await writeFile(path.join(tmpDir, 'server.ts'), 'export const port = 3000;\n', 'utf-8'); + const original = process.env.CODEBASE_ROOT; + process.env.CODEBASE_ROOT = tmpDir; + try { + await handleMapCli(['--export']); + await access(path.join(tmpDir, '.codebase-context', 'index.json')); + const content = await readFile(path.join(tmpDir, 'CODEBASE_MAP.md'), 'utf-8'); + expect(content).toMatch(/Files: [1-9]/); + } finally { + if (original === undefined) delete process.env.CODEBASE_ROOT; + else process.env.CODEBASE_ROOT = original; + } + }); }); diff --git a/tests/cli-version.test.ts b/tests/cli-version.test.ts new file mode 100644 index 0000000..c4c56f6 --- /dev/null +++ b/tests/cli-version.test.ts @@ -0,0 +1,26 @@ +import { spawnSync } from 'node:child_process'; +import { readFileSync } from 'node:fs'; +import { resolve } from 'node:path'; +import { describe, expect, it } from 'vitest'; + +const root = resolve(import.meta.dirname, '..'); +const packageJson = JSON.parse(readFileSync(resolve(root, 'package.json'), 'utf8')) as { + version: string; +}; + +describe('version flag', () => { + it('prints the package version and exits', () => { + const result = spawnSync( + process.execPath, + ['--import', 'tsx', resolve(root, 'src/index.ts'), '--version'], + { + cwd: root, + encoding: 'utf8', + timeout: 30_000 + } + ); + + expect(result.status).toBe(0); + expect(result.stdout.trim()).toBe(packageJson.version); + }); +}); diff --git a/tests/codebase-map.test.ts b/tests/codebase-map.test.ts index c90de78..10f5b49 100644 --- a/tests/codebase-map.test.ts +++ b/tests/codebase-map.test.ts @@ -84,15 +84,19 @@ async function removeTempMapProject(rootPath: string): Promise { } function runMapCli(args: string[], rootPath: string) { - return spawnSync(process.execPath, ['--no-deprecation', '--import', 'tsx', ENTRYPOINT, 'map', ...args], { - cwd: CURRENT_REPO_ROOT, - env: { - ...process.env, - CODEBASE_ROOT: rootPath - }, - encoding: 'utf8', - timeout: 120_000 - }); + return spawnSync( + process.execPath, + ['--no-deprecation', '--import', 'tsx', ENTRYPOINT, 'map', ...args], + { + cwd: CURRENT_REPO_ROOT, + env: { + ...process.env, + CODEBASE_ROOT: rootPath + }, + encoding: 'utf8', + timeout: 120_000 + } + ); } // --------------------------------------------------------------------------- @@ -198,6 +202,21 @@ describe('buildCodebaseMap', () => { expect(map.suggestedNextCalls.length).toBeGreaterThan(0); }); + it('counts indexed files when recorded stats say zero', async () => { + const rootPath = await createTempMapProject({ + chunks: [ + { + relativePath: 'src/server.ts', + content: 'export const port = 3000;', + metadata: {} + } as CodeChunk + ], + graph: { stats: { files: 0, edges: 0, avgDependencies: 0 } } + }); + const map = await buildCodebaseMap(createProjectState(rootPath)); + expect(map.graphStats.files).toBe(1); + }); + it('caps suggested next calls at 3', async () => { const project = createProjectState(FIXTURE_ROOT); const map = await buildCodebaseMap(project, { mode: 'full' }); diff --git a/tests/search-quality.test.ts b/tests/search-quality.test.ts index 52adffe..f3703b0 100644 --- a/tests/search-quality.test.ts +++ b/tests/search-quality.test.ts @@ -26,7 +26,7 @@ describe('assessSearchQuality', () => { expect(quality.status).toBe('low_confidence'); expect(quality.confidence).toBe(0); expect(quality.signals).toContain('no results returned'); - expect(quality.nextSteps?.length).toBeGreaterThan(0); + expect(quality.nextSteps?.[0]).toMatch(/not evidence that the code is missing/i); }); it('flags test-artifact dominance for non-testing queries', () => { From d162ddcffece8d003fe53a68c916c2887703f395 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 30 Oct 2026 22:12:24 +0000 Subject: [PATCH 02/15] feat(embeddings): honor OPENAI_BASE_URL for OpenAI-compatible endpoints --- docs/capabilities.md | 2 +- src/embeddings/types.ts | 13 ++++++++++++- tests/embedding-config-env.test.ts | 20 ++++++++++++++++++++ tests/openai-embeddings.test.ts | 19 +++++++++++++++++++ 4 files changed, 52 insertions(+), 2 deletions(-) create mode 100644 tests/embedding-config-env.test.ts diff --git a/docs/capabilities.md b/docs/capabilities.md index 49fc4f9..8032b7b 100644 --- a/docs/capabilities.md +++ b/docs/capabilities.md @@ -318,5 +318,5 @@ Reproducible evaluation is shipped as a CLI entrypoint backed by shared scoring - **Symbol refs are not a call-graph.** `get_symbol_references` counts identifier-node occurrences in the AST (comments/strings excluded via Tree-sitter). It does not distinguish call sites from type annotations, variable assignments, or imports. Full call-site-specific analysis (`call_expression` nodes only) is a roadmap item. - **Impact is 2-hop max.** `computeImpactCandidates` walks direct importers then their importers. Full BFS reachability is on the roadmap. - **Planned for unpublished 2.4.0:** NestJS analyzer support, React and Next.js detection improvements, and reranker fixes. Dedicated Angular, React, and Next.js analyzers already ship in published 2.2.0. Other languages use the Generic analyzer for indexing and retrieval; available symbol extraction depends on parser support. File import resolution and multi-hop navigation currently support TypeScript and JavaScript. Python and Go import navigation are not supported. -- **The unreleased candidate defaults new indexes to model-free mode. Published 2.2.0 enables embeddings by default.** When semantic mode is enabled, its default model is `bge-small-en-v1.5` (512-token context). Granite (8192 context) is opt-in via `EMBEDDING_MODEL`. OpenAI is opt-in via `EMBEDDING_PROVIDER=openai` — sends code externally. +- **The unreleased candidate defaults new indexes to model-free mode. Published 2.2.0 enables embeddings by default.** When semantic mode is enabled, its default model is `bge-small-en-v1.5` (512-token context). Granite (8192 context) is opt-in via `EMBEDDING_MODEL`. OpenAI is opt-in via `EMBEDDING_PROVIDER=openai` with `OPENAI_API_KEY`; use `OPENAI_BASE_URL` for OpenAI-compatible endpoints (Ollama, LiteLLM, proxies) instead of `https://api.openai.com/v1`. - **Patterns are file-level frequency counts.** Not semantic clustering. Rising/Declining trend is derived from git commit recency for files using each pattern, not from usage semantics. diff --git a/src/embeddings/types.ts b/src/embeddings/types.ts index 66e5a7a..2fbe7aa 100644 --- a/src/embeddings/types.ts +++ b/src/embeddings/types.ts @@ -32,10 +32,21 @@ export function parseEmbeddingProviderName( // better conceptual search at the cost of 5-10x slower indexing and higher RAM usage export const DEFAULT_MODEL = process.env.EMBEDDING_MODEL || 'Xenova/bge-small-en-v1.5'; +const DEFAULT_OPENAI_API_ENDPOINT = 'https://api.openai.com/v1'; + +function resolveOpenAiApiEndpoint(): string { + const fromEnv = process.env.OPENAI_BASE_URL?.trim(); + if (!fromEnv) { + return DEFAULT_OPENAI_API_ENDPOINT; + } + return fromEnv.replace(/\/+$/, ''); +} + export const DEFAULT_EMBEDDING_CONFIG: EmbeddingConfig = { provider: parseEmbeddingProviderName(process.env.EMBEDDING_PROVIDER) ?? 'transformers', model: DEFAULT_MODEL, batchSize: 32, maxRetries: 3, - apiKey: process.env.OPENAI_API_KEY + apiKey: process.env.OPENAI_API_KEY, + apiEndpoint: resolveOpenAiApiEndpoint() }; diff --git a/tests/embedding-config-env.test.ts b/tests/embedding-config-env.test.ts new file mode 100644 index 0000000..b71e669 --- /dev/null +++ b/tests/embedding-config-env.test.ts @@ -0,0 +1,20 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; + +describe('DEFAULT_EMBEDDING_CONFIG environment', () => { + afterEach(() => { + vi.unstubAllEnvs(); + vi.resetModules(); + }); + + it('uses OPENAI_BASE_URL when set', async () => { + vi.stubEnv('OPENAI_BASE_URL', 'http://127.0.0.1:11434/v1/'); + const { DEFAULT_EMBEDDING_CONFIG } = await import('../src/embeddings/types.js'); + expect(DEFAULT_EMBEDDING_CONFIG.apiEndpoint).toBe('http://127.0.0.1:11434/v1'); + }); + + it('defaults to the official OpenAI endpoint when OPENAI_BASE_URL is unset', async () => { + vi.unstubAllEnvs(); + const { DEFAULT_EMBEDDING_CONFIG } = await import('../src/embeddings/types.js'); + expect(DEFAULT_EMBEDDING_CONFIG.apiEndpoint).toBe('https://api.openai.com/v1'); + }); +}); diff --git a/tests/openai-embeddings.test.ts b/tests/openai-embeddings.test.ts index 8763efd..1f22261 100644 --- a/tests/openai-embeddings.test.ts +++ b/tests/openai-embeddings.test.ts @@ -46,6 +46,25 @@ describe('OpenAIEmbeddingProvider', () => { expect(result).toEqual([]); }); + it('embedBatch() posts to a custom apiEndpoint base URL', async () => { + const mockFetch = vi.fn().mockReturnValue( + makeFetchResponse({ + data: [{ embedding: [0.1] }] + }) + ); + vi.stubGlobal('fetch', mockFetch); + + const provider = new OpenAIEmbeddingProvider( + 'text-embedding-3-small', + 'sk-test', + 'http://127.0.0.1:11434/v1' + ); + await provider.embedBatch(['hello']); + + const [url] = mockFetch.mock.calls[0] as [string, RequestInit]; + expect(url).toBe('http://127.0.0.1:11434/v1/embeddings'); + }); + it('embedBatch() sends correct Authorization header, model, and encoding_format', async () => { const mockFetch = vi.fn().mockReturnValue( makeFetchResponse({ From 1d765605252c25b824b1ca7a78cdd1ef7f8837df Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 30 Oct 2026 22:12:25 +0000 Subject: [PATCH 03/15] test(cli): cover reindex via npm-style bin symlink entrypoint --- tests/cli-entrypoint-runtime.test.ts | 30 ++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/tests/cli-entrypoint-runtime.test.ts b/tests/cli-entrypoint-runtime.test.ts index 3413363..314ad8e 100644 --- a/tests/cli-entrypoint-runtime.test.ts +++ b/tests/cli-entrypoint-runtime.test.ts @@ -142,6 +142,36 @@ describe('CLI entrypoint runtime', () => { await rm(projectRoot, { recursive: true, force: true }); } }, 70_000); + it('dispatches reindex when invoked through an npm-style symlink entrypoint', async () => { + const projectRoot = await mkdtemp(join(tmpdir(), 'cbc-cli-reindex-entrypoint-')); + const tempDirectory = join(tmpdir(), `cbc-cli-bin-${Date.now()}`); + const linkedEntrypoint = join(tempDirectory, 'codebase-context'); + + await writeFile(join(projectRoot, 'example.ts'), 'export const value = 1;\n'); + await mkdir(tempDirectory, { recursive: true }); + symlinkSync(entrypoint, linkedEntrypoint, 'file'); + + try { + const result = spawnSync( + process.execPath, + ['--import', import.meta.resolve('tsx'), linkedEntrypoint, 'reindex', '--no-embeddings'], + { + cwd: projectRoot, + env: { ...process.env }, + encoding: 'utf8', + timeout: 120_000 + } + ); + + expect(result.status, result.stderr).toBe(0); + expect(result.stderr).toContain('Complete:'); + expect(result.stderr).not.toContain('@modelcontextprotocol/sdk/server/stdio.js'); + } finally { + rmSync(tempDirectory, { recursive: true, force: true }); + await rm(projectRoot, { recursive: true, force: true }); + } + }, 120_000); + it('dispatches CLI help when invoked through a symlink entrypoint', (context) => { const tempDirectory = mkdtempSync(join(tmpdir(), 'cbc-cli-entrypoint-')); const linkedEntrypoint = join(tempDirectory, 'codebase-context.ts'); From 333f5203da9e65a0810c6aa0397a0f5cee27d45b Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 30 Oct 2026 22:26:41 +0000 Subject: [PATCH 04/15] feat(embeddings): add native Ollama embedding provider Implement OllamaEmbeddingProvider using /api/embed with OLLAMA_HOST and provider-specific model defaults. Wire getEmbeddingProvider, dimension detection, and skip Transformers runtime for ollama. Includes OPENAI_BASE_URL config compatible with PR #145, docs, and HTTP-mocked tests. --- README.md | 12 ++- docs/capabilities.md | 4 +- src/core/indexer.ts | 4 +- src/embeddings/index.ts | 20 +++- src/embeddings/ollama.ts | 155 +++++++++++++++++++++++++++++ src/embeddings/types.ts | 65 +++++++++++- tests/embedding-config-env.test.ts | 39 ++++++++ tests/embedding-mismatch.test.ts | 4 + tests/ollama-embeddings.test.ts | 106 ++++++++++++++++++++ tests/openai-embeddings.test.ts | 19 ++++ 10 files changed, 415 insertions(+), 13 deletions(-) create mode 100644 src/embeddings/ollama.ts create mode 100644 tests/embedding-config-env.test.ts create mode 100644 tests/ollama-embeddings.test.ts diff --git a/README.md b/README.md index d93d577..62e0442 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,7 @@ The unreleased source candidate starts new indexes without a model download. Sem - Better React/Next.js detection and a fix for reranker passage scoring. - Transformers.js v4 for local models. -Known limits: [Intel Mac indexing crash (#68)](https://github.com/PatrickSys/codebase-context/issues/68), [custom OpenAI endpoints (#70)](https://github.com/PatrickSys/codebase-context/issues/70) and [Ollama (#71)](https://github.com/PatrickSys/codebase-context/issues/71) remain open. +Known limits: [Intel Mac indexing crash (#68)](https://github.com/PatrickSys/codebase-context/issues/68) and [custom OpenAI endpoints (#70)](https://github.com/PatrickSys/codebase-context/issues/70) remain open on published npm; master adds native Ollama and `OPENAI_BASE_URL` wiring. The existing [benchmark](https://github.com/PatrickSys/codebase-context/blob/master/docs/benchmark.md) reports 25.7% expected-file recovery versus jCodeMunch's 27.1% in a fixed retrieval comparison; it does not establish a winner or coding quality. @@ -132,7 +132,15 @@ More examples are available in the [CLI gallery](./docs/cli.md) and [walkthrough Code and indexes stay on the machine with the default local embedding provider. Docker, a GPU, and an API key are not required. -Cloud embeddings are optional. If you select a cloud provider, code chunks are sent to that provider to create the search index. The provider, model, project root, and local HTTP port can be changed through environment variables or the project config; see the [capabilities reference](./docs/capabilities.md). +For local embeddings through [Ollama](https://ollama.com) (avoids Transformers.js on affected Macs, see #68), enable semantic indexing and set: + +```bash +EMBEDDING_PROVIDER=ollama +EMBEDDING_MODEL=nomic-embed-text # optional; default when unset +OLLAMA_HOST=http://localhost:11434 # optional +``` + +Cloud embeddings are optional. If you select OpenAI (`EMBEDDING_PROVIDER=openai`), code chunks are sent to that provider to create the search index. Override the API base with `OPENAI_BASE_URL` (compatible with Ollama’s OpenAI layer, but native `ollama` provider is preferred). The provider, model, project root, and local HTTP port can be changed through environment variables or the project config; see the [capabilities reference](./docs/capabilities.md). This is the privacy boundary of Codebase Context itself. Your AI client may send search results or file contents to the model provider configured in that client. Codebase Context does not control that connection. If you commit and push `.codebase-context/memory.json`, the recorded project memory also travels with the repository. diff --git a/docs/capabilities.md b/docs/capabilities.md index 49fc4f9..3b9bebb 100644 --- a/docs/capabilities.md +++ b/docs/capabilities.md @@ -199,7 +199,7 @@ Ordered by execution: - **Chunk size**: 50 lines, 0 overlap - **Reranker trigger**: activates when top-3 results are within 0.08 score of each other - **Embedding mode**: new indexes are model-free. Existing indexes keep their recorded mode unless explicitly changed. `reindex --embeddings` opts into semantic indexing and requires the optional model/storage packages. -- **Semantic model when enabled**: `Xenova/bge-small-en-v1.5` (512 token context, fast, local-first) via `@huggingface/transformers`. Override: `EMBEDDING_MODEL=onnx-community/granite-embedding-small-english-r2-ONNX` for Granite (8192 ctx, slower). +- **Semantic model when enabled**: `Xenova/bge-small-en-v1.5` (512 token context, fast, local-first) via `@huggingface/transformers`. Override: `EMBEDDING_MODEL=onnx-community/granite-embedding-small-english-r2-ONNX` for Granite (8192 ctx, slower). **Ollama**: `EMBEDDING_PROVIDER=ollama`, optional `OLLAMA_HOST` (default `http://localhost:11434`), default model `nomic-embed-text` — embeddings stay on your machine via Ollama’s `/api/embed`. - **Vector DB**: LanceDB with cosine distance ## Decision Card (Edit Intent) @@ -318,5 +318,5 @@ Reproducible evaluation is shipped as a CLI entrypoint backed by shared scoring - **Symbol refs are not a call-graph.** `get_symbol_references` counts identifier-node occurrences in the AST (comments/strings excluded via Tree-sitter). It does not distinguish call sites from type annotations, variable assignments, or imports. Full call-site-specific analysis (`call_expression` nodes only) is a roadmap item. - **Impact is 2-hop max.** `computeImpactCandidates` walks direct importers then their importers. Full BFS reachability is on the roadmap. - **Planned for unpublished 2.4.0:** NestJS analyzer support, React and Next.js detection improvements, and reranker fixes. Dedicated Angular, React, and Next.js analyzers already ship in published 2.2.0. Other languages use the Generic analyzer for indexing and retrieval; available symbol extraction depends on parser support. File import resolution and multi-hop navigation currently support TypeScript and JavaScript. Python and Go import navigation are not supported. -- **The unreleased candidate defaults new indexes to model-free mode. Published 2.2.0 enables embeddings by default.** When semantic mode is enabled, its default model is `bge-small-en-v1.5` (512-token context). Granite (8192 context) is opt-in via `EMBEDDING_MODEL`. OpenAI is opt-in via `EMBEDDING_PROVIDER=openai` — sends code externally. +- **The unreleased candidate defaults new indexes to model-free mode. Published 2.2.0 enables embeddings by default.** When semantic mode is enabled, its default model is `bge-small-en-v1.5` (512-token context). Granite (8192 context) is opt-in via `EMBEDDING_MODEL`. OpenAI is opt-in via `EMBEDDING_PROVIDER=openai` (`OPENAI_BASE_URL` overrides the API host). Ollama is opt-in via `EMBEDDING_PROVIDER=ollama` — local HTTP only, no API key. - **Patterns are file-level frequency counts.** Not semantic clustering. Rising/Declining trend is derived from git commit recency for files using each pattern, not from usage semantics. diff --git a/src/core/indexer.ts b/src/core/indexer.ts index 6d9ed37..5400c52 100644 --- a/src/core/indexer.ts +++ b/src/core/indexer.ts @@ -266,7 +266,9 @@ export class CodebaseIndexer { const defaultModel = defaultEmbeddingProvider === 'openai' && !process.env.EMBEDDING_MODEL ? 'text-embedding-3-small' - : DEFAULT_MODEL; + : defaultEmbeddingProvider === 'ollama' && !process.env.EMBEDDING_MODEL + ? 'nomic-embed-text' + : DEFAULT_MODEL; const defaultConfig: CodebaseConfig = { analyzers: { diff --git a/src/embeddings/index.ts b/src/embeddings/index.ts index f8c1718..af76750 100644 --- a/src/embeddings/index.ts +++ b/src/embeddings/index.ts @@ -6,14 +6,17 @@ import { EmbeddingConfig, DEFAULT_EMBEDDING_CONFIG, DEFAULT_MODEL, - parseEmbeddingProviderName + DEFAULT_OLLAMA_MODEL, + parseEmbeddingProviderName, + resolveOllamaHost } from './types.js'; import { TransformersEmbeddingProvider, MODEL_CONFIGS } from './transformers.js'; +import { getOllamaModelDimensions } from './ollama.js'; import { assertOptionalDependency } from '../utils/optional-dependency.js'; export function assertEmbeddingRuntimeAvailable(config: Partial = {}): void { const provider = config.provider ?? DEFAULT_EMBEDDING_CONFIG.provider; - if (provider !== 'openai' && provider !== 'custom') { + if (provider !== 'openai' && provider !== 'custom' && provider !== 'ollama') { assertOptionalDependency('@huggingface/transformers', '^4.2.0'); } } @@ -30,6 +33,7 @@ export function getConfiguredDimensions(config: Partial = {}): config.provider ?? parseEmbeddingProviderName(process.env.EMBEDDING_PROVIDER) ?? 'transformers'; const model = config.model ?? process.env.EMBEDDING_MODEL ?? DEFAULT_MODEL; if (provider === 'openai') return model.includes('large') ? 3072 : 1536; // text-embedding-3-large: 3072, all others: 1536 + if (provider === 'ollama') return getOllamaModelDimensions(model); // Look up from the same MODEL_CONFIGS the provider uses — avoids stale hardcoded guesses return MODEL_CONFIGS[model]?.dimensions ?? 384; } @@ -62,11 +66,19 @@ export async function getEmbeddingProvider( } if (mergedConfig.provider === 'custom') { - throw new Error("Custom provider not implemented. Use 'openai' or 'transformers'."); + throw new Error("Custom provider not implemented. Use 'openai', 'ollama', or 'transformers'."); } if (mergedConfig.provider === 'ollama') { - console.warn('Ollama provider not yet implemented, falling back to Transformers.js'); + const { OllamaEmbeddingProvider } = await import('./ollama.js'); + const provider = new OllamaEmbeddingProvider( + mergedConfig.model || DEFAULT_OLLAMA_MODEL, + mergedConfig.apiEndpoint ?? resolveOllamaHost() + ); + await provider.initialize(); + cachedProvider = provider; + cachedProviderType = providerKey; + return provider; } const provider = new TransformersEmbeddingProvider(mergedConfig.model); diff --git a/src/embeddings/ollama.ts b/src/embeddings/ollama.ts new file mode 100644 index 0000000..1c5d4c3 --- /dev/null +++ b/src/embeddings/ollama.ts @@ -0,0 +1,155 @@ +import { EmbeddingProvider } from './types.js'; + +/** Common Ollama embedding models and their vector sizes (for LanceDB dimension checks). */ +export const OLLAMA_MODEL_DIMENSIONS: Record = { + 'nomic-embed-text': 768, + 'mxbai-embed-large': 1024, + 'all-minilm': 384, + 'snowflake-arctic-embed:137m': 768, + 'snowflake-arctic-embed:335m': 1024 +}; + +interface OllamaEmbedResponse { + embeddings: number[][]; +} + +interface OllamaShowResponse { + modelfile?: string; + parameters?: string; +} + +function normalizeOllamaHost(host: string): string { + return host.replace(/\/+$/, ''); +} + +export function getOllamaModelDimensions(modelName: string): number { + const direct = OLLAMA_MODEL_DIMENSIONS[modelName]; + if (direct !== undefined) { + return direct; + } + const baseName = modelName.split(':')[0] ?? modelName; + const fromBase = OLLAMA_MODEL_DIMENSIONS[baseName]; + if (fromBase !== undefined) { + return fromBase; + } + const fromEnv = process.env.OLLAMA_EMBEDDING_DIMENSIONS; + if (fromEnv) { + const parsed = parseInt(fromEnv, 10); + if (!Number.isNaN(parsed) && parsed > 0) { + return parsed; + } + } + return 768; +} + +/** + * Ollama embedding provider — HTTP client for a local Ollama server (`/api/embed`). + */ +export class OllamaEmbeddingProvider implements EmbeddingProvider { + readonly name = 'ollama'; + private resolvedDimensions: number; + + constructor( + readonly modelName: string, + private host: string = 'http://localhost:11434' + ) { + this.host = normalizeOllamaHost(host); + this.resolvedDimensions = getOllamaModelDimensions(modelName); + } + + get dimensions(): number { + return this.resolvedDimensions; + } + + async initialize(): Promise { + let response: Response; + try { + response = await fetch(`${this.host}/api/show`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ name: this.modelName }) + }); + } catch (error) { + throw new Error( + `Cannot reach Ollama at ${this.host}. Start the server (ollama serve) or set OLLAMA_HOST. ${String(error)}` + ); + } + + if (response.status === 404) { + throw new Error( + `Ollama model "${this.modelName}" is not available. Pull it first: ollama pull ${this.modelName}` + ); + } + + if (!response.ok) { + const error = await response.text(); + throw new Error(`Ollama API Error ${response.status}: ${error}`); + } + + const show = (await response.json()) as OllamaShowResponse; + const fromModelfile = parseEmbeddingLengthFromShow(show); + if (fromModelfile !== undefined) { + this.resolvedDimensions = fromModelfile; + } + } + + isReady(): boolean { + return true; + } + + async embed(text: string): Promise { + const batch = await this.embedBatch([text]); + return batch[0]; + } + + async embedBatch(texts: string[]): Promise { + if (!texts.length) return []; + + try { + const response = await fetch(`${this.host}/api/embed`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + model: this.modelName, + input: texts.length === 1 ? texts[0] : texts + }) + }); + + if (!response.ok) { + const error = await response.text(); + if (response.status === 404 && error.includes('not found')) { + throw new Error( + `Ollama model "${this.modelName}" is not available. Pull it first: ollama pull ${this.modelName}` + ); + } + throw new Error(`Ollama API Error ${response.status}: ${error}`); + } + + const data = (await response.json()) as OllamaEmbedResponse; + if (!Array.isArray(data.embeddings) || data.embeddings.length !== texts.length) { + throw new Error( + `Ollama returned ${data.embeddings?.length ?? 0} embeddings for ${texts.length} inputs` + ); + } + + if (data.embeddings[0]?.length && data.embeddings[0].length !== this.resolvedDimensions) { + this.resolvedDimensions = data.embeddings[0].length; + } + + return data.embeddings; + } catch (error) { + console.error('Ollama Embedding Failed:', error); + throw error; + } + } +} + +function parseEmbeddingLengthFromShow(show: OllamaShowResponse): number | undefined { + const haystack = `${show.modelfile ?? ''}\n${show.parameters ?? ''}`; + const match = haystack.match(/embedding[_\s-]?length[\s=:]+(\d+)/i); + if (!match) { + return undefined; + } + const parsed = parseInt(match[1] ?? '', 10); + return Number.isNaN(parsed) || parsed <= 0 ? undefined : parsed; +} diff --git a/src/embeddings/types.ts b/src/embeddings/types.ts index 66e5a7a..a78433a 100644 --- a/src/embeddings/types.ts +++ b/src/embeddings/types.ts @@ -27,15 +27,72 @@ export function parseEmbeddingProviderName( return undefined; } +export const DEFAULT_OLLAMA_MODEL = 'nomic-embed-text'; +export const DEFAULT_TRANSFORMERS_MODEL = 'Xenova/bge-small-en-v1.5'; +export const DEFAULT_OPENAI_MODEL = 'text-embedding-3-small'; + // Default: bge-small (fast, ~2min indexing, consumer-hardware safe) // Opt-in: set EMBEDDING_MODEL=onnx-community/granite-embedding-small-english-r2-ONNX for // better conceptual search at the cost of 5-10x slower indexing and higher RAM usage -export const DEFAULT_MODEL = process.env.EMBEDDING_MODEL || 'Xenova/bge-small-en-v1.5'; +export const DEFAULT_MODEL = process.env.EMBEDDING_MODEL || DEFAULT_TRANSFORMERS_MODEL; + +const DEFAULT_OPENAI_API_ENDPOINT = 'https://api.openai.com/v1'; +const DEFAULT_OLLAMA_HOST = 'http://localhost:11434'; + +function stripTrailingSlashes(value: string): string { + return value.replace(/\/+$/, ''); +} + +function resolveOpenAiApiEndpoint(): string { + const fromEnv = process.env.OPENAI_BASE_URL?.trim(); + if (!fromEnv) { + return DEFAULT_OPENAI_API_ENDPOINT; + } + return stripTrailingSlashes(fromEnv); +} + +export function resolveOllamaHost(): string { + const fromEnv = process.env.OLLAMA_HOST?.trim(); + if (!fromEnv) { + return DEFAULT_OLLAMA_HOST; + } + return stripTrailingSlashes(fromEnv); +} + +export function resolveEmbeddingModel( + provider: EmbeddingConfig['provider'], + envModel: string | undefined = process.env.EMBEDDING_MODEL +): string { + if (envModel) { + return envModel; + } + if (provider === 'openai') { + return DEFAULT_OPENAI_MODEL; + } + if (provider === 'ollama') { + return DEFAULT_OLLAMA_MODEL; + } + return DEFAULT_TRANSFORMERS_MODEL; +} + +function resolveDefaultApiEndpoint(provider: EmbeddingConfig['provider']): string | undefined { + if (provider === 'openai') { + return resolveOpenAiApiEndpoint(); + } + if (provider === 'ollama') { + return resolveOllamaHost(); + } + return undefined; +} + +const defaultProvider = + parseEmbeddingProviderName(process.env.EMBEDDING_PROVIDER) ?? 'transformers'; export const DEFAULT_EMBEDDING_CONFIG: EmbeddingConfig = { - provider: parseEmbeddingProviderName(process.env.EMBEDDING_PROVIDER) ?? 'transformers', - model: DEFAULT_MODEL, + provider: defaultProvider, + model: resolveEmbeddingModel(defaultProvider), batchSize: 32, maxRetries: 3, - apiKey: process.env.OPENAI_API_KEY + apiKey: process.env.OPENAI_API_KEY, + apiEndpoint: resolveDefaultApiEndpoint(defaultProvider) }; diff --git a/tests/embedding-config-env.test.ts b/tests/embedding-config-env.test.ts new file mode 100644 index 0000000..650e304 --- /dev/null +++ b/tests/embedding-config-env.test.ts @@ -0,0 +1,39 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; + +describe('DEFAULT_EMBEDDING_CONFIG environment', () => { + afterEach(() => { + vi.unstubAllEnvs(); + vi.resetModules(); + }); + + it('uses OPENAI_BASE_URL when set', async () => { + vi.stubEnv('EMBEDDING_PROVIDER', 'openai'); + vi.stubEnv('OPENAI_BASE_URL', 'http://127.0.0.1:11434/v1/'); + const { DEFAULT_EMBEDDING_CONFIG } = await import('../src/embeddings/types.js'); + expect(DEFAULT_EMBEDDING_CONFIG.apiEndpoint).toBe('http://127.0.0.1:11434/v1'); + }); + + it('defaults to the official OpenAI endpoint when OPENAI_BASE_URL is unset', async () => { + vi.stubEnv('EMBEDDING_PROVIDER', 'openai'); + vi.unstubAllEnvs(); + vi.stubEnv('EMBEDDING_PROVIDER', 'openai'); + const { DEFAULT_EMBEDDING_CONFIG } = await import('../src/embeddings/types.js'); + expect(DEFAULT_EMBEDDING_CONFIG.apiEndpoint).toBe('https://api.openai.com/v1'); + }); + + it('uses OLLAMA_HOST when provider is ollama', async () => { + vi.stubEnv('EMBEDDING_PROVIDER', 'ollama'); + vi.stubEnv('OLLAMA_HOST', 'http://127.0.0.1:11434/'); + const { DEFAULT_EMBEDDING_CONFIG } = await import('../src/embeddings/types.js'); + expect(DEFAULT_EMBEDDING_CONFIG.provider).toBe('ollama'); + expect(DEFAULT_EMBEDDING_CONFIG.apiEndpoint).toBe('http://127.0.0.1:11434'); + expect(DEFAULT_EMBEDDING_CONFIG.model).toBe('nomic-embed-text'); + }); + + it('resolveEmbeddingModel picks provider-specific defaults', async () => { + const { resolveEmbeddingModel } = await import('../src/embeddings/types.js'); + expect(resolveEmbeddingModel('ollama', undefined)).toBe('nomic-embed-text'); + expect(resolveEmbeddingModel('openai', undefined)).toBe('text-embedding-3-small'); + expect(resolveEmbeddingModel('transformers', undefined)).toBe('Xenova/bge-small-en-v1.5'); + }); +}); diff --git a/tests/embedding-mismatch.test.ts b/tests/embedding-mismatch.test.ts index 67e78b3..6b38ade 100644 --- a/tests/embedding-mismatch.test.ts +++ b/tests/embedding-mismatch.test.ts @@ -95,4 +95,8 @@ describe('getConfiguredDimensions', () => { it('returns 384 as fallback for unknown transformers model', () => { expect(getConfiguredDimensions({ provider: 'transformers', model: 'some/unknown-model' })).toBe(384); }); + + it('returns 768 for nomic-embed-text via ollama provider', () => { + expect(getConfiguredDimensions({ provider: 'ollama', model: 'nomic-embed-text' })).toBe(768); + }); }); diff --git a/tests/ollama-embeddings.test.ts b/tests/ollama-embeddings.test.ts new file mode 100644 index 0000000..c73c889 --- /dev/null +++ b/tests/ollama-embeddings.test.ts @@ -0,0 +1,106 @@ +import { describe, it, expect, vi, afterEach } from 'vitest'; +import { OllamaEmbeddingProvider } from '../src/embeddings/ollama.js'; + +function makeFetchResponse(body: unknown, ok = true, status = 200) { + return Promise.resolve({ + ok, + status, + json: () => Promise.resolve(body), + text: () => Promise.resolve(JSON.stringify(body)) + } as Response); +} + +describe('OllamaEmbeddingProvider', () => { + afterEach(() => { + vi.unstubAllGlobals(); + vi.unstubAllEnvs(); + }); + + it('uses known dimensions for nomic-embed-text', () => { + const provider = new OllamaEmbeddingProvider('nomic-embed-text'); + expect(provider.dimensions).toBe(768); + }); + + it('initialize() throws when the model is missing on the server', async () => { + vi.stubGlobal( + 'fetch', + vi.fn().mockReturnValue(makeFetchResponse({ error: 'not found' }, false, 404)) + ); + + const provider = new OllamaEmbeddingProvider('nomic-embed-text', 'http://127.0.0.1:11434'); + await expect(provider.initialize()).rejects.toThrow('not available'); + }); + + it('initialize() resolves when /api/show succeeds', async () => { + vi.stubGlobal( + 'fetch', + vi.fn().mockReturnValue(makeFetchResponse({ modelfile: 'FROM nomic-embed-text' })) + ); + + const provider = new OllamaEmbeddingProvider('nomic-embed-text'); + await expect(provider.initialize()).resolves.toBeUndefined(); + }); + + it('embedBatch() posts to /api/embed with model and input', async () => { + const mockFetch = vi.fn().mockReturnValue( + makeFetchResponse({ + embeddings: [[0.1, 0.2, 0.3]] + }) + ); + vi.stubGlobal('fetch', mockFetch); + + const provider = new OllamaEmbeddingProvider('nomic-embed-text', 'http://localhost:11434'); + const result = await provider.embedBatch(['hello world']); + + expect(result).toEqual([[0.1, 0.2, 0.3]]); + expect(mockFetch).toHaveBeenCalledOnce(); + const [url, init] = mockFetch.mock.calls[0] as [string, RequestInit]; + expect(url).toBe('http://localhost:11434/api/embed'); + const body = JSON.parse(init.body as string); + expect(body.model).toBe('nomic-embed-text'); + expect(body.input).toBe('hello world'); + }); + + it('embedBatch() sends array input for multiple texts', async () => { + const mockFetch = vi.fn().mockReturnValue( + makeFetchResponse({ + embeddings: [ + [0.1, 0.2], + [0.3, 0.4] + ] + }) + ); + vi.stubGlobal('fetch', mockFetch); + + const provider = new OllamaEmbeddingProvider('nomic-embed-text'); + await provider.embedBatch(['a', 'b']); + + const body = JSON.parse((mockFetch.mock.calls[0] as [string, RequestInit])[1].body as string); + expect(body.input).toEqual(['a', 'b']); + }); + + it('embedBatch() returns [] for empty input', async () => { + const provider = new OllamaEmbeddingProvider('nomic-embed-text'); + await expect(provider.embedBatch([])).resolves.toEqual([]); + }); + + it('embedBatch() throws on non-ok API response', async () => { + vi.stubGlobal( + 'fetch', + vi.fn().mockReturnValue(makeFetchResponse({ error: 'server error' }, false, 500)) + ); + + const provider = new OllamaEmbeddingProvider('nomic-embed-text'); + await expect(provider.embedBatch(['test'])).rejects.toThrow('500'); + }); + + it('embed() delegates to embedBatch', async () => { + vi.stubGlobal( + 'fetch', + vi.fn().mockReturnValue(makeFetchResponse({ embeddings: [[0.5, 0.6]] })) + ); + + const provider = new OllamaEmbeddingProvider('nomic-embed-text'); + await expect(provider.embed('hello')).resolves.toEqual([0.5, 0.6]); + }); +}); diff --git a/tests/openai-embeddings.test.ts b/tests/openai-embeddings.test.ts index 8763efd..4e722a2 100644 --- a/tests/openai-embeddings.test.ts +++ b/tests/openai-embeddings.test.ts @@ -46,6 +46,25 @@ describe('OpenAIEmbeddingProvider', () => { expect(result).toEqual([]); }); + it('embedBatch() uses custom apiEndpoint base URL', async () => { + const mockFetch = vi.fn().mockReturnValue( + makeFetchResponse({ + data: [{ embedding: [0.1] }] + }) + ); + vi.stubGlobal('fetch', mockFetch); + + const provider = new OpenAIEmbeddingProvider( + 'text-embedding-3-small', + 'sk-test', + 'http://127.0.0.1:11434/v1' + ); + await provider.embedBatch(['hello']); + + const [url] = mockFetch.mock.calls[0] as [string, RequestInit]; + expect(url).toBe('http://127.0.0.1:11434/v1/embeddings'); + }); + it('embedBatch() sends correct Authorization header, model, and encoding_format', async () => { const mockFetch = vi.fn().mockReturnValue( makeFetchResponse({ From 7b823d2720e4c5ba5f72fb125671f94ed649b433 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 30 Oct 2026 22:28:43 +0000 Subject: [PATCH 05/15] fix(embeddings): guard against macOS onnxruntime mutex crash Pin onnxruntime-node 1.24.3 for npm installs, assert a minimum runtime before Transformers.js embedding/rerank work, and add regression tests. Fixes #68 --- package.json | 3 ++ src/core/reranker.ts | 3 ++ src/embeddings/transformers.ts | 2 + src/utils/onnxruntime-version.ts | 65 +++++++++++++++++++++++++++++++ tests/onnxruntime-version.test.ts | 47 ++++++++++++++++++++++ 5 files changed, 120 insertions(+) create mode 100644 src/utils/onnxruntime-version.ts create mode 100644 tests/onnxruntime-version.test.ts diff --git a/package.json b/package.json index a855889..fa92a6a 100644 --- a/package.json +++ b/package.json @@ -181,6 +181,9 @@ "typescript-eslint": "^8.51.0", "vitest": "^4.0.18" }, + "overrides": { + "@huggingface/transformers>onnxruntime-node": "1.24.3" + }, "pnpm": { "onlyBuiltDependencies": [ "esbuild", diff --git a/src/core/reranker.ts b/src/core/reranker.ts index 9dabeac..f579289 100644 --- a/src/core/reranker.ts +++ b/src/core/reranker.ts @@ -11,6 +11,7 @@ import type { SearchResult } from '../types/index.js'; import type { PreTrainedTokenizer } from '@huggingface/transformers'; import os from 'os'; +import { assertMinimumOnnxRuntimeNodeVersion } from '../utils/onnxruntime-version.js'; const DEFAULT_RERANKER_MODEL = 'Xenova/ms-marco-MiniLM-L-6-v2'; @@ -48,6 +49,8 @@ async function ensureModelLoaded(): Promise { if (initPromise) return initPromise; initPromise = (async () => { + assertMinimumOnnxRuntimeNodeVersion(); + const { AutoTokenizer, AutoModelForSequenceClassification, env } = await import('@huggingface/transformers'); diff --git a/src/embeddings/transformers.ts b/src/embeddings/transformers.ts index bb358d1..db0b2b4 100644 --- a/src/embeddings/transformers.ts +++ b/src/embeddings/transformers.ts @@ -1,6 +1,7 @@ import { EmbeddingProvider, DEFAULT_MODEL } from './types.js'; import type { FeatureExtractionPipeline } from '@huggingface/transformers'; import os from 'os'; +import { assertMinimumOnnxRuntimeNodeVersion } from '../utils/onnxruntime-version.js'; import { assertOptionalDependency } from '../utils/optional-dependency.js'; /** @@ -58,6 +59,7 @@ export class TransformersEmbeddingProvider implements EmbeddingProvider { if (this.initPromise) return this.initPromise; assertOptionalDependency('@huggingface/transformers', '^4.2.0'); + assertMinimumOnnxRuntimeNodeVersion(); this.initPromise = this._initialize(); return this.initPromise; diff --git a/src/utils/onnxruntime-version.ts b/src/utils/onnxruntime-version.ts new file mode 100644 index 0000000..2e0df77 --- /dev/null +++ b/src/utils/onnxruntime-version.ts @@ -0,0 +1,65 @@ +import { readFileSync } from 'node:fs'; +import { createRequire } from 'node:module'; +import path from 'node:path'; + +/** microsoft/onnxruntime#24579 — mutex crash on macOS in 1.21.x and earlier. */ +export const MIN_ONNXRUNTIME_NODE_VERSION = '1.24.0'; + +const resolve = createRequire(import.meta.url).resolve; + +export function parseSemverTriple(version: string): [number, number, number] { + const match = /^(\d+)\.(\d+)\.(\d+)/.exec(version.trim()); + if (!match) { + return [0, 0, 0]; + } + return [Number(match[1]), Number(match[2]), Number(match[3])]; +} + +export function isSemverAtLeast(actual: string, minimum: string): boolean { + const [aMajor, aMinor, aPatch] = parseSemverTriple(actual); + const [mMajor, mMinor, mPatch] = parseSemverTriple(minimum); + if (aMajor !== mMajor) { + return aMajor > mMajor; + } + if (aMinor !== mMinor) { + return aMinor > mMinor; + } + return aPatch >= mPatch; +} + +function readInstalledOnnxRuntimeVersion(): string | null { + try { + const transformersPkg = resolve('@huggingface/transformers/package.json'); + const onnxPkg = resolve('onnxruntime-node/package.json', { + paths: [path.dirname(transformersPkg)] + }); + const pkg = JSON.parse(readFileSync(onnxPkg, 'utf8')) as { version?: string }; + return typeof pkg.version === 'string' ? pkg.version : null; + } catch { + try { + const onnxPkg = resolve('onnxruntime-node/package.json'); + const pkg = JSON.parse(readFileSync(onnxPkg, 'utf8')) as { version?: string }; + return typeof pkg.version === 'string' ? pkg.version : null; + } catch { + return null; + } + } +} + +/** Fail fast before ONNX embedding/rerank work when the native runtime is too old. */ +export function assertMinimumOnnxRuntimeNodeVersion(minimum = MIN_ONNXRUNTIME_NODE_VERSION): void { + const installed = readInstalledOnnxRuntimeVersion(); + if (!installed) { + return; + } + + if (isSemverAtLeast(installed, minimum)) { + return; + } + + throw new Error( + `onnxruntime-node@${installed} can crash on macOS during embedding (mutex lock failed; see microsoft/onnxruntime#24579). ` + + `Install @huggingface/transformers@^4.2.0 so onnxruntime-node resolves to >=${minimum}, ` + + `or set EMBEDDING_PROVIDER=openai to bypass local Transformers.js.` + ); +} diff --git a/tests/onnxruntime-version.test.ts b/tests/onnxruntime-version.test.ts new file mode 100644 index 0000000..f3ae407 --- /dev/null +++ b/tests/onnxruntime-version.test.ts @@ -0,0 +1,47 @@ +import { readFileSync } from 'node:fs'; +import { resolve } from 'node:path'; +import { describe, expect, it } from 'vitest'; + +import { + MIN_ONNXRUNTIME_NODE_VERSION, + assertMinimumOnnxRuntimeNodeVersion, + isSemverAtLeast, + parseSemverTriple +} from '../src/utils/onnxruntime-version.js'; + +const root = resolve(import.meta.dirname, '..'); + +type PackageJson = { + peerDependencies?: Record; + overrides?: Record; + pnpm?: { overrides?: Record }; +}; + +describe('onnxruntime-node macOS mutex regression (#68)', () => { + const packageJson = JSON.parse(readFileSync(resolve(root, 'package.json'), 'utf8')) as PackageJson; + const lockfile = readFileSync(resolve(root, 'pnpm-lock.yaml'), 'utf8'); + + it('requires Transformers.js v4 peers (ships onnxruntime-node 1.24.x)', () => { + expect(packageJson.peerDependencies?.['@huggingface/transformers']).toBe('^4.2.0'); + }); + + it('pins onnxruntime-node 1.24.3 for npm and pnpm installs', () => { + expect(packageJson.overrides?.['@huggingface/transformers>onnxruntime-node']).toBe('1.24.3'); + expect(packageJson.pnpm?.overrides?.['@huggingface/transformers>onnxruntime-node']).toBe('1.24.3'); + }); + + it('locks onnxruntime-node 1.24.3 in pnpm-lock.yaml', () => { + expect(lockfile).toContain("'@huggingface/transformers>onnxruntime-node': 1.24.3"); + expect(lockfile).toMatch(/onnxruntime-node@1\.24\.3/); + }); + + it('compares semver triples for the minimum safe runtime', () => { + expect(parseSemverTriple('1.21.0')).toEqual([1, 21, 0]); + expect(isSemverAtLeast('1.24.3', MIN_ONNXRUNTIME_NODE_VERSION)).toBe(true); + expect(isSemverAtLeast('1.21.0', MIN_ONNXRUNTIME_NODE_VERSION)).toBe(false); + }); + + it('accepts onnxruntime-node 1.24.x when installed with Transformers.js', () => { + expect(() => assertMinimumOnnxRuntimeNodeVersion()).not.toThrow(); + }); +}); From e823cdfcb2bf87e29024141b0d458e79725c3a38 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 30 Oct 2026 22:29:18 +0000 Subject: [PATCH 06/15] docs: prepare honest 2.3.0 release surfaces Align README and docs with team-conventions plus edit-preflight positioning, credit Aider/Repomix/jCodeMunch, and separate shipped 2.3.0 from planned 2.4.0. Add 2.3.0 CHANGELOG release notes including the npm 2.2.0 silent CLI fix. --- CHANGELOG.md | 8 +++++++- MOTIVATION.md | 3 +++ README.md | 31 ++++++++++++++++--------------- docs/capabilities.md | 8 ++++---- docs/cli.md | 12 ++++++------ docs/client-setup.md | 2 +- docs/comparison-table.md | 2 +- 7 files changed, 38 insertions(+), 28 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 33f0294..8bd7bae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,8 +17,14 @@ - Updated the runtime parser and watcher to remove the legacy braces dependency chain. Watcher exclusions use exact directory names; quoted import and export names retain their public symbol names. - Known limits: model-free conceptual relevance is not guaranteed. Short requests can miss implementations when query words differ from source names; syntactic edges are not runtime calls; workspace/language-specific resolution can remain unresolved. Packed/protocol tests and scoped negative cases do not establish general agent-quality improvement or customer adoption. No npm publication is implied by this entry. -## [2.3.0](https://github.com/PatrickSys/codebase-context/compare/v2.2.0...v2.3.0) (2026-04-30) +## [2.3.0](https://github.com/PatrickSys/codebase-context/compare/v2.2.0...v2.3.0) (2026-10-30) +### Release notes + +- **CLI:** Fix npm **2.2.0** regression where `npx codebase-context --version`, `status`, and `reindex` could exit with code 0 and no output (direct CLI subcommands lazy-load the MCP runtime; canonical entrypoint detection includes npm symlink launchers). +- **Indexing:** Fresh project folders default to keyword-only indexing without downloading embedding models; opt in with `reindex --embeddings` and optional peer packages. +- **Setup:** `init` wizard for Claude Code, Codex, Cursor, and OpenCode (multi-client registration from one run). +- **Docs:** Position around local team conventions and edit preflight; cite Aider repo map, Repomix, and jCodeMunch as related tools; report 25.7% vs 27.1% retrieval without leadership claims; keep **2.4.0** features out of shipped language until published. ### Features diff --git a/MOTIVATION.md b/MOTIVATION.md index c83fcd8..916498b 100644 --- a/MOTIVATION.md +++ b/MOTIVATION.md @@ -24,6 +24,9 @@ The root cause isn't the model. It's the context layer. Agents get raw code from | ---------------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------- | | `.cursorrules` / `AGENTS.md` | Static rules the agent may or may not always follow | Can't express migration states ("use A for new code, B for old"). Goes stale. | | Semantic search / RAG | Finds similar code | Blind to quality. Can't tell the best example to follow from hacky workarounds. | +| **Aider repo map** | Compact structural map for the model | No team pattern frequencies, memory, or edit preflight card. | +| **Repomix** | Pack repo (or slices) into a prompt | One-shot context, not live search, conventions, or session memory. | +| **jCodeMunch** | Fast MCP code search | Strong retrieval in our fixed benchmark; less convention/memory/edit gating. | | Linters / formatters | Enforce static rules after writing | Don't guide before or during generation. Agent still writes wrong code first. | | Tool-heavy MCPs (50+ tools) | Expose everything | Agents don't know which tool to call. More tools = more confusion. | diff --git a/README.md b/README.md index d93d577..15a5068 100644 --- a/README.md +++ b/README.md @@ -2,11 +2,13 @@ [![npm version](https://img.shields.io/npm/v/codebase-context)](https://www.npmjs.com/package/codebase-context) [![license](https://img.shields.io/npm/l/codebase-context)](./LICENSE) [![node](https://img.shields.io/node/v/codebase-context)](https://github.com/PatrickSys/codebase-context/blob/master/package.json) -Codebase Context gives coding agents a map of your repository through MCP, for developers who want their agent to find relevant code and follow the team's patterns. Search, examples and project memory run locally by default. +Codebase Context is a local MCP server for **team conventions and edit preflight**: a bounded map of how your repo is organized, ranked search with pattern and memory context, and a decision card before risky edits. It is not a retrieval leaderboard; frozen benchmarks put it at **25.7%** expected-file recovery versus **jCodeMunch 27.1%** in the same adapter run ([details](https://github.com/PatrickSys/codebase-context/blob/master/docs/benchmark.md)). Search, examples, and project memory stay on your machine by default. + +**Colleagues, not competitors:** [Aider's repo map](https://aider.chat/docs/repomap.html) for structure, [Repomix](https://repomix.com/) for pack-and-prompt workflows, and jCodeMunch for fast MCP code search (see [benchmark comparators](https://github.com/PatrickSys/codebase-context/blob/master/docs/benchmark.md)). Codebase Context combines map-first routing, convention detection, scoped memory, and edit-intent gating in one local tool surface. ## Set up your AI client -Use Node.js 22 or newer. These commands use the current npm release of Codebase Context: +Use Node.js 22 or newer. Unpinned `npx` installs the **current npm release (2.2.0 today)**; **2.3.0** is the pending release from this repository (Patrick publishes manually). Pin `@2.3.0` after it lands. npm **2.2.0** has a regression where `npx codebase-context --version`, `status`, and `reindex` can exit silently; **2.3.0** fixes that. ```bash # Claude Code @@ -19,9 +21,7 @@ codex mcp add codebase-context -- npx -y codebase-context opencode mcp add codebase-context '--' npx -y codebase-context ``` -In the unreleased candidate, `--version` prints the installed package version and `map` builds a missing index before printing the map. A low-confidence search includes a hint that weak matches do not establish that code is absent. These additions require the built candidate; the unpinned commands above install published 2.2.0. - -This repository's `init` command detects Claude Code, Codex, Cursor, and OpenCode and writes a project config for each selected tool. The saved launch is the Node binary and the package file from the npx cache. That command is in this source and is not part of published 2.2.0. Until it is published, use the copy-paste lines above. The [client setup guide](./docs/client-setup.md) marks the unpublished command. +**Ships in 2.3.0 (not on npm until published):** `--version`, non-silent `status` / `reindex`, `init` for supported clients, fresh indexes without a model download (use `reindex --embeddings` when you want local semantic search), `map` that builds a missing index first, and low-confidence search hints that weak matches do not prove absence. Until 2.3.0 is on npm, use the manual MCP lines below or install from a built checkout / packed tarball. Start a new agent session in your project, then ask: @@ -29,13 +29,14 @@ Start a new agent session in your project, then ask: Replace `[feature]` with something you want to find. The first run downloads a local model and indexes the repository; time depends on repository size, CPU and model. -### Planned for 2.4.0 (not yet published) +### Planned for 2.4.0 (not in 2.3.0) -The unreleased source candidate starts new indexes without a model download. Semantic embeddings are optional; existing indexes keep their recorded mode. Its setup wizard can configure Claude Code, Codex, Cursor and OpenCode using the Node binary and package file that ran setup. These changes are not in the current npm release. +Do not treat these as shipped until 2.4.0 is published: -- NestJS support from community contributor Aaron Olin. See the [analyzer details](./docs/capabilities.md#analyzers). -- Better React/Next.js detection and a fix for reranker passage scoring. -- Transformers.js v4 for local models. +- NestJS analyzer from community contributor Aaron Olin ([analyzer details](./docs/capabilities.md#analyzers)). +- React/Next.js detection improvements and reranker passage-scoring fix. +- Transformers.js v4 for optional local embeddings. +- Per-project `parsing.maxChunks` in config and packaged `codebase-context-review` polish called out in [review-context](./docs/review-context.md). Known limits: [Intel Mac indexing crash (#68)](https://github.com/PatrickSys/codebase-context/issues/68), [custom OpenAI endpoints (#70)](https://github.com/PatrickSys/codebase-context/issues/70) and [Ollama (#71)](https://github.com/PatrickSys/codebase-context/issues/71) remain open. @@ -76,7 +77,7 @@ The default connection is `stdio` (standard input/output): your client starts th ### Team patterns and examples -`get_team_patterns` shows the approaches used in the repository and points to representative files. Dedicated analyzers for Angular, React, and Next.js ship in published 2.2.0; NestJS support (along with React and Next.js detection improvements and a reranker fix) is planned for unpublished 2.4.0. +`get_team_patterns` shows the approaches used in the repository and points to representative files. Angular, React, and Next.js analyzers ship today on npm 2.2.0 and remain in 2.3.0. NestJS and the 2.4.0 analyzer/reranker work above are not published yet. ### Project memory @@ -88,11 +89,11 @@ The `codebase-context-review` CLI creates a bounded context packet from a commit ## How it works -1. **Index locally.** Codebase Context scans the project, builds a keyword index, and creates local semantic embeddings - numeric representations used to match code by meaning as well as exact words. +1. **Index locally.** Codebase Context scans the project and builds a keyword index; opt into local semantic embeddings with `reindex --embeddings` when you need meaning-based search (npm **2.2.0** still pulled embeddings on first index). 2. **Understand the repository.** The agent can request a compact codebase map with structure, patterns, and representative files. 3. **Find the code for the task.** Search returns ranked files and symbols; the agent reads the selected files before editing. -The same information is available from the terminal. Run these commands from your project root. The first run downloads a local model and indexes the repository; time depends on repository size, CPU and model. Once it is ready, inspect the map and search for the code you need: +The same information is available from the terminal. Run these commands from your project root. **2.3.0** can index without a model download; semantic search is optional. Once the index is ready, inspect the map and search: ```bash # Build or refresh the local index @@ -161,8 +162,8 @@ The method and failures are documented so the measurements can be inspected with - Retrieval measurements describe expected-file coverage, file precision, and reported `peakPrivateGb`, not patch correctness or end-to-end coding quality. - The paired token observation covers two frozen investigation tasks and records observed agent behavior; it is not a universal token or time guarantee. - Setup checks differ by client. The detailed guide distinguishes a written config, a config recognized by the client, a local connection, and instructions checked only against official docs. -- Dedicated Angular, React, and Next.js analyzers ship in published 2.2.0; NestJS support is planned for unpublished 2.4.0. Other projects use the generic analyzer and the language parsers available for that stack. -- The default searchable-chunk limit is 5,000 per project. Customizing this via `projects[].parsing.maxChunks` in `~/.codebase-context/config.json` requires 2.4.0 (not yet published). +- Angular, React, and Next.js analyzers are on npm 2.2.0; NestJS lands in 2.4.0. Other stacks use the generic analyzer and available parsers. +- Default searchable-chunk cap is 5,000 per project. `projects[].parsing.maxChunks` in `~/.codebase-context/config.json` requires 2.4.0 (not yet published). - The agent must identify its repository in tool calls when no default or unambiguous client root is available. Concurrent HTTP client isolation is not established by the stdio routing checks. ## Reference diff --git a/docs/capabilities.md b/docs/capabilities.md index 49fc4f9..0e52321 100644 --- a/docs/capabilities.md +++ b/docs/capabilities.md @@ -1,6 +1,6 @@ # Capabilities Reference -Unpinned `npx codebase-context` commands install the published 2.2.0 package. Sections marked as unreleased describe the source candidate and require a built checkout or a local candidate installation. +Unpinned `npx codebase-context` installs **npm 2.2.0 today**. **2.3.0** is the pending release from this repo; **2.4.0** is not published. Sections marked **2.4.0** or **planned** describe future work only. Start with the repository map, then search for relevant code. For an overview, see [README.md](../README.md). Features marked as planned for 2.4.0 are not yet available from npm. @@ -65,7 +65,7 @@ npx -y codebase-context reindex --incremental ## Tool Surface -10 MCP tools + active/project-scoped context resources. +10 MCP tools plus project-scoped context resources in published 2.2.0/2.3.0. **`get_code_relationships`** and related navigation improvements are documented below for the in-repo build; treat them as **2.4.0** until that release is on npm unless you install from source. Shared selector inputs: @@ -317,6 +317,6 @@ Reproducible evaluation is shipped as a CLI entrypoint backed by shared scoring - **Symbol refs are not a call-graph.** `get_symbol_references` counts identifier-node occurrences in the AST (comments/strings excluded via Tree-sitter). It does not distinguish call sites from type annotations, variable assignments, or imports. Full call-site-specific analysis (`call_expression` nodes only) is a roadmap item. - **Impact is 2-hop max.** `computeImpactCandidates` walks direct importers then their importers. Full BFS reachability is on the roadmap. -- **Planned for unpublished 2.4.0:** NestJS analyzer support, React and Next.js detection improvements, and reranker fixes. Dedicated Angular, React, and Next.js analyzers already ship in published 2.2.0. Other languages use the Generic analyzer for indexing and retrieval; available symbol extraction depends on parser support. File import resolution and multi-hop navigation currently support TypeScript and JavaScript. Python and Go import navigation are not supported. -- **The unreleased candidate defaults new indexes to model-free mode. Published 2.2.0 enables embeddings by default.** When semantic mode is enabled, its default model is `bge-small-en-v1.5` (512-token context). Granite (8192 context) is opt-in via `EMBEDDING_MODEL`. OpenAI is opt-in via `EMBEDDING_PROVIDER=openai` — sends code externally. +- **Planned for 2.4.0 (not on npm):** NestJS analyzer, React/Next.js detection improvements, reranker fixes, optional `maxChunks` config, and full `get_code_relationships` / review-CLI publication called out in [review-context](./review-context.md). Angular, React, and Next.js analyzers ship on npm 2.2.0 and remain in 2.3.0. +- **Indexing defaults in 2.3.0:** new project folders index keyword-only until you run `reindex --embeddings` (optional `@huggingface/transformers` + `@lancedb/lancedb` peers). npm **2.2.0** still pulled embeddings on first index. When semantic mode is enabled, default model is `bge-small-en-v1.5` (512-token context). Granite (8192 context) is opt-in via `EMBEDDING_MODEL`. OpenAI is opt-in via `EMBEDDING_PROVIDER=openai` — sends code externally. - **Patterns are file-level frequency counts.** Not semantic clustering. Rising/Declining trend is derived from git commit recency for files using each pattern, not from usage semantics. diff --git a/docs/cli.md b/docs/cli.md index 9ae65f6..7d7f443 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -46,11 +46,11 @@ CODEBASE_CONTEXT_ASCII=1 npx -y codebase-context patterns - `reindex` — rebuild index (full or incremental) - `style-guide` — find style guide sections in docs - `memory list|add|remove` — manage team memory (stored in `.codebase-context/memory.json`) -- MCP client registration — use the recipes in the [client setup guide](./client-setup.md). Published 2.2.0 has the older interactive HTTP wizard; the four-client wizard below requires the unreleased candidate. +- MCP client registration — use the [client setup guide](./client-setup.md). npm **2.2.0** ships the older HTTP-only wizard path; the multi-client `init` flow ships in **2.3.0** (pending npm publish). -## `init` in the unreleased candidate +## `init` (2.3.0 — pending npm publish) -`init` detects Claude Code, Codex, Cursor and OpenCode, then configures the selected tools together. In a terminal, detected tools start selected; use Space to change the selection and Enter to apply it. Without a terminal, it configures all detected tools. +`init` detects Claude Code, Codex, Cursor, and OpenCode, then configures the selected tools together. In a terminal, detected tools start selected; use Space to change the selection and Enter to apply it. Without a terminal, it configures all detected tools. Not available in npm **2.2.0**. ```powershell codebase-context init --help @@ -59,15 +59,15 @@ codebase-context init --yes codebase-context init --client codex --client cursor --root "C:\path\to\project" ``` -Use repeatable `--client` options to choose tools explicitly. `--yes` configures every detected tool when no clients are named. `--root` selects another repository; otherwise `CODEBASE_ROOT` or the current directory is used. Setup writes client registrations but does not build an index or change agent instructions. The first search creates the index. The multi-tool flow is not in published npm `2.2.0` yet; see the [main setup guide](../README.md#set-up-your-ai-client) for its publication status and the [client setup guide](./client-setup.md) for current manual recipes. +Use repeatable `--client` options to choose tools explicitly. `--yes` configures every detected tool when no clients are named. `--root` selects another repository; otherwise `CODEBASE_ROOT` or the current directory is used. Setup writes client registrations but does not build an index or change agent instructions. The first search creates the index. Until **2.3.0** is on npm, use the manual recipes in the [client setup guide](./client-setup.md). -The built candidate also supports `--version` and builds a missing index when you run `map`. New indexes use keyword search without a model download; `reindex --embeddings` explicitly enables local semantic search. Preview, reindex, and map apply the same project exclusions and analyzer hints. A low-confidence search result means the matches are weak; it does not establish that code is absent. +**Also in 2.3.0:** `--version`; non-silent `status` / `reindex` (fixes npm 2.2.0 silent exit); fresh indexes without a model download; `map` builds a missing index first; low-confidence search hints. Use `reindex --embeddings` for local semantic search. --- ## First use -Run these CLI commands from the project root. Build or refresh the local index first; the first run downloads a local model and indexes the repository; time depends on repository size, CPU and model. Then inspect the conventions map and search for the code you need: +Run these CLI commands from the project root. **2.3.0** indexes keyword-only on first run (no model download); add `reindex --embeddings` when you want local semantic search. npm **2.2.0** still downloaded a model on first index. ```bash npx -y codebase-context reindex diff --git a/docs/client-setup.md b/docs/client-setup.md index f43ab54..0d44385 100644 --- a/docs/client-setup.md +++ b/docs/client-setup.md @@ -1,6 +1,6 @@ # Client Setup -These commands use the current npm release because the package name is unpinned. The current published version is 2.2.0; 2.4.0 is not yet published. +These commands use the current npm release (**2.2.0** on npm today; **2.3.0** pending). **2.4.0** is not published. Historical verification notes below reference **2.2.0** unless another version is named. Codebase Context runs as a local MCP server: a tool your AI editor or command-line client can call while it works. Register it without a fixed folder, then let the agent supply its current repository through the `project` tool parameter. diff --git a/docs/comparison-table.md b/docs/comparison-table.md index 1b55f11..96b11ba 100644 --- a/docs/comparison-table.md +++ b/docs/comparison-table.md @@ -1,6 +1,6 @@ # Corrected retrieval comparison -This table summarizes the corrected retrieval-only family across five local code-context tools. In this fixed setup, Codebase Context recovered 25.7% of expected gold files at 11.5% file precision; this does not show better completed coding tasks. Expected-file coverage means expected gold files recovered divided by expected gold files. File precision means expected gold files recovered divided by all files returned. The reported `peakPrivateGb` field is copied from retained telemetry; its underlying unit was not independently verified. +This table summarizes the corrected retrieval-only family across five local code-context tools. **It is not a product ranking:** Codebase Context optimizes for team conventions and edit preflight; jCodeMunch leads this fixed expected-file metric at 27.1% versus Codebase Context at 25.7%. Expected-file coverage means expected gold files recovered divided by expected gold files. File precision means expected gold files recovered divided by all files returned. The reported `peakPrivateGb` field is copied from retained telemetry; its underlying unit was not independently verified. | Tool | Completed | Failed | Expected-file coverage | File precision | Reported `peakPrivateGb` | | --- | ---: | ---: | ---: | ---: | ---: | From 83698db7852806a29cc792caafa176205d2f0866 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 30 Oct 2026 22:31:14 +0000 Subject: [PATCH 07/15] feat(cli): extend init to six MCP clients with dry-run Configure Claude Code, Cursor, Codex, VS Code, Windsurf, and OpenCode in one pass with idempotent merges, user-scoped Windsurf paths, and explicit create/ update output. Add temp-HOME tests and document init in the README. --- README.md | 17 ++- docs/cli.md | 5 +- docs/client-setup.md | 7 +- src/cli-init.ts | 220 +++++++++++++++++++++------- src/cli.ts | 2 +- tests/cli-init-dry-run-home.test.ts | 91 ++++++++++++ tests/cli-init.test.ts | 36 ++++- 7 files changed, 315 insertions(+), 63 deletions(-) create mode 100644 tests/cli-init-dry-run-home.test.ts diff --git a/README.md b/README.md index d93d577..0e62882 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,14 @@ opencode mcp add codebase-context '--' npx -y codebase-context In the unreleased candidate, `--version` prints the installed package version and `map` builds a missing index before printing the map. A low-confidence search includes a hint that weak matches do not establish that code is absent. These additions require the built candidate; the unpinned commands above install published 2.2.0. -This repository's `init` command detects Claude Code, Codex, Cursor, and OpenCode and writes a project config for each selected tool. The saved launch is the Node binary and the package file from the npx cache. That command is in this source and is not part of published 2.2.0. Until it is published, use the copy-paste lines above. The [client setup guide](./docs/client-setup.md) marks the unpublished command. +This repository's `init` command detects Claude Code, Codex, Cursor, VS Code (Copilot), Windsurf, and OpenCode, then writes or merges each tool's MCP config in one pass. It follows the official VS Code `servers` and Windsurf user-config shapes (same pattern as the [@modelcontextprotocol/server-filesystem](https://www.npmjs.com/package/@modelcontextprotocol/server-filesystem) workspace examples). Use `--dry-run` to preview changes before writing. + +```bash +npx codebase-context init --yes # configure detected tools +npx codebase-context init --dry-run --yes # show Would create/update lines only +``` + +The saved launch is the Node binary and the package file from the npx cache. That command is in this source and is not part of published 2.2.0. Until it is published, use the copy-paste lines above. The [client setup guide](./docs/client-setup.md) marks the unpublished command. Start a new agent session in your project, then ask: @@ -31,7 +38,7 @@ Replace `[feature]` with something you want to find. The first run downloads a l ### Planned for 2.4.0 (not yet published) -The unreleased source candidate starts new indexes without a model download. Semantic embeddings are optional; existing indexes keep their recorded mode. Its setup wizard can configure Claude Code, Codex, Cursor and OpenCode using the Node binary and package file that ran setup. These changes are not in the current npm release. +The unreleased source candidate starts new indexes without a model download. Semantic embeddings are optional; existing indexes keep their recorded mode. Its setup wizard configures Claude Code, Codex, Cursor, VS Code (Copilot), Windsurf, and OpenCode using the Node binary and package file that ran setup, with `--dry-run` to preview MCP config changes. These changes are not in the current npm release. - NestJS support from community contributor Aaron Olin. See the [analyzer details](./docs/capabilities.md#analyzers). - Better React/Next.js detection and a fix for reranker passage scoring. @@ -57,10 +64,10 @@ Other clients use their own setup commands: | Client | Shortest current setup | | --------------------------- | ---------------------------------------------------------------------------- | | Gemini CLI | `gemini mcp add --scope user codebase-context npx -y codebase-context` | -| Cursor | Add `.cursor/mcp.json` | -| VS Code with GitHub Copilot | Add `.vscode/mcp.json` | +| Cursor | `codebase-context init --client cursor` or add `.cursor/mcp.json` | +| VS Code with GitHub Copilot | `codebase-context init --client vscode` or add `.vscode/mcp.json` | | GitHub Copilot CLI | `copilot mcp add codebase-context -- npx -y codebase-context` | -| Windsurf | Add `~/.codeium/windsurf/mcp_config.json` | +| Windsurf | `codebase-context init --client windsurf` or add `~/.codeium/windsurf/mcp_config.json` | Check an existing same-name entry before replacing it. To give the server a default folder, append that folder's absolute path to the `npx` arguments. The [client setup guide](./docs/client-setup.md) covers scopes, optional fixed-folder configuration and the interactive setup wizard's limitations. diff --git a/docs/cli.md b/docs/cli.md index 9ae65f6..16dae76 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -46,16 +46,17 @@ CODEBASE_CONTEXT_ASCII=1 npx -y codebase-context patterns - `reindex` — rebuild index (full or incremental) - `style-guide` — find style guide sections in docs - `memory list|add|remove` — manage team memory (stored in `.codebase-context/memory.json`) -- MCP client registration — use the recipes in the [client setup guide](./client-setup.md). Published 2.2.0 has the older interactive HTTP wizard; the four-client wizard below requires the unreleased candidate. +- MCP client registration — use the recipes in the [client setup guide](./client-setup.md). Published 2.2.0 has the older interactive HTTP wizard; the six-client wizard below requires the unreleased candidate. ## `init` in the unreleased candidate -`init` detects Claude Code, Codex, Cursor and OpenCode, then configures the selected tools together. In a terminal, detected tools start selected; use Space to change the selection and Enter to apply it. Without a terminal, it configures all detected tools. +`init` detects Claude Code, Codex, Cursor, VS Code (Copilot), Windsurf, and OpenCode, then configures the selected tools together. In a terminal, detected tools start selected; use Space to change the selection and Enter to apply it. Without a terminal, it configures all detected tools. Add `--dry-run` to print `Would create`, `Would update`, or `Unchanged` without writing files. ```powershell codebase-context init --help codebase-context init codebase-context init --yes +codebase-context init --dry-run --yes codebase-context init --client codex --client cursor --root "C:\path\to\project" ``` diff --git a/docs/client-setup.md b/docs/client-setup.md index f43ab54..d992460 100644 --- a/docs/client-setup.md +++ b/docs/client-setup.md @@ -106,9 +106,12 @@ On October 6 the exact published `2.2.0` `npx` entrypoint was tested without a f The four-client wizard is not in published 2.2.0. From a built checkout or a locally installed candidate, run its actual entrypoint from the project you want to search: ```bash -# Detect Claude Code, Codex, Cursor and OpenCode, then configure the detected clients. +# Detect Claude Code, Codex, Cursor, VS Code, Windsurf, and OpenCode, then configure the detected clients. node /path/to/codebase-context/dist/index.js init --yes +# Preview MCP config changes without writing files. +node /path/to/codebase-context/dist/index.js init --dry-run --yes + # Configure one client for an explicit project. node /path/to/codebase-context/dist/index.js init --client codex --root /path/to/project --yes @@ -118,7 +121,7 @@ node /path/to/codebase-context/dist/index.js init --root /path/to/project --prev Replace the paths with the candidate installation and project folders. With an interactive terminal, omit `--yes` to select clients together. `--client` is repeatable. `--root` takes priority over `CODEBASE_ROOT` and the current folder. -Claude Code uses `.mcp.json`, Codex uses `.codex/config.toml`, Cursor uses `.cursor/mcp.json`, and OpenCode uses `opencode.json`. Generated configs save the running Node binary, the actual package entrypoint and the project folder. If setup ran through `npx`, that entrypoint is in its package cache. Rerun setup if that installation or cache is removed or moved. +Claude Code uses `.mcp.json`, Codex uses `.codex/config.toml`, Cursor uses `.cursor/mcp.json`, VS Code uses `.vscode/mcp.json` (`servers`), Windsurf uses `~/.codeium/windsurf/mcp_config.json`, and OpenCode uses `opencode.json`. Generated configs save the running Node binary, the actual package entrypoint and the project folder. If setup ran through `npx`, that entrypoint is in its package cache. Rerun setup if that installation or cache is removed or moved. Existing unrelated settings are preserved. A conflicting custom Codebase Context launch stops setup before selected config files are written. To replace a custom launch, remove only its Codebase Context entry from the client config, keep the other settings, and rerun setup. Restart the client and approve the server or project trust if asked. Config generation and SDK launch tests do not establish acceptance in each native client. diff --git a/src/cli-init.ts b/src/cli-init.ts index 4641e84..1888986 100644 --- a/src/cli-init.ts +++ b/src/cli-init.ts @@ -4,7 +4,11 @@ import { loadProjectConfig } from './server/config.js'; /** * Interactive setup wizard for codebase-context. * Handles `codebase-context init` — detects clients and merges project MCP configs - * for the four first-wave AI clients: Claude Code, Cursor, Codex, OpenCode. + * for Claude Code, Cursor, Codex, VS Code (Copilot), Windsurf, and OpenCode. + * Config merge shapes follow each client's official docs (VS Code uses a top-level + * `servers` object per the MCP filesystem server workspace examples; Windsurf uses + * `~/.codeium/windsurf/mcp_config.json`). Use `--dry-run` to preview writes first, + * matching the preview-before-apply pattern from MCP filesystem `edit_file`. * * Config writes run only after all selected registrations pass preflight. * File previews share the indexer's selection rules and project configuration. @@ -19,12 +23,12 @@ import { checkbox } from '@inquirer/prompts'; import { homedir } from 'node:os'; import { constants } from 'node:fs'; -export type Client = (typeof CLIENTS)[number]['id']; - export type McpConfigResult = { kind: 'file'; path: string; content: string }; export type McpConfigOptions = { rootPath?: string; + /** Override home for user-scoped configs (tests). Defaults to os.homedir(). */ + homeDirectory?: string; /** Node binary written into the client config. Defaults to the current process. */ nodeExecutable?: string; /** @@ -37,19 +41,79 @@ export type McpConfigOptions = { type JsonObject = Record; export const CLIENTS = [ - { id: 'claude-code', name: 'Claude Code', executable: 'claude', directory: '.claude' }, - { id: 'codex', name: 'Codex', executable: 'codex', directory: '.codex' }, - { id: 'cursor', name: 'Cursor', executable: 'cursor', directory: '.cursor' }, - { id: 'opencode', name: 'OpenCode', executable: 'opencode', directory: '.opencode' } + { + id: 'claude-code', + name: 'Claude Code', + executable: 'claude', + directory: '.claude', + scope: 'project' + }, + { id: 'codex', name: 'Codex', executable: 'codex', directory: '.codex', scope: 'project' }, + { id: 'cursor', name: 'Cursor', executable: 'cursor', directory: '.cursor', scope: 'project' }, + { + id: 'vscode', + name: 'VS Code (Copilot)', + executable: 'code', + directory: '.vscode', + scope: 'project' + }, + { + id: 'windsurf', + name: 'Windsurf', + executable: 'windsurf', + directory: '.codeium/windsurf', + scope: 'user' + }, + { + id: 'opencode', + name: 'OpenCode', + executable: 'opencode', + directory: '.opencode', + scope: 'project' + } ] as const; +export type Client = (typeof CLIENTS)[number]['id']; + +function resolveHome( + env: NodeJS.ProcessEnv = process.env, + platform: NodeJS.Platform = process.platform +): string { + const paths = platform === 'win32' ? path.win32 : path.posix; + return env.HOME || env.USERPROFILE || homedir(); +} + +function jsonMcpServerKey( + client: Extract +): string { + if (client === 'opencode') return 'mcp'; + if (client === 'vscode') return 'servers'; + return 'mcpServers'; +} + +export function resolveClientConfigPath( + client: Client, + rootPath: string, + env: NodeJS.ProcessEnv = process.env, + platform: NodeJS.Platform = process.platform +): string { + const paths = platform === 'win32' ? path.win32 : path.posix; + const home = resolveHome(env, platform); + if (client === 'codex') return path.join(rootPath, '.codex', 'config.toml'); + if (client === 'windsurf') return path.join(home, '.codeium', 'windsurf', 'mcp_config.json'); + if (client === 'claude-code') return path.join(rootPath, '.mcp.json'); + if (client === 'cursor') return path.join(rootPath, '.cursor', 'mcp.json'); + if (client === 'vscode') return path.join(rootPath, '.vscode', 'mcp.json'); + return path.join(rootPath, 'opencode.json'); +} + export async function detectClients( rootPath: string, env: NodeJS.ProcessEnv = process.env, platform: NodeJS.Platform = process.platform ): Promise { const paths = platform === 'win32' ? path.win32 : path.posix; - const home = env.HOME || env.USERPROFILE || homedir(); + const home = resolveHome(env, platform); const configHome = env.XDG_CONFIG_HOME || paths.join(home, '.config'); const searchPath = (env.PATH || env.Path || '').split(platform === 'win32' ? ';' : ':'); const extensions = platform === 'win32' ? ['', '.exe', '.cmd', '.bat', '.ps1'] : ['']; @@ -62,8 +126,15 @@ export async function detectClients( ? env.CLAUDE_CONFIG_DIR || paths.join(home, '.claude') : client.id === 'opencode' ? paths.join(configHome, 'opencode') - : paths.join(home, client.directory); - const directories = [paths.join(rootPath, client.directory), userDirectory]; + : client.id === 'windsurf' + ? paths.join(home, '.codeium', 'windsurf') + : paths.join(home, client.directory.split('/').at(-1) ?? client.directory); + const projectDirectory = + client.id === 'windsurf' + ? paths.join(home, '.codeium', 'windsurf') + : paths.join(rootPath, client.directory); + const directories = + client.scope === 'user' ? [userDirectory] : [projectDirectory, userDirectory]; const executables = searchPath .filter(Boolean) .flatMap((directory) => @@ -116,47 +187,45 @@ function resolveServerEntrypoint(explicit?: string): string { */ export function generateMcpConfig(client: Client, options: McpConfigOptions = {}): McpConfigResult { const rootPath = path.resolve(options.rootPath ?? process.cwd()); + const homeDirectory = options.homeDirectory ?? resolveHome(); const nodeExecutable = path.resolve(options.nodeExecutable ?? process.execPath); const serverEntrypoint = resolveServerEntrypoint(options.serverEntrypoint); const args = [serverEntrypoint, rootPath]; if (client === 'codex') { return { kind: 'file', - path: path.join(rootPath, '.codex', 'config.toml'), + path: resolveClientConfigPath('codex', rootPath, { HOME: homeDirectory }), content: `[mcp_servers.codebase-context]\ncommand = ${JSON.stringify(nodeExecutable)}\n` + `args = [${args.map((value) => JSON.stringify(value)).join(', ')}]\nstartup_timeout_sec = 120\n` }; } - const configPath = - client === 'claude-code' - ? '.mcp.json' + const configPath = resolveClientConfigPath(client, rootPath, { HOME: homeDirectory }); + const serverPayload = + client === 'opencode' + ? { type: 'local', command: [nodeExecutable, ...args] } : client === 'cursor' - ? '.cursor/mcp.json' - : 'opencode.json'; + ? { type: 'stdio', command: nodeExecutable, args } + : { command: nodeExecutable, args }; + const body = + client === 'opencode' + ? { + $schema: 'https://opencode.ai/config.json', + [jsonMcpServerKey('opencode')]: { 'codebase-context': serverPayload } + } + : client === 'vscode' + ? { [jsonMcpServerKey('vscode')]: { 'codebase-context': serverPayload } } + : client === 'windsurf' + ? { [jsonMcpServerKey('windsurf')]: { 'codebase-context': serverPayload } } + : { + mcpServers: { + 'codebase-context': serverPayload + } + }; return { kind: 'file', - path: path.join(rootPath, configPath), - content: JSON.stringify( - client === 'opencode' - ? { - $schema: 'https://opencode.ai/config.json', - mcp: { - 'codebase-context': { type: 'local', command: [nodeExecutable, ...args] } - } - } - : { - mcpServers: { - 'codebase-context': { - ...(client === 'cursor' ? { type: 'stdio' } : {}), - command: nodeExecutable, - args - } - } - }, - null, - 2 - ) + path: configPath, + content: JSON.stringify(body, null, 2) }; } @@ -281,13 +350,12 @@ function mergeGeneratedServer(existing: unknown, generated: unknown, client: Cli if (jsonValuesEqual(existing, generated)) return existing; if (!isJsonObject(existing) || !isJsonObject(generated) || 'url' in existing) return null; const local = client === 'opencode'; + const stdio = client === 'cursor'; const command = local && Array.isArray(existing.command) ? existing.command[0] : existing.command; const args = local && Array.isArray(existing.command) ? existing.command.slice(1) : existing.args; - if ( - (existing.type !== undefined && existing.type !== (local ? 'local' : 'stdio')) || - !isGeneratedLaunch(command, args, generatedEntrypoint(generated, client)) - ) - return null; + if (local && existing.type !== undefined && existing.type !== 'local') return null; + if (stdio && existing.type !== undefined && existing.type !== 'stdio') return null; + if (!isGeneratedLaunch(command, args, generatedEntrypoint(generated, client))) return null; return { ...existing, command: generated.command, @@ -303,7 +371,7 @@ function mergeGeneratedServer(existing: unknown, generated: unknown, client: Cli export async function _buildMergedMcpContent( filePath: string, generatedContent: string, - client: Extract + client: Extract ): Promise<{ content: string; mergedFromExisting: boolean }> { const existingText = await readExistingConfig(filePath); if (existingText === null) { @@ -324,7 +392,7 @@ export async function _buildMergedMcpContent( throw new Error(`Cannot merge ${client} config at ${filePath}: expected a JSON object.`); } - const serverKey = client === 'opencode' ? 'mcp' : 'mcpServers'; + const serverKey = jsonMcpServerKey(client); const existingServersValue = existing[serverKey]; if (existingServersValue !== undefined && !isJsonObject(existingServersValue)) { throw new Error( @@ -461,8 +529,40 @@ type InitOptions = { yes: boolean; help: boolean; preview: boolean; + dryRun: boolean; }; +function formatDisplayPath(configPath: string, rootPath: string): string { + const relative = path.relative(rootPath, configPath); + if (relative && !relative.startsWith('..') && !path.isAbsolute(relative)) return relative; + return configPath; +} + +function classifyInitChange( + existingText: string | null, + mergedContent: string +): 'create' | 'update' | 'unchanged' { + if (existingText === null) return 'create'; + if (existingText === mergedContent) return 'unchanged'; + return 'update'; +} + +function describeInitChange( + dryRun: boolean, + action: 'create' | 'update' | 'unchanged', + displayPath: string +): string { + if (action === 'unchanged') return `Unchanged: ${displayPath}`; + const prefix = dryRun + ? action === 'create' + ? 'Would create' + : 'Would update' + : action === 'create' + ? 'Created' + : 'Updated'; + return `${prefix}: ${displayPath}`; +} + function parseClient(value: string): Client { const client = CLIENTS.find((client) => client.id === value); if (client) return client.id; @@ -473,13 +573,22 @@ function parseClient(value: string): Client { function parseInitOptions(args: string[]): InitOptions { if (args.includes('--help') || args.includes('-h')) - return { clients: [], rootPath: process.cwd(), yes: false, help: true, preview: false }; + return { + clients: [], + rootPath: process.cwd(), + yes: false, + help: true, + preview: false, + dryRun: false + }; const clients: Client[] = []; let yes = false; let preview = false; + let dryRun = false; for (let index = 0; index < args.length; index++) { const arg = args[index]; if (arg === '--preview') preview = true; + else if (arg === '--dry-run') dryRun = true; else if (arg === '--yes' || arg === '-y') yes = true; else if (arg === '--client') { const value = args[++index]; @@ -495,7 +604,8 @@ function parseInitOptions(args: string[]): InitOptions { rootPath: resolveInitRoot(args, process.env.CODEBASE_ROOT), yes, help: false, - preview + preview, + dryRun }; } @@ -514,6 +624,7 @@ function printInitUsage(): void { ); console.log(' --root Project root (default: CODEBASE_ROOT or cwd)'); console.log(' --preview Show selected files and exclusions without writing config'); + console.log(' --dry-run Show MCP config changes without writing files'); console.log(' --yes, -y Configure every detected tool without prompts'); console.log(' --help, -h Show this help'); } @@ -554,30 +665,35 @@ export async function handleInitCli(args: string[]): Promise { } if (!clients.length) throw new Error( - 'No coding tools selected or detected. Choose with --client claude-code, --client codex, --client cursor, or --client opencode.' + `No coding tools selected or detected. Choose with --client (${CLIENTS.map((client) => client.id).join(', ')}).` ); const prepared = []; for (const client of clients) { const config = generateMcpConfig(client, { rootPath: options.rootPath }); + const existingText = await readExistingConfig(config.path); const merged = client === 'codex' ? await _buildMergedCodexContent(config.path, config.content) : await _buildMergedMcpContent(config.path, config.content, client); + const action = classifyInitChange(existingText, merged.content); prepared.push({ ...config, content: merged.content, - unchanged: (await readExistingConfig(config.path)) === merged.content + action, + displayPath: formatDisplayPath(config.path, options.rootPath) }); } for (const config of prepared) { - if (!config.unchanged) { + if (config.action !== 'unchanged' && !options.dryRun) { await fs.mkdir(path.dirname(config.path), { recursive: true }); await atomicWrite(config.path, config.content); } - console.log( - `${config.unchanged ? 'Unchanged' : 'Configured'}: ${path.relative(options.rootPath, config.path)}` - ); + console.log(describeInitChange(options.dryRun, config.action, config.displayPath)); + } + if (options.dryRun) { + console.log('Dry run complete; no files written.'); + return; } console.log('Reopen your tools; approve the server or project trust if asked.'); console.log("The first search builds this repository's index."); diff --git a/src/cli.ts b/src/cli.ts index 345eaec..b074f43 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -83,7 +83,7 @@ function printUsage(): void { console.log(' patterns [--category all|di|state|testing|libraries] Team patterns'); console.log(' refs --symbol [--limit ] Symbol references'); console.log(' cycles [--scope ] Circular dependency detection'); - console.log(' init [--client ] [--yes] Detect and configure coding tools'); + console.log(' init [--client ] [--yes] [--dry-run] Configure detected coding tools'); console.log(' map [--json] [--pretty] Conventions map (indexes first if needed)'); console.log(''); console.log('Global flags:'); diff --git a/tests/cli-init-dry-run-home.test.ts b/tests/cli-init-dry-run-home.test.ts new file mode 100644 index 0000000..ff95b43 --- /dev/null +++ b/tests/cli-init-dry-run-home.test.ts @@ -0,0 +1,91 @@ +import { describe, it, expect, afterEach, vi } from 'vitest'; +import { promises as fs } from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { handleInitCli } from '../src/cli-init.js'; + +describe('init dry-run and user-scoped configs', () => { + const previousHome = process.env.HOME; + const previousUserProfile = process.env.USERPROFILE; + let tempHome = ''; + let projectRoot = ''; + let logs: string[] = []; + + afterEach(async () => { + process.env.HOME = previousHome; + process.env.USERPROFILE = previousUserProfile; + if (tempHome) await fs.rm(tempHome, { recursive: true, force: true }); + tempHome = ''; + projectRoot = ''; + logs = []; + }); + + async function runInit(args: string[]) { + logs = []; + const logSpy = vi.spyOn(console, 'log').mockImplementation((...parts: unknown[]) => { + logs.push(parts.map(String).join(' ')); + }); + try { + await handleInitCli(args); + } finally { + logSpy.mockRestore(); + } + return logs.join('\n'); + } + + it('dry-run writes Windsurf user config under a temp HOME without touching disk', async () => { + tempHome = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-init-home-')); + projectRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-init-project-')); + process.env.HOME = tempHome; + process.env.USERPROFILE = tempHome; + await fs.mkdir(path.join(tempHome, '.codeium', 'windsurf'), { recursive: true }); + + const output = await runInit([ + '--client', + 'windsurf', + '--yes', + '--dry-run', + '--root', + projectRoot + ]); + + expect(output).toContain(`Would create: ${path.join(tempHome, '.codeium', 'windsurf', 'mcp_config.json')}`); + expect(output).toContain('Dry run complete; no files written.'); + await expect( + fs.access(path.join(tempHome, '.codeium', 'windsurf', 'mcp_config.json')) + ).rejects.toMatchObject({ code: 'ENOENT' }); + }); + + it('merges VS Code servers and reports an update on repeat dry-run', async () => { + tempHome = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-init-home-')); + projectRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-init-project-')); + process.env.HOME = tempHome; + process.env.USERPROFILE = tempHome; + const vscodeDir = path.join(projectRoot, '.vscode'); + await fs.mkdir(vscodeDir, { recursive: true }); + await fs.writeFile( + path.join(vscodeDir, 'mcp.json'), + JSON.stringify( + { + servers: { + other: { command: 'echo', args: ['hi'] } + } + }, + null, + 2 + ) + ); + + const first = await runInit(['--client', 'vscode', '--yes', '--root', projectRoot]); + expect(first).toContain('Updated: .vscode/mcp.json'); + + const second = await runInit(['--client', 'vscode', '--yes', '--dry-run', '--root', projectRoot]); + expect(second).toContain('Unchanged: .vscode/mcp.json'); + + const saved = JSON.parse(await fs.readFile(path.join(vscodeDir, 'mcp.json'), 'utf8')) as { + servers: Record; + }; + expect(saved.servers.other).toEqual({ command: 'echo', args: ['hi'] }); + expect(saved.servers['codebase-context']).toBeDefined(); + }); +}); diff --git a/tests/cli-init.test.ts b/tests/cli-init.test.ts index bfff804..b397c7f 100644 --- a/tests/cli-init.test.ts +++ b/tests/cli-init.test.ts @@ -120,6 +120,33 @@ describe('generateMcpConfig', () => { command: [nodeExecutable, cacheEntrypoint, projectRoot] }); }); + + it('VS Code writes workspace servers config with the saved launch', () => { + const result = generateMcpConfig('vscode', savedLaunch(projectRoot)); + expect(result.path).toBe(path.join(projectRoot, '.vscode', 'mcp.json')); + expect(JSON.parse(result.content)).toEqual({ + servers: { + 'codebase-context': { + command: nodeExecutable, + args: [cacheEntrypoint, projectRoot] + } + } + }); + }); + + it('Windsurf writes user mcp_config.json with the saved launch', () => { + const home = path.resolve('C:/Users/Test User'); + const result = generateMcpConfig('windsurf', { ...savedLaunch(projectRoot), homeDirectory: home }); + expect(result.path).toBe(path.join(home, '.codeium', 'windsurf', 'mcp_config.json')); + expect(JSON.parse(result.content)).toEqual({ + mcpServers: { + 'codebase-context': { + command: nodeExecutable, + args: [cacheEntrypoint, projectRoot] + } + } + }); + }); }); describe('resolveInitRoot', () => { @@ -382,7 +409,14 @@ describe('detectClients', () => { const clients = await detectClients('/project', { PATH: 'C:/tools' }, 'win32'); - expect(CLIENTS.map(({ id }) => id)).toEqual(['claude-code', 'codex', 'cursor', 'opencode']); + expect(CLIENTS.map(({ id }) => id)).toEqual([ + 'claude-code', + 'codex', + 'cursor', + 'vscode', + 'windsurf', + 'opencode' + ]); expect(clients).toContain('codex'); expect(clients).not.toContain('claude-code'); expect( From 0265d4d8fcbbfdbed8960a8fbede81463a57014c Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 30 Oct 2026 22:32:16 +0000 Subject: [PATCH 08/15] fix(cli): detect npm bin symlinks for --version and subcommands Port canonical realpath entrypoint matching from master so npx-style launchers named codebase-context dispatch the CLI instead of exiting silently. --- src/index.ts | 19 +++++++++++++++---- tests/cli-version.test.ts | 25 +++++++++++++++++++++++-- 2 files changed, 38 insertions(+), 6 deletions(-) diff --git a/src/index.ts b/src/index.ts index 038f8f6..e20871d 100644 --- a/src/index.ts +++ b/src/index.ts @@ -5,7 +5,7 @@ * Provides codebase indexing and semantic search capabilities */ -import { promises as fs } from 'fs'; +import { promises as fs, realpathSync } from 'fs'; import path from 'path'; import { fileURLToPath } from 'url'; @@ -97,10 +97,21 @@ const CLI_SUBCOMMANDS = [ 'help' ]; -// Check if this module is the entry point. +function canonicalEntrypointPath(filePath: string | undefined): string | undefined { + if (!filePath) return undefined; + try { + const canonicalPath = path.normalize(realpathSync(filePath)); + return process.platform === 'win32' ? canonicalPath.toLowerCase() : canonicalPath; + } catch { + return undefined; + } +} + +// Resolve symlink launchers while keeping imports and unrelated index files inert. +const moduleEntrypointPath = canonicalEntrypointPath(fileURLToPath(import.meta.url)); +const processEntrypointPath = canonicalEntrypointPath(process.argv[1]); const isDirectRun = - process.argv[1]?.replace(/\\/g, '/').endsWith('index.js') || - process.argv[1]?.replace(/\\/g, '/').endsWith('index.ts'); + moduleEntrypointPath !== undefined && moduleEntrypointPath === processEntrypointPath; const directSubcommand = process.argv[2]; const isDirectCliSubcommand = isDirectRun && diff --git a/tests/cli-version.test.ts b/tests/cli-version.test.ts index c4c56f6..be57d93 100644 --- a/tests/cli-version.test.ts +++ b/tests/cli-version.test.ts @@ -1,6 +1,7 @@ import { spawnSync } from 'node:child_process'; -import { readFileSync } from 'node:fs'; -import { resolve } from 'node:path'; +import { mkdtempSync, readFileSync, rmSync, symlinkSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join, resolve } from 'node:path'; import { describe, expect, it } from 'vitest'; const root = resolve(import.meta.dirname, '..'); @@ -23,4 +24,24 @@ describe('version flag', () => { expect(result.status).toBe(0); expect(result.stdout.trim()).toBe(packageJson.version); }); + + it('prints the package version when invoked through an npm-style symlink', () => { + const entrypoint = resolve(root, 'src', 'index.ts'); + const tempDirectory = mkdtempSync(join(tmpdir(), 'cbc-cli-version-bin-')); + const linkedEntrypoint = join(tempDirectory, 'codebase-context'); + + symlinkSync(entrypoint, linkedEntrypoint, 'file'); + try { + const result = spawnSync( + process.execPath, + ['--import', 'tsx', linkedEntrypoint, '--version'], + { cwd: root, encoding: 'utf8', timeout: 30_000 } + ); + + expect(result.status, result.stderr).toBe(0); + expect(result.stdout.trim()).toBe(packageJson.version); + } finally { + rmSync(tempDirectory, { recursive: true, force: true }); + } + }); }); From 28e71e31a21a09d14e83ca335ac5745ae1e64216 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 30 Oct 2026 22:33:16 +0000 Subject: [PATCH 09/15] fix: triage quick wins for chunk cap, map, and path search Clarify searchable chunk limits, exclude vendored bundles from bounded map, and boost keyword hits for file-like query tokens. Adds regression tests. --- README.md | 2 +- docs/capabilities.md | 2 +- src/cli.ts | 8 +-- src/core/codebase-map.ts | 6 +- src/core/indexer.ts | 36 ++++++++-- src/core/search.ts | 75 +++++++++++++++++++-- src/index.ts | 8 +-- src/types/index.ts | 4 ++ tests/codebase-map.test.ts | 27 ++++++++ tests/indexer-chunk-limit-reporting.test.ts | 51 ++++++++++++++ tests/search-path-token-boost.test.ts | 23 +++++++ 11 files changed, 219 insertions(+), 23 deletions(-) create mode 100644 tests/indexer-chunk-limit-reporting.test.ts create mode 100644 tests/search-path-token-boost.test.ts diff --git a/README.md b/README.md index d93d577..a61facd 100644 --- a/README.md +++ b/README.md @@ -162,7 +162,7 @@ The method and failures are documented so the measurements can be inspected with - The paired token observation covers two frozen investigation tasks and records observed agent behavior; it is not a universal token or time guarantee. - Setup checks differ by client. The detailed guide distinguishes a written config, a config recognized by the client, a local connection, and instructions checked only against official docs. - Dedicated Angular, React, and Next.js analyzers ship in published 2.2.0; NestJS support is planned for unpublished 2.4.0. Other projects use the generic analyzer and the language parsers available for that stack. -- The default searchable-chunk limit is 5,000 per project. Customizing this via `projects[].parsing.maxChunks` in `~/.codebase-context/config.json` requires 2.4.0 (not yet published). +- The default searchable-chunk limit is 5,000 per project. Raise it with `projects[].parsing.maxChunks` in `~/.codebase-context/config.json` (see [capabilities](docs/capabilities.md)). - The agent must identify its repository in tool calls when no default or unambiguous client root is available. Concurrent HTTP client isolation is not established by the stdio routing checks. ## Reference diff --git a/docs/capabilities.md b/docs/capabilities.md index 49fc4f9..8211c65 100644 --- a/docs/capabilities.md +++ b/docs/capabilities.md @@ -22,7 +22,7 @@ Per-project config overrides supported today: - `projects[].excludePatterns`: merged with the built-in exclusion set for that project at index time - `projects[].analyzerHints.analyzer`: prefers a registered analyzer by name for that project and falls back safely when the name is missing or invalid - `projects[].analyzerHints.extensions`: adds project-local source extensions for indexing and auto-refresh watching without changing defaults for other projects -- `projects[].parsing.maxChunks`: sets the maximum number of searchable chunks indexed for that project (requires 2.4.0, not yet published) +- `projects[].parsing.maxChunks`: sets the maximum number of searchable chunks indexed for that project (default 5000; configurable via `~/.codebase-context/config.json`) Copy-pasteable client config templates are shipped in the package: diff --git a/src/cli.ts b/src/cli.ts index 345eaec..2121e73 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -9,7 +9,7 @@ import path from 'path'; import { promises as fs } from 'fs'; import { KEYWORD_INDEX_FILENAME } from './constants/codebase-context.js'; import { makePaths } from './project-state.js'; -import { CodebaseIndexer } from './core/indexer.js'; +import { CodebaseIndexer, formatIndexingCompleteLine } from './core/indexer.js'; import { dispatchTool } from './tools/index.js'; import type { ToolContext } from './tools/index.js'; import type { IndexState } from './tools/types.js'; @@ -155,11 +155,7 @@ async function initToolContext(): Promise { indexState.lastIndexed = new Date(); indexState.stats = stats; - console.error( - `Complete: ${stats.indexedFiles} files, ${stats.totalChunks} chunks in ${( - stats.duration / 1000 - ).toFixed(2)}s` - ); + console.error(formatIndexingCompleteLine(stats)); } catch (error) { indexState.status = 'error'; indexState.error = error instanceof Error ? error.message : String(error); diff --git a/src/core/codebase-map.ts b/src/core/codebase-map.ts index 87b491f..8ae5b05 100644 --- a/src/core/codebase-map.ts +++ b/src/core/codebase-map.ts @@ -78,6 +78,9 @@ const MAP_EXCLUDED_PATH_PATTERNS = [ /(?:^|\/)(?:fixtures?|__fixtures__)(?:\/|$)/i, /(?:^|\/)(?:generated|__generated__)(?:\/|$)/i, /(?:^|\/)[^/]*\.(?:generated|gen|min)\.[^/]+$/i, + /(?:^|\/)raw(?:\/|$)/i, + /(?:^|\/)live-from-ai(?:\/|$)/i, + /[^/]+-[A-Za-z0-9]{6,}\.(?:js|css|mjs|cjs)$/i, /\.snap$/i ] as const; @@ -180,6 +183,7 @@ async function buildPinnedCodebaseMap( const entrypoints: string[] = []; for (const [file, imports] of Object.entries(graphImports)) { if (imports.length === 0) continue; // no imports — not an entrypoint + if (!isMapEligiblePath(file, mode)) continue; const importers = graphImportedBy[file]; if (!importers || importers.length === 0) { entrypoints.push(file); @@ -193,7 +197,7 @@ async function buildPinnedCodebaseMap( graphImportedBy ).map(([file, importers]) => ({ file, count: importers.length })); const hubFiles: string[] = sortByCountThenAlpha( - importedByCounts, + importedByCounts.filter(({ file }) => isMapEligiblePath(file, mode)), (x) => x.count, (x) => x.file ) diff --git a/src/core/indexer.ts b/src/core/indexer.ts index 6d9ed37..d3a04e9 100644 --- a/src/core/indexer.ts +++ b/src/core/indexer.ts @@ -141,6 +141,8 @@ export interface IndexerOptions { interface PersistedIndexingStats { indexedFiles: number; totalChunks: number; + searchableChunks?: number; + chunkLimitApplied?: boolean; totalFiles: number; generatedAt: string; } @@ -956,7 +958,9 @@ export class CodebaseIndexer { let chunksToEmbed = chunksForEmbedding; if (chunksForEmbedding.length > maxChunks) { console.warn( - `WARNING: ${chunksForEmbedding.length} chunks exceed limit. Indexing first ${maxChunks} chunks.` + `WARNING: ${chunksForEmbedding.length} chunks exceed the searchable limit (${maxChunks}). ` + + `Only the first ${maxChunks} chunks are indexed for search. ` + + `Raise projects[].parsing.maxChunks in ~/.codebase-context/config.json for this repo root.` ); chunksToEmbed = chunksForEmbedding.slice(0, maxChunks); } @@ -1098,8 +1102,10 @@ export class CodebaseIndexer { // Keyword index always uses ALL chunks (full regen) const indexPath = path.join(generationDirectory, KEYWORD_INDEX_FILENAME); // Memory safety: cap keyword index too - const keywordChunks = - allChunks.length > maxChunks ? allChunks.slice(0, maxChunks) : allChunks; + const chunkLimitApplied = allChunks.length > maxChunks; + const keywordChunks = chunkLimitApplied ? allChunks.slice(0, maxChunks) : allChunks; + stats.searchableChunks = keywordChunks.length; + stats.chunkLimitApplied = chunkLimitApplied; await fs.writeFile( indexPath, JSON.stringify({ @@ -1206,6 +1212,8 @@ export class CodebaseIndexer { const persistedStats: PersistedIndexingStats = { indexedFiles: stats.indexedFiles, totalChunks: stats.totalChunks, + searchableChunks: stats.searchableChunks, + chunkLimitApplied: stats.chunkLimitApplied, totalFiles: stats.totalFiles, generatedAt }; @@ -1264,7 +1272,7 @@ export class CodebaseIndexer { ); } else { console.error(`Indexing complete in ${stats.duration}ms`); - console.error(`Indexed ${stats.indexedFiles} files, ${stats.totalChunks} chunks`); + console.error(formatIndexingProgressSummary(stats)); } return stats; @@ -1589,3 +1597,23 @@ export class CodebaseIndexer { return { ...this.progress }; } } + +export function formatIndexingProgressSummary(stats: IndexingStats): string { + const searchable = stats.searchableChunks ?? stats.totalChunks; + if (stats.chunkLimitApplied && searchable < stats.totalChunks) { + return ( + `Indexed ${stats.indexedFiles} files, ${stats.totalChunks} chunks analyzed, ` + + `${searchable} chunks searchable (limit applied)` + ); + } + return `Indexed ${stats.indexedFiles} files, ${stats.totalChunks} chunks`; +} + +export function formatIndexingCompleteLine(stats: IndexingStats): string { + const searchable = stats.searchableChunks ?? stats.totalChunks; + const durationSec = (stats.duration / 1000).toFixed(2); + if (stats.chunkLimitApplied && searchable < stats.totalChunks) { + return `Complete: ${stats.indexedFiles} files, ${searchable}/${stats.totalChunks} chunks searchable in ${durationSec}s`; + } + return `Complete: ${stats.indexedFiles} files, ${stats.totalChunks} chunks in ${durationSec}s`; +} diff --git a/src/core/search.ts b/src/core/search.ts index 743a3b2..ce1d72b 100644 --- a/src/core/search.ts +++ b/src/core/search.ts @@ -39,6 +39,24 @@ import { VECTOR_DB_DIRNAME } from '../constants/codebase-context.js'; +/** File/path-like tokens in a natural-language query (e.g. NOW.md, check-agent-instructions.mjs). */ +export function extractQueryPathTokens(query: string): string[] { + const tokens = new Set(); + for (const match of query.matchAll(/\b([A-Za-z0-9][A-Za-z0-9._-]*\.(?:md|mdx|txt))\b/g)) { + tokens.add(match[1].toLowerCase()); + } + for (const match of query.matchAll(/\b([a-z][a-z0-9-]*(?:-[a-z0-9]+)+)\.(?:mjs|cjs|js|ts)\b/gi)) { + tokens.add(match[0].toLowerCase()); + tokens.add(match[1].toLowerCase()); + } + for (const match of query.matchAll(/\b([a-z][a-z0-9-]*(?:-[a-z0-9]+){2,})\b/gi)) { + if (match[1].length >= 12) { + tokens.add(match[1].toLowerCase()); + } + } + return [...tokens]; +} + export interface SearchOptions { useSemanticSearch?: boolean; useKeywordSearch?: boolean; @@ -1941,11 +1959,23 @@ export class CodebaseSearcher { } if (fuseResults.length === 0 || (fuseResults[0].score ?? 0) > KEYWORD_FUSE_OPTIONS.threshold!) { - const rescue = await this.rescueKeywordMatches(query, limit, filters, validation); - const seen = new Set(rescue.map(({ item }) => item.id)); - fuseResults = [...rescue, ...fuseResults.filter(({ item }) => !seen.has(item.id))].sort( - (a, b) => (a.score ?? 0) - (b.score ?? 0) + const pathRescue: FuseResult[] = this.matchChunksByPathTokens(query, filters).map( + (chunk, refIndex) => ({ + item: chunk, + refIndex, + score: 0 + }) ); + const rescue = await this.rescueKeywordMatches(query, limit, filters, validation); + const seen = new Set([ + ...pathRescue.map(({ item }) => item.id), + ...rescue.map(({ item }) => item.id) + ]); + fuseResults = [ + ...pathRescue, + ...rescue, + ...fuseResults.filter(({ item }) => !seen.has(item.id)) + ].sort((a, b) => (a.score ?? 0) - (b.score ?? 0)); } const current = await this.selectCurrentCandidates( fuseResults, @@ -1984,6 +2014,20 @@ export class CodebaseSearcher { score = Math.min(1.0, score + 0.1); } + for (const token of extractQueryPathTokens(query)) { + const base = path.basename(relativePathLower); + const stem = base.replace(/\.[^.]+$/, ''); + if ( + base === token || + relativePathLower.endsWith(`/${token}`) || + stem === token.replace(/\.[^.]+$/, '') + ) { + score = Math.min(1.0, score + 0.45); + } else if (stem.includes(token) || relativePathLower.includes(token)) { + score = Math.min(1.0, score + 0.25); + } + } + return { chunk, score @@ -1996,6 +2040,29 @@ export class CodebaseSearcher { ); } + private matchChunksByPathTokens(query: string, filters?: SearchFilters): CodeChunk[] { + const tokens = extractQueryPathTokens(query); + if (tokens.length === 0) { + return []; + } + return this.chunks.filter((item) => { + if (!this.matchesSearchFilters(item, filters)) { + return false; + } + const relativePathLower = item.relativePath.toLowerCase(); + const base = path.basename(relativePathLower); + const stem = base.replace(/\.[^.]+$/, ''); + return tokens.some( + (token) => + base === token || + relativePathLower.endsWith(`/${token}`) || + stem === token.replace(/\.[^.]+$/, '') || + stem.includes(token) || + relativePathLower.includes(token) + ); + }); + } + private generateRelevanceReason(chunk: CodeChunk, query: string): string { const reasons: string[] = []; diff --git a/src/index.ts b/src/index.ts index a3bd53b..3373378 100644 --- a/src/index.ts +++ b/src/index.ts @@ -13,7 +13,7 @@ import { fileURLToPath } from 'url'; import type { Server } from '@modelcontextprotocol/sdk/server/index.js'; import type { ProjectConfig } from './server/config.js'; import type { Resource } from '@modelcontextprotocol/sdk/types.js'; -import { CodebaseIndexer } from './core/indexer.js'; +import { CodebaseIndexer, formatIndexingCompleteLine } from './core/indexer.js'; import { analyzerRegistry } from './core/analyzer-registry.js'; import { AngularAnalyzer } from './analyzers/angular/index.js'; @@ -1389,11 +1389,7 @@ async function performIndexingOnce( project.indexState.lastIndexed = new Date(); project.indexState.stats = stats; - console.error( - `Complete: ${stats.indexedFiles} files, ${stats.totalChunks} chunks in ${( - stats.duration / 1000 - ).toFixed(2)}s` - ); + console.error(formatIndexingCompleteLine(stats)); // Auto-extract memories from git history (non-blocking, best-effort) try { diff --git a/src/types/index.ts b/src/types/index.ts index 619b5e9..230e603 100644 --- a/src/types/index.ts +++ b/src/types/index.ts @@ -433,6 +433,10 @@ export interface IndexingStats { indexedFiles: number; skippedFiles: number; totalChunks: number; + /** Chunks written to the keyword index (may be lower than totalChunks when capped). */ + searchableChunks?: number; + /** True when parsing.maxChunks (default 5000) truncated the searchable index. */ + chunkLimitApplied?: boolean; totalLines: number; duration: number; // milliseconds avgChunkSize: number; diff --git a/tests/codebase-map.test.ts b/tests/codebase-map.test.ts index 10f5b49..e390c99 100644 --- a/tests/codebase-map.test.ts +++ b/tests/codebase-map.test.ts @@ -126,6 +126,33 @@ describe('buildCodebaseMap', () => { expect(map.architecture.entrypoints).toEqual(['src/cli.ts', 'src/index.ts']); }); + it('excludes vendored raw bundles from bounded map entrypoints and layers', async () => { + const rootPath = await createTempMapProject({ + graph: { + imports: { + 'live-from-ai/raw/index-BMtpUqT3.js': ['live-from-ai/raw/vendor.js'], + 'NOW.md': [], + 'scripts/context-route.mjs': ['NOW.md'] + }, + importedBy: { + 'live-from-ai/raw/index-BMtpUqT3.js': [], + 'NOW.md': ['scripts/context-route.mjs'], + 'scripts/context-route.mjs': [] + }, + exports: {} + } + }); + try { + const project = createProjectState(rootPath); + const map = await buildCodebaseMap(project); + expect(map.architecture.layers.map((layer) => layer.name)).not.toContain('live-from-ai'); + expect(map.architecture.entrypoints.some((ep) => ep.includes('index-BMtpUqT3'))).toBe(false); + expect(map.architecture.entrypoints).toContain('scripts/context-route.mjs'); + } finally { + await removeTempMapProject(rootPath); + } + }); + it('derives hub files: top 5 by importedBy count, sorted count-desc then alpha', async () => { const project = createProjectState(FIXTURE_ROOT); const map = await buildCodebaseMap(project, { mode: 'full' }); diff --git a/tests/indexer-chunk-limit-reporting.test.ts b/tests/indexer-chunk-limit-reporting.test.ts new file mode 100644 index 0000000..0156b0a --- /dev/null +++ b/tests/indexer-chunk-limit-reporting.test.ts @@ -0,0 +1,51 @@ +import { describe, expect, it } from 'vitest'; +import { + formatIndexingCompleteLine, + formatIndexingProgressSummary +} from '../src/core/indexer.js'; +import type { IndexingStats } from '../src/types/index.js'; + +function baseStats(overrides: Partial = {}): IndexingStats { + return { + totalFiles: 100, + indexedFiles: 90, + skippedFiles: 10, + totalChunks: 88424, + searchableChunks: 5000, + chunkLimitApplied: true, + totalLines: 1, + duration: 2500, + avgChunkSize: 100, + componentsByType: {}, + componentsByLayer: { + core: 0, + feature: 0, + shared: 0, + infrastructure: 0, + unknown: 0 + }, + errors: [], + startedAt: new Date(), + ...overrides + }; +} + +describe('indexer chunk limit reporting', () => { + it('formatIndexingProgressSummary distinguishes analyzed vs searchable counts', () => { + const line = formatIndexingProgressSummary(baseStats()); + expect(line).toContain('88424 chunks analyzed'); + expect(line).toContain('5000 chunks searchable'); + }); + + it('formatIndexingCompleteLine shows searchable fraction when capped', () => { + const line = formatIndexingCompleteLine(baseStats()); + expect(line).toBe('Complete: 90 files, 5000/88424 chunks searchable in 2.50s'); + }); + + it('formatIndexingCompleteLine stays compact when no cap applied', () => { + const line = formatIndexingCompleteLine( + baseStats({ totalChunks: 120, searchableChunks: 120, chunkLimitApplied: false }) + ); + expect(line).toBe('Complete: 90 files, 120 chunks in 2.50s'); + }); +}); diff --git a/tests/search-path-token-boost.test.ts b/tests/search-path-token-boost.test.ts new file mode 100644 index 0000000..74a0ff2 --- /dev/null +++ b/tests/search-path-token-boost.test.ts @@ -0,0 +1,23 @@ +import { describe, expect, it } from 'vitest'; +import { extractQueryPathTokens } from '../src/core/search.js'; + +describe('extractQueryPathTokens', () => { + it('extracts markdown filenames from natural-language queries', () => { + const tokens = extractQueryPathTokens( + 'NOW.md live root entrypoint current state for routing' + ); + expect(tokens).toContain('now.md'); + }); + + it('extracts script paths and hyphenated stems', () => { + const tokens = extractQueryPathTokens( + 'what must I check before editing AGENTS.md protected agent instructions' + ); + expect(tokens).toContain('agents.md'); + }); + + it('extracts explicit script module names', () => { + const tokens = extractQueryPathTokens('run check-agent-instructions before commit'); + expect(tokens).toContain('check-agent-instructions'); + }); +}); From 4f9867d25ac5bd036e618c05f6cd8dec1c133d74 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 30 Oct 2026 22:34:06 +0000 Subject: [PATCH 10/15] fix(embeddings): bust provider cache when endpoint or API key changes OPENAI_BASE_URL wiring is useless if a cached OpenAI client keeps the first endpoint. --- src/embeddings/index.ts | 7 ++++- tests/embedding-provider-cache.test.ts | 39 ++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 1 deletion(-) create mode 100644 tests/embedding-provider-cache.test.ts diff --git a/src/embeddings/index.ts b/src/embeddings/index.ts index f8c1718..d419976 100644 --- a/src/embeddings/index.ts +++ b/src/embeddings/index.ts @@ -42,7 +42,12 @@ export async function getEmbeddingProvider( ): Promise { assertEmbeddingRuntimeAvailable(config); const mergedConfig = { ...DEFAULT_EMBEDDING_CONFIG, ...config }; - const providerKey = `${mergedConfig.provider}:${mergedConfig.model}`; + const providerKey = [ + mergedConfig.provider, + mergedConfig.model, + mergedConfig.apiEndpoint ?? '', + mergedConfig.apiKey ?? '' + ].join(':'); if (cachedProvider && cachedProviderType === providerKey) { return cachedProvider; diff --git a/tests/embedding-provider-cache.test.ts b/tests/embedding-provider-cache.test.ts new file mode 100644 index 0000000..f4e0228 --- /dev/null +++ b/tests/embedding-provider-cache.test.ts @@ -0,0 +1,39 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; + +describe('getEmbeddingProvider cache key', () => { + afterEach(() => { + vi.resetModules(); + vi.doUnmock('../src/embeddings/openai.js'); + }); + + it('rebuilds the OpenAI provider when apiEndpoint changes', async () => { + const endpoints: string[] = []; + vi.doMock('../src/embeddings/openai.js', () => ({ + OpenAIEmbeddingProvider: class { + constructor(_model: string, _apiKey?: string, apiEndpoint?: string) { + endpoints.push(apiEndpoint ?? ''); + } + + async initialize(): Promise { + return undefined; + } + } + })); + + const { getEmbeddingProvider } = await import('../src/embeddings/index.js'); + await getEmbeddingProvider({ + provider: 'openai', + model: 'text-embedding-3-small', + apiKey: 'sk-test', + apiEndpoint: 'http://127.0.0.1:11434/v1' + }); + await getEmbeddingProvider({ + provider: 'openai', + model: 'text-embedding-3-small', + apiKey: 'sk-test', + apiEndpoint: 'http://127.0.0.1:8080/v1' + }); + + expect(endpoints).toEqual(['http://127.0.0.1:11434/v1', 'http://127.0.0.1:8080/v1']); + }); +}); From 814dcdec5d4a3256705301341bd13221ba8fcaf2 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 30 Oct 2026 22:36:20 +0000 Subject: [PATCH 11/15] docs: address PR #148 review (indexing copy, changelog, credit) Fix stale first-run model-download wording; fold product-candidate notes into 2.3.0 (unreleased/TBD); thank @rothnic for #68/#70 analysis. --- CHANGELOG.md | 29 +++++++---------------------- README.md | 2 +- docs/client-setup.md | 2 +- 3 files changed, 9 insertions(+), 24 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8bd7bae..0fe0440 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,30 +1,15 @@ # Changelog -## Product candidate (not published) - -- New projects index and search without embedding or reranker models. Existing index modes are preserved; explicit mode switches rebuild fully. Transformers and LanceDB are optional peers with actionable preflight errors. -- Setup detects supported coding tools and configures the selected tools together. It preserves existing settings, saves version-pinned launches for the selected repository, and leaves indexing to the first search. -- Added `get_code_relationships`: directed import/re-export paths up to three hops, current resolution and source checks, confined endpoints, unresolved/stale diagnostics, and a 4,000-byte response cap. Resolution selects the root's `tsconfig.json` or, when absent, `tsconfig.base.json`, without inheriting an unrelated parent configuration. -- AST chunks preserve contiguous source and static property ownership, including concise descriptor methods, callback bodies and same-line siblings. Generated scope labels remain presentation metadata. Index format 4 rebuilds incompatible indexes even when sources are unchanged, preserves the selected embedding mode, and retains previous artifacts if rebuilding fails. -- Angular TypeScript eligibility uses parsed Angular imports; ordinary `resolve` or `validate` methods, comments, strings and imports from other packages no longer select the Angular analyzer. Other eligible analyzers retain the source and declaration context. The registry order and existing template/style routing are unchanged. -- Compact `signaturePreview` uses a contiguous window of up to three lines around the strongest query-term match in the returned snippet, falling back to its opening lines. Default and explicit exploration searches omit edit preflight while retaining search quality and incomplete-source warnings. Explicit `edit`, `refactor`, and `migrate` intents retain mutation guards. -- Search excludes ordinary stale/unavailable nominees without discarding verified current lower results or triggering an automatic full rebuild. Partial coverage, bounded exhaustion and failed semantic replenishment are explicit; corrupt metadata, build identity, paths and source ranges still fail closed. -- Symbol references scan same-build manifest sources in bounded read batches and skip parsing when a simple ASCII symbol occurs only inside longer identifiers. Unicode or arbitrary query spellings retain raw-text nomination. Coverage keeps raw-text absence separate from parser work and uncertain lexical fallback. Keyword artifact identity uses the shared index validator. Graph traversal verifies edges before consuming output caps, with bounded inspection and honest continuation. -- Watcher shutdown awaits handle closure and cancels pending callbacks. Connected stdio sessions have no default inactivity shutdown; explicit positive idle timeouts remain supported alongside handshake, EOF and parent-death guards. -- Chunk paths are persisted relative to the selected project root, including explicit-root indexing from another working directory. Previously incorrect relative paths can require a full reindex. -- Query advice is limited to grounded returned source, and abstained searches omit global pattern advice. Same-named symbols in different files remain eligible after file-level deduplication; TypeScript type arguments alone no longer select the JSX analyzer. -- The candidate requires Node 22+. CLI dispatch compares canonical entrypoint identity, including npm symlink launchers, while preserving the exported server's initialization on imports. -- Updated the runtime parser and watcher to remove the legacy braces dependency chain. Watcher exclusions use exact directory names; quoted import and export names retain their public symbol names. -- Known limits: model-free conceptual relevance is not guaranteed. Short requests can miss implementations when query words differ from source names; syntactic edges are not runtime calls; workspace/language-specific resolution can remain unresolved. Packed/protocol tests and scoped negative cases do not establish general agent-quality improvement or customer adoption. No npm publication is implied by this entry. - -## [2.3.0](https://github.com/PatrickSys/codebase-context/compare/v2.2.0...v2.3.0) (2026-10-30) +## [2.3.0](https://github.com/PatrickSys/codebase-context/compare/v2.2.0...v2.3.0) (unreleased — TBD) ### Release notes -- **CLI:** Fix npm **2.2.0** regression where `npx codebase-context --version`, `status`, and `reindex` could exit with code 0 and no output (direct CLI subcommands lazy-load the MCP runtime; canonical entrypoint detection includes npm symlink launchers). -- **Indexing:** Fresh project folders default to keyword-only indexing without downloading embedding models; opt in with `reindex --embeddings` and optional peer packages. -- **Setup:** `init` wizard for Claude Code, Codex, Cursor, and OpenCode (multi-client registration from one run). -- **Docs:** Position around local team conventions and edit preflight; cite Aider repo map, Repomix, and jCodeMunch as related tools; report 25.7% vs 27.1% retrieval without leadership claims; keep **2.4.0** features out of shipped language until published. +- **CLI:** Fix npm **2.2.0** regression where `npx codebase-context --version`, `status`, and `reindex` could exit with code 0 and no output (direct CLI subcommands lazy-load the MCP runtime; canonical entrypoint detection includes npm symlink launchers). Requires Node 22+. +- **Indexing:** Fresh project folders default to keyword-only indexing without downloading embedding or reranker models; existing index modes are preserved; opt in with `reindex --embeddings` and optional Transformers/LanceDB peers. Explicit mode switches rebuild fully. +- **Setup:** `init` wizard detects Claude Code, Codex, Cursor, and OpenCode, configures selected tools together, preserves existing settings, and leaves indexing to the first search. +- **Search & index pipeline:** AST-aligned chunks, index format metadata with atomic rebuilds, exploration vs edit/refactor/migrate preflight behavior, bounded symbol references, and source-validation hardening (see Features below). +- **Docs:** Position around local team conventions and edit preflight; cite Aider repo map, Repomix, and jCodeMunch as related tools; report **25.7%** vs **jCodeMunch 27.1%** expected-file recovery per [docs/benchmark.md](./docs/benchmark.md) without leadership claims; keep **2.4.0** features out of shipped language until published. +- **Thanks:** Nick Roth ([@rothnic](https://github.com/rothnic)) for the Intel Mac embedding crash analysis ([#68](https://github.com/PatrickSys/codebase-context/issues/68)) and custom OpenAI-compatible endpoint requirements ([#70](https://github.com/PatrickSys/codebase-context/issues/70)) that shaped workaround guidance in this release. ### Features diff --git a/README.md b/README.md index 15a5068..c1337f6 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,7 @@ Start a new agent session in your project, then ask: > Use Codebase Context to find [feature] in this repository. Pass this repository's absolute path as project, wait for indexing if needed, and open the relevant source files. -Replace `[feature]` with something you want to find. The first run downloads a local model and indexes the repository; time depends on repository size, CPU and model. +Replace `[feature]` with something you want to find. The first index pass scans the repository locally; **2.3.0** uses keyword search by default (no embedding model download). npm **2.2.0** still downloads a local embedding model on first index unless you change providers. Duration depends on repository size and CPU; optional semantic search adds model download time when enabled. ### Planned for 2.4.0 (not in 2.3.0) diff --git a/docs/client-setup.md b/docs/client-setup.md index 0d44385..c404d56 100644 --- a/docs/client-setup.md +++ b/docs/client-setup.md @@ -27,7 +27,7 @@ After registration, start a new agent session in your project and ask: > Use Codebase Context to find [feature] in this repository. Pass this repository's absolute path as project when checking get_indexing_status and searching. Wait for indexing if needed, read codebase://context, then search_codebase and open a returned source file. Show me the relevant files. -The first run downloads a local model and indexes the repository; time depends on repository size, CPU and model. You can prepare it separately with `npx -y codebase-context reindex` from the project folder. Index readiness, client connection and useful agent behavior are separate checks. +The first index pass scans the repository locally. Pending **2.3.0** uses keyword search by default (no model download); npm **2.2.0** still downloads embedding models on first index. You can prepare the index separately with `npx -y codebase-context reindex` (add `--no-embeddings` on 2.3.0, or `--embeddings` when you want semantic search). Index readiness, client connection, and useful agent behavior are separate checks. ## Claude Code From 8fd2c49b2c178edd71e394f3f208769d073441fb Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 30 Oct 2026 22:40:48 +0000 Subject: [PATCH 12/15] fix(frontend): index Vue/Svelte sources and tighten Next route detection Add .vue/.svelte to scan and language detection, extract script blocks for generic analysis, detect Nuxt/SvelteKit from workspace deps, and ignore SvelteKit + routes and non-Next app/pages paths. Document five-repo smoke results in docs/benchmark.md with fixture tests. --- docs/benchmark.md | 33 ++++++ src/analyzers/generic/index.ts | 34 +++++- src/analyzers/nextjs/index.ts | 26 ++++- src/core/indexer.ts | 1 + src/utils/frontend-framework-metadata.ts | 98 +++++++++++++++++ src/utils/language-detection.ts | 4 + tests/fixtures/frontend-vue-mini/Button.vue | 12 ++ tests/frontend-spa-indexing.test.ts | 115 ++++++++++++++++++++ tests/nextjs-analyzer.test.ts | 14 +++ 9 files changed, 333 insertions(+), 4 deletions(-) create mode 100644 src/utils/frontend-framework-metadata.ts create mode 100644 tests/fixtures/frontend-vue-mini/Button.vue create mode 100644 tests/frontend-spa-indexing.test.ts diff --git a/docs/benchmark.md b/docs/benchmark.md index 73d073b..99aeb63 100644 --- a/docs/benchmark.md +++ b/docs/benchmark.md @@ -1,5 +1,38 @@ # What our local code-context experiments show +## Front-end repo matrix (plan) + +### Now + +- Ship `.vue` / `.svelte` indexing, Nuxt/SvelteKit framework claims, and tighter Next.js route detection (SvelteKit `+` routes, Nuxt `app/pages/*.vue`). +- Re-run the five-repo smoke matrix after each fix; keep this section honest. + +### Next + +- Dedicated Vue/Nuxt and Svelte analyzers (patterns, route conventions) instead of generic-only SPA handling. +- Path-alias coverage for Vite/Nuxt `~/` and monorepo package exports beyond TypeScript `paths`. +- Markdown-heavy repos (docs sites) as a separate benchmark series. + +### Done + +- 2026-10-30: Cloned and exercised five OSS front-end repos (see table below); fixed SPA file gaps and Next.js false positives; added fixture tests in `tests/frontend-spa-indexing.test.ts`. + +### Five-repo smoke (2026-10-30, `reindex --no-embeddings`, Node 22, this branch) + +Clones: [shadcn-ui/taxonomy](https://github.com/shadcn-ui/taxonomy) (Next.js), [nuxt/ui](https://github.com/nuxt/ui), [angular/components](https://github.com/angular/components), [sveltejs/kit](https://github.com/sveltejs/kit), [vercel/turbo](https://github.com/vercel/turbo) (pnpm/turbo monorepo with UI apps). + +| Repo | Init / reindex / status | Framework metadata | SPA sources in index | Symbol refs | Edit preflight (realistic query) | +| --- | --- | --- | --- | --- | --- | +| taxonomy (Next.js) | pass / pass / ready | Next.js detected | 94 `.tsx` indexed | `cn` useful (241 hits) | not ready — low confidence; top hit was relevant layout | +| nuxt/ui | pass / pass / ready | Nuxt detected (was missing) | **816** `.vue` indexed (was **0**) | `UButton` finds `Button.vue` imports | not ready — low confidence; stale-source hint on some queries | +| angular/components | pass / pass / ready | Angular patterns | TS/Angular only | `MatDialog` static refs OK | not ready — low confidence; dialog hits sensible | +| sveltejs/kit | pass / pass / ready | SvelteKit detected (monorepo deps) | **1011** `.svelte` indexed (was **0**) | `load` very noisy (587 lexical) | not ready — no longer mislabels SvelteKit tests as Next.js pages | +| vercel/turbo | pass / pass / ready; chunk cap warning | monorepo packages listed | UI via Next apps in `examples/` | `Button` finds UI components | not ready — low confidence; 5k chunk cap drops most of repo | + +Known limits surfaced here (not fixed in this pass): default 5k chunk cap on very large repos, lexical `load`/`Button` ambiguity, edit preflight still conservative on broad natural-language queries, one Rust-generated TS parse warning in turbo. + +--- + This report describes local evidence from five ways of giving an AI coding agent context: Raw-native, Codebase Context, jCodeMunch, Repowise, and context-mode. The evidence is split into separate families because retrieval, repeated runs, full-agent observations, and a metered replay do not answer the same question. The families are not pooled into one score. The clearest current finding is narrow: in this fixed retrieval setup, Codebase Context recovered 25.7% of expected gold files at 11.5% file precision, while fresh indexing had material local cost. These measurements do not show better completed coding tasks. diff --git a/src/analyzers/generic/index.ts b/src/analyzers/generic/index.ts index 78ed320..6fb0dd2 100644 --- a/src/analyzers/generic/index.ts +++ b/src/analyzers/generic/index.ts @@ -32,6 +32,10 @@ import { categorizeDependency } from '../../utils/dependency-detection.js'; import type { WorkspacePackageJson } from '../../utils/workspace-detection.js'; +import { + detectFrontendFrameworkFromDeps, + extractSingleFileComponentScript +} from '../../utils/frontend-framework-metadata.js'; export class GenericAnalyzer implements FrameworkAnalyzer { readonly name = 'generic'; @@ -91,7 +95,9 @@ export class GenericAnalyzer implements FrameworkAnalyzer { '.css', '.scss', '.sass', - '.less' + '.less', + '.vue', + '.svelte' ]; readonly priority = 10; // Low priority - fallback analyzer @@ -126,8 +132,23 @@ export class GenericAnalyzer implements FrameworkAnalyzer { usesTreeSitterSymbols = true; } - if (language === 'typescript' || language === 'javascript') { - const parsed = await this.parseJSTSFile(filePath, content, language); + let scriptLanguage: 'typescript' | 'javascript' | null = null; + let scriptBody: string | null = null; + if (language === 'vue' || language === 'svelte') { + const extracted = extractSingleFileComponentScript(content, language); + if (extracted) { + scriptBody = extracted.body; + scriptLanguage = extracted.language; + } + } + + const jstsLanguage = + scriptLanguage ?? + (language === 'typescript' || language === 'javascript' ? language : null); + const jstsContent = scriptBody ?? content; + + if (jstsLanguage) { + const parsed = await this.parseJSTSFile(filePath, jstsContent, jstsLanguage); imports = parsed.imports; exports = parsed.exports; @@ -247,6 +268,7 @@ export class GenericAnalyzer implements FrameworkAnalyzer { let workspaceType: string = 'single'; let workspacePackages: WorkspacePackageJson[] = []; + let frameworkClaim: CodebaseMetadata['framework']; try { workspaceType = await detectWorkspaceType(rootPath); @@ -276,6 +298,11 @@ export class GenericAnalyzer implements FrameworkAnalyzer { version: version as string, category: categorizeDependency(name) })); + + const framework = detectFrontendFrameworkFromDeps(rawDeps); + if (framework) { + frameworkClaim = framework; + } } catch (_error) { // skip } @@ -285,6 +312,7 @@ export class GenericAnalyzer implements FrameworkAnalyzer { rootPath, languages: [], dependencies, + framework: frameworkClaim, architecture: { type: 'mixed', layers: { diff --git a/src/analyzers/nextjs/index.ts b/src/analyzers/nextjs/index.ts index ea99dab..cf9e8ca 100644 --- a/src/analyzers/nextjs/index.ts +++ b/src/analyzers/nextjs/index.ts @@ -71,7 +71,19 @@ export class NextJsAnalyzer implements FrameworkAnalyzer { return false; } - if (isInAppRouter(filePath) || isInPagesRouter(filePath) || isProxyFile(filePath)) { + if (isSvelteKitRouteFile(filePath)) { + return false; + } + + if (isProxyFile(filePath)) { + return true; + } + + if (matchesNextAppRouterConvention(filePath)) { + return true; + } + + if (isInPagesRouter(filePath)) { return true; } @@ -342,6 +354,18 @@ function analyzeRouting(filePath: string, content: string): NextRoutingInfo { }; } +function isSvelteKitRouteFile(filePath: string): boolean { + const baseName = path.basename(filePath, path.extname(filePath)); + return baseName.startsWith('+'); +} + +function matchesNextAppRouterConvention(filePath: string): boolean { + if (!isInAppRouter(filePath)) { + return false; + } + return getAppRouteFileKind(path.basename(filePath)) !== null; +} + function isInAppRouter(filePath: string): boolean { const normalizedPath = filePath.replace(/\\/g, '/'); return normalizedPath.includes('/app/') || normalizedPath.includes('/src/app/'); diff --git a/src/core/indexer.ts b/src/core/indexer.ts index 6d9ed37..397c3cb 100644 --- a/src/core/indexer.ts +++ b/src/core/indexer.ts @@ -279,6 +279,7 @@ export class CodebaseIndexer { }, include: [ '**/*.{ts,tsx,js,jsx,mjs,cjs,mts,cts}', + '**/*.{vue,svelte}', '**/*.{html,htm,css,scss,sass,less}', '**/*.{py,pyi,rb,php}', '**/*.{java,kt,kts,scala,swift,cs}', diff --git a/src/utils/frontend-framework-metadata.ts b/src/utils/frontend-framework-metadata.ts new file mode 100644 index 0000000..69de53b --- /dev/null +++ b/src/utils/frontend-framework-metadata.ts @@ -0,0 +1,98 @@ +import type { FrameworkInfo } from '../types/index.js'; +import { normalizeAnalyzerVersion } from '../analyzers/shared/metadata.js'; + +function hasDep(deps: Record, name: string): boolean { + return Object.prototype.hasOwnProperty.call(deps, name); +} + +/** + * Detect Vue / Nuxt / Svelte / SvelteKit from merged package dependencies. + * Uses the same MIN_INDICATORS=3 gate as indexer.selectFramework. + */ +export function detectFrontendFrameworkFromDeps( + deps: Record +): FrameworkInfo | undefined { + const indicators: string[] = []; + + if (hasDep(deps, 'nuxt')) indicators.push('dep:nuxt'); + if (hasDep(deps, '@nuxt/kit')) indicators.push('dep:@nuxt/kit'); + if (hasDep(deps, '@nuxt/schema')) indicators.push('dep:@nuxt/schema'); + if (hasDep(deps, 'vue')) indicators.push('dep:vue'); + if (hasDep(deps, '@sveltejs/kit')) indicators.push('dep:@sveltejs/kit'); + if (hasDep(deps, '@sveltejs/vite-plugin-svelte')) { + indicators.push('dep:@sveltejs/vite-plugin-svelte'); + } + if (hasDep(deps, 'svelte')) indicators.push('dep:svelte'); + if (hasDep(deps, 'vite')) indicators.push('dep:vite'); + + if (indicators.length < 3) { + return undefined; + } + + if (indicators.includes('dep:nuxt') || indicators.includes('dep:@nuxt/kit')) { + return { + name: 'Nuxt', + version: normalizeAnalyzerVersion(deps.nuxt ?? deps['@nuxt/kit'] ?? 'unknown'), + type: 'vue', + variant: 'nuxt', + uiLibraries: hasDep(deps, 'tailwindcss') ? ['Tailwind'] : undefined, + indicators + }; + } + + if (indicators.includes('dep:@sveltejs/kit')) { + return { + name: 'SvelteKit', + version: normalizeAnalyzerVersion(deps['@sveltejs/kit']), + type: 'svelte', + indicators + }; + } + + if (indicators.includes('dep:vue')) { + return { + name: 'Vue', + version: normalizeAnalyzerVersion(deps.vue), + type: 'vue', + indicators + }; + } + + if (indicators.includes('dep:svelte')) { + return { + name: 'Svelte', + version: normalizeAnalyzerVersion(deps.svelte), + type: 'svelte', + indicators + }; + } + + return undefined; +} + +export function extractSingleFileComponentScript( + content: string, + kind: 'vue' | 'svelte' +): { body: string; language: 'typescript' | 'javascript' } | null { + const tag = 'script'; + const match = content.match(new RegExp(`<${tag}([^>]*)>([\\s\\S]*?)<\\/${tag}>`, 'i')); + if (!match?.[2]) { + return null; + } + + const attrs = match[1] ?? ''; + const body = match[2].trim(); + if (!body) { + return null; + } + + const isTypeScript = + /\blang\s*=\s*['"]ts['"]/i.test(attrs) || + /\blang\s*=\s*['"]typescript['"]/i.test(attrs) || + (kind === 'svelte' && attrs.includes('lang="ts"')); + + return { + body, + language: isTypeScript ? 'typescript' : 'javascript' + }; +} diff --git a/src/utils/language-detection.ts b/src/utils/language-detection.ts index 2cc8bac..5df0fae 100644 --- a/src/utils/language-detection.ts +++ b/src/utils/language-detection.ts @@ -16,6 +16,8 @@ const extensionToLanguage: Record = { '.tsx': 'typescriptreact', '.mts': 'typescript', '.cts': 'typescript', + '.vue': 'vue', + '.svelte': 'svelte', // Web '.html': 'html', @@ -115,6 +117,8 @@ const baseCodeExtensions = new Set([ '.tsx', '.mts', '.cts', + '.vue', + '.svelte', '.html', '.htm', '.css', diff --git a/tests/fixtures/frontend-vue-mini/Button.vue b/tests/fixtures/frontend-vue-mini/Button.vue new file mode 100644 index 0000000..b14d667 --- /dev/null +++ b/tests/fixtures/frontend-vue-mini/Button.vue @@ -0,0 +1,12 @@ + + + diff --git a/tests/frontend-spa-indexing.test.ts b/tests/frontend-spa-indexing.test.ts new file mode 100644 index 0000000..ff7c63c --- /dev/null +++ b/tests/frontend-spa-indexing.test.ts @@ -0,0 +1,115 @@ +import { mkdir, rm, writeFile } from 'node:fs/promises'; +import path from 'node:path'; +import { randomUUID } from 'node:crypto'; +import { describe, expect, it } from 'vitest'; +import { isCodeFile, detectLanguage } from '../src/utils/language-detection.js'; +import { GenericAnalyzer } from '../src/analyzers/generic/index.js'; +import { NextJsAnalyzer } from '../src/analyzers/nextjs/index.js'; +import { + detectFrontendFrameworkFromDeps, + extractSingleFileComponentScript +} from '../src/utils/frontend-framework-metadata.js'; + +describe('frontend single-file component support', () => { + it('treats .vue and .svelte as code files', () => { + expect(isCodeFile('src/Button.vue')).toBe(true); + expect(isCodeFile('routes/+page.svelte')).toBe(true); + expect(detectLanguage('App.vue')).toBe('vue'); + expect(detectLanguage('Page.svelte')).toBe('svelte'); + }); + + it('extracts script blocks from vue and svelte files', () => { + const vue = extractSingleFileComponentScript( + '', + 'vue' + ); + expect(vue).toEqual({ body: 'export const x = 1', language: 'typescript' }); + + const svelte = extractSingleFileComponentScript( + '', + 'svelte' + ); + expect(svelte?.language).toBe('typescript'); + }); + + it('indexes vue components through the generic analyzer', async () => { + const fixture = path.join(process.cwd(), 'tests/fixtures/frontend-vue-mini/Button.vue'); + const analyzer = new GenericAnalyzer(); + expect(analyzer.canAnalyze(fixture)).toBe(true); + + const content = await import('node:fs/promises').then((fs) => fs.readFile(fixture, 'utf-8')); + const result = await analyzer.analyze(fixture, content); + expect(result.chunks.length).toBeGreaterThan(0); + const serialized = JSON.stringify(result); + expect(serialized.includes('ButtonProps')).toBe(true); + }); + + it('does not classify SvelteKit +page files or Nuxt app/pages vue as Next.js', () => { + const analyzer = new NextJsAnalyzer(); + expect( + analyzer.canAnalyze( + '/repo/packages/kit/test/+page.server.js', + 'export async function load() { return {}; }' + ) + ).toBe(false); + expect( + analyzer.canAnalyze( + '/repo/playgrounds/nuxt/app/pages/typography.vue', + '' + ) + ).toBe(false); + expect( + analyzer.canAnalyze( + path.join(process.cwd(), 'app', 'dashboard', 'page.tsx'), + 'export default function Page() { return null; }' + ) + ).toBe(true); + }); + + it('detects Nuxt and SvelteKit framework metadata from dependencies', () => { + const nuxt = detectFrontendFrameworkFromDeps({ + nuxt: '^4.0.0', + '@nuxt/kit': 'catalog:', + vue: '^3.5.0' + }); + expect(nuxt?.type).toBe('vue'); + expect(nuxt?.name).toBe('Nuxt'); + + const kit = detectFrontendFrameworkFromDeps({ + '@sveltejs/kit': '^2.0.0', + svelte: '^5.0.0', + vite: '^6.0.0' + }); + expect(kit?.type).toBe('svelte'); + expect(kit?.name).toBe('SvelteKit'); + }); +}); + +describe('frontend fixture indexing smoke', () => { + it('writes vue files into a temp project manifest via glob include patterns', async () => { + const root = path.join(process.cwd(), 'tests/fixtures/frontend-vue-mini'); + const analyzer = new GenericAnalyzer(); + const filePath = path.join(root, 'Button.vue'); + const content = await import('node:fs/promises').then((fs) => fs.readFile(filePath, 'utf-8')); + const result = await analyzer.analyze(filePath, content); + expect(result.language).toBe('vue'); + expect(result.chunks[0]?.metadata.analyzer).toBe('generic'); + }); + + it('creates an isolated vue file for incremental tests', async () => { + const dir = path.join(process.cwd(), 'tests/fixtures/.tmp', randomUUID()); + const filePath = path.join(dir, 'Widget.vue'); + const source = + ''; + await mkdir(dir, { recursive: true }); + try { + await writeFile(filePath, source); + const analyzer = new GenericAnalyzer(); + expect(analyzer.canAnalyze(filePath)).toBe(true); + const result = await analyzer.analyze(filePath, source); + expect(result.chunks.length).toBeGreaterThan(0); + } finally { + await rm(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/tests/nextjs-analyzer.test.ts b/tests/nextjs-analyzer.test.ts index 4f6d9e6..1fbb508 100644 --- a/tests/nextjs-analyzer.test.ts +++ b/tests/nextjs-analyzer.test.ts @@ -15,6 +15,20 @@ describe('NextJsAnalyzer', () => { expect(analyzer.canAnalyze('/tmp/file.tsx', 'import React from "react";')).toBe(false); }); + it('ignores SvelteKit route files and Nuxt app/pages vue files', () => { + const analyzer = new NextJsAnalyzer(); + + expect( + analyzer.canAnalyze( + '/repo/routes/+page.server.js', + 'export async function load() { return {}; }' + ) + ).toBe(false); + expect( + analyzer.canAnalyze('/repo/app/pages/home.vue', '') + ).toBe(false); + }); + it('detects app router pages, client components, and metadata exports', async () => { const analyzer = new NextJsAnalyzer(); const filePath = path.join(process.cwd(), 'app', 'settings', 'page.tsx'); From 1ba2a56e69697cb4568d69419537043488774224 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 30 Oct 2026 22:40:52 +0000 Subject: [PATCH 13/15] Add reproducible niche prior-art benchmark harness and results Extend the existing eval fixtures with a frozen 18-task manifest across three public repos, a one-command runner comparing Codebase Context to jCodeMunch, Aider repo map, Repomix, and ripgrep, and checked-in evidence plus benchmark.md reporting including known losses. --- .gitignore | 1 + docs/benchmark.md | 48 + package.json | 1 + results/niche-benchmark-evidence.json | 1885 +++++++++++++++++ .../niche-benchmark/aider-repomap-query.py | 102 + scripts/lib/niche-benchmark/lanes.mjs | 408 ++++ scripts/lib/niche-benchmark/scoring.mjs | 134 ++ scripts/run-niche-benchmark.mjs | 283 +++ tests/fixtures/niche-benchmark-manifest.json | 204 ++ tests/fixtures/niche-benchmark-protocol.json | 73 + tests/niche-benchmark.test.ts | 86 + 11 files changed, 3225 insertions(+) create mode 100644 results/niche-benchmark-evidence.json create mode 100644 scripts/lib/niche-benchmark/aider-repomap-query.py create mode 100644 scripts/lib/niche-benchmark/lanes.mjs create mode 100644 scripts/lib/niche-benchmark/scoring.mjs create mode 100644 scripts/run-niche-benchmark.mjs create mode 100644 tests/fixtures/niche-benchmark-manifest.json create mode 100644 tests/fixtures/niche-benchmark-protocol.json create mode 100644 tests/niche-benchmark.test.ts diff --git a/.gitignore b/.gitignore index cee0f89..837b700 100644 --- a/.gitignore +++ b/.gitignore @@ -34,6 +34,7 @@ repos/ # Local benchmark/lab outputs benchmark-runs/ +benchmark-repos/ /outputs/ /results/*/raw/ /results/*/local/ diff --git a/docs/benchmark.md b/docs/benchmark.md index 73d073b..a9ae0fb 100644 --- a/docs/benchmark.md +++ b/docs/benchmark.md @@ -1,5 +1,53 @@ # What our local code-context experiments show +## Reproducible niche benchmark (prior art colleagues) + +This section reports a **frozen, rerunnable** comparison on **18 tasks** (team-convention lookup, edit-preflight navigation, symbol retrieval) across **three public repos**: [angular-spotify](https://github.com/trungk18/angular-spotify) (TypeScript), [Excalidraw](https://github.com/excalidraw/excalidraw) (TypeScript), and [FastAPI](https://github.com/fastapi/fastapi) (Python). Tasks and expected patterns live in `tests/fixtures/niche-benchmark-manifest.json`; comparator credits and rules live in `tests/fixtures/niche-benchmark-protocol.json`. Full machine output is checked in at `results/niche-benchmark-evidence.json` (generated **2026-10-30** on Node **v22.23.4**). + +**Primary accuracy metric:** top-3 file hit rate (expected path substring appears in the first three ranked paths returned for the same natural-language query). + +**Efficiency metrics:** estimated tokens returned (UTF-8 bytes ÷ 4) and query latency; index/setup time is reported separately in the evidence JSON. + +### One-command rerun + +```bash +pnpm install && pnpm run build +npm run benchmark:niche +# or: node scripts/run-niche-benchmark.mjs --validate-fixtures +``` + +Optional: `--lanes codebase-context,jCodeMunch,aider-repo-map,repomix,ripgrep` and `--output results/niche-benchmark-evidence.json`. + +**Prerequisites (not installed via npm publish):** `git`, `rg`, `python3` with `jcodemunch-mcp` and `aider-chat`, and network access to clone the three pinned repos into `benchmark-repos/`. + +### Aggregate results (18 tasks) + +| Colleague tool | Top-3 hit rate | Avg est. tokens | Avg query ms | Avg index ms | +| --- | ---: | ---: | ---: | ---: | +| **Codebase Context** | **50.0%** | 642 | 928 | 673 | +| [Aider](https://github.com/Aider-AI/aider) repo map | 22.2% | **95** | 1442 | 350 | +| [Repomix](https://github.com/yamadashy/repomix) (pack + section rank) | 11.1% | 5,559,174 | **114** | 286 | +| [jCodeMunch](https://github.com/jgravelle/jcodemunch-mcp) | 5.6% | **202** | 701 | **69** | +| [ripgrep](https://github.com/BurntSushi/ripgrep) (lexical baseline) | 5.6% | 5,001* | **76** | 0 | + +\*Ripgrep and Repomix token figures include large truncated raw payloads in this harness; treat them as upper-bound context cost, not model-ready bundles. + +### Where Codebase Context loses (same run) + +- **Team-convention file ranking:** 0/6 top-3 file hits — `patterns` answers matched expected **signals** (for example `dependencyInjection`, `stateManagement`) but did not return ranked repo paths, so file-based scoring stays at zero. **Aider repo map** reached **33%** top-3 on the same team-convention slice. +- **Payload size vs Aider / jCodeMunch:** median-style averages **642** estimated tokens vs **95** (Aider) and **202** (jCodeMunch) on the same queries. +- **Query latency vs ripgrep:** **928 ms** average query time vs **76 ms** for the ripgrep baseline (no index). +- **Edit-preflight slice:** **67%** top-3 here, but still below perfect; broad unsafe tasks are out of scope for this table (see edit-preflight fixtures for abstention tests). +- **Cross-repo variance:** symbol-retrieval top-3 was **83%** on the two TypeScript repos but **50%** on FastAPI in this run (see `byNiche` in the evidence JSON). + +### What this does not claim + +- No patch correctness, agent productivity, or npm-package performance guarantees. +- Repomix is not a search engine; the lane documents pack-generation cost plus section ranking. +- Earlier ContextBench / discovery families below remain separate evidence; they are not merged into this table. + +--- + This report describes local evidence from five ways of giving an AI coding agent context: Raw-native, Codebase Context, jCodeMunch, Repowise, and context-mode. The evidence is split into separate families because retrieval, repeated runs, full-agent observations, and a metered replay do not answer the same question. The families are not pooled into one score. The clearest current finding is narrow: in this fixed retrieval setup, Codebase Context recovered 25.7% of expected gold files at 11.5% file precision, while fresh indexing had material local cost. These measurements do not show better completed coding tasks. diff --git a/package.json b/package.json index a855889..efbbda6 100644 --- a/package.json +++ b/package.json @@ -132,6 +132,7 @@ "type-check": "tsc --noEmit", "mcp:inspect": "npx -y @modelcontextprotocol/inspector node dist/index.js .", "eval": "pnpm run build && node scripts/run-eval.mjs", + "benchmark:niche": "node scripts/run-niche-benchmark.mjs", "prepare": "husky" }, "dependencies": { diff --git a/results/niche-benchmark-evidence.json b/results/niche-benchmark-evidence.json new file mode 100644 index 0000000..a058157 --- /dev/null +++ b/results/niche-benchmark-evidence.json @@ -0,0 +1,1885 @@ +{ + "artifactType": "niche-prior-art-benchmark-evidence", + "generatedAt": "2026-10-30T22:39:55.772Z", + "protocol": { + "name": "niche-prior-art-benchmark", + "frozenDate": "2026-10-30", + "manifestFrozenDate": "2026-10-30" + }, + "runMeta": { + "manifestPath": "/workspace/tests/fixtures/niche-benchmark-manifest.json", + "protocolPath": "/workspace/tests/fixtures/niche-benchmark-protocol.json", + "manifestSha256": "9dd48fb64b7e81e6a16e694be031134f7a1506ff9a259b4cd084ab44827c704c", + "protocolSha256": "0b666b5ff5ed86f9a31cd0cc2857ef7dcd233ddf9c85b93091eb92488522aed0", + "nodeVersion": "v22.23.4", + "lanes": [ + "codebase-context", + "jCodeMunch", + "aider-repo-map", + "repomix", + "ripgrep" + ], + "repoCheckouts": { + "angular-spotify": { + "ref": "ff9efa765c53cfde78c9a172c62d515ae8ef9fe0", + "path": "/workspace/benchmark-repos/angular-spotify" + }, + "excalidraw": { + "ref": "e18c1dd213000dde0ae94ef7eb00aab537b39708", + "path": "/workspace/benchmark-repos/excalidraw" + }, + "fastapi": { + "ref": "628c34e0cae200564d191c95d7edea78c88c4b5e", + "path": "/workspace/benchmark-repos/fastapi" + } + } + }, + "comparators": [ + { + "id": "codebase-context", + "displayName": "Codebase Context", + "credit": "This repository (local dist CLI / MCP surfaces).", + "projectUrl": "https://github.com/PatrickSys/codebase-context", + "status": "completed", + "tasksTotal": 18, + "tasksCompleted": 18, + "tasksFailed": 0, + "top3HitRate": 0.5, + "averageUsefulness": 0.6666666666666666, + "averageEstimatedTokens": 641.7222222222222, + "averageQueryElapsedMs": 927.6111111111111, + "averageIndexElapsedMs": 672.7777777777779, + "byNiche": { + "team-convention": { + "tasks": 6, + "top3HitRate": 0 + }, + "edit-preflight": { + "tasks": 6, + "top3HitRate": 0.6666666666666666 + }, + "symbol-retrieval": { + "tasks": 6, + "top3HitRate": 0.8333333333333334 + } + }, + "taskResults": [ + { + "taskId": "as-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 1, + "matchedSignals": [ + "dependencyInjection" + ], + "payloadBytes": 1295, + "estimatedTokens": 324, + "queryElapsedMs": 518, + "indexElapsedMs": 512.5, + "topFiles": [] + }, + { + "taskId": "as-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 1, + "matchedSignals": [ + "stateManagement" + ], + "payloadBytes": 1220, + "estimatedTokens": 305, + "queryElapsedMs": 492, + "indexElapsedMs": 512.5, + "topFiles": [] + }, + { + "taskId": "as-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 4017, + "estimatedTokens": 1005, + "queryElapsedMs": 978, + "indexElapsedMs": 512.5, + "topFiles": [ + "libs/web/auth/data-access/src/lib/store/auth.store.ts", + "libs/web/auth/util/src/lib/interceptors/auth.interceptor.ts", + "libs/web/visualizer/data-access/src/lib/audio-context.ts", + "libs/web/visualizer/data-access/src/lib/particles-renderer.ts", + "libs/web/visualizer/data-access/src/lib/sound-lines-renderer.ts" + ] + }, + { + "taskId": "as-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": true, + "firstRelevantHit": 2, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 4114, + "estimatedTokens": 1029, + "queryElapsedMs": 719, + "indexElapsedMs": 512.5, + "topFiles": [ + "libs/web/auth/util/src/lib/interceptors/auth.interceptor.ts", + "libs/web/shell/ui/player-controls/src/lib/player-controls.component.ts", + "libs/web/shared/ui/media-summary/src/lib/media-summary.component.ts", + "libs/web/shell/ui/player-playback/src/lib/player-playback.component.ts", + "libs/web/lyrics/ui/lyrics-pip/src/lib/lyrics-pip.component.ts" + ] + }, + { + "taskId": "as-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 2420, + "estimatedTokens": 605, + "queryElapsedMs": 645, + "indexElapsedMs": 512.5, + "topFiles": [ + "libs/web/auth/util/src/lib/interceptors/auth.interceptor.ts", + "libs/web/auth/util/src/lib/interceptors/unauthorized.interceptor.ts", + "libs/web/future-responsive/src/lib/albums.mock.json", + "libs/web/container-queries/src/lib/albums.mock.json", + "tsconfig.base.json" + ] + }, + { + "taskId": "as-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 2834, + "estimatedTokens": 709, + "queryElapsedMs": 906, + "indexElapsedMs": 512.5, + "topFiles": [ + "libs/web/auth/util/src/lib/interceptors/auth.interceptor.ts", + "libs/web/auth/data-access/src/lib/store/auth.store.ts", + "libs/web/shared/data-access/store/src/lib/ui/ui-store.ts", + "libs/web/auth/util/src/lib/interceptors/unauthorized.interceptor.ts", + "libs/web/auth/ui/unauthorized-modal/src/lib/unauthorized-modal.component.html" + ] + }, + { + "taskId": "ex-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 1, + "matchedSignals": [ + "stateManagement" + ], + "payloadBytes": 1124, + "estimatedTokens": 281, + "queryElapsedMs": 500, + "indexElapsedMs": 1122, + "topFiles": [] + }, + { + "taskId": "ex-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 51, + "estimatedTokens": 13, + "queryElapsedMs": 491, + "indexElapsedMs": 1122, + "topFiles": [] + }, + { + "taskId": "ex-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 4389, + "estimatedTokens": 1098, + "queryElapsedMs": 1336, + "indexElapsedMs": 1122, + "topFiles": [ + "packages/excalidraw/history.ts", + "excalidraw-app/data/FileManager.ts", + "excalidraw-app/app_constants.ts", + "excalidraw-app/collab/Collab.tsx", + "excalidraw-app/App.tsx" + ] + }, + { + "taskId": "ex-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": true, + "firstRelevantHit": 3, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 4537, + "estimatedTokens": 1135, + "queryElapsedMs": 1261, + "indexElapsedMs": 1122, + "topFiles": [ + "packages/excalidraw/components/App.tsx", + "packages/element/src/types.ts", + "packages/excalidraw/data/blob.ts", + "packages/excalidraw/types.ts", + "excalidraw-app/data/firebase.ts" + ] + }, + { + "taskId": "ex-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 3763, + "estimatedTokens": 941, + "queryElapsedMs": 1224, + "indexElapsedMs": 1122, + "topFiles": [ + "packages/math/src/types.ts", + "packages/utils/src/withinBounds.ts", + "packages/excalidraw/clipboard.ts", + "excalidraw-app/data/index.ts", + "packages/excalidraw/tests/test-utils.ts" + ] + }, + { + "taskId": "ex-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": true, + "firstRelevantHit": 2, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 3568, + "estimatedTokens": 892, + "queryElapsedMs": 1488, + "indexElapsedMs": 1122, + "topFiles": [ + "packages/element/src/types.ts", + "packages/excalidraw/data/blob.ts", + "excalidraw-app/data/firebase.ts", + "packages/excalidraw/data/index.ts", + "packages/excalidraw/data/image.ts" + ] + }, + { + "taskId": "fa-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 51, + "estimatedTokens": 13, + "queryElapsedMs": 477, + "indexElapsedMs": 383.8333333333333, + "topFiles": [] + }, + { + "taskId": "fa-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 51, + "estimatedTokens": 13, + "queryElapsedMs": 452, + "indexElapsedMs": 383.8333333333333, + "topFiles": [] + }, + { + "taskId": "fa-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 3144, + "estimatedTokens": 786, + "queryElapsedMs": 913, + "indexElapsedMs": 383.8333333333333, + "topFiles": [ + "fastapi/security/oauth2.py", + "docs_src/security/tutorial005_py39.py", + "docs_src/security/tutorial005_py310.py", + "fastapi/security/http.py", + "fastapi/applications.py" + ] + }, + { + "taskId": "fa-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 3476, + "estimatedTokens": 869, + "queryElapsedMs": 1313, + "indexElapsedMs": 383.8333333333333, + "topFiles": [ + "tests/test_response_model_as_return_annotation.py", + "tests/test_tuples.py", + "tests/test_schema_extra_examples.py", + "tests/test_response_by_alias.py", + "tests/test_response_model_sub_types.py" + ] + }, + { + "taskId": "fa-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 2727, + "estimatedTokens": 682, + "queryElapsedMs": 1520, + "indexElapsedMs": 383.8333333333333, + "topFiles": [ + "fastapi/routing.py", + "fastapi/utils.py", + "tests/test_multi_body_errors.py", + "scripts/contributors.py", + "tests/test_router_events.py" + ] + }, + { + "taskId": "fa-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 3403, + "estimatedTokens": 851, + "queryElapsedMs": 1464, + "indexElapsedMs": 383.8333333333333, + "topFiles": [ + "tests/test_starlette_exception.py", + "tests/test_security_http_basic_optional.py", + "tests/test_multi_body_errors.py", + "tests/test_webhooks_security.py", + "tests/test_union_body.py" + ] + } + ] + }, + { + "id": "jCodeMunch", + "displayName": "jCodeMunch", + "credit": "jCodeMunch MCP symbol search and repo indexing (jcodemunch-mcp on PyPI).", + "projectUrl": "https://github.com/jgrahamc/jcodemunch", + "status": "completed", + "tasksTotal": 18, + "tasksCompleted": 18, + "tasksFailed": 0, + "top3HitRate": 0.05555555555555555, + "averageUsefulness": 0.1111111111111111, + "averageEstimatedTokens": 202.38888888888889, + "averageQueryElapsedMs": 701.4444444444445, + "averageIndexElapsedMs": 69.27777777777777, + "byNiche": { + "team-convention": { + "tasks": 6, + "top3HitRate": 0 + }, + "edit-preflight": { + "tasks": 6, + "top3HitRate": 0 + }, + "symbol-retrieval": { + "tasks": 6, + "top3HitRate": 0.16666666666666666 + } + }, + "taskResults": [ + { + "taskId": "as-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 787, + "estimatedTokens": 197, + "queryElapsedMs": 221, + "indexElapsedMs": 55.166666666666664, + "topFiles": [ + "constant" + ] + }, + { + "taskId": "as-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 1141, + "estimatedTokens": 286, + "queryElapsedMs": 222, + "indexElapsedMs": 55.166666666666664, + "topFiles": [ + "method", + "constant", + "type" + ] + }, + { + "taskId": "as-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 905, + "estimatedTokens": 227, + "queryElapsedMs": 225, + "indexElapsedMs": 55.166666666666664, + "topFiles": [ + "type", + "method" + ] + }, + { + "taskId": "as-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 1159, + "estimatedTokens": 290, + "queryElapsedMs": 222, + "indexElapsedMs": 55.166666666666664, + "topFiles": [ + "method" + ] + }, + { + "taskId": "as-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 1072, + "estimatedTokens": 268, + "queryElapsedMs": 221, + "indexElapsedMs": 55.166666666666664, + "topFiles": [ + "method" + ] + }, + { + "taskId": "as-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 905, + "estimatedTokens": 227, + "queryElapsedMs": 224, + "indexElapsedMs": 55.166666666666664, + "topFiles": [ + "type", + "method" + ] + }, + { + "taskId": "ex-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 947, + "estimatedTokens": 237, + "queryElapsedMs": 1036, + "indexElapsedMs": 50.833333333333336, + "topFiles": [ + "type", + "function" + ] + }, + { + "taskId": "ex-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 818, + "estimatedTokens": 205, + "queryElapsedMs": 1022, + "indexElapsedMs": 50.833333333333336, + "topFiles": [ + "function" + ] + }, + { + "taskId": "ex-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 797, + "estimatedTokens": 200, + "queryElapsedMs": 1195, + "indexElapsedMs": 50.833333333333336, + "topFiles": [ + "function", + "method", + "type" + ] + }, + { + "taskId": "ex-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 772, + "estimatedTokens": 193, + "queryElapsedMs": 1123, + "indexElapsedMs": 50.833333333333336, + "topFiles": [ + "method", + "function" + ] + }, + { + "taskId": "ex-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 592, + "estimatedTokens": 148, + "queryElapsedMs": 1081, + "indexElapsedMs": 50.833333333333336, + "topFiles": [ + "type" + ] + }, + { + "taskId": "ex-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 626, + "estimatedTokens": 157, + "queryElapsedMs": 1201, + "indexElapsedMs": 50.833333333333336, + "topFiles": [ + "function" + ] + }, + { + "taskId": "fa-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 850, + "estimatedTokens": 213, + "queryElapsedMs": 733, + "indexElapsedMs": 101.83333333333333, + "topFiles": [ + "method" + ] + }, + { + "taskId": "fa-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 1, + "matchedSignals": [ + "APIRouter", + "router" + ], + "payloadBytes": 510, + "estimatedTokens": 128, + "queryElapsedMs": 939, + "indexElapsedMs": 101.83333333333333, + "topFiles": [ + "method" + ] + }, + { + "taskId": "fa-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 721, + "estimatedTokens": 181, + "queryElapsedMs": 760, + "indexElapsedMs": 101.83333333333333, + "topFiles": [ + "class", + "function" + ] + }, + { + "taskId": "fa-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 496, + "estimatedTokens": 124, + "queryElapsedMs": 764, + "indexElapsedMs": 101.83333333333333, + "topFiles": [ + "class", + "method" + ] + }, + { + "taskId": "fa-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 634, + "estimatedTokens": 159, + "queryElapsedMs": 706, + "indexElapsedMs": 101.83333333333333, + "topFiles": [ + "function", + "field", + "class" + ] + }, + { + "taskId": "fa-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 810, + "estimatedTokens": 203, + "queryElapsedMs": 731, + "indexElapsedMs": 101.83333333333333, + "topFiles": [ + "function" + ] + } + ] + }, + { + "id": "aider-repo-map", + "displayName": "Aider repo map", + "credit": "Paul Gauthier's Aider RepoMap ranking (aider-chat on PyPI). Query tokens are passed as mentioned identifiers; there is no separate natural-language search API.", + "projectUrl": "https://github.com/Aider-AI/aider", + "status": "completed", + "tasksTotal": 18, + "tasksCompleted": 18, + "tasksFailed": 0, + "top3HitRate": 0.2222222222222222, + "averageUsefulness": 0.16666666666666666, + "averageEstimatedTokens": 95.05555555555556, + "averageQueryElapsedMs": 1441.8654441833496, + "averageIndexElapsedMs": 349.8888888888889, + "byNiche": { + "team-convention": { + "tasks": 6, + "top3HitRate": 0.3333333333333333 + }, + "edit-preflight": { + "tasks": 6, + "top3HitRate": 0 + }, + "symbol-retrieval": { + "tasks": 6, + "top3HitRate": 0.3333333333333333 + } + }, + "taskResults": [ + { + "taskId": "as-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 468, + "estimatedTokens": 117, + "queryElapsedMs": 1425.365924835205, + "indexElapsedMs": 351.3333333333333, + "topFiles": [ + "libs/web/shared/app-config/src/lib/app.config.ts", + "libs/web/shared/data-access/models/src/lib/api/play-request.ts", + "libs/web/shared/data-access/models/src/lib/api/spotify-api-params.ts", + "libs/web/visualizer/data-access/src/lib/visualizer-renderer.interface.ts", + "libs/web/shared/data-access/models/src/lib/api/track-extended.ts" + ] + }, + { + "taskId": "as-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 505, + "estimatedTokens": 127, + "queryElapsedMs": 1442.3494338989258, + "indexElapsedMs": 351.3333333333333, + "topFiles": [ + "libs/web/shared/app-config/src/lib/app.config.ts", + "libs/web/shared/data-access/models/src/lib/api/play-request.ts", + "libs/web/shared/data-access/models/src/lib/api/spotify-api-params.ts", + "libs/web/visualizer/data-access/src/lib/visualizer-renderer.interface.ts", + "libs/web/shared/data-access/models/src/lib/api/track-extended.ts" + ] + }, + { + "taskId": "as-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 530, + "estimatedTokens": 133, + "queryElapsedMs": 1495.1844215393066, + "indexElapsedMs": 351.3333333333333, + "topFiles": [ + "libs/web/shared/app-config/src/lib/app.config.ts", + "libs/web/shared/data-access/models/src/lib/api/play-request.ts", + "libs/web/shared/data-access/models/src/lib/api/spotify-api-params.ts", + "libs/web/visualizer/data-access/src/lib/visualizer-renderer.interface.ts", + "libs/web/shared/data-access/models/src/lib/api/track-extended.ts" + ] + }, + { + "taskId": "as-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 488, + "estimatedTokens": 122, + "queryElapsedMs": 1396.5404033660889, + "indexElapsedMs": 351.3333333333333, + "topFiles": [ + "libs/web/shared/app-config/src/lib/app.config.ts", + "libs/web/shared/data-access/models/src/lib/api/play-request.ts", + "libs/web/shared/data-access/models/src/lib/api/spotify-api-params.ts", + "libs/web/visualizer/data-access/src/lib/visualizer-renderer.interface.ts", + "libs/web/shared/data-access/models/src/lib/api/track-extended.ts" + ] + }, + { + "taskId": "as-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 460, + "estimatedTokens": 115, + "queryElapsedMs": 1381.8109035491943, + "indexElapsedMs": 351.3333333333333, + "topFiles": [ + "libs/web/shared/app-config/src/lib/app.config.ts", + "libs/web/shared/data-access/models/src/lib/api/play-request.ts", + "libs/web/shared/data-access/models/src/lib/api/spotify-api-params.ts", + "libs/web/visualizer/data-access/src/lib/visualizer-renderer.interface.ts", + "libs/web/shared/data-access/models/src/lib/api/track-extended.ts" + ] + }, + { + "taskId": "as-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 496, + "estimatedTokens": 124, + "queryElapsedMs": 1579.0975093841553, + "indexElapsedMs": 351.3333333333333, + "topFiles": [ + "libs/web/shared/app-config/src/lib/app.config.ts", + "libs/web/shared/data-access/models/src/lib/api/play-request.ts", + "libs/web/shared/data-access/models/src/lib/api/spotify-api-params.ts", + "libs/web/visualizer/data-access/src/lib/visualizer-renderer.interface.ts", + "libs/web/shared/data-access/models/src/lib/api/track-extended.ts" + ] + }, + { + "taskId": "ex-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 335, + "estimatedTokens": 84, + "queryElapsedMs": 1385.5478763580322, + "indexElapsedMs": 333.8333333333333, + "topFiles": [ + "packages/excalidraw/charts/charts.types.ts", + "packages/excalidraw/global.d.ts", + "packages/math/src/types.ts", + "packages/element/src/types.ts", + "packages/excalidraw/types.ts" + ] + }, + { + "taskId": "ex-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": true, + "firstRelevantHit": 3, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 364, + "estimatedTokens": 91, + "queryElapsedMs": 1377.885341644287, + "indexElapsedMs": 333.8333333333333, + "topFiles": [ + "packages/excalidraw/charts/charts.types.ts", + "packages/excalidraw/global.d.ts", + "packages/math/src/types.ts", + "packages/element/src/types.ts", + "packages/excalidraw/types.ts" + ] + }, + { + "taskId": "ex-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 331, + "estimatedTokens": 83, + "queryElapsedMs": 1376.3632774353027, + "indexElapsedMs": 333.8333333333333, + "topFiles": [ + "packages/excalidraw/types.ts", + "packages/math/src/types.ts", + "packages/excalidraw/charts/charts.types.ts", + "packages/element/src/types.ts", + "packages/common/src/editorInterface.ts" + ] + }, + { + "taskId": "ex-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 348, + "estimatedTokens": 87, + "queryElapsedMs": 1386.0201835632324, + "indexElapsedMs": 333.8333333333333, + "topFiles": [ + "packages/excalidraw/types.ts", + "packages/math/src/types.ts", + "packages/excalidraw/charts/charts.types.ts", + "packages/element/src/types.ts", + "packages/common/src/editorInterface.ts" + ] + }, + { + "taskId": "ex-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 308, + "estimatedTokens": 77, + "queryElapsedMs": 1516.0613059997559, + "indexElapsedMs": 333.8333333333333, + "topFiles": [ + "packages/math/src/types.ts", + "packages/element/src/types.ts", + "packages/excalidraw/tests/helpers/ui.ts", + "packages/common/src/bounds.ts", + "packages/excalidraw/types.ts" + ] + }, + { + "taskId": "ex-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 334, + "estimatedTokens": 84, + "queryElapsedMs": 1381.5414905548096, + "indexElapsedMs": 333.8333333333333, + "topFiles": [ + "packages/excalidraw/types.ts", + "packages/math/src/types.ts", + "packages/excalidraw/charts/charts.types.ts", + "packages/element/src/types.ts", + "packages/common/src/editorInterface.ts" + ] + }, + { + "taskId": "fa-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0.5, + "matchedSignals": [ + "dependency" + ], + "payloadBytes": 307, + "estimatedTokens": 77, + "queryElapsedMs": 1467.3211574554443, + "indexElapsedMs": 364.5, + "topFiles": [ + "fastapi/applications.py", + "fastapi/encoders.py", + "fastapi/routing.py", + "fastapi/_compat.py", + "docs/en/docs/js/termynal.js" + ] + }, + { + "taskId": "fa-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 0.5, + "matchedSignals": [ + "router" + ], + "payloadBytes": 292, + "estimatedTokens": 73, + "queryElapsedMs": 1499.6511936187744, + "indexElapsedMs": 364.5, + "topFiles": [ + "fastapi/routing.py", + "fastapi/exceptions.py", + "fastapi/encoders.py", + "fastapi/_compat.py", + "fastapi/utils.py" + ] + }, + { + "taskId": "fa-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 308, + "estimatedTokens": 77, + "queryElapsedMs": 1510.103464126587, + "indexElapsedMs": 364.5, + "topFiles": [ + "fastapi/encoders.py", + "fastapi/applications.py", + "docs/en/docs/js/termynal.js", + "fastapi/_compat.py", + "fastapi/exceptions.py" + ] + }, + { + "taskId": "fa-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 317, + "estimatedTokens": 80, + "queryElapsedMs": 1470.5393314361572, + "indexElapsedMs": 364.5, + "topFiles": [ + "fastapi/dependencies/models.py", + "fastapi/_compat.py", + "fastapi/exceptions.py", + "docs/en/docs/js/termynal.js", + "fastapi/encoders.py" + ] + }, + { + "taskId": "fa-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": true, + "firstRelevantHit": 3, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 320, + "estimatedTokens": 80, + "queryElapsedMs": 1429.5134544372559, + "indexElapsedMs": 364.5, + "topFiles": [ + "fastapi/encoders.py", + "fastapi/applications.py", + "fastapi/routing.py", + "tests/test_additional_properties_bool.py", + "fastapi/_compat.py" + ] + }, + { + "taskId": "fa-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": 5, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 317, + "estimatedTokens": 80, + "queryElapsedMs": 1432.6813220977783, + "indexElapsedMs": 364.5, + "topFiles": [ + "fastapi/encoders.py", + "fastapi/applications.py", + "docs/en/docs/js/termynal.js", + "fastapi/_compat.py", + "fastapi/exceptions.py" + ] + } + ] + }, + { + "id": "repomix", + "displayName": "Repomix", + "credit": "Repomix repository packing (repomix on npm). Repomix does not expose query-specific retrieval; the lane charges pack generation to index time and ranks files by scanning pack sections for query tokens.", + "projectUrl": "https://github.com/yamadashy/repomix", + "status": "completed", + "tasksTotal": 18, + "tasksCompleted": 18, + "tasksFailed": 0, + "top3HitRate": 0.1111111111111111, + "averageUsefulness": 0.1111111111111111, + "averageEstimatedTokens": 5559173.666666667, + "averageQueryElapsedMs": 113.88888888888889, + "averageIndexElapsedMs": 286.27777777777777, + "byNiche": { + "team-convention": { + "tasks": 6, + "top3HitRate": 0 + }, + "edit-preflight": { + "tasks": 6, + "top3HitRate": 0.16666666666666666 + }, + "symbol-retrieval": { + "tasks": 6, + "top3HitRate": 0.16666666666666666 + } + }, + "taskResults": [ + { + "taskId": "as-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 1847698, + "queryElapsedMs": 26, + "indexElapsedMs": 178.66666666666666, + "topFiles": [ + ".codebase-context/.generations/a2cd3546-363b-4787-910e-11cce05901af/index/index-build.json" + ] + }, + { + "taskId": "as-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 1847698, + "queryElapsedMs": 28, + "indexElapsedMs": 178.66666666666666, + "topFiles": [ + ".codebase-context/.generations/a2cd3546-363b-4787-910e-11cce05901af/index/index-build.json" + ] + }, + { + "taskId": "as-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 1847698, + "queryElapsedMs": 21, + "indexElapsedMs": 178.66666666666666, + "topFiles": [ + ".codebase-context/.generations/a2cd3546-363b-4787-910e-11cce05901af/index/index-build.json" + ] + }, + { + "taskId": "as-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 1847698, + "queryElapsedMs": 21, + "indexElapsedMs": 178.66666666666666, + "topFiles": [ + ".codebase-context/.generations/a2cd3546-363b-4787-910e-11cce05901af/index/index-build.json" + ] + }, + { + "taskId": "as-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 1847698, + "queryElapsedMs": 14, + "indexElapsedMs": 178.66666666666666, + "topFiles": [ + ".codebase-context/.generations/a2cd3546-363b-4787-910e-11cce05901af/index/index-build.json" + ] + }, + { + "taskId": "as-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 1847698, + "queryElapsedMs": 18, + "indexElapsedMs": 178.66666666666666, + "topFiles": [ + ".codebase-context/.generations/a2cd3546-363b-4787-910e-11cce05901af/index/index-build.json" + ] + }, + { + "taskId": "ex-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 8754051, + "queryElapsedMs": 235, + "indexElapsedMs": 346.6666666666667, + "topFiles": [ + ".codebase-context/.generations/7ef9aa97-4e56-4dbe-aaa5-ba2bfb795092/index/index-build.json" + ] + }, + { + "taskId": "ex-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 8754051, + "queryElapsedMs": 235, + "indexElapsedMs": 346.6666666666667, + "topFiles": [ + ".codebase-context/.generations/7ef9aa97-4e56-4dbe-aaa5-ba2bfb795092/index/index-build.json" + ] + }, + { + "taskId": "ex-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 8754051, + "queryElapsedMs": 162, + "indexElapsedMs": 346.6666666666667, + "topFiles": [ + ".codebase-context/.generations/7ef9aa97-4e56-4dbe-aaa5-ba2bfb795092/index/index-build.json" + ] + }, + { + "taskId": "ex-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 8754051, + "queryElapsedMs": 194, + "indexElapsedMs": 346.6666666666667, + "topFiles": [ + ".codebase-context/.generations/7ef9aa97-4e56-4dbe-aaa5-ba2bfb795092/index/index-build.json" + ] + }, + { + "taskId": "ex-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 8754051, + "queryElapsedMs": 187, + "indexElapsedMs": 346.6666666666667, + "topFiles": [ + ".codebase-context/.generations/7ef9aa97-4e56-4dbe-aaa5-ba2bfb795092/index/index-build.json" + ] + }, + { + "taskId": "ex-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 8754051, + "queryElapsedMs": 189, + "indexElapsedMs": 346.6666666666667, + "topFiles": [ + ".codebase-context/.generations/7ef9aa97-4e56-4dbe-aaa5-ba2bfb795092/index/index-build.json" + ] + }, + { + "taskId": "fa-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 6075772, + "queryElapsedMs": 133, + "indexElapsedMs": 333.5, + "topFiles": [ + ".codebase-context/.generations/93a21abb-70cf-4b9c-8b23-c234ecf4165f/index/index-build.json" + ] + }, + { + "taskId": "fa-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 6075772, + "queryElapsedMs": 111, + "indexElapsedMs": 333.5, + "topFiles": [ + ".codebase-context/.generations/93a21abb-70cf-4b9c-8b23-c234ecf4165f/index/index-build.json" + ] + }, + { + "taskId": "fa-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 6075772, + "queryElapsedMs": 121, + "indexElapsedMs": 333.5, + "topFiles": [ + ".codebase-context/.generations/93a21abb-70cf-4b9c-8b23-c234ecf4165f/index/index-build.json" + ] + }, + { + "taskId": "fa-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 6075772, + "queryElapsedMs": 113, + "indexElapsedMs": 333.5, + "topFiles": [ + ".codebase-context/.generations/93a21abb-70cf-4b9c-8b23-c234ecf4165f/index/index-build.json" + ] + }, + { + "taskId": "fa-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 6075772, + "queryElapsedMs": 122, + "indexElapsedMs": 333.5, + "topFiles": [ + ".codebase-context/.generations/93a21abb-70cf-4b9c-8b23-c234ecf4165f/index/index-build.json" + ] + }, + { + "taskId": "fa-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 6075772, + "queryElapsedMs": 120, + "indexElapsedMs": 333.5, + "topFiles": [ + ".codebase-context/.generations/93a21abb-70cf-4b9c-8b23-c234ecf4165f/index/index-build.json" + ] + } + ] + }, + { + "id": "ripgrep", + "displayName": "ripgrep (lexical baseline)", + "credit": "BurntSushi ripgrep JSON search over the repo checkout. No index; measures lexical retrieval only.", + "projectUrl": "https://github.com/BurntSushi/ripgrep", + "status": "completed", + "tasksTotal": 18, + "tasksCompleted": 18, + "tasksFailed": 0, + "top3HitRate": 0.05555555555555555, + "averageUsefulness": 0.1111111111111111, + "averageEstimatedTokens": 5000.5, + "averageQueryElapsedMs": 76.16666666666667, + "averageIndexElapsedMs": 0, + "byNiche": { + "team-convention": { + "tasks": 6, + "top3HitRate": 0 + }, + "edit-preflight": { + "tasks": 6, + "top3HitRate": 0.16666666666666666 + }, + "symbol-retrieval": { + "tasks": 6, + "top3HitRate": 0 + } + }, + "taskResults": [ + { + "taskId": "as-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20008, + "estimatedTokens": 5002, + "queryElapsedMs": 24, + "indexElapsedMs": 0, + "topFiles": [ + "apps/angular-spotify/src/polyfills.ts", + "README.md", + "yarn.lock", + "docs/plans/2026-03-07-infinite-scroll-pagination-design.md", + "docs/plans/2026-03-01-visualizer-selector-design.md" + ] + }, + { + "taskId": "as-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 23, + "indexElapsedMs": 0, + "topFiles": [ + "apps/angular-spotify/src/environments/environment.ts", + "apps/angular-spotify/src/app/app.component.spec.ts", + "apps/angular-spotify/src/polyfills.ts", + ".codebase-context/.generations/e369989f-3d39-444e-a743-4ed8fb5ce7f5/intelligence.json", + "README.md" + ] + }, + { + "taskId": "as-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 33, + "indexElapsedMs": 0, + "topFiles": [ + "tsconfig.base.json", + "apps/angular-spotify/ngsw-config.json", + "apps/angular-spotify/project.json", + "apps/angular-spotify/src/environments/environment.prod.ts", + "apps/angular-spotify/src/environments/environment.ts" + ] + }, + { + "taskId": "as-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 26, + "indexElapsedMs": 0, + "topFiles": [ + "tsconfig.base.json", + "apps/angular-spotify/src/environments/environment.ts", + "apps/angular-spotify/src/manifest.webmanifest", + "apps/angular-spotify/src/index.prod.html", + "apps/angular-spotify/src/app/app.component.spec.ts" + ] + }, + { + "taskId": "as-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20008, + "estimatedTokens": 5002, + "queryElapsedMs": 22, + "indexElapsedMs": 0, + "topFiles": [ + "tsconfig.base.json", + ".codebase-context/.generations/e369989f-3d39-444e-a743-4ed8fb5ce7f5/intelligence.json", + "README.md", + "migrations.json", + "docs/plans/2026-03-07-infinite-scroll-pagination-design.md" + ] + }, + { + "taskId": "as-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 24, + "indexElapsedMs": 0, + "topFiles": [ + "tsconfig.base.json", + "apps/angular-spotify/src/environments/environment.prod.ts", + "apps/angular-spotify/src/environments/environment.ts", + "apps/angular-spotify/src/polyfills.ts", + "apps/angular-spotify/src/custom/ant/_button.scss" + ] + }, + { + "taskId": "ex-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20006, + "estimatedTokens": 5002, + "queryElapsedMs": 84, + "indexElapsedMs": 0, + "topFiles": [ + "vitest.config.mts", + ".github/copilot-instructions.md", + "yarn.lock", + ".eslintrc.json", + "examples/with-script-in-browser/components/CustomFooter.tsx" + ] + }, + { + "taskId": "ex-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": 5, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 93, + "indexElapsedMs": 0, + "topFiles": [ + ".env.development", + ".eslintrc.json", + "scripts/release.js", + "scripts/build-node.js", + "excalidraw-app/collab/Collab.tsx" + ] + }, + { + "taskId": "ex-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20002, + "estimatedTokens": 5001, + "queryElapsedMs": 105, + "indexElapsedMs": 0, + "topFiles": [ + "examples/with-nextjs/README.md", + "examples/with-script-in-browser/index.tsx", + ".eslintrc.json", + "README.md", + "examples/with-script-in-browser/components/ExampleApp.tsx" + ] + }, + { + "taskId": "ex-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": true, + "firstRelevantHit": 3, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 20004, + "estimatedTokens": 5001, + "queryElapsedMs": 119, + "indexElapsedMs": 0, + "topFiles": [ + ".dockerignore", + ".gitignore", + ".eslintrc.json", + "scripts/release.js", + "scripts/buildWasm.js" + ] + }, + { + "taskId": "ex-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 158, + "indexElapsedMs": 0, + "topFiles": [ + ".gitignore", + "examples/with-nextjs/.gitignore", + "examples/with-nextjs/next.config.js", + "examples/with-nextjs/tsconfig.json", + "examples/with-nextjs/yarn.lock" + ] + }, + { + "taskId": "ex-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 110, + "indexElapsedMs": 0, + "topFiles": [ + ".gitignore", + "examples/with-nextjs/next.config.js", + "examples/with-nextjs/src/pages/excalidraw-in-pages.tsx", + "examples/with-nextjs/src/excalidrawWrapper.tsx", + "examples/with-nextjs/src/app/layout.tsx" + ] + }, + { + "taskId": "fa-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0.5, + "matchedSignals": [ + "dependency" + ], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 99, + "indexElapsedMs": 0, + "topFiles": [ + "requirements-docs.txt", + "pyproject.toml", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/index/index-build.json", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/intelligence.json", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/relationships.json" + ] + }, + { + "taskId": "fa-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0.5, + "matchedSignals": [ + "router" + ], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 105, + "indexElapsedMs": 0, + "topFiles": [ + "pyproject.toml", + "requirements-docs-tests.txt", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/intelligence.json", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/health.json", + "LICENSE" + ] + }, + { + "taskId": "fa-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20002, + "estimatedTokens": 5001, + "queryElapsedMs": 82, + "indexElapsedMs": 0, + "topFiles": [ + "pyproject.toml", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/intelligence.json", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/relationships.json", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/health.json", + "LICENSE" + ] + }, + { + "taskId": "fa-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 96, + "indexElapsedMs": 0, + "topFiles": [ + "requirements-docs.txt", + "pyproject.toml", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/index/index-build.json", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/intelligence.json", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/relationships.json" + ] + }, + { + "taskId": "fa-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 76, + "indexElapsedMs": 0, + "topFiles": [ + "requirements-docs.txt", + "pyproject.toml", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/health.json", + "LICENSE", + "docs_src/body/tutorial001_py310.py" + ] + }, + { + "taskId": "fa-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 92, + "indexElapsedMs": 0, + "topFiles": [ + "pyproject.toml", + "requirements-docs-tests.txt", + "requirements-docs-insiders.txt", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/health.json", + "scripts/notify_translations.py" + ] + } + ] + } + ], + "summaryTable": [ + { + "laneId": "codebase-context", + "status": "completed", + "top3HitRate": 0.5, + "averageEstimatedTokens": 641.7222222222222, + "averageQueryElapsedMs": 927.6111111111111, + "averageIndexElapsedMs": 672.7777777777779 + }, + { + "laneId": "jCodeMunch", + "status": "completed", + "top3HitRate": 0.05555555555555555, + "averageEstimatedTokens": 202.38888888888889, + "averageQueryElapsedMs": 701.4444444444445, + "averageIndexElapsedMs": 69.27777777777777 + }, + { + "laneId": "aider-repo-map", + "status": "completed", + "top3HitRate": 0.2222222222222222, + "averageEstimatedTokens": 95.05555555555556, + "averageQueryElapsedMs": 1441.8654441833496, + "averageIndexElapsedMs": 349.8888888888889 + }, + { + "laneId": "repomix", + "status": "completed", + "top3HitRate": 0.1111111111111111, + "averageEstimatedTokens": 5559173.666666667, + "averageQueryElapsedMs": 113.88888888888889, + "averageIndexElapsedMs": 286.27777777777777 + }, + { + "laneId": "ripgrep", + "status": "completed", + "top3HitRate": 0.05555555555555555, + "averageEstimatedTokens": 5000.5, + "averageQueryElapsedMs": 76.16666666666667, + "averageIndexElapsedMs": 0 + } + ] +} diff --git a/scripts/lib/niche-benchmark/aider-repomap-query.py b/scripts/lib/niche-benchmark/aider-repomap-query.py new file mode 100644 index 0000000..a366133 --- /dev/null +++ b/scripts/lib/niche-benchmark/aider-repomap-query.py @@ -0,0 +1,102 @@ +#!/usr/bin/env python3 +"""Aider RepoMap lane for the niche prior-art benchmark.""" +from __future__ import annotations + +import argparse +import json +import os +import re +import subprocess +import sys +import time + + +def tokenize(text: str) -> list[str]: + return [ + token + for token in re.findall(r"[A-Za-z_][A-Za-z0-9_+#.-]{2,}", text) + if token.lower() + not in {"the", "and", "for", "with", "from", "that", "this", "what", "how", "does", "are"} + ] + + +def git_tracked_files(root: str) -> list[str]: + output = subprocess.check_output(["git", "ls-files"], cwd=root, text=True) + return [line.strip() for line in output.splitlines() if line.strip()] + + +def ranked_files(root: str, query: str, limit: int) -> tuple[list[str], str, float]: + from aider.io import InputOutput + from aider.models import Model + from aider.repomap import RepoMap + + started = time.time() + io = InputOutput(yes=True) + model = Model("gpt-4o-mini") + repo_map = RepoMap(map_tokens=4096, root=root, main_model=model, io=io) + tracked = git_tracked_files(root) + source_files = [ + path + for path in tracked + if any( + path.endswith(ext) + for ext in (".py", ".ts", ".tsx", ".js", ".jsx", ".go", ".rs", ".java", ".cs", ".cpp", ".c", ".h") + ) + ] + mentioned = tokenize(query) + tags = repo_map.get_ranked_tags([], source_files, [], mentioned) + top_files: list[str] = [] + for tag in tags: + rel = getattr(tag, "rel_fname", None) or getattr(tag, "fname", None) + if not isinstance(rel, str): + continue + normalized = rel.replace("\\", "/").lstrip("./") + if normalized and normalized not in top_files: + top_files.append(normalized) + if len(top_files) >= limit: + break + payload = json.dumps( + {"mentionedIdents": mentioned, "rankedTags": len(tags), "topFiles": top_files}, + indent=2, + ) + return top_files, payload, (time.time() - started) * 1000 + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--root", required=True) + parser.add_argument("--query", default="") + parser.add_argument("--limit", type=int, default=5) + parser.add_argument("--warm-index", action="store_true") + args = parser.parse_args() + + root = os.path.abspath(args.root) + if not os.path.isdir(os.path.join(root, ".git")): + print(json.dumps({"error": "missing_git_metadata", "topFiles": [], "payload": ""})) + return 2 + + try: + if args.warm_index: + _, _, elapsed_ms = ranked_files(root, "warm index scan", min(args.limit, 3)) + print(json.dumps({"status": "indexed", "indexElapsedMs": elapsed_ms})) + return 0 + + top_files, payload, elapsed_ms = ranked_files(root, args.query, args.limit) + print( + json.dumps( + { + "topFiles": top_files, + "payload": payload, + "queryElapsedMs": elapsed_ms, + "indexElapsedMs": 0, + } + ) + ) + return 0 + except Exception as exc: # noqa: BLE001 - benchmark boundary + print(json.dumps({"error": str(exc), "topFiles": [], "payload": ""})) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/lib/niche-benchmark/lanes.mjs b/scripts/lib/niche-benchmark/lanes.mjs new file mode 100644 index 0000000..b4b296f --- /dev/null +++ b/scripts/lib/niche-benchmark/lanes.mjs @@ -0,0 +1,408 @@ +import { spawnSync } from 'node:child_process'; +import { existsSync, mkdirSync, readFileSync } from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { withManagedStdioClientSession } from '../managed-mcp-session.mjs'; +import { + countUtf8Bytes, + estimateTokens, + normalizeRelativePath, + scoreTaskResult +} from './scoring.mjs'; + +const __dirname = path.dirname(fileURLToPath(import.meta.url)); +const aiderScript = path.join(__dirname, 'aider-repomap-query.py'); +const pythonCmd = process.platform === 'win32' ? 'python' : 'python3'; + +function runCommand(command, args, options = {}) { + const started = Date.now(); + const result = spawnSync(command, args, { + encoding: 'utf8', + maxBuffer: 256 * 1024 * 1024, + timeout: options.timeoutMs ?? 300_000, + cwd: options.cwd ?? process.cwd(), + env: options.env ?? process.env + }); + return { + status: typeof result.status === 'number' ? result.status : 1, + stdout: result.stdout ?? '', + stderr: result.stderr ?? '', + durationMs: Date.now() - started, + error: result.error?.message ?? null + }; +} + +function extractSearchFilesFromJson(value) { + const topFiles = []; + if (!value || typeof value !== 'object') return topFiles; + const results = Array.isArray(value.results) ? value.results : []; + for (const entry of results) { + const normalized = normalizeRelativePath(entry.file ?? entry.path ?? ''); + if (normalized && !topFiles.includes(normalized)) topFiles.push(normalized); + } + return topFiles; +} + +function codebaseContextCommand(projectRoot) { + const distIndex = path.join(projectRoot, 'dist', 'index.js'); + if (existsSync(distIndex)) { + return { command: process.execPath, args: [distIndex] }; + } + return { + command: process.platform === 'win32' ? 'npm.cmd' : 'npm', + args: ['exec', '--', 'codebase-context'] + }; +} + +export function ensureRepoCheckout(repo, projectRoot) { + const cloneRoot = path.join(projectRoot, 'benchmark-repos'); + const checkout = path.join(cloneRoot, repo.id); + mkdirSync(cloneRoot, { recursive: true }); + if (!existsSync(path.join(checkout, '.git'))) { + const clone = runCommand('git', ['clone', '--filter=blob:none', repo.repositoryUrl, checkout], { + timeoutMs: 600_000 + }); + if (clone.status !== 0) { + throw new Error(`git clone failed for ${repo.id}: ${clone.stderr || clone.error}`); + } + } + const fetch = runCommand( + 'git', + ['fetch', '--depth', '1', 'origin', repo.repositoryRef], + { cwd: checkout, timeoutMs: 600_000 } + ); + if (fetch.status !== 0) { + throw new Error(`git fetch failed for ${repo.id}: ${fetch.stderr || fetch.error}`); + } + const checkoutRun = runCommand('git', ['checkout', '--force', repo.repositoryRef], { + cwd: checkout, + timeoutMs: 120_000 + }); + if (checkoutRun.status !== 0) { + throw new Error(`git checkout failed for ${repo.id}: ${checkoutRun.stderr || checkoutRun.error}`); + } + return checkout; +} + +export async function warmCodebaseContextIndex(repoPath, projectRoot) { + const base = codebaseContextCommand(projectRoot); + const env = { + ...process.env, + CODEBASE_ROOT: repoPath, + CODEBASE_CONTEXT_ASCII: '1' + }; + return runCommand( + base.command, + [...base.args, 'reindex', '--no-embeddings', '--json'], + { env, timeoutMs: 900_000 } + ); +} + +export async function runCodebaseContextTask(task, repoPath, projectRoot, indexElapsedMs) { + const base = codebaseContextCommand(projectRoot); + const env = { + ...process.env, + CODEBASE_ROOT: repoPath, + CODEBASE_CONTEXT_ASCII: '1' + }; + const cc = task.codebaseContext; + const args = [...base.args, cc.command]; + if (cc.command === 'search') { + args.push('--query', task.query); + } + args.push(...cc.args); + const started = Date.now(); + const result = runCommand(base.command, args, { env, timeoutMs: 180_000 }); + const queryElapsedMs = Date.now() - started; + let payload = `${result.stdout}\n${result.stderr}`.trim(); + let topFiles = []; + if (result.status === 0) { + try { + const parsed = JSON.parse(result.stdout); + payload = JSON.stringify(parsed); + topFiles = extractSearchFilesFromJson(parsed).map((filePath) => { + const resolvedRepo = path.resolve(repoPath); + const resolvedFile = path.resolve(filePath); + if (resolvedFile.startsWith(resolvedRepo + path.sep)) { + return normalizeRelativePath(path.relative(resolvedRepo, resolvedFile)); + } + return normalizeRelativePath(filePath); + }); + if (topFiles.length === 0 && Array.isArray(parsed.patterns)) { + for (const pattern of parsed.patterns) { + const example = normalizeRelativePath(pattern?.bestExample ?? pattern?.example ?? ''); + if (example) topFiles.push(example); + } + } + } catch { + topFiles = []; + } + } + const score = scoreTaskResult(task, { payload, topFiles }); + return { + status: result.status === 0 ? 'completed' : 'tool_error', + niche: task.niche, + ...score, + payloadBytes: countUtf8Bytes(payload), + estimatedTokens: estimateTokens(countUtf8Bytes(payload)), + queryElapsedMs, + indexElapsedMs: indexElapsedMs ?? 0, + topFiles + }; +} + +function tokenizeQuery(query) { + return query + .toLowerCase() + .split(/[^a-z0-9_+#./-]+/i) + .map((token) => token.trim()) + .filter((token) => token.length > 2); +} + +export function runRipgrepTask(task, repoPath, indexElapsedMs = 0) { + const tokens = tokenizeQuery(task.query); + const pattern = tokens.slice(0, 8).join('|') || task.query.slice(0, 80); + const started = Date.now(); + const result = runCommand( + process.env.RG_PATH || 'rg', + ['--json', '-n', '--hidden', '--glob', '!.git/**', '--glob', '!node_modules/**', '--glob', '!dist/**', pattern, '.'], + { cwd: repoPath, timeoutMs: 120_000 } + ); + const queryElapsedMs = Date.now() - started; + const topFiles = []; + for (const line of result.stdout.split('\n')) { + if (!line.trim()) continue; + let event; + try { + event = JSON.parse(line); + } catch { + continue; + } + if (event.type !== 'match') continue; + const normalized = normalizeRelativePath(event.data?.path?.text ?? ''); + if (normalized && !topFiles.includes(normalized)) topFiles.push(normalized); + if (topFiles.length >= 5) break; + } + const payload = result.stdout.slice(0, 20_000); + const score = scoreTaskResult(task, { payload, topFiles }); + return { + status: result.status === 0 || result.status === 1 ? 'completed' : 'tool_error', + niche: task.niche, + ...score, + payloadBytes: countUtf8Bytes(payload), + estimatedTokens: estimateTokens(countUtf8Bytes(payload)), + queryElapsedMs, + indexElapsedMs, + topFiles + }; +} + +function parseJCodeMunchPayload(result) { + const textParts = Array.isArray(result?.content) + ? result.content + .map((item) => (typeof item?.text === 'string' ? item.text.trim() : '')) + .filter(Boolean) + : []; + const payload = textParts.join('\n'); + let topFiles = []; + try { + const parsed = JSON.parse(payload); + const results = Array.isArray(parsed.results) ? parsed.results : []; + for (const entry of results) { + const normalized = normalizeRelativePath(entry.file ?? entry.path ?? ''); + if (normalized && !topFiles.includes(normalized)) topFiles.push(normalized); + } + } catch { + for (const line of payload.split('\n')) { + if (!line.startsWith('s,')) continue; + const columns = line.split(','); + const fileColumn = columns[3] ?? columns[4]; + const normalized = normalizeRelativePath(fileColumn ?? ''); + if (normalized && !topFiles.includes(normalized)) topFiles.push(normalized); + } + } + return { payload, topFiles }; +} + +export async function warmJCodeMunchRepo(repoPath) { + return withManagedStdioClientSession( + { + serverCommand: pythonCmd, + serverArgs: ['-m', 'jcodemunch_mcp.server', 'serve', '--transport', 'stdio', '--log-level', 'ERROR'], + serverEnv: { ...process.env, JCODEMUNCH_USE_AI_SUMMARIES: 'false' }, + cwd: repoPath, + connectTimeoutMs: 30_000 + }, + async ({ client }) => { + await new Promise((resolve) => setTimeout(resolve, 2_000)); + const started = Date.now(); + const indexResult = await client.callTool({ + name: 'index_folder', + arguments: { + path: path.resolve(repoPath), + use_ai_summaries: false, + incremental: true, + follow_symlinks: false, + extra_ignore_patterns: ['.codebase-context/**', 'node_modules/**'] + } + }); + const { payload } = parseJCodeMunchPayload(indexResult); + let repoName = null; + try { + const parsed = JSON.parse(payload || '{}'); + repoName = typeof parsed.repo === 'string' ? parsed.repo : parsed.name ?? null; + } catch { + repoName = null; + } + return { repoName, indexElapsedMs: Date.now() - started }; + } + ); +} + +export async function runJCodeMunchTask(task, repoPath, repoName, indexElapsedMs) { + if (!repoName) { + return { + status: 'setup_failed', + niche: task.niche, + top3Hit: false, + usefulnessScore: 0, + queryElapsedMs: 0, + indexElapsedMs, + payloadBytes: 0, + estimatedTokens: 0, + topFiles: [] + }; + } + return withManagedStdioClientSession( + { + serverCommand: pythonCmd, + serverArgs: ['-m', 'jcodemunch_mcp.server', 'serve', '--transport', 'stdio', '--log-level', 'ERROR'], + serverEnv: { ...process.env, JCODEMUNCH_USE_AI_SUMMARIES: 'false' }, + cwd: repoPath, + connectTimeoutMs: 30_000 + }, + async ({ client }) => { + await new Promise((resolve) => setTimeout(resolve, 1_000)); + const started = Date.now(); + const result = await client.callTool({ + name: 'search_symbols', + arguments: { + repo: repoName, + query: task.query, + max_results: 5, + detail_level: 'compact', + semantic: false + } + }); + const queryElapsedMs = Date.now() - started; + const { payload, topFiles } = parseJCodeMunchPayload(result); + const score = scoreTaskResult(task, { payload, topFiles }); + return { + status: 'completed', + niche: task.niche, + ...score, + payloadBytes: countUtf8Bytes(payload), + estimatedTokens: estimateTokens(countUtf8Bytes(payload)), + queryElapsedMs, + indexElapsedMs, + topFiles + }; + } + ); +} + +export function warmAiderRepo(repoPath) { + return runCommand( + pythonCmd, + [aiderScript, '--root', repoPath, '--warm-index', '--limit', '3'], + { cwd: repoPath, timeoutMs: 900_000 } + ); +} + +export function runAiderTask(task, repoPath, indexElapsedMs) { + const result = runCommand( + pythonCmd, + [aiderScript, '--root', repoPath, '--query', task.query, '--limit', '5'], + { cwd: repoPath, timeoutMs: 300_000 } + ); + let topFiles = []; + let payload = result.stdout; + try { + const parsed = JSON.parse(result.stdout); + topFiles = Array.isArray(parsed.topFiles) ? parsed.topFiles : []; + payload = parsed.payload ?? result.stdout; + } catch { + topFiles = []; + } + const score = scoreTaskResult(task, { payload, topFiles }); + return { + status: result.status === 0 ? 'completed' : 'tool_error', + niche: task.niche, + ...score, + payloadBytes: countUtf8Bytes(payload), + estimatedTokens: estimateTokens(countUtf8Bytes(payload)), + queryElapsedMs: (() => { + try { + return JSON.parse(result.stdout || '{}').queryElapsedMs ?? result.durationMs; + } catch { + return result.durationMs; + } + })(), + indexElapsedMs, + topFiles + }; +} + +function rankRepomixPack(packText, query, limit) { + const tokens = tokenizeQuery(query); + const sections = packText.split(/\n={32}\n/); + const scores = new Map(); + for (const section of sections) { + const fileMatch = section.match(/^File:\s+(.+)$/m); + if (!fileMatch) continue; + const filePath = fileMatch[1].trim(); + const lower = section.toLowerCase(); + let score = 0; + for (const token of tokens) { + if (lower.includes(token)) score += 1; + } + if (score > 0) scores.set(filePath, score); + } + return [...scores.entries()] + .sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0])) + .slice(0, limit) + .map(([filePath]) => filePath.replace(/\\/g, '/')); +} + +export function warmRepomixRepo(repoPath) { + const started = Date.now(); + const result = runCommand( + process.platform === 'win32' ? 'npx.cmd' : 'npx', + ['--yes', 'repomix', '--stdout', '--quiet', '--style', 'plain'], + { cwd: repoPath, timeoutMs: 900_000 } + ); + return { + packText: result.stdout, + indexElapsedMs: Date.now() - started, + status: result.status === 0 ? 'completed' : 'setup_failed', + error: result.status === 0 ? null : result.stderr || result.error + }; +} + +export function runRepomixTask(task, packText, indexElapsedMs) { + const started = Date.now(); + const topFiles = rankRepomixPack(packText, task.query, 5); + const payload = topFiles.join('\n'); + const queryElapsedMs = Date.now() - started; + const score = scoreTaskResult(task, { payload, topFiles }); + return { + status: packText ? 'completed' : 'setup_failed', + niche: task.niche, + ...score, + payloadBytes: countUtf8Bytes(packText.slice(0, 50_000)), + estimatedTokens: estimateTokens(countUtf8Bytes(packText)), + queryElapsedMs, + indexElapsedMs, + topFiles + }; +} diff --git a/scripts/lib/niche-benchmark/scoring.mjs b/scripts/lib/niche-benchmark/scoring.mjs new file mode 100644 index 0000000..c3ad84c --- /dev/null +++ b/scripts/lib/niche-benchmark/scoring.mjs @@ -0,0 +1,134 @@ +export function normalizeText(value) { + return String(value ?? '') + .toLowerCase() + .replace(/\\/g, '/'); +} + +export function normalizeRelativePath(candidate) { + if (typeof candidate !== 'string') return null; + const trimmed = candidate.trim().replace(/^["']|["']$/g, ''); + if (!trimmed) return null; + const normalized = trimmed.replace(/\\/g, '/').replace(/^\.\//, ''); + if (/^[A-Za-z]:\//.test(normalized)) { + return normalized.replace(/^[A-Za-z]:\//, ''); + } + return normalized.split(':')[0]; +} + +export function matchesPatterns(candidate, patterns) { + if (!patterns?.length) return false; + const normalizedCandidate = normalizeText(candidate); + return patterns.some((pattern) => normalizedCandidate.includes(normalizeText(pattern))); +} + +export function firstRelevantHit(topFiles, patterns) { + if (!patterns?.length) return null; + for (let index = 0; index < topFiles.length; index++) { + if (matchesPatterns(topFiles[index], patterns)) return index + 1; + } + return null; +} + +export function scoreTaskResult(task, { payload = '', topFiles = [] }) { + const hit = firstRelevantHit(topFiles, task.expectedFilePatterns); + const top3Hit = hit !== null && hit <= 3; + const normalizedPayload = normalizeText(payload); + const matchedSignals = (task.expectedSignals ?? []).filter((signal) => + normalizedPayload.includes(normalizeText(signal)) + ); + const usefulnessScore = + task.expectedSignals?.length > 0 + ? matchedSignals.length / Math.max(task.expectedSignals.length, 1) + : top3Hit + ? 1 + : 0; + + return { + top3Hit, + firstRelevantHit: hit, + usefulnessScore, + matchedSignals + }; +} + +export function countUtf8Bytes(str) { + return Buffer.byteLength(String(str ?? ''), 'utf8'); +} + +export function estimateTokens(bytes) { + return Math.ceil(bytes / 4); +} + +export function aggregateLaneResults(taskResults) { + const completed = taskResults.filter((result) => result.status === 'completed'); + const n = taskResults.length; + const top3Hits = completed.filter((result) => result.top3Hit).length; + const avgUsefulness = + completed.length > 0 + ? completed.reduce((sum, result) => sum + result.usefulnessScore, 0) / completed.length + : null; + const avgTokens = + completed.length > 0 + ? completed.reduce((sum, result) => sum + result.estimatedTokens, 0) / completed.length + : null; + const avgQueryMs = + completed.length > 0 + ? completed.reduce((sum, result) => sum + result.queryElapsedMs, 0) / completed.length + : null; + const avgIndexMs = + completed.length > 0 + ? completed.reduce((sum, result) => sum + (result.indexElapsedMs ?? 0), 0) / completed.length + : null; + + const byNiche = {}; + for (const niche of ['team-convention', 'edit-preflight', 'symbol-retrieval']) { + const nicheRows = completed.filter((result) => result.niche === niche); + byNiche[niche] = { + tasks: nicheRows.length, + top3HitRate: nicheRows.length > 0 ? nicheRows.filter((row) => row.top3Hit).length / nicheRows.length : null + }; + } + + const failed = taskResults.filter((result) => result.status !== 'completed'); + return { + status: completed.length === 0 ? 'setup_failed' : failed.length > 0 ? 'partial' : 'completed', + tasksTotal: n, + tasksCompleted: completed.length, + tasksFailed: failed.length, + top3HitRate: completed.length > 0 ? top3Hits / completed.length : null, + averageUsefulness: avgUsefulness, + averageEstimatedTokens: avgTokens, + averageQueryElapsedMs: avgQueryMs, + averageIndexElapsedMs: avgIndexMs, + byNiche, + taskResults + }; +} + +export function aggregateEvidence(laneEvidence, manifest, protocol, runMeta) { + return { + artifactType: 'niche-prior-art-benchmark-evidence', + generatedAt: new Date().toISOString(), + protocol: { + name: protocol.name, + frozenDate: protocol.frozenDate, + manifestFrozenDate: manifest.frozenDate + }, + runMeta, + comparators: protocol.comparators.map((comparator) => ({ + id: comparator.id, + displayName: comparator.displayName, + credit: comparator.credit, + projectUrl: comparator.projectUrl, + ...(laneEvidence[comparator.id] ?? { status: 'not_run' }) + })), + summaryTable: Object.entries(laneEvidence).map(([laneId, lane]) => ({ + laneId, + status: lane.status, + top3HitRate: lane.top3HitRate, + averageEstimatedTokens: lane.averageEstimatedTokens, + averageQueryElapsedMs: lane.averageQueryElapsedMs, + averageIndexElapsedMs: lane.averageIndexElapsedMs + })) + }; +} diff --git a/scripts/run-niche-benchmark.mjs b/scripts/run-niche-benchmark.mjs new file mode 100644 index 0000000..06200d3 --- /dev/null +++ b/scripts/run-niche-benchmark.mjs @@ -0,0 +1,283 @@ +#!/usr/bin/env node +/** + * One-command niche prior-art benchmark runner. + * + * Usage: + * node scripts/run-niche-benchmark.mjs + * node scripts/run-niche-benchmark.mjs --lanes codebase-context,ripgrep --output results/niche-benchmark-evidence.json + * node scripts/run-niche-benchmark.mjs --validate-fixtures + */ + +import { createHash } from 'node:crypto'; +import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { parseArgs } from 'node:util'; +import { spawnSync } from 'node:child_process'; +import { + ensureRepoCheckout, + runAiderTask, + runCodebaseContextTask, + runJCodeMunchTask, + runRepomixTask, + runRipgrepTask, + warmAiderRepo, + warmCodebaseContextIndex, + warmJCodeMunchRepo, + warmRepomixRepo +} from './lib/niche-benchmark/lanes.mjs'; +import { aggregateEvidence, aggregateLaneResults } from './lib/niche-benchmark/scoring.mjs'; + +const __dirname = path.dirname(fileURLToPath(import.meta.url)); +const projectRoot = path.join(__dirname, '..'); +const defaultManifest = path.join(projectRoot, 'tests', 'fixtures', 'niche-benchmark-manifest.json'); +const defaultProtocol = path.join(projectRoot, 'tests', 'fixtures', 'niche-benchmark-protocol.json'); +const defaultOutput = path.join(projectRoot, 'results', 'niche-benchmark-evidence.json'); + +const ALL_LANES = ['codebase-context', 'jCodeMunch', 'aider-repo-map', 'repomix', 'ripgrep']; + +function sha256File(filePath) { + return createHash('sha256').update(readFileSync(filePath)).digest('hex'); +} + +function loadJson(filePath) { + return JSON.parse(readFileSync(filePath, 'utf8')); +} + +function ensureBuilt() { + const distIndex = path.join(projectRoot, 'dist', 'index.js'); + if (existsSync(distIndex)) return; + console.log('Building codebase-context (dist missing)...'); + const build = spawnSync('pnpm', ['run', 'build'], { + cwd: projectRoot, + stdio: 'inherit', + encoding: 'utf8' + }); + if (build.status !== 0) { + throw new Error('pnpm run build failed; dist/index.js is required for the codebase-context lane'); + } +} + +function validateFixtures(manifestPath, protocolPath) { + const manifest = loadJson(manifestPath); + const protocol = loadJson(protocolPath); + const errors = []; + if (manifest.tasks.length !== 18) errors.push(`expected 18 tasks, found ${manifest.tasks.length}`); + if (manifest.repos.length !== 3) errors.push(`expected 3 repos, found ${manifest.repos.length}`); + const nicheCounts = {}; + for (const task of manifest.tasks) { + nicheCounts[task.niche] = (nicheCounts[task.niche] ?? 0) + 1; + } + for (const niche of protocol.niches) { + if (nicheCounts[niche] !== 6) { + errors.push(`expected 6 ${niche} tasks, found ${nicheCounts[niche] ?? 0}`); + } + } + const repoIds = new Set(manifest.repos.map((repo) => repo.id)); + for (const task of manifest.tasks) { + if (!repoIds.has(task.repoId)) errors.push(`task ${task.id} references unknown repo ${task.repoId}`); + } + if (errors.length > 0) { + console.error(errors.join('\n')); + process.exit(1); + } + console.log('Fixture validation passed.'); + console.log(` manifest: ${manifestPath}`); + console.log(` protocol: ${protocolPath}`); + console.log(` manifest sha256: ${sha256File(manifestPath)}`); + console.log(` protocol sha256: ${sha256File(protocolPath)}`); +} + +async function runLane(laneId, manifest, repoCheckouts) { + const tasksByRepo = manifest.repos.map((repo) => ({ + repo, + checkout: repoCheckouts[repo.id], + tasks: manifest.tasks.filter((task) => task.repoId === repo.id) + })); + + const taskResults = []; + + if (laneId === 'ripgrep') { + for (const group of tasksByRepo) { + for (const task of group.tasks) { + taskResults.push({ + taskId: task.id, + ...runRipgrepTask(task, group.checkout, 0) + }); + } + } + return aggregateLaneResults(taskResults.map((row) => ({ ...row, status: row.status ?? 'completed' }))); + } + + for (const group of tasksByRepo) { + let indexElapsedMs = 0; + let repomixPack = ''; + let jcodemunchRepo = null; + + if (laneId === 'codebase-context') { + const index = await warmCodebaseContextIndex(group.checkout, projectRoot); + indexElapsedMs = index.durationMs; + if (index.status !== 0) { + return { + status: 'setup_failed', + reason: `codebase-context reindex failed on ${group.repo.id}`, + taskResults: [] + }; + } + } else if (laneId === 'jCodeMunch') { + try { + const warmed = await warmJCodeMunchRepo(group.checkout); + jcodemunchRepo = warmed.repoName; + indexElapsedMs = warmed.indexElapsedMs; + } catch (error) { + return { + status: 'setup_failed', + reason: `jCodeMunch setup failed on ${group.repo.id}: ${error.message}`, + taskResults: [] + }; + } + } else if (laneId === 'aider-repo-map') { + const warm = warmAiderRepo(group.checkout); + indexElapsedMs = warm.durationMs; + if (warm.status !== 0) { + return { + status: 'setup_failed', + reason: `Aider repo map warm index failed on ${group.repo.id}`, + taskResults: [] + }; + } + } else if (laneId === 'repomix') { + const warm = warmRepomixRepo(group.checkout); + indexElapsedMs = warm.indexElapsedMs; + if (warm.status !== 'completed') { + return { + status: 'setup_failed', + reason: warm.error ?? `Repomix pack failed on ${group.repo.id}`, + taskResults: [] + }; + } + repomixPack = warm.packText; + } + + for (const task of group.tasks) { + let row; + if (laneId === 'codebase-context') { + row = await runCodebaseContextTask(task, group.checkout, projectRoot, indexElapsedMs / group.tasks.length); + } else if (laneId === 'jCodeMunch') { + row = await runJCodeMunchTask(task, group.checkout, jcodemunchRepo, indexElapsedMs / group.tasks.length); + } else if (laneId === 'aider-repo-map') { + row = runAiderTask(task, group.checkout, indexElapsedMs / group.tasks.length); + } else if (laneId === 'repomix') { + row = runRepomixTask(task, repomixPack, indexElapsedMs / group.tasks.length); + } + taskResults.push({ taskId: task.id, ...row }); + } + } + + return aggregateLaneResults( + taskResults.map((row) => ({ + ...row, + status: row.status ?? 'completed' + })) + ); +} + +async function main() { + const { values } = parseArgs({ + options: { + help: { type: 'boolean', default: false }, + 'validate-fixtures': { type: 'boolean', default: false }, + manifest: { type: 'string' }, + protocol: { type: 'string' }, + lanes: { type: 'string' }, + output: { type: 'string' }, + 'skip-build': { type: 'boolean', default: false } + } + }); + + const manifestPath = path.resolve(values.manifest ?? defaultManifest); + const protocolPath = path.resolve(values.protocol ?? defaultProtocol); + const outputPath = path.resolve(values.output ?? defaultOutput); + + if (values.help) { + console.log(`Usage: node scripts/run-niche-benchmark.mjs [options] + + --validate-fixtures Validate frozen manifest/protocol and exit + --lanes Default: ${ALL_LANES.join(',')} + --output Default: ${defaultOutput} + --skip-build Do not auto-build dist before running +`); + process.exit(0); + } + + if (values['validate-fixtures']) { + validateFixtures(manifestPath, protocolPath); + process.exit(0); + } + + const manifest = loadJson(manifestPath); + const protocol = loadJson(protocolPath); + validateFixtures(manifestPath, protocolPath); + + const lanes = (values.lanes ?? ALL_LANES.join(',')) + .split(',') + .map((lane) => lane.trim()) + .filter(Boolean); + for (const lane of lanes) { + if (!ALL_LANES.includes(lane)) { + throw new Error(`Unknown lane ${lane}. Supported: ${ALL_LANES.join(', ')}`); + } + } + + if (!values['skip-build'] && lanes.includes('codebase-context')) { + ensureBuilt(); + } + + console.log(`\nNiche prior-art benchmark (${manifest.tasks.length} tasks, ${lanes.length} lanes)`); + const repoCheckouts = {}; + for (const repo of manifest.repos) { + console.log(`Checking out ${repo.id} @ ${repo.repositoryRef.slice(0, 12)}...`); + repoCheckouts[repo.id] = ensureRepoCheckout(repo, projectRoot); + } + + const laneEvidence = {}; + for (const laneId of lanes) { + console.log(`\n=== Lane: ${laneId} ===`); + try { + laneEvidence[laneId] = await runLane(laneId, manifest, repoCheckouts); + console.log( + ` top-3 hit rate: ${laneEvidence[laneId].top3HitRate === null ? 'n/a' : (laneEvidence[laneId].top3HitRate * 100).toFixed(1) + '%'}` + ); + console.log( + ` avg tokens: ${laneEvidence[laneId].averageEstimatedTokens === null ? 'n/a' : Math.round(laneEvidence[laneId].averageEstimatedTokens)}` + ); + console.log( + ` avg query ms: ${laneEvidence[laneId].averageQueryElapsedMs === null ? 'n/a' : Math.round(laneEvidence[laneId].averageQueryElapsedMs)}` + ); + } catch (error) { + laneEvidence[laneId] = { status: 'setup_failed', reason: error.message, taskResults: [] }; + console.error(` SETUP FAILED: ${error.message}`); + } + } + + const evidence = aggregateEvidence(laneEvidence, manifest, protocol, { + manifestPath, + protocolPath, + manifestSha256: sha256File(manifestPath), + protocolSha256: sha256File(protocolPath), + nodeVersion: process.version, + lanes, + repoCheckouts: Object.fromEntries( + manifest.repos.map((repo) => [repo.id, { ref: repo.repositoryRef, path: repoCheckouts[repo.id] }]) + ) + }); + + mkdirSync(path.dirname(outputPath), { recursive: true }); + writeFileSync(outputPath, `${JSON.stringify(evidence, null, 2)}\n`); + console.log(`\nEvidence written to ${outputPath}`); +} + +main().catch((error) => { + console.error('Fatal:', error); + process.exit(2); +}); diff --git a/tests/fixtures/niche-benchmark-manifest.json b/tests/fixtures/niche-benchmark-manifest.json new file mode 100644 index 0000000..fde60c6 --- /dev/null +++ b/tests/fixtures/niche-benchmark-manifest.json @@ -0,0 +1,204 @@ +{ + "description": "Frozen 18-task niche benchmark across three public repos, two tasks per niche per repo. Expected patterns are path substring checks, not full paths.", + "frozenDate": "2026-10-30", + "cloneRoot": "benchmark-repos", + "repos": [ + { + "id": "angular-spotify", + "repository": "trungk18/angular-spotify", + "repositoryUrl": "https://github.com/trungk18/angular-spotify", + "repositoryRef": "ff9efa765c53cfde78c9a172c62d515ae8ef9fe0", + "language": "typescript", + "sizeClass": "medium", + "sourceFixture": "tests/fixtures/discovery-angular-spotify.json" + }, + { + "id": "excalidraw", + "repository": "excalidraw/excalidraw", + "repositoryUrl": "https://github.com/excalidraw/excalidraw", + "repositoryRef": "e18c1dd213000dde0ae94ef7eb00aab537b39708", + "language": "typescript", + "sizeClass": "large", + "sourceFixture": "tests/fixtures/discovery-excalidraw.json" + }, + { + "id": "fastapi", + "repository": "fastapi/fastapi", + "repositoryUrl": "https://github.com/fastapi/fastapi", + "repositoryRef": "628c34e0cae200564d191c95d7edea78c88c4b5e", + "language": "python", + "sizeClass": "medium", + "sourceFixture": null + } + ], + "tasks": [ + { + "id": "as-team-01", + "repoId": "angular-spotify", + "niche": "team-convention", + "sourceTaskId": "as-find-01", + "query": "What DI pattern dominates here?", + "expectedFilePatterns": ["service", "inject"], + "expectedSignals": ["dependencyInjection"], + "codebaseContext": { "command": "patterns", "args": ["--category", "di", "--json"] } + }, + { + "id": "as-team-02", + "repoId": "angular-spotify", + "niche": "team-convention", + "sourceTaskId": "as-find-02", + "query": "What state-management pattern should an agent imitate?", + "expectedFilePatterns": ["store", "state"], + "expectedSignals": ["stateManagement"], + "codebaseContext": { "command": "patterns", "args": ["--category", "state", "--json"] } + }, + { + "id": "as-ep-01", + "repoId": "angular-spotify", + "niche": "edit-preflight", + "sourceTaskId": "as-ep-01", + "query": "update how authorization token headers are attached to API requests", + "expectedFilePatterns": ["auth", "interceptor"], + "codebaseContext": { "command": "search", "args": ["--intent", "edit", "--limit", "5", "--json"] } + }, + { + "id": "as-ep-02", + "repoId": "angular-spotify", + "niche": "edit-preflight", + "sourceTaskId": "as-ep-02", + "query": "change the logic that skips to the next song", + "expectedFilePatterns": ["player-api", "player/api", "player"], + "codebaseContext": { "command": "search", "args": ["--intent", "edit", "--limit", "5", "--json"] } + }, + { + "id": "as-sym-01", + "repoId": "angular-spotify", + "niche": "symbol-retrieval", + "sourceTaskId": "as-search-01", + "query": "skip to next song", + "expectedFilePatterns": ["player-api", "player/api", "player"], + "codebaseContext": { "command": "search", "args": ["--intent", "explore", "--limit", "5", "--json"] } + }, + { + "id": "as-sym-02", + "repoId": "angular-spotify", + "niche": "symbol-retrieval", + "sourceTaskId": "as-search-03", + "query": "add authorization token to API requests", + "expectedFilePatterns": ["auth", "interceptor"], + "codebaseContext": { "command": "search", "args": ["--intent", "explore", "--limit", "5", "--json"] } + }, + { + "id": "ex-team-01", + "repoId": "excalidraw", + "niche": "team-convention", + "sourceTaskId": "ex-find-01", + "query": "What state-management pattern dominates here?", + "expectedFilePatterns": ["appstate", "state", "store"], + "expectedSignals": ["stateManagement"], + "codebaseContext": { "command": "patterns", "args": ["--category", "state", "--json"] } + }, + { + "id": "ex-team-02", + "repoId": "excalidraw", + "niche": "team-convention", + "sourceTaskId": "ex-find-04", + "query": "What dependency injection or dependency pattern dominates here?", + "expectedFilePatterns": ["component", "app", "src"], + "expectedSignals": ["dependencyInjection"], + "codebaseContext": { "command": "patterns", "args": ["--category", "di", "--json"] } + }, + { + "id": "ex-ep-01", + "repoId": "excalidraw", + "niche": "edit-preflight", + "sourceTaskId": "ex-ep-01", + "query": "change how scene updates are applied", + "expectedFilePatterns": ["scene"], + "codebaseContext": { "command": "search", "args": ["--intent", "edit", "--limit", "5", "--json"] } + }, + { + "id": "ex-ep-02", + "repoId": "excalidraw", + "niche": "edit-preflight", + "sourceTaskId": "ex-ep-03", + "query": "modify scene serialization to json export", + "expectedFilePatterns": ["scene", "json", "data"], + "codebaseContext": { "command": "search", "args": ["--intent", "edit", "--limit", "5", "--json"] } + }, + { + "id": "ex-sym-01", + "repoId": "excalidraw", + "niche": "symbol-retrieval", + "sourceTaskId": "ex-search-01", + "query": "element type definitions", + "expectedFilePatterns": ["element", "type"], + "codebaseContext": { "command": "search", "args": ["--intent", "explore", "--limit", "5", "--json"] } + }, + { + "id": "ex-sym-02", + "repoId": "excalidraw", + "niche": "symbol-retrieval", + "sourceTaskId": "ex-search-02", + "query": "scene serialization export json", + "expectedFilePatterns": ["scene", "json", "data"], + "codebaseContext": { "command": "search", "args": ["--intent", "explore", "--limit", "5", "--json"] } + }, + { + "id": "fa-team-01", + "repoId": "fastapi", + "niche": "team-convention", + "sourceTaskId": null, + "query": "How does FastAPI wire dependency injection for route handlers?", + "expectedFilePatterns": ["param_functions", "depends"], + "expectedSignals": ["Depends", "dependency"], + "codebaseContext": { "command": "patterns", "args": ["--category", "di", "--json"] } + }, + { + "id": "fa-team-02", + "repoId": "fastapi", + "niche": "team-convention", + "sourceTaskId": null, + "query": "Where are API routers defined and included in the application?", + "expectedFilePatterns": ["routing", "applications"], + "expectedSignals": ["APIRouter", "router"], + "codebaseContext": { "command": "patterns", "args": ["--category", "all", "--json"] } + }, + { + "id": "fa-ep-01", + "repoId": "fastapi", + "niche": "edit-preflight", + "sourceTaskId": null, + "query": "change how OAuth2 bearer tokens are extracted from requests", + "expectedFilePatterns": ["oauth2", "security"], + "codebaseContext": { "command": "search", "args": ["--intent", "edit", "--limit", "5", "--json"] } + }, + { + "id": "fa-ep-02", + "repoId": "fastapi", + "niche": "edit-preflight", + "sourceTaskId": null, + "query": "modify OpenAPI schema generation for response models", + "expectedFilePatterns": ["openapi"], + "codebaseContext": { "command": "search", "args": ["--intent", "edit", "--limit", "5", "--json"] } + }, + { + "id": "fa-sym-01", + "repoId": "fastapi", + "niche": "symbol-retrieval", + "sourceTaskId": null, + "query": "validate request body with pydantic field constraints", + "expectedFilePatterns": ["routing", "param"], + "codebaseContext": { "command": "search", "args": ["--intent", "explore", "--limit", "5", "--json"] } + }, + { + "id": "fa-sym-02", + "repoId": "fastapi", + "niche": "symbol-retrieval", + "sourceTaskId": null, + "query": "HTTP exception handler returns JSON error detail", + "expectedFilePatterns": ["exception", "routing"], + "codebaseContext": { "command": "search", "args": ["--intent", "explore", "--limit", "5", "--json"] } + } + ] +} diff --git a/tests/fixtures/niche-benchmark-protocol.json b/tests/fixtures/niche-benchmark-protocol.json new file mode 100644 index 0000000..2490015 --- /dev/null +++ b/tests/fixtures/niche-benchmark-protocol.json @@ -0,0 +1,73 @@ +{ + "name": "niche-prior-art-benchmark", + "frozenDate": "2026-10-30", + "scope": "local-context-retrieval", + "claimAllowed": false, + "purpose": "Reproducible comparison of codebase-context against credited prior art on team-convention lookup, edit-preflight navigation, and symbol retrieval. Not patch quality or agent productivity.", + "niches": ["team-convention", "edit-preflight", "symbol-retrieval"], + "comparators": [ + { + "id": "codebase-context", + "displayName": "Codebase Context", + "kind": "product", + "credit": "This repository (local dist CLI / MCP surfaces).", + "projectUrl": "https://github.com/PatrickSys/codebase-context" + }, + { + "id": "jCodeMunch", + "displayName": "jCodeMunch", + "kind": "mcp-comparator", + "credit": "jCodeMunch MCP symbol search and repo indexing (jcodemunch-mcp on PyPI).", + "projectUrl": "https://github.com/jgravelle/jcodemunch-mcp" + }, + { + "id": "aider-repo-map", + "displayName": "Aider repo map", + "kind": "cli-comparator", + "credit": "Paul Gauthier's Aider RepoMap ranking (aider-chat on PyPI). Query tokens are passed as mentioned identifiers; there is no separate natural-language search API.", + "projectUrl": "https://github.com/Aider-AI/aider" + }, + { + "id": "repomix", + "displayName": "Repomix", + "kind": "cli-comparator", + "credit": "Repomix repository packing (repomix on npm). Repomix does not expose query-specific retrieval; the lane charges pack generation to index time and ranks files by scanning pack sections for query tokens.", + "projectUrl": "https://github.com/yamadashy/repomix" + }, + { + "id": "ripgrep", + "displayName": "ripgrep (lexical baseline)", + "kind": "baseline", + "credit": "BurntSushi ripgrep JSON search over the repo checkout. No index; measures lexical retrieval only.", + "projectUrl": "https://github.com/BurntSushi/ripgrep" + } + ], + "metrics": { + "accuracy": { + "primary": "top3HitRate", + "definition": "Share of tasks where any expectedFilePattern appears in the first three ranked paths returned by the lane." + }, + "tokens": { + "primary": "estimatedTokens", + "definition": "UTF-8 payload bytes divided by four (same heuristic as discovery benchmark)." + }, + "latency": { + "primary": "queryElapsedMs", + "secondary": ["indexElapsedMs", "setupElapsedMs"], + "definition": "Wall time for the lane query step; index and setup are reported separately and amortized per repo where applicable." + } + }, + "fairnessRules": [ + "Task queries and expected patterns are frozen in tests/fixtures/niche-benchmark-manifest.json before benchmark code ships.", + "All lanes receive the same natural-language query string per task.", + "Do not adjust expected patterns after observing lane outputs.", + "Do not add task-specific ranking heuristics to codebase-context for this benchmark.", + "Report setup_failed lanes honestly; do not treat missing competitors as wins.", + "Credit prior art by display name and project URL in public docs." + ], + "blockedClaims": [ + "codebase_context_beats_all_competitors", + "universal_token_savings", + "patch_or_productivity_improvement" + ] +} diff --git a/tests/niche-benchmark.test.ts b/tests/niche-benchmark.test.ts new file mode 100644 index 0000000..0d92dcf --- /dev/null +++ b/tests/niche-benchmark.test.ts @@ -0,0 +1,86 @@ +import { readFileSync } from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { describe, expect, it } from 'vitest'; +import { aggregateLaneResults, scoreTaskResult } from '../scripts/lib/niche-benchmark/scoring.mjs'; + +const __dirname = path.dirname(fileURLToPath(import.meta.url)); +const manifest = JSON.parse( + readFileSync(path.join(__dirname, 'fixtures', 'niche-benchmark-manifest.json'), 'utf8') +); +const protocol = JSON.parse( + readFileSync(path.join(__dirname, 'fixtures', 'niche-benchmark-protocol.json'), 'utf8') +); + +describe('niche benchmark fixtures', () => { + it('freezes 18 tasks across three repos and three niches', () => { + expect(manifest.tasks).toHaveLength(18); + expect(manifest.repos).toHaveLength(3); + const languages = new Set(manifest.repos.map((repo) => repo.language)); + expect(languages.size).toBeGreaterThanOrEqual(2); + for (const niche of protocol.niches) { + expect(manifest.tasks.filter((task) => task.niche === niche)).toHaveLength(6); + } + }); + + it('lists credited prior-art comparators including ripgrep baseline', () => { + const ids = protocol.comparators.map((comparator) => comparator.id); + expect(ids).toEqual([ + 'codebase-context', + 'jCodeMunch', + 'aider-repo-map', + 'repomix', + 'ripgrep' + ]); + for (const comparator of protocol.comparators) { + expect(comparator.projectUrl).toMatch(/^https:\/\//); + expect(comparator.credit.length).toBeGreaterThan(10); + } + expect(protocol.claimAllowed).toBe(false); + }); +}); + +describe('niche benchmark scoring', () => { + it('marks top-3 pattern hits without tuning scores upward', () => { + const task = manifest.tasks.find((entry) => entry.id === 'as-sym-01'); + expect(task).toBeDefined(); + const hit = scoreTaskResult(task, { + payload: 'player-api next track', + topFiles: ['libs/shared/player-api.ts', 'other.ts'] + }); + expect(hit.top3Hit).toBe(true); + const miss = scoreTaskResult(task, { + payload: 'nothing useful', + topFiles: ['README.md'] + }); + expect(miss.top3Hit).toBe(false); + }); + + it('aggregates lane metrics with failures visible', () => { + const aggregated = aggregateLaneResults([ + { + taskId: 't1', + niche: 'symbol-retrieval', + status: 'completed', + top3Hit: true, + usefulnessScore: 1, + estimatedTokens: 100, + queryElapsedMs: 10, + indexElapsedMs: 50 + }, + { + taskId: 't2', + niche: 'edit-preflight', + status: 'tool_error', + top3Hit: false, + usefulnessScore: 0, + estimatedTokens: 0, + queryElapsedMs: 0, + indexElapsedMs: 0 + } + ]); + expect(aggregated.tasksCompleted).toBe(1); + expect(aggregated.tasksFailed).toBe(1); + expect(aggregated.top3HitRate).toBe(1); + }); +}); From 453d2712ec29a3af9d09947fac1785bc0ce30b31 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 30 Oct 2026 22:40:54 +0000 Subject: [PATCH 14/15] Ignore Aider tag cache created by repo-map lane --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 837b700..af67add 100644 --- a/.gitignore +++ b/.gitignore @@ -35,6 +35,7 @@ repos/ # Local benchmark/lab outputs benchmark-runs/ benchmark-repos/ +.aider.tags.cache.v4/ /outputs/ /results/*/raw/ /results/*/local/ From ee291d9ac18be99a98f0360be0f71cafe28487e6 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 30 Oct 2026 22:48:56 +0000 Subject: [PATCH 15/15] docs: integration preview log for 2.3.0 relaunch (Agent: cursor-cloud/cbc-integration) --- INTEGRATION-PREVIEW.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 INTEGRATION-PREVIEW.md diff --git a/INTEGRATION-PREVIEW.md b/INTEGRATION-PREVIEW.md new file mode 100644 index 0000000..33334b5 --- /dev/null +++ b/INTEGRATION-PREVIEW.md @@ -0,0 +1,25 @@ +Agent: cursor-cloud/cbc-integration + +| PR | Merge result | Tests after | Conflicts and fixes | +| --- | --- | --- | --- | +| [#154](https://github.com/PatrickSys/codebase-context/pull/154) | fast-forward | pass (131 files, build, `pnpm pack`) | none | +| [#145](https://github.com/PatrickSys/codebase-context/pull/145) | merged | pass | auto-merge `docs/capabilities.md` only | +| [#142](https://github.com/PatrickSys/codebase-context/pull/142) | merged (PR closed, branch merged) | pass | none | +| [#146](https://github.com/PatrickSys/codebase-context/pull/146) | merged | pass | `capabilities.md` (OpenAI + Ollama bullets); `types.ts` (`resolveDefaultApiEndpoint`, dedupe); embedding env + OpenAI tests | +| [#147](https://github.com/PatrickSys/codebase-context/pull/147) | merged | pass | none | +| [#153](https://github.com/PatrickSys/codebase-context/pull/153) | merged | pass | none | +| [#155](https://github.com/PatrickSys/codebase-context/pull/155) | merged | pass | none | +| [#156](https://github.com/PatrickSys/codebase-context/pull/156) | merged | pass | `docs/benchmark.md`: kept front-end matrix (#155) + niche benchmark (#156) | +| [#148](https://github.com/PatrickSys/codebase-context/pull/148) | merged | pass | README/cli/capabilities release copy; **README/CHANGELOG aligned to `docs/benchmark.md` after #156** (see below) | + +**Integration checks (final):** `pnpm test` (1348 passed), `pnpm run build`, `pnpm pack --dry-run`, MCP stdio `search_codebase` on `/tmp/vite-react-smoke` (Vite React template). + +**#148 follow-up (not applied on #148 branch):** add to README Evidence / Limits and CHANGELOG 2.3.0 Docs bullet: + +- README (after 25.7% retrieval sentence): *The reproducible niche prior-art slice (18 tasks, `npm run benchmark:niche`) scored **50.0%** top-3 file hit rate overall but **0/6** top-3 file hits on team-convention lookup (pattern signals matched; ranked paths did not).* +- CHANGELOG Docs bullet append: *…and the separate niche prior-art run (**50.0%** top-3 overall, **0/6** team-convention file hits) per docs/benchmark.md…* +- README Evidence paragraph: note niche table is a **separate frozen family**, not pooled with 25.7%. + +**Recommended merge order to master:** #154 → #145 → #142 → #146 → #147 → #153 → #155 → #156 → #148 (with #148 doc patch above). + +**Do not merge this PR.** Integration preview only.