diff --git a/.gitignore b/.gitignore index cee0f89..af67add 100644 --- a/.gitignore +++ b/.gitignore @@ -34,6 +34,8 @@ repos/ # Local benchmark/lab outputs benchmark-runs/ +benchmark-repos/ +.aider.tags.cache.v4/ /outputs/ /results/*/raw/ /results/*/local/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 33f0294..fc0a865 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,24 +1,15 @@ # Changelog -## Product candidate (not published) - -- New projects index and search without embedding or reranker models. Existing index modes are preserved; explicit mode switches rebuild fully. Transformers and LanceDB are optional peers with actionable preflight errors. -- Setup detects supported coding tools and configures the selected tools together. It preserves existing settings, saves version-pinned launches for the selected repository, and leaves indexing to the first search. -- Added `get_code_relationships`: directed import/re-export paths up to three hops, current resolution and source checks, confined endpoints, unresolved/stale diagnostics, and a 4,000-byte response cap. Resolution selects the root's `tsconfig.json` or, when absent, `tsconfig.base.json`, without inheriting an unrelated parent configuration. -- AST chunks preserve contiguous source and static property ownership, including concise descriptor methods, callback bodies and same-line siblings. Generated scope labels remain presentation metadata. Index format 4 rebuilds incompatible indexes even when sources are unchanged, preserves the selected embedding mode, and retains previous artifacts if rebuilding fails. -- Angular TypeScript eligibility uses parsed Angular imports; ordinary `resolve` or `validate` methods, comments, strings and imports from other packages no longer select the Angular analyzer. Other eligible analyzers retain the source and declaration context. The registry order and existing template/style routing are unchanged. -- Compact `signaturePreview` uses a contiguous window of up to three lines around the strongest query-term match in the returned snippet, falling back to its opening lines. Default and explicit exploration searches omit edit preflight while retaining search quality and incomplete-source warnings. Explicit `edit`, `refactor`, and `migrate` intents retain mutation guards. -- Search excludes ordinary stale/unavailable nominees without discarding verified current lower results or triggering an automatic full rebuild. Partial coverage, bounded exhaustion and failed semantic replenishment are explicit; corrupt metadata, build identity, paths and source ranges still fail closed. -- Symbol references scan same-build manifest sources in bounded read batches and skip parsing when a simple ASCII symbol occurs only inside longer identifiers. Unicode or arbitrary query spellings retain raw-text nomination. Coverage keeps raw-text absence separate from parser work and uncertain lexical fallback. Keyword artifact identity uses the shared index validator. Graph traversal verifies edges before consuming output caps, with bounded inspection and honest continuation. -- Watcher shutdown awaits handle closure and cancels pending callbacks. Connected stdio sessions have no default inactivity shutdown; explicit positive idle timeouts remain supported alongside handshake, EOF and parent-death guards. -- Chunk paths are persisted relative to the selected project root, including explicit-root indexing from another working directory. Previously incorrect relative paths can require a full reindex. -- Query advice is limited to grounded returned source, and abstained searches omit global pattern advice. Same-named symbols in different files remain eligible after file-level deduplication; TypeScript type arguments alone no longer select the JSX analyzer. -- The candidate requires Node 22+. CLI dispatch compares canonical entrypoint identity, including npm symlink launchers, while preserving the exported server's initialization on imports. -- Updated the runtime parser and watcher to remove the legacy braces dependency chain. Watcher exclusions use exact directory names; quoted import and export names retain their public symbol names. -- Known limits: model-free conceptual relevance is not guaranteed. Short requests can miss implementations when query words differ from source names; syntactic edges are not runtime calls; workspace/language-specific resolution can remain unresolved. Packed/protocol tests and scoped negative cases do not establish general agent-quality improvement or customer adoption. No npm publication is implied by this entry. - -## [2.3.0](https://github.com/PatrickSys/codebase-context/compare/v2.2.0...v2.3.0) (2026-04-30) +## [2.3.0](https://github.com/PatrickSys/codebase-context/compare/v2.2.0...v2.3.0) (unreleased — TBD) +### Release notes + +- **CLI:** Fix npm **2.2.0** regression where `npx codebase-context --version`, `status`, and `reindex` could exit with code 0 and no output (direct CLI subcommands lazy-load the MCP runtime; canonical entrypoint detection includes npm symlink launchers). Requires Node 22+. +- **Indexing:** Fresh project folders default to keyword-only indexing without downloading embedding or reranker models; existing index modes are preserved; opt in with `reindex --embeddings` and optional Transformers/LanceDB peers. Explicit mode switches rebuild fully. +- **Setup:** `init` wizard detects Claude Code, Codex, Cursor, VS Code (Copilot), Windsurf, and OpenCode, configures selected tools together, preserves existing settings, and leaves indexing to the first search. +- **Search & index pipeline:** AST-aligned chunks, index format metadata with atomic rebuilds, exploration vs edit/refactor/migrate preflight behavior, bounded symbol references, and source-validation hardening (see Features below). +- **Docs:** Position around local team conventions and edit preflight; cite Aider repo map, Repomix, and jCodeMunch as related tools; report **25.7%** vs **jCodeMunch 27.1%** expected-file recovery and the separate niche prior-art run (**50.0%** top-3 overall, **0/6** team-convention file hits) per [docs/benchmark.md](./docs/benchmark.md) without leadership claims; keep **2.4.0** features out of shipped language until published. +- **Thanks:** Nick Roth ([@rothnic](https://github.com/rothnic)) for the Intel Mac embedding crash analysis ([#68](https://github.com/PatrickSys/codebase-context/issues/68)) and custom OpenAI-compatible endpoint requirements ([#70](https://github.com/PatrickSys/codebase-context/issues/70)) that shaped workaround guidance in this release. ### Features diff --git a/INTEGRATION-PREVIEW.md b/INTEGRATION-PREVIEW.md new file mode 100644 index 0000000..33334b5 --- /dev/null +++ b/INTEGRATION-PREVIEW.md @@ -0,0 +1,25 @@ +Agent: cursor-cloud/cbc-integration + +| PR | Merge result | Tests after | Conflicts and fixes | +| --- | --- | --- | --- | +| [#154](https://github.com/PatrickSys/codebase-context/pull/154) | fast-forward | pass (131 files, build, `pnpm pack`) | none | +| [#145](https://github.com/PatrickSys/codebase-context/pull/145) | merged | pass | auto-merge `docs/capabilities.md` only | +| [#142](https://github.com/PatrickSys/codebase-context/pull/142) | merged (PR closed, branch merged) | pass | none | +| [#146](https://github.com/PatrickSys/codebase-context/pull/146) | merged | pass | `capabilities.md` (OpenAI + Ollama bullets); `types.ts` (`resolveDefaultApiEndpoint`, dedupe); embedding env + OpenAI tests | +| [#147](https://github.com/PatrickSys/codebase-context/pull/147) | merged | pass | none | +| [#153](https://github.com/PatrickSys/codebase-context/pull/153) | merged | pass | none | +| [#155](https://github.com/PatrickSys/codebase-context/pull/155) | merged | pass | none | +| [#156](https://github.com/PatrickSys/codebase-context/pull/156) | merged | pass | `docs/benchmark.md`: kept front-end matrix (#155) + niche benchmark (#156) | +| [#148](https://github.com/PatrickSys/codebase-context/pull/148) | merged | pass | README/cli/capabilities release copy; **README/CHANGELOG aligned to `docs/benchmark.md` after #156** (see below) | + +**Integration checks (final):** `pnpm test` (1348 passed), `pnpm run build`, `pnpm pack --dry-run`, MCP stdio `search_codebase` on `/tmp/vite-react-smoke` (Vite React template). + +**#148 follow-up (not applied on #148 branch):** add to README Evidence / Limits and CHANGELOG 2.3.0 Docs bullet: + +- README (after 25.7% retrieval sentence): *The reproducible niche prior-art slice (18 tasks, `npm run benchmark:niche`) scored **50.0%** top-3 file hit rate overall but **0/6** top-3 file hits on team-convention lookup (pattern signals matched; ranked paths did not).* +- CHANGELOG Docs bullet append: *…and the separate niche prior-art run (**50.0%** top-3 overall, **0/6** team-convention file hits) per docs/benchmark.md…* +- README Evidence paragraph: note niche table is a **separate frozen family**, not pooled with 25.7%. + +**Recommended merge order to master:** #154 → #145 → #142 → #146 → #147 → #153 → #155 → #156 → #148 (with #148 doc patch above). + +**Do not merge this PR.** Integration preview only. diff --git a/MOTIVATION.md b/MOTIVATION.md index c83fcd8..916498b 100644 --- a/MOTIVATION.md +++ b/MOTIVATION.md @@ -24,6 +24,9 @@ The root cause isn't the model. It's the context layer. Agents get raw code from | ---------------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------- | | `.cursorrules` / `AGENTS.md` | Static rules the agent may or may not always follow | Can't express migration states ("use A for new code, B for old"). Goes stale. | | Semantic search / RAG | Finds similar code | Blind to quality. Can't tell the best example to follow from hacky workarounds. | +| **Aider repo map** | Compact structural map for the model | No team pattern frequencies, memory, or edit preflight card. | +| **Repomix** | Pack repo (or slices) into a prompt | One-shot context, not live search, conventions, or session memory. | +| **jCodeMunch** | Fast MCP code search | Strong retrieval in our fixed benchmark; less convention/memory/edit gating. | | Linters / formatters | Enforce static rules after writing | Don't guide before or during generation. Agent still writes wrong code first. | | Tool-heavy MCPs (50+ tools) | Expose everything | Agents don't know which tool to call. More tools = more confusion. | diff --git a/README.md b/README.md index d93d577..4fd0e01 100644 --- a/README.md +++ b/README.md @@ -2,11 +2,13 @@ [![npm version](https://img.shields.io/npm/v/codebase-context)](https://www.npmjs.com/package/codebase-context) [![license](https://img.shields.io/npm/l/codebase-context)](./LICENSE) [![node](https://img.shields.io/node/v/codebase-context)](https://github.com/PatrickSys/codebase-context/blob/master/package.json) -Codebase Context gives coding agents a map of your repository through MCP, for developers who want their agent to find relevant code and follow the team's patterns. Search, examples and project memory run locally by default. +Codebase Context is a local MCP server for **team conventions and edit preflight**: a bounded map of how your repo is organized, ranked search with pattern and memory context, and a decision card before risky edits. It is not a retrieval leaderboard; frozen benchmarks put it at **25.7%** expected-file recovery versus **jCodeMunch 27.1%** in the same adapter run ([details](https://github.com/PatrickSys/codebase-context/blob/master/docs/benchmark.md)). Search, examples, and project memory stay on your machine by default. + +**Colleagues, not competitors:** [Aider's repo map](https://aider.chat/docs/repomap.html) for structure, [Repomix](https://repomix.com/) for pack-and-prompt workflows, and jCodeMunch for fast MCP code search (see [benchmark comparators](https://github.com/PatrickSys/codebase-context/blob/master/docs/benchmark.md)). Codebase Context combines map-first routing, convention detection, scoped memory, and edit-intent gating in one local tool surface. ## Set up your AI client -Use Node.js 22 or newer. These commands use the current npm release of Codebase Context: +Use Node.js 22 or newer. Unpinned `npx` installs the **current npm release (2.2.0 today)**; **2.3.0** is the pending release from this repository (Patrick publishes manually). Pin `@2.3.0` after it lands. npm **2.2.0** has a regression where `npx codebase-context --version`, `status`, and `reindex` can exit silently; **2.3.0** fixes that. ```bash # Claude Code @@ -19,27 +21,40 @@ codex mcp add codebase-context -- npx -y codebase-context opencode mcp add codebase-context '--' npx -y codebase-context ``` -In the unreleased candidate, `--version` prints the installed package version and `map` builds a missing index before printing the map. A low-confidence search includes a hint that weak matches do not establish that code is absent. These additions require the built candidate; the unpinned commands above install published 2.2.0. +**Ships in 2.3.0 (not on npm until published):** `--version`, non-silent `status` / `reindex`, `init` for six supported clients, fresh indexes without a model download (use `reindex --embeddings` when you want local semantic search), `map` that builds a missing index first, and low-confidence search hints that weak matches do not prove absence. Until 2.3.0 is on npm, use the manual MCP lines above or install from a built checkout / packed tarball. + +This repository's `init` command detects Claude Code, Codex, Cursor, VS Code (Copilot), Windsurf, and OpenCode, then writes or merges each tool's MCP config in one pass. Use `--dry-run` to preview changes before writing. + +```bash +npx codebase-context init --yes # configure detected tools +npx codebase-context init --dry-run --yes # show Would create/update lines only +``` + +The [client setup guide](./docs/client-setup.md) marks unpublished npm behavior and lists manual recipes for npm **2.2.0**. -This repository's `init` command detects Claude Code, Codex, Cursor, and OpenCode and writes a project config for each selected tool. The saved launch is the Node binary and the package file from the npx cache. That command is in this source and is not part of published 2.2.0. Until it is published, use the copy-paste lines above. The [client setup guide](./docs/client-setup.md) marks the unpublished command. Start a new agent session in your project, then ask: > Use Codebase Context to find [feature] in this repository. Pass this repository's absolute path as project, wait for indexing if needed, and open the relevant source files. -Replace `[feature]` with something you want to find. The first run downloads a local model and indexes the repository; time depends on repository size, CPU and model. +Replace `[feature]` with something you want to find. The first index pass scans the repository locally; **2.3.0** uses keyword search by default (no embedding model download). npm **2.2.0** still downloads a local embedding model on first index unless you change providers. Duration depends on repository size and CPU; optional semantic search adds model download time when enabled. + +### Planned for 2.4.0 (not in 2.3.0) + +Do not treat these as shipped until 2.4.0 is published: -### Planned for 2.4.0 (not yet published) +The unreleased source candidate starts new indexes without a model download. Semantic embeddings are optional; existing indexes keep their recorded mode. -The unreleased source candidate starts new indexes without a model download. Semantic embeddings are optional; existing indexes keep their recorded mode. Its setup wizard can configure Claude Code, Codex, Cursor and OpenCode using the Node binary and package file that ran setup. These changes are not in the current npm release. -- NestJS support from community contributor Aaron Olin. See the [analyzer details](./docs/capabilities.md#analyzers). -- Better React/Next.js detection and a fix for reranker passage scoring. -- Transformers.js v4 for local models. +- NestJS analyzer from community contributor Aaron Olin ([analyzer details](./docs/capabilities.md#analyzers)). +- React/Next.js detection improvements and reranker passage-scoring fix. +- Transformers.js v4 for optional local embeddings. +- Per-project `parsing.maxChunks` in config and packaged `codebase-context-review` polish called out in [review-context](./docs/review-context.md). -Known limits: [Intel Mac indexing crash (#68)](https://github.com/PatrickSys/codebase-context/issues/68), [custom OpenAI endpoints (#70)](https://github.com/PatrickSys/codebase-context/issues/70) and [Ollama (#71)](https://github.com/PatrickSys/codebase-context/issues/71) remain open. +Known limits: Intel Mac Transformers.js crashes ([#68](https://github.com/PatrickSys/codebase-context/issues/68)) are mitigated in this branch via pinned ONNX runtime and Ollama/OpenAI embedding paths; custom OpenAI endpoints use `OPENAI_BASE_URL` ([#70](https://github.com/PatrickSys/codebase-context/issues/70)). + +The existing [benchmark](https://github.com/PatrickSys/codebase-context/blob/master/docs/benchmark.md) reports 25.7% expected-file recovery versus jCodeMunch's 27.1% in a fixed retrieval comparison; it does not establish a winner or coding quality. The reproducible **niche prior-art** slice (18 tasks, `npm run benchmark:niche`) scored **50.0%** top-3 file hit rate overall but **0/6** top-3 file hits on team-convention lookup (pattern signals matched; ranked paths did not); see [Where Codebase Context loses](https://github.com/PatrickSys/codebase-context/blob/master/docs/benchmark.md#where-codebase-context-loses-same-run). -The existing [benchmark](https://github.com/PatrickSys/codebase-context/blob/master/docs/benchmark.md) reports 25.7% expected-file recovery versus jCodeMunch's 27.1% in a fixed retrieval comparison; it does not establish a winner or coding quality. For Codex Desktop, create or merge `.codex/config.toml` in the project you want to search: @@ -57,10 +72,10 @@ Other clients use their own setup commands: | Client | Shortest current setup | | --------------------------- | ---------------------------------------------------------------------------- | | Gemini CLI | `gemini mcp add --scope user codebase-context npx -y codebase-context` | -| Cursor | Add `.cursor/mcp.json` | -| VS Code with GitHub Copilot | Add `.vscode/mcp.json` | +| Cursor | `codebase-context init --client cursor` or add `.cursor/mcp.json` | +| VS Code with GitHub Copilot | `codebase-context init --client vscode` or add `.vscode/mcp.json` | | GitHub Copilot CLI | `copilot mcp add codebase-context -- npx -y codebase-context` | -| Windsurf | Add `~/.codeium/windsurf/mcp_config.json` | +| Windsurf | `codebase-context init --client windsurf` or add `~/.codeium/windsurf/mcp_config.json` | Check an existing same-name entry before replacing it. To give the server a default folder, append that folder's absolute path to the `npx` arguments. The [client setup guide](./docs/client-setup.md) covers scopes, optional fixed-folder configuration and the interactive setup wizard's limitations. @@ -76,7 +91,7 @@ The default connection is `stdio` (standard input/output): your client starts th ### Team patterns and examples -`get_team_patterns` shows the approaches used in the repository and points to representative files. Dedicated analyzers for Angular, React, and Next.js ship in published 2.2.0; NestJS support (along with React and Next.js detection improvements and a reranker fix) is planned for unpublished 2.4.0. +`get_team_patterns` shows the approaches used in the repository and points to representative files. Angular, React, and Next.js analyzers ship today on npm 2.2.0 and remain in 2.3.0. NestJS and the 2.4.0 analyzer/reranker work above are not published yet. ### Project memory @@ -88,11 +103,11 @@ The `codebase-context-review` CLI creates a bounded context packet from a commit ## How it works -1. **Index locally.** Codebase Context scans the project, builds a keyword index, and creates local semantic embeddings - numeric representations used to match code by meaning as well as exact words. +1. **Index locally.** Codebase Context scans the project and builds a keyword index; opt into local semantic embeddings with `reindex --embeddings` when you need meaning-based search (npm **2.2.0** still pulled embeddings on first index). 2. **Understand the repository.** The agent can request a compact codebase map with structure, patterns, and representative files. 3. **Find the code for the task.** Search returns ranked files and symbols; the agent reads the selected files before editing. -The same information is available from the terminal. Run these commands from your project root. The first run downloads a local model and indexes the repository; time depends on repository size, CPU and model. Once it is ready, inspect the map and search for the code you need: +The same information is available from the terminal. Run these commands from your project root. **2.3.0** can index without a model download; semantic search is optional. Once the index is ready, inspect the map and search: ```bash # Build or refresh the local index @@ -132,7 +147,15 @@ More examples are available in the [CLI gallery](./docs/cli.md) and [walkthrough Code and indexes stay on the machine with the default local embedding provider. Docker, a GPU, and an API key are not required. -Cloud embeddings are optional. If you select a cloud provider, code chunks are sent to that provider to create the search index. The provider, model, project root, and local HTTP port can be changed through environment variables or the project config; see the [capabilities reference](./docs/capabilities.md). +For local embeddings through [Ollama](https://ollama.com) (avoids Transformers.js on affected Macs, see #68), enable semantic indexing and set: + +```bash +EMBEDDING_PROVIDER=ollama +EMBEDDING_MODEL=nomic-embed-text # optional; default when unset +OLLAMA_HOST=http://localhost:11434 # optional +``` + +Cloud embeddings are optional. If you select OpenAI (`EMBEDDING_PROVIDER=openai`), code chunks are sent to that provider to create the search index. Override the API base with `OPENAI_BASE_URL` (compatible with Ollama’s OpenAI layer, but native `ollama` provider is preferred). The provider, model, project root, and local HTTP port can be changed through environment variables or the project config; see the [capabilities reference](./docs/capabilities.md). This is the privacy boundary of Codebase Context itself. Your AI client may send search results or file contents to the model provider configured in that client. Codebase Context does not control that connection. If you commit and push `.codebase-context/memory.json`, the recorded project memory also travels with the repository. @@ -151,7 +174,8 @@ Its ranked code search combines **Match words**, **Search by meaning**, and **Ra The [benchmark](https://github.com/PatrickSys/codebase-context/blob/master/docs/benchmark.md) reports a corrected 100-attempt retrieval family across five local code-context tools: 99 completed attempts and 1 failed attempt. Codebase Context recovered 25.7% of expected gold files with 11.5% file precision in that fixed adapter run; jCodeMunch recovered 27.1% with 11.0% file precision. Raw-native is a deterministic lexical adapter here, not a full normal coding-agent baseline. These are retrieval observations, not a winner or a coding-quality claim. The [sanitized evidence extract](https://github.com/PatrickSys/codebase-context/blob/master/results/benchmark-presentation-evidence.json) records the exact values and source hashes. -The same report retains a separate 300-attempt repeatability history, a 30-run two-task full-agent pilot, and a metered replay whose tool-use validity remains unresolved. A historical Codebase Context indexing observation with embeddings enabled took about 12 minutes 31 seconds, but it does not establish warm-use speed, query latency, install cost, or a cross-tool index-speed ranking. The report does not combine these families into a pooled score and does not establish patch correctness or end-to-end coding quality. Earlier public reports are preserved separately in the [benchmark archive](https://github.com/PatrickSys/codebase-context/blob/master/docs/benchmark-prior-public-report.md). +The same report retains a separate 300-attempt repeatability history, a 30-run two-task full-agent pilot, and a metered replay whose tool-use validity remains unresolved. The niche prior-art table (50.0% top-3 overall, **0/6** team-convention file hits, higher token cost than Aider/jCodeMunch on the same queries) is a separate frozen family — not pooled with the 25.7% retrieval line. A historical Codebase Context indexing observation with embeddings enabled took about 12 minutes 31 seconds, but it does not establish warm-use speed, query latency, install cost, or a cross-tool index-speed ranking. The report does not combine these families into a pooled score and does not establish patch correctness or end-to-end coding quality. Earlier public reports are preserved separately in the [benchmark archive](https://github.com/PatrickSys/codebase-context/blob/master/docs/benchmark-prior-public-report.md). + The method and failures are documented so the measurements can be inspected with their limits. @@ -162,7 +186,8 @@ The method and failures are documented so the measurements can be inspected with - The paired token observation covers two frozen investigation tasks and records observed agent behavior; it is not a universal token or time guarantee. - Setup checks differ by client. The detailed guide distinguishes a written config, a config recognized by the client, a local connection, and instructions checked only against official docs. - Dedicated Angular, React, and Next.js analyzers ship in published 2.2.0; NestJS support is planned for unpublished 2.4.0. Other projects use the generic analyzer and the language parsers available for that stack. -- The default searchable-chunk limit is 5,000 per project. Customizing this via `projects[].parsing.maxChunks` in `~/.codebase-context/config.json` requires 2.4.0 (not yet published). +- The default searchable-chunk limit is 5,000 per project. Raise it with `projects[].parsing.maxChunks` in `~/.codebase-context/config.json` (see [capabilities](docs/capabilities.md)). + - The agent must identify its repository in tool calls when no default or unambiguous client root is available. Concurrent HTTP client isolation is not established by the stdio routing checks. ## Reference diff --git a/docs/benchmark.md b/docs/benchmark.md index 73d073b..d018ad4 100644 --- a/docs/benchmark.md +++ b/docs/benchmark.md @@ -1,5 +1,84 @@ # What our local code-context experiments show +## Front-end repo matrix (plan) + +### Now + +- Ship `.vue` / `.svelte` indexing, Nuxt/SvelteKit framework claims, and tighter Next.js route detection (SvelteKit `+` routes, Nuxt `app/pages/*.vue`). +- Re-run the five-repo smoke matrix after each fix; keep this section honest. + +### Next + +- Dedicated Vue/Nuxt and Svelte analyzers (patterns, route conventions) instead of generic-only SPA handling. +- Path-alias coverage for Vite/Nuxt `~/` and monorepo package exports beyond TypeScript `paths`. +- Markdown-heavy repos (docs sites) as a separate benchmark series. + +### Done + +- 2026-10-30: Cloned and exercised five OSS front-end repos (see table below); fixed SPA file gaps and Next.js false positives; added fixture tests in `tests/frontend-spa-indexing.test.ts`. + +### Five-repo smoke (2026-10-30, `reindex --no-embeddings`, Node 22, this branch) + +Clones: [shadcn-ui/taxonomy](https://github.com/shadcn-ui/taxonomy) (Next.js), [nuxt/ui](https://github.com/nuxt/ui), [angular/components](https://github.com/angular/components), [sveltejs/kit](https://github.com/sveltejs/kit), [vercel/turbo](https://github.com/vercel/turbo) (pnpm/turbo monorepo with UI apps). + +| Repo | Init / reindex / status | Framework metadata | SPA sources in index | Symbol refs | Edit preflight (realistic query) | +| --- | --- | --- | --- | --- | --- | +| taxonomy (Next.js) | pass / pass / ready | Next.js detected | 94 `.tsx` indexed | `cn` useful (241 hits) | not ready — low confidence; top hit was relevant layout | +| nuxt/ui | pass / pass / ready | Nuxt detected (was missing) | **816** `.vue` indexed (was **0**) | `UButton` finds `Button.vue` imports | not ready — low confidence; stale-source hint on some queries | +| angular/components | pass / pass / ready | Angular patterns | TS/Angular only | `MatDialog` static refs OK | not ready — low confidence; dialog hits sensible | +| sveltejs/kit | pass / pass / ready | SvelteKit detected (monorepo deps) | **1011** `.svelte` indexed (was **0**) | `load` very noisy (587 lexical) | not ready — no longer mislabels SvelteKit tests as Next.js pages | +| vercel/turbo | pass / pass / ready; chunk cap warning | monorepo packages listed | UI via Next apps in `examples/` | `Button` finds UI components | not ready — low confidence; 5k chunk cap drops most of repo | + +Known limits surfaced here (not fixed in this pass): default 5k chunk cap on very large repos, lexical `load`/`Button` ambiguity, edit preflight still conservative on broad natural-language queries, one Rust-generated TS parse warning in turbo. + +## Reproducible niche benchmark (prior art colleagues) + +This section reports a **frozen, rerunnable** comparison on **18 tasks** (team-convention lookup, edit-preflight navigation, symbol retrieval) across **three public repos**: [angular-spotify](https://github.com/trungk18/angular-spotify) (TypeScript), [Excalidraw](https://github.com/excalidraw/excalidraw) (TypeScript), and [FastAPI](https://github.com/fastapi/fastapi) (Python). Tasks and expected patterns live in `tests/fixtures/niche-benchmark-manifest.json`; comparator credits and rules live in `tests/fixtures/niche-benchmark-protocol.json`. Full machine output is checked in at `results/niche-benchmark-evidence.json` (generated **2026-10-30** on Node **v22.23.4**). + +**Primary accuracy metric:** top-3 file hit rate (expected path substring appears in the first three ranked paths returned for the same natural-language query). + +**Efficiency metrics:** estimated tokens returned (UTF-8 bytes ÷ 4) and query latency; index/setup time is reported separately in the evidence JSON. + +### One-command rerun + +```bash +pnpm install && pnpm run build +npm run benchmark:niche +# or: node scripts/run-niche-benchmark.mjs --validate-fixtures +``` + +Optional: `--lanes codebase-context,jCodeMunch,aider-repo-map,repomix,ripgrep` and `--output results/niche-benchmark-evidence.json`. + +**Prerequisites (not installed via npm publish):** `git`, `rg`, `python3` with `jcodemunch-mcp` and `aider-chat`, and network access to clone the three pinned repos into `benchmark-repos/`. + +### Aggregate results (18 tasks) + +| Colleague tool | Top-3 hit rate | Avg est. tokens | Avg query ms | Avg index ms | +| --- | ---: | ---: | ---: | ---: | +| **Codebase Context** | **50.0%** | 642 | 928 | 673 | +| [Aider](https://github.com/Aider-AI/aider) repo map | 22.2% | **95** | 1442 | 350 | +| [Repomix](https://github.com/yamadashy/repomix) (pack + section rank) | 11.1% | 5,559,174 | **114** | 286 | +| [jCodeMunch](https://github.com/jgravelle/jcodemunch-mcp) | 5.6% | **202** | 701 | **69** | +| [ripgrep](https://github.com/BurntSushi/ripgrep) (lexical baseline) | 5.6% | 5,001* | **76** | 0 | + +\*Ripgrep and Repomix token figures include large truncated raw payloads in this harness; treat them as upper-bound context cost, not model-ready bundles. + +### Where Codebase Context loses (same run) + +- **Team-convention file ranking:** 0/6 top-3 file hits — `patterns` answers matched expected **signals** (for example `dependencyInjection`, `stateManagement`) but did not return ranked repo paths, so file-based scoring stays at zero. **Aider repo map** reached **33%** top-3 on the same team-convention slice. +- **Payload size vs Aider / jCodeMunch:** median-style averages **642** estimated tokens vs **95** (Aider) and **202** (jCodeMunch) on the same queries. +- **Query latency vs ripgrep:** **928 ms** average query time vs **76 ms** for the ripgrep baseline (no index). +- **Edit-preflight slice:** **67%** top-3 here, but still below perfect; broad unsafe tasks are out of scope for this table (see edit-preflight fixtures for abstention tests). +- **Cross-repo variance:** symbol-retrieval top-3 was **83%** on the two TypeScript repos but **50%** on FastAPI in this run (see `byNiche` in the evidence JSON). + +### What this does not claim + +- No patch correctness, agent productivity, or npm-package performance guarantees. +- Repomix is not a search engine; the lane documents pack-generation cost plus section ranking. +- Earlier ContextBench / discovery families below remain separate evidence; they are not merged into this table. + +--- + This report describes local evidence from five ways of giving an AI coding agent context: Raw-native, Codebase Context, jCodeMunch, Repowise, and context-mode. The evidence is split into separate families because retrieval, repeated runs, full-agent observations, and a metered replay do not answer the same question. The families are not pooled into one score. The clearest current finding is narrow: in this fixed retrieval setup, Codebase Context recovered 25.7% of expected gold files at 11.5% file precision, while fresh indexing had material local cost. These measurements do not show better completed coding tasks. diff --git a/docs/capabilities.md b/docs/capabilities.md index 49fc4f9..e44d5ef 100644 --- a/docs/capabilities.md +++ b/docs/capabilities.md @@ -1,6 +1,6 @@ # Capabilities Reference -Unpinned `npx codebase-context` commands install the published 2.2.0 package. Sections marked as unreleased describe the source candidate and require a built checkout or a local candidate installation. +Unpinned `npx codebase-context` installs **npm 2.2.0 today**. **2.3.0** is the pending release from this repo; **2.4.0** is not published. Sections marked **2.4.0** or **planned** describe future work only. Start with the repository map, then search for relevant code. For an overview, see [README.md](../README.md). Features marked as planned for 2.4.0 are not yet available from npm. @@ -22,7 +22,7 @@ Per-project config overrides supported today: - `projects[].excludePatterns`: merged with the built-in exclusion set for that project at index time - `projects[].analyzerHints.analyzer`: prefers a registered analyzer by name for that project and falls back safely when the name is missing or invalid - `projects[].analyzerHints.extensions`: adds project-local source extensions for indexing and auto-refresh watching without changing defaults for other projects -- `projects[].parsing.maxChunks`: sets the maximum number of searchable chunks indexed for that project (requires 2.4.0, not yet published) +- `projects[].parsing.maxChunks`: sets the maximum number of searchable chunks indexed for that project (default 5000; configurable via `~/.codebase-context/config.json`) Copy-pasteable client config templates are shipped in the package: @@ -65,7 +65,7 @@ npx -y codebase-context reindex --incremental ## Tool Surface -10 MCP tools + active/project-scoped context resources. +10 MCP tools plus project-scoped context resources in published 2.2.0/2.3.0. **`get_code_relationships`** and related navigation improvements are documented below for the in-repo build; treat them as **2.4.0** until that release is on npm unless you install from source. Shared selector inputs: @@ -199,7 +199,7 @@ Ordered by execution: - **Chunk size**: 50 lines, 0 overlap - **Reranker trigger**: activates when top-3 results are within 0.08 score of each other - **Embedding mode**: new indexes are model-free. Existing indexes keep their recorded mode unless explicitly changed. `reindex --embeddings` opts into semantic indexing and requires the optional model/storage packages. -- **Semantic model when enabled**: `Xenova/bge-small-en-v1.5` (512 token context, fast, local-first) via `@huggingface/transformers`. Override: `EMBEDDING_MODEL=onnx-community/granite-embedding-small-english-r2-ONNX` for Granite (8192 ctx, slower). +- **Semantic model when enabled**: `Xenova/bge-small-en-v1.5` (512 token context, fast, local-first) via `@huggingface/transformers`. Override: `EMBEDDING_MODEL=onnx-community/granite-embedding-small-english-r2-ONNX` for Granite (8192 ctx, slower). **Ollama**: `EMBEDDING_PROVIDER=ollama`, optional `OLLAMA_HOST` (default `http://localhost:11434`), default model `nomic-embed-text` — embeddings stay on your machine via Ollama’s `/api/embed`. - **Vector DB**: LanceDB with cosine distance ## Decision Card (Edit Intent) @@ -317,6 +317,7 @@ Reproducible evaluation is shipped as a CLI entrypoint backed by shared scoring - **Symbol refs are not a call-graph.** `get_symbol_references` counts identifier-node occurrences in the AST (comments/strings excluded via Tree-sitter). It does not distinguish call sites from type annotations, variable assignments, or imports. Full call-site-specific analysis (`call_expression` nodes only) is a roadmap item. - **Impact is 2-hop max.** `computeImpactCandidates` walks direct importers then their importers. Full BFS reachability is on the roadmap. -- **Planned for unpublished 2.4.0:** NestJS analyzer support, React and Next.js detection improvements, and reranker fixes. Dedicated Angular, React, and Next.js analyzers already ship in published 2.2.0. Other languages use the Generic analyzer for indexing and retrieval; available symbol extraction depends on parser support. File import resolution and multi-hop navigation currently support TypeScript and JavaScript. Python and Go import navigation are not supported. -- **The unreleased candidate defaults new indexes to model-free mode. Published 2.2.0 enables embeddings by default.** When semantic mode is enabled, its default model is `bge-small-en-v1.5` (512-token context). Granite (8192 context) is opt-in via `EMBEDDING_MODEL`. OpenAI is opt-in via `EMBEDDING_PROVIDER=openai` — sends code externally. +- **Planned for 2.4.0 (not on npm):** NestJS analyzer, React/Next.js detection improvements, reranker fixes, optional `maxChunks` config, and full `get_code_relationships` / review-CLI publication called out in [review-context](./review-context.md). Angular, React, and Next.js analyzers ship on npm 2.2.0 and remain in 2.3.0. Other languages use the Generic analyzer for indexing and retrieval; available symbol extraction depends on parser support. File import resolution and multi-hop navigation currently support TypeScript and JavaScript. Python and Go import navigation are not supported. +- **Indexing defaults in 2.3.0:** new project folders index keyword-only until you run `reindex --embeddings` (optional `@huggingface/transformers` + `@lancedb/lancedb` peers). npm **2.2.0** still pulled embeddings on first index. When semantic mode is enabled, default model is `bge-small-en-v1.5` (512-token context). Granite (8192 context) is opt-in via `EMBEDDING_MODEL`. OpenAI is opt-in via `EMBEDDING_PROVIDER=openai` with `OPENAI_API_KEY`; use `OPENAI_BASE_URL` for OpenAI-compatible endpoints (LiteLLM, proxies). Ollama is opt-in via `EMBEDDING_PROVIDER=ollama` — local HTTP only, no API key (`OLLAMA_HOST` overrides the default host). + - **Patterns are file-level frequency counts.** Not semantic clustering. Rising/Declining trend is derived from git commit recency for files using each pattern, not from usage semantics. diff --git a/docs/cli.md b/docs/cli.md index 9ae65f6..d034cb5 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -36,7 +36,7 @@ CODEBASE_CONTEXT_ASCII=1 npx -y codebase-context patterns ## Commands -- `map` — conventions map: architecture layers, patterns, golden files +- `map` — conventions map: architecture layers, patterns, golden files. Builds a file index first when the project does not have one, so a fresh checkout does not report 0 files - `metadata` — tech stack overview - `patterns` — team conventions + adoption/trends - `search --query ` — ranked results; add `--intent edit` for a preflight card @@ -46,28 +46,35 @@ CODEBASE_CONTEXT_ASCII=1 npx -y codebase-context patterns - `reindex` — rebuild index (full or incremental) - `style-guide` — find style guide sections in docs - `memory list|add|remove` — manage team memory (stored in `.codebase-context/memory.json`) -- MCP client registration — use the recipes in the [client setup guide](./client-setup.md). Published 2.2.0 has the older interactive HTTP wizard; the four-client wizard below requires the unreleased candidate. +- MCP client registration — use the recipes in the [client setup guide](./client-setup.md). npm **2.2.0** ships the older HTTP-only wizard path; the six-client `init` flow ships in **2.3.0** (pending npm publish). -## `init` in the unreleased candidate -`init` detects Claude Code, Codex, Cursor and OpenCode, then configures the selected tools together. In a terminal, detected tools start selected; use Space to change the selection and Enter to apply it. Without a terminal, it configures all detected tools. +## `init` (2.3.0 — pending npm publish) + +`init` detects Claude Code, Codex, Cursor, VS Code (Copilot), Windsurf, and OpenCode, then configures the selected tools together. In a terminal, detected tools start selected; use Space to change the selection and Enter to apply it. Without a terminal, it configures all detected tools. Add `--dry-run` to print `Would create`, `Would update`, or `Unchanged` without writing files. Not available in npm **2.2.0**. + ```powershell codebase-context init --help codebase-context init codebase-context init --yes +codebase-context init --dry-run --yes codebase-context init --client codex --client cursor --root "C:\path\to\project" ``` -Use repeatable `--client` options to choose tools explicitly. `--yes` configures every detected tool when no clients are named. `--root` selects another repository; otherwise `CODEBASE_ROOT` or the current directory is used. Setup writes client registrations but does not build an index or change agent instructions. The first search creates the index. The multi-tool flow is not in published npm `2.2.0` yet; see the [main setup guide](../README.md#set-up-your-ai-client) for its publication status and the [client setup guide](./client-setup.md) for current manual recipes. +Codex one-liner (works with published npm): `codex mcp add codebase-context -- npx -y codebase-context` + +Use repeatable `--client` options to choose tools explicitly. `--yes` configures every detected tool when no clients are named. `--root` selects another repository; otherwise `CODEBASE_ROOT` or the current directory is used. Setup writes client registrations but does not build an index or change agent instructions. The first search creates the index. Until **2.3.0** is on npm, use the manual recipes in the [client setup guide](./client-setup.md) and the [main setup guide](../README.md#set-up-your-ai-client). -The built candidate also supports `--version` and builds a missing index when you run `map`. New indexes use keyword search without a model download; `reindex --embeddings` explicitly enables local semantic search. Preview, reindex, and map apply the same project exclusions and analyzer hints. A low-confidence search result means the matches are weak; it does not establish that code is absent. + +**Also in 2.3.0:** `--version`; non-silent `status` / `reindex` (fixes npm 2.2.0 silent exit); fresh indexes without a model download; `map` builds a missing index first; low-confidence search hints. Use `reindex --embeddings` for local semantic search. --- ## First use -Run these CLI commands from the project root. Build or refresh the local index first; the first run downloads a local model and indexes the repository; time depends on repository size, CPU and model. Then inspect the conventions map and search for the code you need: +Run these CLI commands from the project root. **2.3.0** indexes keyword-only on first run (no model download); add `reindex --embeddings` when you want local semantic search. npm **2.2.0** still downloaded a model on first index. `map` indexes the project files when no index exists yet. + ```bash npx -y codebase-context reindex @@ -84,7 +91,7 @@ npx -y codebase-context reindex npx -y codebase-context map ``` -The conventions map - run this first on an unfamiliar repo. It shows architecture layers, active patterns with adoption rates and trend direction, and the golden files the team treats as the strongest examples. This is also what the MCP server delivers to AI agents via the `codebase://context` resource on first call, before search narrows to a specific local example. +The conventions map - run this first on an unfamiliar repo. If the project has no index, `map` builds the file index before it prints, so the file count is not stuck at 0. It shows architecture layers, active patterns with adoption rates and trend direction, and the golden files the team treats as the strongest examples. This is also what the MCP server delivers to AI agents via the `codebase://context` resource on first call, before search narrows to a specific local example. That automatic index skips embeddings; run `reindex` when you want semantic search. Example output (truncated): @@ -163,6 +170,8 @@ npx -y codebase-context reindex npx -y codebase-context search --query "file watcher" --intent edit --limit 3 ``` +A low-confidence result includes a hint. The hint means the matches are weak. It is not a finding that the code is absent. + Example output (truncated): ```text diff --git a/docs/client-setup.md b/docs/client-setup.md index f43ab54..2a025d8 100644 --- a/docs/client-setup.md +++ b/docs/client-setup.md @@ -1,6 +1,6 @@ # Client Setup -These commands use the current npm release because the package name is unpinned. The current published version is 2.2.0; 2.4.0 is not yet published. +These commands use the current npm release (**2.2.0** on npm today; **2.3.0** pending). **2.4.0** is not published. Historical verification notes below reference **2.2.0** unless another version is named. Codebase Context runs as a local MCP server: a tool your AI editor or command-line client can call while it works. Register it without a fixed folder, then let the agent supply its current repository through the `project` tool parameter. @@ -27,7 +27,7 @@ After registration, start a new agent session in your project and ask: > Use Codebase Context to find [feature] in this repository. Pass this repository's absolute path as project when checking get_indexing_status and searching. Wait for indexing if needed, read codebase://context, then search_codebase and open a returned source file. Show me the relevant files. -The first run downloads a local model and indexes the repository; time depends on repository size, CPU and model. You can prepare it separately with `npx -y codebase-context reindex` from the project folder. Index readiness, client connection and useful agent behavior are separate checks. +The first index pass scans the repository locally. Pending **2.3.0** uses keyword search by default (no model download); npm **2.2.0** still downloads embedding models on first index. You can prepare the index separately with `npx -y codebase-context reindex` (add `--no-embeddings` on 2.3.0, or `--embeddings` when you want semantic search). Index readiness, client connection, and useful agent behavior are separate checks. ## Claude Code @@ -106,9 +106,12 @@ On October 6 the exact published `2.2.0` `npx` entrypoint was tested without a f The four-client wizard is not in published 2.2.0. From a built checkout or a locally installed candidate, run its actual entrypoint from the project you want to search: ```bash -# Detect Claude Code, Codex, Cursor and OpenCode, then configure the detected clients. +# Detect Claude Code, Codex, Cursor, VS Code, Windsurf, and OpenCode, then configure the detected clients. node /path/to/codebase-context/dist/index.js init --yes +# Preview MCP config changes without writing files. +node /path/to/codebase-context/dist/index.js init --dry-run --yes + # Configure one client for an explicit project. node /path/to/codebase-context/dist/index.js init --client codex --root /path/to/project --yes @@ -116,13 +119,15 @@ node /path/to/codebase-context/dist/index.js init --client codex --root /path/to node /path/to/codebase-context/dist/index.js init --root /path/to/project --preview ``` +Codex one-liner (published npm): `codex mcp add codebase-context -- npx -y codebase-context` + Replace the paths with the candidate installation and project folders. With an interactive terminal, omit `--yes` to select clients together. `--client` is repeatable. `--root` takes priority over `CODEBASE_ROOT` and the current folder. -Claude Code uses `.mcp.json`, Codex uses `.codex/config.toml`, Cursor uses `.cursor/mcp.json`, and OpenCode uses `opencode.json`. Generated configs save the running Node binary, the actual package entrypoint and the project folder. If setup ran through `npx`, that entrypoint is in its package cache. Rerun setup if that installation or cache is removed or moved. +Claude Code uses `.mcp.json`, Codex uses `.codex/config.toml`, Cursor uses `.cursor/mcp.json`, VS Code uses `.vscode/mcp.json` (`servers`), Windsurf uses `~/.codeium/windsurf/mcp_config.json`, and OpenCode uses `opencode.json`. Generated configs save the running Node binary, the actual package entrypoint and the project folder. If setup ran through `npx`, that entrypoint is in its package cache. Rerun setup if that installation or cache is removed or moved. Existing unrelated settings are preserved. A conflicting custom Codebase Context launch stops setup before selected config files are written. To replace a custom launch, remove only its Codebase Context entry from the client config, keep the other settings, and rerun setup. Restart the client and approve the server or project trust if asked. Config generation and SDK launch tests do not establish acceptance in each native client. -New candidate indexes default to model-free search and do not initialize the optional model packages. Existing indexes keep their recorded mode; semantic indexing remains an explicit option. +New candidate indexes default to model-free search and do not initialize the optional model packages. Existing indexes keep their recorded mode; semantic indexing remains an explicit option. `map` builds a missing index before printing the map. ## Optional fixed-folder setup and CLI preparation diff --git a/docs/comparison-table.md b/docs/comparison-table.md index 1b55f11..96b11ba 100644 --- a/docs/comparison-table.md +++ b/docs/comparison-table.md @@ -1,6 +1,6 @@ # Corrected retrieval comparison -This table summarizes the corrected retrieval-only family across five local code-context tools. In this fixed setup, Codebase Context recovered 25.7% of expected gold files at 11.5% file precision; this does not show better completed coding tasks. Expected-file coverage means expected gold files recovered divided by expected gold files. File precision means expected gold files recovered divided by all files returned. The reported `peakPrivateGb` field is copied from retained telemetry; its underlying unit was not independently verified. +This table summarizes the corrected retrieval-only family across five local code-context tools. **It is not a product ranking:** Codebase Context optimizes for team conventions and edit preflight; jCodeMunch leads this fixed expected-file metric at 27.1% versus Codebase Context at 25.7%. Expected-file coverage means expected gold files recovered divided by expected gold files. File precision means expected gold files recovered divided by all files returned. The reported `peakPrivateGb` field is copied from retained telemetry; its underlying unit was not independently verified. | Tool | Completed | Failed | Expected-file coverage | File precision | Reported `peakPrivateGb` | | --- | ---: | ---: | ---: | ---: | ---: | diff --git a/package.json b/package.json index a855889..407cb7d 100644 --- a/package.json +++ b/package.json @@ -132,6 +132,7 @@ "type-check": "tsc --noEmit", "mcp:inspect": "npx -y @modelcontextprotocol/inspector node dist/index.js .", "eval": "pnpm run build && node scripts/run-eval.mjs", + "benchmark:niche": "node scripts/run-niche-benchmark.mjs", "prepare": "husky" }, "dependencies": { @@ -181,6 +182,9 @@ "typescript-eslint": "^8.51.0", "vitest": "^4.0.18" }, + "overrides": { + "@huggingface/transformers>onnxruntime-node": "1.24.3" + }, "pnpm": { "onlyBuiltDependencies": [ "esbuild", diff --git a/results/niche-benchmark-evidence.json b/results/niche-benchmark-evidence.json new file mode 100644 index 0000000..a058157 --- /dev/null +++ b/results/niche-benchmark-evidence.json @@ -0,0 +1,1885 @@ +{ + "artifactType": "niche-prior-art-benchmark-evidence", + "generatedAt": "2026-10-30T22:39:55.772Z", + "protocol": { + "name": "niche-prior-art-benchmark", + "frozenDate": "2026-10-30", + "manifestFrozenDate": "2026-10-30" + }, + "runMeta": { + "manifestPath": "/workspace/tests/fixtures/niche-benchmark-manifest.json", + "protocolPath": "/workspace/tests/fixtures/niche-benchmark-protocol.json", + "manifestSha256": "9dd48fb64b7e81e6a16e694be031134f7a1506ff9a259b4cd084ab44827c704c", + "protocolSha256": "0b666b5ff5ed86f9a31cd0cc2857ef7dcd233ddf9c85b93091eb92488522aed0", + "nodeVersion": "v22.23.4", + "lanes": [ + "codebase-context", + "jCodeMunch", + "aider-repo-map", + "repomix", + "ripgrep" + ], + "repoCheckouts": { + "angular-spotify": { + "ref": "ff9efa765c53cfde78c9a172c62d515ae8ef9fe0", + "path": "/workspace/benchmark-repos/angular-spotify" + }, + "excalidraw": { + "ref": "e18c1dd213000dde0ae94ef7eb00aab537b39708", + "path": "/workspace/benchmark-repos/excalidraw" + }, + "fastapi": { + "ref": "628c34e0cae200564d191c95d7edea78c88c4b5e", + "path": "/workspace/benchmark-repos/fastapi" + } + } + }, + "comparators": [ + { + "id": "codebase-context", + "displayName": "Codebase Context", + "credit": "This repository (local dist CLI / MCP surfaces).", + "projectUrl": "https://github.com/PatrickSys/codebase-context", + "status": "completed", + "tasksTotal": 18, + "tasksCompleted": 18, + "tasksFailed": 0, + "top3HitRate": 0.5, + "averageUsefulness": 0.6666666666666666, + "averageEstimatedTokens": 641.7222222222222, + "averageQueryElapsedMs": 927.6111111111111, + "averageIndexElapsedMs": 672.7777777777779, + "byNiche": { + "team-convention": { + "tasks": 6, + "top3HitRate": 0 + }, + "edit-preflight": { + "tasks": 6, + "top3HitRate": 0.6666666666666666 + }, + "symbol-retrieval": { + "tasks": 6, + "top3HitRate": 0.8333333333333334 + } + }, + "taskResults": [ + { + "taskId": "as-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 1, + "matchedSignals": [ + "dependencyInjection" + ], + "payloadBytes": 1295, + "estimatedTokens": 324, + "queryElapsedMs": 518, + "indexElapsedMs": 512.5, + "topFiles": [] + }, + { + "taskId": "as-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 1, + "matchedSignals": [ + "stateManagement" + ], + "payloadBytes": 1220, + "estimatedTokens": 305, + "queryElapsedMs": 492, + "indexElapsedMs": 512.5, + "topFiles": [] + }, + { + "taskId": "as-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 4017, + "estimatedTokens": 1005, + "queryElapsedMs": 978, + "indexElapsedMs": 512.5, + "topFiles": [ + "libs/web/auth/data-access/src/lib/store/auth.store.ts", + "libs/web/auth/util/src/lib/interceptors/auth.interceptor.ts", + "libs/web/visualizer/data-access/src/lib/audio-context.ts", + "libs/web/visualizer/data-access/src/lib/particles-renderer.ts", + "libs/web/visualizer/data-access/src/lib/sound-lines-renderer.ts" + ] + }, + { + "taskId": "as-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": true, + "firstRelevantHit": 2, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 4114, + "estimatedTokens": 1029, + "queryElapsedMs": 719, + "indexElapsedMs": 512.5, + "topFiles": [ + "libs/web/auth/util/src/lib/interceptors/auth.interceptor.ts", + "libs/web/shell/ui/player-controls/src/lib/player-controls.component.ts", + "libs/web/shared/ui/media-summary/src/lib/media-summary.component.ts", + "libs/web/shell/ui/player-playback/src/lib/player-playback.component.ts", + "libs/web/lyrics/ui/lyrics-pip/src/lib/lyrics-pip.component.ts" + ] + }, + { + "taskId": "as-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 2420, + "estimatedTokens": 605, + "queryElapsedMs": 645, + "indexElapsedMs": 512.5, + "topFiles": [ + "libs/web/auth/util/src/lib/interceptors/auth.interceptor.ts", + "libs/web/auth/util/src/lib/interceptors/unauthorized.interceptor.ts", + "libs/web/future-responsive/src/lib/albums.mock.json", + "libs/web/container-queries/src/lib/albums.mock.json", + "tsconfig.base.json" + ] + }, + { + "taskId": "as-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 2834, + "estimatedTokens": 709, + "queryElapsedMs": 906, + "indexElapsedMs": 512.5, + "topFiles": [ + "libs/web/auth/util/src/lib/interceptors/auth.interceptor.ts", + "libs/web/auth/data-access/src/lib/store/auth.store.ts", + "libs/web/shared/data-access/store/src/lib/ui/ui-store.ts", + "libs/web/auth/util/src/lib/interceptors/unauthorized.interceptor.ts", + "libs/web/auth/ui/unauthorized-modal/src/lib/unauthorized-modal.component.html" + ] + }, + { + "taskId": "ex-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 1, + "matchedSignals": [ + "stateManagement" + ], + "payloadBytes": 1124, + "estimatedTokens": 281, + "queryElapsedMs": 500, + "indexElapsedMs": 1122, + "topFiles": [] + }, + { + "taskId": "ex-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 51, + "estimatedTokens": 13, + "queryElapsedMs": 491, + "indexElapsedMs": 1122, + "topFiles": [] + }, + { + "taskId": "ex-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 4389, + "estimatedTokens": 1098, + "queryElapsedMs": 1336, + "indexElapsedMs": 1122, + "topFiles": [ + "packages/excalidraw/history.ts", + "excalidraw-app/data/FileManager.ts", + "excalidraw-app/app_constants.ts", + "excalidraw-app/collab/Collab.tsx", + "excalidraw-app/App.tsx" + ] + }, + { + "taskId": "ex-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": true, + "firstRelevantHit": 3, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 4537, + "estimatedTokens": 1135, + "queryElapsedMs": 1261, + "indexElapsedMs": 1122, + "topFiles": [ + "packages/excalidraw/components/App.tsx", + "packages/element/src/types.ts", + "packages/excalidraw/data/blob.ts", + "packages/excalidraw/types.ts", + "excalidraw-app/data/firebase.ts" + ] + }, + { + "taskId": "ex-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 3763, + "estimatedTokens": 941, + "queryElapsedMs": 1224, + "indexElapsedMs": 1122, + "topFiles": [ + "packages/math/src/types.ts", + "packages/utils/src/withinBounds.ts", + "packages/excalidraw/clipboard.ts", + "excalidraw-app/data/index.ts", + "packages/excalidraw/tests/test-utils.ts" + ] + }, + { + "taskId": "ex-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": true, + "firstRelevantHit": 2, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 3568, + "estimatedTokens": 892, + "queryElapsedMs": 1488, + "indexElapsedMs": 1122, + "topFiles": [ + "packages/element/src/types.ts", + "packages/excalidraw/data/blob.ts", + "excalidraw-app/data/firebase.ts", + "packages/excalidraw/data/index.ts", + "packages/excalidraw/data/image.ts" + ] + }, + { + "taskId": "fa-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 51, + "estimatedTokens": 13, + "queryElapsedMs": 477, + "indexElapsedMs": 383.8333333333333, + "topFiles": [] + }, + { + "taskId": "fa-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 51, + "estimatedTokens": 13, + "queryElapsedMs": 452, + "indexElapsedMs": 383.8333333333333, + "topFiles": [] + }, + { + "taskId": "fa-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 3144, + "estimatedTokens": 786, + "queryElapsedMs": 913, + "indexElapsedMs": 383.8333333333333, + "topFiles": [ + "fastapi/security/oauth2.py", + "docs_src/security/tutorial005_py39.py", + "docs_src/security/tutorial005_py310.py", + "fastapi/security/http.py", + "fastapi/applications.py" + ] + }, + { + "taskId": "fa-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 3476, + "estimatedTokens": 869, + "queryElapsedMs": 1313, + "indexElapsedMs": 383.8333333333333, + "topFiles": [ + "tests/test_response_model_as_return_annotation.py", + "tests/test_tuples.py", + "tests/test_schema_extra_examples.py", + "tests/test_response_by_alias.py", + "tests/test_response_model_sub_types.py" + ] + }, + { + "taskId": "fa-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 2727, + "estimatedTokens": 682, + "queryElapsedMs": 1520, + "indexElapsedMs": 383.8333333333333, + "topFiles": [ + "fastapi/routing.py", + "fastapi/utils.py", + "tests/test_multi_body_errors.py", + "scripts/contributors.py", + "tests/test_router_events.py" + ] + }, + { + "taskId": "fa-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 3403, + "estimatedTokens": 851, + "queryElapsedMs": 1464, + "indexElapsedMs": 383.8333333333333, + "topFiles": [ + "tests/test_starlette_exception.py", + "tests/test_security_http_basic_optional.py", + "tests/test_multi_body_errors.py", + "tests/test_webhooks_security.py", + "tests/test_union_body.py" + ] + } + ] + }, + { + "id": "jCodeMunch", + "displayName": "jCodeMunch", + "credit": "jCodeMunch MCP symbol search and repo indexing (jcodemunch-mcp on PyPI).", + "projectUrl": "https://github.com/jgrahamc/jcodemunch", + "status": "completed", + "tasksTotal": 18, + "tasksCompleted": 18, + "tasksFailed": 0, + "top3HitRate": 0.05555555555555555, + "averageUsefulness": 0.1111111111111111, + "averageEstimatedTokens": 202.38888888888889, + "averageQueryElapsedMs": 701.4444444444445, + "averageIndexElapsedMs": 69.27777777777777, + "byNiche": { + "team-convention": { + "tasks": 6, + "top3HitRate": 0 + }, + "edit-preflight": { + "tasks": 6, + "top3HitRate": 0 + }, + "symbol-retrieval": { + "tasks": 6, + "top3HitRate": 0.16666666666666666 + } + }, + "taskResults": [ + { + "taskId": "as-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 787, + "estimatedTokens": 197, + "queryElapsedMs": 221, + "indexElapsedMs": 55.166666666666664, + "topFiles": [ + "constant" + ] + }, + { + "taskId": "as-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 1141, + "estimatedTokens": 286, + "queryElapsedMs": 222, + "indexElapsedMs": 55.166666666666664, + "topFiles": [ + "method", + "constant", + "type" + ] + }, + { + "taskId": "as-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 905, + "estimatedTokens": 227, + "queryElapsedMs": 225, + "indexElapsedMs": 55.166666666666664, + "topFiles": [ + "type", + "method" + ] + }, + { + "taskId": "as-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 1159, + "estimatedTokens": 290, + "queryElapsedMs": 222, + "indexElapsedMs": 55.166666666666664, + "topFiles": [ + "method" + ] + }, + { + "taskId": "as-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 1072, + "estimatedTokens": 268, + "queryElapsedMs": 221, + "indexElapsedMs": 55.166666666666664, + "topFiles": [ + "method" + ] + }, + { + "taskId": "as-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 905, + "estimatedTokens": 227, + "queryElapsedMs": 224, + "indexElapsedMs": 55.166666666666664, + "topFiles": [ + "type", + "method" + ] + }, + { + "taskId": "ex-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 947, + "estimatedTokens": 237, + "queryElapsedMs": 1036, + "indexElapsedMs": 50.833333333333336, + "topFiles": [ + "type", + "function" + ] + }, + { + "taskId": "ex-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 818, + "estimatedTokens": 205, + "queryElapsedMs": 1022, + "indexElapsedMs": 50.833333333333336, + "topFiles": [ + "function" + ] + }, + { + "taskId": "ex-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 797, + "estimatedTokens": 200, + "queryElapsedMs": 1195, + "indexElapsedMs": 50.833333333333336, + "topFiles": [ + "function", + "method", + "type" + ] + }, + { + "taskId": "ex-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 772, + "estimatedTokens": 193, + "queryElapsedMs": 1123, + "indexElapsedMs": 50.833333333333336, + "topFiles": [ + "method", + "function" + ] + }, + { + "taskId": "ex-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 592, + "estimatedTokens": 148, + "queryElapsedMs": 1081, + "indexElapsedMs": 50.833333333333336, + "topFiles": [ + "type" + ] + }, + { + "taskId": "ex-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 626, + "estimatedTokens": 157, + "queryElapsedMs": 1201, + "indexElapsedMs": 50.833333333333336, + "topFiles": [ + "function" + ] + }, + { + "taskId": "fa-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 850, + "estimatedTokens": 213, + "queryElapsedMs": 733, + "indexElapsedMs": 101.83333333333333, + "topFiles": [ + "method" + ] + }, + { + "taskId": "fa-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 1, + "matchedSignals": [ + "APIRouter", + "router" + ], + "payloadBytes": 510, + "estimatedTokens": 128, + "queryElapsedMs": 939, + "indexElapsedMs": 101.83333333333333, + "topFiles": [ + "method" + ] + }, + { + "taskId": "fa-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 721, + "estimatedTokens": 181, + "queryElapsedMs": 760, + "indexElapsedMs": 101.83333333333333, + "topFiles": [ + "class", + "function" + ] + }, + { + "taskId": "fa-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 496, + "estimatedTokens": 124, + "queryElapsedMs": 764, + "indexElapsedMs": 101.83333333333333, + "topFiles": [ + "class", + "method" + ] + }, + { + "taskId": "fa-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 634, + "estimatedTokens": 159, + "queryElapsedMs": 706, + "indexElapsedMs": 101.83333333333333, + "topFiles": [ + "function", + "field", + "class" + ] + }, + { + "taskId": "fa-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 810, + "estimatedTokens": 203, + "queryElapsedMs": 731, + "indexElapsedMs": 101.83333333333333, + "topFiles": [ + "function" + ] + } + ] + }, + { + "id": "aider-repo-map", + "displayName": "Aider repo map", + "credit": "Paul Gauthier's Aider RepoMap ranking (aider-chat on PyPI). Query tokens are passed as mentioned identifiers; there is no separate natural-language search API.", + "projectUrl": "https://github.com/Aider-AI/aider", + "status": "completed", + "tasksTotal": 18, + "tasksCompleted": 18, + "tasksFailed": 0, + "top3HitRate": 0.2222222222222222, + "averageUsefulness": 0.16666666666666666, + "averageEstimatedTokens": 95.05555555555556, + "averageQueryElapsedMs": 1441.8654441833496, + "averageIndexElapsedMs": 349.8888888888889, + "byNiche": { + "team-convention": { + "tasks": 6, + "top3HitRate": 0.3333333333333333 + }, + "edit-preflight": { + "tasks": 6, + "top3HitRate": 0 + }, + "symbol-retrieval": { + "tasks": 6, + "top3HitRate": 0.3333333333333333 + } + }, + "taskResults": [ + { + "taskId": "as-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 468, + "estimatedTokens": 117, + "queryElapsedMs": 1425.365924835205, + "indexElapsedMs": 351.3333333333333, + "topFiles": [ + "libs/web/shared/app-config/src/lib/app.config.ts", + "libs/web/shared/data-access/models/src/lib/api/play-request.ts", + "libs/web/shared/data-access/models/src/lib/api/spotify-api-params.ts", + "libs/web/visualizer/data-access/src/lib/visualizer-renderer.interface.ts", + "libs/web/shared/data-access/models/src/lib/api/track-extended.ts" + ] + }, + { + "taskId": "as-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 505, + "estimatedTokens": 127, + "queryElapsedMs": 1442.3494338989258, + "indexElapsedMs": 351.3333333333333, + "topFiles": [ + "libs/web/shared/app-config/src/lib/app.config.ts", + "libs/web/shared/data-access/models/src/lib/api/play-request.ts", + "libs/web/shared/data-access/models/src/lib/api/spotify-api-params.ts", + "libs/web/visualizer/data-access/src/lib/visualizer-renderer.interface.ts", + "libs/web/shared/data-access/models/src/lib/api/track-extended.ts" + ] + }, + { + "taskId": "as-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 530, + "estimatedTokens": 133, + "queryElapsedMs": 1495.1844215393066, + "indexElapsedMs": 351.3333333333333, + "topFiles": [ + "libs/web/shared/app-config/src/lib/app.config.ts", + "libs/web/shared/data-access/models/src/lib/api/play-request.ts", + "libs/web/shared/data-access/models/src/lib/api/spotify-api-params.ts", + "libs/web/visualizer/data-access/src/lib/visualizer-renderer.interface.ts", + "libs/web/shared/data-access/models/src/lib/api/track-extended.ts" + ] + }, + { + "taskId": "as-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 488, + "estimatedTokens": 122, + "queryElapsedMs": 1396.5404033660889, + "indexElapsedMs": 351.3333333333333, + "topFiles": [ + "libs/web/shared/app-config/src/lib/app.config.ts", + "libs/web/shared/data-access/models/src/lib/api/play-request.ts", + "libs/web/shared/data-access/models/src/lib/api/spotify-api-params.ts", + "libs/web/visualizer/data-access/src/lib/visualizer-renderer.interface.ts", + "libs/web/shared/data-access/models/src/lib/api/track-extended.ts" + ] + }, + { + "taskId": "as-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 460, + "estimatedTokens": 115, + "queryElapsedMs": 1381.8109035491943, + "indexElapsedMs": 351.3333333333333, + "topFiles": [ + "libs/web/shared/app-config/src/lib/app.config.ts", + "libs/web/shared/data-access/models/src/lib/api/play-request.ts", + "libs/web/shared/data-access/models/src/lib/api/spotify-api-params.ts", + "libs/web/visualizer/data-access/src/lib/visualizer-renderer.interface.ts", + "libs/web/shared/data-access/models/src/lib/api/track-extended.ts" + ] + }, + { + "taskId": "as-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 496, + "estimatedTokens": 124, + "queryElapsedMs": 1579.0975093841553, + "indexElapsedMs": 351.3333333333333, + "topFiles": [ + "libs/web/shared/app-config/src/lib/app.config.ts", + "libs/web/shared/data-access/models/src/lib/api/play-request.ts", + "libs/web/shared/data-access/models/src/lib/api/spotify-api-params.ts", + "libs/web/visualizer/data-access/src/lib/visualizer-renderer.interface.ts", + "libs/web/shared/data-access/models/src/lib/api/track-extended.ts" + ] + }, + { + "taskId": "ex-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 335, + "estimatedTokens": 84, + "queryElapsedMs": 1385.5478763580322, + "indexElapsedMs": 333.8333333333333, + "topFiles": [ + "packages/excalidraw/charts/charts.types.ts", + "packages/excalidraw/global.d.ts", + "packages/math/src/types.ts", + "packages/element/src/types.ts", + "packages/excalidraw/types.ts" + ] + }, + { + "taskId": "ex-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": true, + "firstRelevantHit": 3, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 364, + "estimatedTokens": 91, + "queryElapsedMs": 1377.885341644287, + "indexElapsedMs": 333.8333333333333, + "topFiles": [ + "packages/excalidraw/charts/charts.types.ts", + "packages/excalidraw/global.d.ts", + "packages/math/src/types.ts", + "packages/element/src/types.ts", + "packages/excalidraw/types.ts" + ] + }, + { + "taskId": "ex-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 331, + "estimatedTokens": 83, + "queryElapsedMs": 1376.3632774353027, + "indexElapsedMs": 333.8333333333333, + "topFiles": [ + "packages/excalidraw/types.ts", + "packages/math/src/types.ts", + "packages/excalidraw/charts/charts.types.ts", + "packages/element/src/types.ts", + "packages/common/src/editorInterface.ts" + ] + }, + { + "taskId": "ex-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 348, + "estimatedTokens": 87, + "queryElapsedMs": 1386.0201835632324, + "indexElapsedMs": 333.8333333333333, + "topFiles": [ + "packages/excalidraw/types.ts", + "packages/math/src/types.ts", + "packages/excalidraw/charts/charts.types.ts", + "packages/element/src/types.ts", + "packages/common/src/editorInterface.ts" + ] + }, + { + "taskId": "ex-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 308, + "estimatedTokens": 77, + "queryElapsedMs": 1516.0613059997559, + "indexElapsedMs": 333.8333333333333, + "topFiles": [ + "packages/math/src/types.ts", + "packages/element/src/types.ts", + "packages/excalidraw/tests/helpers/ui.ts", + "packages/common/src/bounds.ts", + "packages/excalidraw/types.ts" + ] + }, + { + "taskId": "ex-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 334, + "estimatedTokens": 84, + "queryElapsedMs": 1381.5414905548096, + "indexElapsedMs": 333.8333333333333, + "topFiles": [ + "packages/excalidraw/types.ts", + "packages/math/src/types.ts", + "packages/excalidraw/charts/charts.types.ts", + "packages/element/src/types.ts", + "packages/common/src/editorInterface.ts" + ] + }, + { + "taskId": "fa-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0.5, + "matchedSignals": [ + "dependency" + ], + "payloadBytes": 307, + "estimatedTokens": 77, + "queryElapsedMs": 1467.3211574554443, + "indexElapsedMs": 364.5, + "topFiles": [ + "fastapi/applications.py", + "fastapi/encoders.py", + "fastapi/routing.py", + "fastapi/_compat.py", + "docs/en/docs/js/termynal.js" + ] + }, + { + "taskId": "fa-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 0.5, + "matchedSignals": [ + "router" + ], + "payloadBytes": 292, + "estimatedTokens": 73, + "queryElapsedMs": 1499.6511936187744, + "indexElapsedMs": 364.5, + "topFiles": [ + "fastapi/routing.py", + "fastapi/exceptions.py", + "fastapi/encoders.py", + "fastapi/_compat.py", + "fastapi/utils.py" + ] + }, + { + "taskId": "fa-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 308, + "estimatedTokens": 77, + "queryElapsedMs": 1510.103464126587, + "indexElapsedMs": 364.5, + "topFiles": [ + "fastapi/encoders.py", + "fastapi/applications.py", + "docs/en/docs/js/termynal.js", + "fastapi/_compat.py", + "fastapi/exceptions.py" + ] + }, + { + "taskId": "fa-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 317, + "estimatedTokens": 80, + "queryElapsedMs": 1470.5393314361572, + "indexElapsedMs": 364.5, + "topFiles": [ + "fastapi/dependencies/models.py", + "fastapi/_compat.py", + "fastapi/exceptions.py", + "docs/en/docs/js/termynal.js", + "fastapi/encoders.py" + ] + }, + { + "taskId": "fa-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": true, + "firstRelevantHit": 3, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 320, + "estimatedTokens": 80, + "queryElapsedMs": 1429.5134544372559, + "indexElapsedMs": 364.5, + "topFiles": [ + "fastapi/encoders.py", + "fastapi/applications.py", + "fastapi/routing.py", + "tests/test_additional_properties_bool.py", + "fastapi/_compat.py" + ] + }, + { + "taskId": "fa-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": 5, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 317, + "estimatedTokens": 80, + "queryElapsedMs": 1432.6813220977783, + "indexElapsedMs": 364.5, + "topFiles": [ + "fastapi/encoders.py", + "fastapi/applications.py", + "docs/en/docs/js/termynal.js", + "fastapi/_compat.py", + "fastapi/exceptions.py" + ] + } + ] + }, + { + "id": "repomix", + "displayName": "Repomix", + "credit": "Repomix repository packing (repomix on npm). Repomix does not expose query-specific retrieval; the lane charges pack generation to index time and ranks files by scanning pack sections for query tokens.", + "projectUrl": "https://github.com/yamadashy/repomix", + "status": "completed", + "tasksTotal": 18, + "tasksCompleted": 18, + "tasksFailed": 0, + "top3HitRate": 0.1111111111111111, + "averageUsefulness": 0.1111111111111111, + "averageEstimatedTokens": 5559173.666666667, + "averageQueryElapsedMs": 113.88888888888889, + "averageIndexElapsedMs": 286.27777777777777, + "byNiche": { + "team-convention": { + "tasks": 6, + "top3HitRate": 0 + }, + "edit-preflight": { + "tasks": 6, + "top3HitRate": 0.16666666666666666 + }, + "symbol-retrieval": { + "tasks": 6, + "top3HitRate": 0.16666666666666666 + } + }, + "taskResults": [ + { + "taskId": "as-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 1847698, + "queryElapsedMs": 26, + "indexElapsedMs": 178.66666666666666, + "topFiles": [ + ".codebase-context/.generations/a2cd3546-363b-4787-910e-11cce05901af/index/index-build.json" + ] + }, + { + "taskId": "as-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 1847698, + "queryElapsedMs": 28, + "indexElapsedMs": 178.66666666666666, + "topFiles": [ + ".codebase-context/.generations/a2cd3546-363b-4787-910e-11cce05901af/index/index-build.json" + ] + }, + { + "taskId": "as-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 1847698, + "queryElapsedMs": 21, + "indexElapsedMs": 178.66666666666666, + "topFiles": [ + ".codebase-context/.generations/a2cd3546-363b-4787-910e-11cce05901af/index/index-build.json" + ] + }, + { + "taskId": "as-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 1847698, + "queryElapsedMs": 21, + "indexElapsedMs": 178.66666666666666, + "topFiles": [ + ".codebase-context/.generations/a2cd3546-363b-4787-910e-11cce05901af/index/index-build.json" + ] + }, + { + "taskId": "as-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 1847698, + "queryElapsedMs": 14, + "indexElapsedMs": 178.66666666666666, + "topFiles": [ + ".codebase-context/.generations/a2cd3546-363b-4787-910e-11cce05901af/index/index-build.json" + ] + }, + { + "taskId": "as-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 1847698, + "queryElapsedMs": 18, + "indexElapsedMs": 178.66666666666666, + "topFiles": [ + ".codebase-context/.generations/a2cd3546-363b-4787-910e-11cce05901af/index/index-build.json" + ] + }, + { + "taskId": "ex-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 8754051, + "queryElapsedMs": 235, + "indexElapsedMs": 346.6666666666667, + "topFiles": [ + ".codebase-context/.generations/7ef9aa97-4e56-4dbe-aaa5-ba2bfb795092/index/index-build.json" + ] + }, + { + "taskId": "ex-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 8754051, + "queryElapsedMs": 235, + "indexElapsedMs": 346.6666666666667, + "topFiles": [ + ".codebase-context/.generations/7ef9aa97-4e56-4dbe-aaa5-ba2bfb795092/index/index-build.json" + ] + }, + { + "taskId": "ex-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 8754051, + "queryElapsedMs": 162, + "indexElapsedMs": 346.6666666666667, + "topFiles": [ + ".codebase-context/.generations/7ef9aa97-4e56-4dbe-aaa5-ba2bfb795092/index/index-build.json" + ] + }, + { + "taskId": "ex-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 8754051, + "queryElapsedMs": 194, + "indexElapsedMs": 346.6666666666667, + "topFiles": [ + ".codebase-context/.generations/7ef9aa97-4e56-4dbe-aaa5-ba2bfb795092/index/index-build.json" + ] + }, + { + "taskId": "ex-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 8754051, + "queryElapsedMs": 187, + "indexElapsedMs": 346.6666666666667, + "topFiles": [ + ".codebase-context/.generations/7ef9aa97-4e56-4dbe-aaa5-ba2bfb795092/index/index-build.json" + ] + }, + { + "taskId": "ex-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": true, + "firstRelevantHit": 1, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 8754051, + "queryElapsedMs": 189, + "indexElapsedMs": 346.6666666666667, + "topFiles": [ + ".codebase-context/.generations/7ef9aa97-4e56-4dbe-aaa5-ba2bfb795092/index/index-build.json" + ] + }, + { + "taskId": "fa-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 6075772, + "queryElapsedMs": 133, + "indexElapsedMs": 333.5, + "topFiles": [ + ".codebase-context/.generations/93a21abb-70cf-4b9c-8b23-c234ecf4165f/index/index-build.json" + ] + }, + { + "taskId": "fa-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 6075772, + "queryElapsedMs": 111, + "indexElapsedMs": 333.5, + "topFiles": [ + ".codebase-context/.generations/93a21abb-70cf-4b9c-8b23-c234ecf4165f/index/index-build.json" + ] + }, + { + "taskId": "fa-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 6075772, + "queryElapsedMs": 121, + "indexElapsedMs": 333.5, + "topFiles": [ + ".codebase-context/.generations/93a21abb-70cf-4b9c-8b23-c234ecf4165f/index/index-build.json" + ] + }, + { + "taskId": "fa-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 6075772, + "queryElapsedMs": 113, + "indexElapsedMs": 333.5, + "topFiles": [ + ".codebase-context/.generations/93a21abb-70cf-4b9c-8b23-c234ecf4165f/index/index-build.json" + ] + }, + { + "taskId": "fa-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 6075772, + "queryElapsedMs": 122, + "indexElapsedMs": 333.5, + "topFiles": [ + ".codebase-context/.generations/93a21abb-70cf-4b9c-8b23-c234ecf4165f/index/index-build.json" + ] + }, + { + "taskId": "fa-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 50000, + "estimatedTokens": 6075772, + "queryElapsedMs": 120, + "indexElapsedMs": 333.5, + "topFiles": [ + ".codebase-context/.generations/93a21abb-70cf-4b9c-8b23-c234ecf4165f/index/index-build.json" + ] + } + ] + }, + { + "id": "ripgrep", + "displayName": "ripgrep (lexical baseline)", + "credit": "BurntSushi ripgrep JSON search over the repo checkout. No index; measures lexical retrieval only.", + "projectUrl": "https://github.com/BurntSushi/ripgrep", + "status": "completed", + "tasksTotal": 18, + "tasksCompleted": 18, + "tasksFailed": 0, + "top3HitRate": 0.05555555555555555, + "averageUsefulness": 0.1111111111111111, + "averageEstimatedTokens": 5000.5, + "averageQueryElapsedMs": 76.16666666666667, + "averageIndexElapsedMs": 0, + "byNiche": { + "team-convention": { + "tasks": 6, + "top3HitRate": 0 + }, + "edit-preflight": { + "tasks": 6, + "top3HitRate": 0.16666666666666666 + }, + "symbol-retrieval": { + "tasks": 6, + "top3HitRate": 0 + } + }, + "taskResults": [ + { + "taskId": "as-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20008, + "estimatedTokens": 5002, + "queryElapsedMs": 24, + "indexElapsedMs": 0, + "topFiles": [ + "apps/angular-spotify/src/polyfills.ts", + "README.md", + "yarn.lock", + "docs/plans/2026-03-07-infinite-scroll-pagination-design.md", + "docs/plans/2026-03-01-visualizer-selector-design.md" + ] + }, + { + "taskId": "as-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 23, + "indexElapsedMs": 0, + "topFiles": [ + "apps/angular-spotify/src/environments/environment.ts", + "apps/angular-spotify/src/app/app.component.spec.ts", + "apps/angular-spotify/src/polyfills.ts", + ".codebase-context/.generations/e369989f-3d39-444e-a743-4ed8fb5ce7f5/intelligence.json", + "README.md" + ] + }, + { + "taskId": "as-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 33, + "indexElapsedMs": 0, + "topFiles": [ + "tsconfig.base.json", + "apps/angular-spotify/ngsw-config.json", + "apps/angular-spotify/project.json", + "apps/angular-spotify/src/environments/environment.prod.ts", + "apps/angular-spotify/src/environments/environment.ts" + ] + }, + { + "taskId": "as-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 26, + "indexElapsedMs": 0, + "topFiles": [ + "tsconfig.base.json", + "apps/angular-spotify/src/environments/environment.ts", + "apps/angular-spotify/src/manifest.webmanifest", + "apps/angular-spotify/src/index.prod.html", + "apps/angular-spotify/src/app/app.component.spec.ts" + ] + }, + { + "taskId": "as-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20008, + "estimatedTokens": 5002, + "queryElapsedMs": 22, + "indexElapsedMs": 0, + "topFiles": [ + "tsconfig.base.json", + ".codebase-context/.generations/e369989f-3d39-444e-a743-4ed8fb5ce7f5/intelligence.json", + "README.md", + "migrations.json", + "docs/plans/2026-03-07-infinite-scroll-pagination-design.md" + ] + }, + { + "taskId": "as-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 24, + "indexElapsedMs": 0, + "topFiles": [ + "tsconfig.base.json", + "apps/angular-spotify/src/environments/environment.prod.ts", + "apps/angular-spotify/src/environments/environment.ts", + "apps/angular-spotify/src/polyfills.ts", + "apps/angular-spotify/src/custom/ant/_button.scss" + ] + }, + { + "taskId": "ex-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20006, + "estimatedTokens": 5002, + "queryElapsedMs": 84, + "indexElapsedMs": 0, + "topFiles": [ + "vitest.config.mts", + ".github/copilot-instructions.md", + "yarn.lock", + ".eslintrc.json", + "examples/with-script-in-browser/components/CustomFooter.tsx" + ] + }, + { + "taskId": "ex-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": 5, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 93, + "indexElapsedMs": 0, + "topFiles": [ + ".env.development", + ".eslintrc.json", + "scripts/release.js", + "scripts/build-node.js", + "excalidraw-app/collab/Collab.tsx" + ] + }, + { + "taskId": "ex-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20002, + "estimatedTokens": 5001, + "queryElapsedMs": 105, + "indexElapsedMs": 0, + "topFiles": [ + "examples/with-nextjs/README.md", + "examples/with-script-in-browser/index.tsx", + ".eslintrc.json", + "README.md", + "examples/with-script-in-browser/components/ExampleApp.tsx" + ] + }, + { + "taskId": "ex-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": true, + "firstRelevantHit": 3, + "usefulnessScore": 1, + "matchedSignals": [], + "payloadBytes": 20004, + "estimatedTokens": 5001, + "queryElapsedMs": 119, + "indexElapsedMs": 0, + "topFiles": [ + ".dockerignore", + ".gitignore", + ".eslintrc.json", + "scripts/release.js", + "scripts/buildWasm.js" + ] + }, + { + "taskId": "ex-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 158, + "indexElapsedMs": 0, + "topFiles": [ + ".gitignore", + "examples/with-nextjs/.gitignore", + "examples/with-nextjs/next.config.js", + "examples/with-nextjs/tsconfig.json", + "examples/with-nextjs/yarn.lock" + ] + }, + { + "taskId": "ex-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 110, + "indexElapsedMs": 0, + "topFiles": [ + ".gitignore", + "examples/with-nextjs/next.config.js", + "examples/with-nextjs/src/pages/excalidraw-in-pages.tsx", + "examples/with-nextjs/src/excalidrawWrapper.tsx", + "examples/with-nextjs/src/app/layout.tsx" + ] + }, + { + "taskId": "fa-team-01", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0.5, + "matchedSignals": [ + "dependency" + ], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 99, + "indexElapsedMs": 0, + "topFiles": [ + "requirements-docs.txt", + "pyproject.toml", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/index/index-build.json", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/intelligence.json", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/relationships.json" + ] + }, + { + "taskId": "fa-team-02", + "status": "completed", + "niche": "team-convention", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0.5, + "matchedSignals": [ + "router" + ], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 105, + "indexElapsedMs": 0, + "topFiles": [ + "pyproject.toml", + "requirements-docs-tests.txt", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/intelligence.json", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/health.json", + "LICENSE" + ] + }, + { + "taskId": "fa-ep-01", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20002, + "estimatedTokens": 5001, + "queryElapsedMs": 82, + "indexElapsedMs": 0, + "topFiles": [ + "pyproject.toml", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/intelligence.json", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/relationships.json", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/health.json", + "LICENSE" + ] + }, + { + "taskId": "fa-ep-02", + "status": "completed", + "niche": "edit-preflight", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 96, + "indexElapsedMs": 0, + "topFiles": [ + "requirements-docs.txt", + "pyproject.toml", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/index/index-build.json", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/intelligence.json", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/relationships.json" + ] + }, + { + "taskId": "fa-sym-01", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 76, + "indexElapsedMs": 0, + "topFiles": [ + "requirements-docs.txt", + "pyproject.toml", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/health.json", + "LICENSE", + "docs_src/body/tutorial001_py310.py" + ] + }, + { + "taskId": "fa-sym-02", + "status": "completed", + "niche": "symbol-retrieval", + "top3Hit": false, + "firstRelevantHit": null, + "usefulnessScore": 0, + "matchedSignals": [], + "payloadBytes": 20000, + "estimatedTokens": 5000, + "queryElapsedMs": 92, + "indexElapsedMs": 0, + "topFiles": [ + "pyproject.toml", + "requirements-docs-tests.txt", + "requirements-docs-insiders.txt", + ".codebase-context/.generations/b4709add-8981-4a99-af8d-5bce4ff4b916/health.json", + "scripts/notify_translations.py" + ] + } + ] + } + ], + "summaryTable": [ + { + "laneId": "codebase-context", + "status": "completed", + "top3HitRate": 0.5, + "averageEstimatedTokens": 641.7222222222222, + "averageQueryElapsedMs": 927.6111111111111, + "averageIndexElapsedMs": 672.7777777777779 + }, + { + "laneId": "jCodeMunch", + "status": "completed", + "top3HitRate": 0.05555555555555555, + "averageEstimatedTokens": 202.38888888888889, + "averageQueryElapsedMs": 701.4444444444445, + "averageIndexElapsedMs": 69.27777777777777 + }, + { + "laneId": "aider-repo-map", + "status": "completed", + "top3HitRate": 0.2222222222222222, + "averageEstimatedTokens": 95.05555555555556, + "averageQueryElapsedMs": 1441.8654441833496, + "averageIndexElapsedMs": 349.8888888888889 + }, + { + "laneId": "repomix", + "status": "completed", + "top3HitRate": 0.1111111111111111, + "averageEstimatedTokens": 5559173.666666667, + "averageQueryElapsedMs": 113.88888888888889, + "averageIndexElapsedMs": 286.27777777777777 + }, + { + "laneId": "ripgrep", + "status": "completed", + "top3HitRate": 0.05555555555555555, + "averageEstimatedTokens": 5000.5, + "averageQueryElapsedMs": 76.16666666666667, + "averageIndexElapsedMs": 0 + } + ] +} diff --git a/scripts/lib/niche-benchmark/aider-repomap-query.py b/scripts/lib/niche-benchmark/aider-repomap-query.py new file mode 100644 index 0000000..a366133 --- /dev/null +++ b/scripts/lib/niche-benchmark/aider-repomap-query.py @@ -0,0 +1,102 @@ +#!/usr/bin/env python3 +"""Aider RepoMap lane for the niche prior-art benchmark.""" +from __future__ import annotations + +import argparse +import json +import os +import re +import subprocess +import sys +import time + + +def tokenize(text: str) -> list[str]: + return [ + token + for token in re.findall(r"[A-Za-z_][A-Za-z0-9_+#.-]{2,}", text) + if token.lower() + not in {"the", "and", "for", "with", "from", "that", "this", "what", "how", "does", "are"} + ] + + +def git_tracked_files(root: str) -> list[str]: + output = subprocess.check_output(["git", "ls-files"], cwd=root, text=True) + return [line.strip() for line in output.splitlines() if line.strip()] + + +def ranked_files(root: str, query: str, limit: int) -> tuple[list[str], str, float]: + from aider.io import InputOutput + from aider.models import Model + from aider.repomap import RepoMap + + started = time.time() + io = InputOutput(yes=True) + model = Model("gpt-4o-mini") + repo_map = RepoMap(map_tokens=4096, root=root, main_model=model, io=io) + tracked = git_tracked_files(root) + source_files = [ + path + for path in tracked + if any( + path.endswith(ext) + for ext in (".py", ".ts", ".tsx", ".js", ".jsx", ".go", ".rs", ".java", ".cs", ".cpp", ".c", ".h") + ) + ] + mentioned = tokenize(query) + tags = repo_map.get_ranked_tags([], source_files, [], mentioned) + top_files: list[str] = [] + for tag in tags: + rel = getattr(tag, "rel_fname", None) or getattr(tag, "fname", None) + if not isinstance(rel, str): + continue + normalized = rel.replace("\\", "/").lstrip("./") + if normalized and normalized not in top_files: + top_files.append(normalized) + if len(top_files) >= limit: + break + payload = json.dumps( + {"mentionedIdents": mentioned, "rankedTags": len(tags), "topFiles": top_files}, + indent=2, + ) + return top_files, payload, (time.time() - started) * 1000 + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--root", required=True) + parser.add_argument("--query", default="") + parser.add_argument("--limit", type=int, default=5) + parser.add_argument("--warm-index", action="store_true") + args = parser.parse_args() + + root = os.path.abspath(args.root) + if not os.path.isdir(os.path.join(root, ".git")): + print(json.dumps({"error": "missing_git_metadata", "topFiles": [], "payload": ""})) + return 2 + + try: + if args.warm_index: + _, _, elapsed_ms = ranked_files(root, "warm index scan", min(args.limit, 3)) + print(json.dumps({"status": "indexed", "indexElapsedMs": elapsed_ms})) + return 0 + + top_files, payload, elapsed_ms = ranked_files(root, args.query, args.limit) + print( + json.dumps( + { + "topFiles": top_files, + "payload": payload, + "queryElapsedMs": elapsed_ms, + "indexElapsedMs": 0, + } + ) + ) + return 0 + except Exception as exc: # noqa: BLE001 - benchmark boundary + print(json.dumps({"error": str(exc), "topFiles": [], "payload": ""})) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/lib/niche-benchmark/lanes.mjs b/scripts/lib/niche-benchmark/lanes.mjs new file mode 100644 index 0000000..b4b296f --- /dev/null +++ b/scripts/lib/niche-benchmark/lanes.mjs @@ -0,0 +1,408 @@ +import { spawnSync } from 'node:child_process'; +import { existsSync, mkdirSync, readFileSync } from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { withManagedStdioClientSession } from '../managed-mcp-session.mjs'; +import { + countUtf8Bytes, + estimateTokens, + normalizeRelativePath, + scoreTaskResult +} from './scoring.mjs'; + +const __dirname = path.dirname(fileURLToPath(import.meta.url)); +const aiderScript = path.join(__dirname, 'aider-repomap-query.py'); +const pythonCmd = process.platform === 'win32' ? 'python' : 'python3'; + +function runCommand(command, args, options = {}) { + const started = Date.now(); + const result = spawnSync(command, args, { + encoding: 'utf8', + maxBuffer: 256 * 1024 * 1024, + timeout: options.timeoutMs ?? 300_000, + cwd: options.cwd ?? process.cwd(), + env: options.env ?? process.env + }); + return { + status: typeof result.status === 'number' ? result.status : 1, + stdout: result.stdout ?? '', + stderr: result.stderr ?? '', + durationMs: Date.now() - started, + error: result.error?.message ?? null + }; +} + +function extractSearchFilesFromJson(value) { + const topFiles = []; + if (!value || typeof value !== 'object') return topFiles; + const results = Array.isArray(value.results) ? value.results : []; + for (const entry of results) { + const normalized = normalizeRelativePath(entry.file ?? entry.path ?? ''); + if (normalized && !topFiles.includes(normalized)) topFiles.push(normalized); + } + return topFiles; +} + +function codebaseContextCommand(projectRoot) { + const distIndex = path.join(projectRoot, 'dist', 'index.js'); + if (existsSync(distIndex)) { + return { command: process.execPath, args: [distIndex] }; + } + return { + command: process.platform === 'win32' ? 'npm.cmd' : 'npm', + args: ['exec', '--', 'codebase-context'] + }; +} + +export function ensureRepoCheckout(repo, projectRoot) { + const cloneRoot = path.join(projectRoot, 'benchmark-repos'); + const checkout = path.join(cloneRoot, repo.id); + mkdirSync(cloneRoot, { recursive: true }); + if (!existsSync(path.join(checkout, '.git'))) { + const clone = runCommand('git', ['clone', '--filter=blob:none', repo.repositoryUrl, checkout], { + timeoutMs: 600_000 + }); + if (clone.status !== 0) { + throw new Error(`git clone failed for ${repo.id}: ${clone.stderr || clone.error}`); + } + } + const fetch = runCommand( + 'git', + ['fetch', '--depth', '1', 'origin', repo.repositoryRef], + { cwd: checkout, timeoutMs: 600_000 } + ); + if (fetch.status !== 0) { + throw new Error(`git fetch failed for ${repo.id}: ${fetch.stderr || fetch.error}`); + } + const checkoutRun = runCommand('git', ['checkout', '--force', repo.repositoryRef], { + cwd: checkout, + timeoutMs: 120_000 + }); + if (checkoutRun.status !== 0) { + throw new Error(`git checkout failed for ${repo.id}: ${checkoutRun.stderr || checkoutRun.error}`); + } + return checkout; +} + +export async function warmCodebaseContextIndex(repoPath, projectRoot) { + const base = codebaseContextCommand(projectRoot); + const env = { + ...process.env, + CODEBASE_ROOT: repoPath, + CODEBASE_CONTEXT_ASCII: '1' + }; + return runCommand( + base.command, + [...base.args, 'reindex', '--no-embeddings', '--json'], + { env, timeoutMs: 900_000 } + ); +} + +export async function runCodebaseContextTask(task, repoPath, projectRoot, indexElapsedMs) { + const base = codebaseContextCommand(projectRoot); + const env = { + ...process.env, + CODEBASE_ROOT: repoPath, + CODEBASE_CONTEXT_ASCII: '1' + }; + const cc = task.codebaseContext; + const args = [...base.args, cc.command]; + if (cc.command === 'search') { + args.push('--query', task.query); + } + args.push(...cc.args); + const started = Date.now(); + const result = runCommand(base.command, args, { env, timeoutMs: 180_000 }); + const queryElapsedMs = Date.now() - started; + let payload = `${result.stdout}\n${result.stderr}`.trim(); + let topFiles = []; + if (result.status === 0) { + try { + const parsed = JSON.parse(result.stdout); + payload = JSON.stringify(parsed); + topFiles = extractSearchFilesFromJson(parsed).map((filePath) => { + const resolvedRepo = path.resolve(repoPath); + const resolvedFile = path.resolve(filePath); + if (resolvedFile.startsWith(resolvedRepo + path.sep)) { + return normalizeRelativePath(path.relative(resolvedRepo, resolvedFile)); + } + return normalizeRelativePath(filePath); + }); + if (topFiles.length === 0 && Array.isArray(parsed.patterns)) { + for (const pattern of parsed.patterns) { + const example = normalizeRelativePath(pattern?.bestExample ?? pattern?.example ?? ''); + if (example) topFiles.push(example); + } + } + } catch { + topFiles = []; + } + } + const score = scoreTaskResult(task, { payload, topFiles }); + return { + status: result.status === 0 ? 'completed' : 'tool_error', + niche: task.niche, + ...score, + payloadBytes: countUtf8Bytes(payload), + estimatedTokens: estimateTokens(countUtf8Bytes(payload)), + queryElapsedMs, + indexElapsedMs: indexElapsedMs ?? 0, + topFiles + }; +} + +function tokenizeQuery(query) { + return query + .toLowerCase() + .split(/[^a-z0-9_+#./-]+/i) + .map((token) => token.trim()) + .filter((token) => token.length > 2); +} + +export function runRipgrepTask(task, repoPath, indexElapsedMs = 0) { + const tokens = tokenizeQuery(task.query); + const pattern = tokens.slice(0, 8).join('|') || task.query.slice(0, 80); + const started = Date.now(); + const result = runCommand( + process.env.RG_PATH || 'rg', + ['--json', '-n', '--hidden', '--glob', '!.git/**', '--glob', '!node_modules/**', '--glob', '!dist/**', pattern, '.'], + { cwd: repoPath, timeoutMs: 120_000 } + ); + const queryElapsedMs = Date.now() - started; + const topFiles = []; + for (const line of result.stdout.split('\n')) { + if (!line.trim()) continue; + let event; + try { + event = JSON.parse(line); + } catch { + continue; + } + if (event.type !== 'match') continue; + const normalized = normalizeRelativePath(event.data?.path?.text ?? ''); + if (normalized && !topFiles.includes(normalized)) topFiles.push(normalized); + if (topFiles.length >= 5) break; + } + const payload = result.stdout.slice(0, 20_000); + const score = scoreTaskResult(task, { payload, topFiles }); + return { + status: result.status === 0 || result.status === 1 ? 'completed' : 'tool_error', + niche: task.niche, + ...score, + payloadBytes: countUtf8Bytes(payload), + estimatedTokens: estimateTokens(countUtf8Bytes(payload)), + queryElapsedMs, + indexElapsedMs, + topFiles + }; +} + +function parseJCodeMunchPayload(result) { + const textParts = Array.isArray(result?.content) + ? result.content + .map((item) => (typeof item?.text === 'string' ? item.text.trim() : '')) + .filter(Boolean) + : []; + const payload = textParts.join('\n'); + let topFiles = []; + try { + const parsed = JSON.parse(payload); + const results = Array.isArray(parsed.results) ? parsed.results : []; + for (const entry of results) { + const normalized = normalizeRelativePath(entry.file ?? entry.path ?? ''); + if (normalized && !topFiles.includes(normalized)) topFiles.push(normalized); + } + } catch { + for (const line of payload.split('\n')) { + if (!line.startsWith('s,')) continue; + const columns = line.split(','); + const fileColumn = columns[3] ?? columns[4]; + const normalized = normalizeRelativePath(fileColumn ?? ''); + if (normalized && !topFiles.includes(normalized)) topFiles.push(normalized); + } + } + return { payload, topFiles }; +} + +export async function warmJCodeMunchRepo(repoPath) { + return withManagedStdioClientSession( + { + serverCommand: pythonCmd, + serverArgs: ['-m', 'jcodemunch_mcp.server', 'serve', '--transport', 'stdio', '--log-level', 'ERROR'], + serverEnv: { ...process.env, JCODEMUNCH_USE_AI_SUMMARIES: 'false' }, + cwd: repoPath, + connectTimeoutMs: 30_000 + }, + async ({ client }) => { + await new Promise((resolve) => setTimeout(resolve, 2_000)); + const started = Date.now(); + const indexResult = await client.callTool({ + name: 'index_folder', + arguments: { + path: path.resolve(repoPath), + use_ai_summaries: false, + incremental: true, + follow_symlinks: false, + extra_ignore_patterns: ['.codebase-context/**', 'node_modules/**'] + } + }); + const { payload } = parseJCodeMunchPayload(indexResult); + let repoName = null; + try { + const parsed = JSON.parse(payload || '{}'); + repoName = typeof parsed.repo === 'string' ? parsed.repo : parsed.name ?? null; + } catch { + repoName = null; + } + return { repoName, indexElapsedMs: Date.now() - started }; + } + ); +} + +export async function runJCodeMunchTask(task, repoPath, repoName, indexElapsedMs) { + if (!repoName) { + return { + status: 'setup_failed', + niche: task.niche, + top3Hit: false, + usefulnessScore: 0, + queryElapsedMs: 0, + indexElapsedMs, + payloadBytes: 0, + estimatedTokens: 0, + topFiles: [] + }; + } + return withManagedStdioClientSession( + { + serverCommand: pythonCmd, + serverArgs: ['-m', 'jcodemunch_mcp.server', 'serve', '--transport', 'stdio', '--log-level', 'ERROR'], + serverEnv: { ...process.env, JCODEMUNCH_USE_AI_SUMMARIES: 'false' }, + cwd: repoPath, + connectTimeoutMs: 30_000 + }, + async ({ client }) => { + await new Promise((resolve) => setTimeout(resolve, 1_000)); + const started = Date.now(); + const result = await client.callTool({ + name: 'search_symbols', + arguments: { + repo: repoName, + query: task.query, + max_results: 5, + detail_level: 'compact', + semantic: false + } + }); + const queryElapsedMs = Date.now() - started; + const { payload, topFiles } = parseJCodeMunchPayload(result); + const score = scoreTaskResult(task, { payload, topFiles }); + return { + status: 'completed', + niche: task.niche, + ...score, + payloadBytes: countUtf8Bytes(payload), + estimatedTokens: estimateTokens(countUtf8Bytes(payload)), + queryElapsedMs, + indexElapsedMs, + topFiles + }; + } + ); +} + +export function warmAiderRepo(repoPath) { + return runCommand( + pythonCmd, + [aiderScript, '--root', repoPath, '--warm-index', '--limit', '3'], + { cwd: repoPath, timeoutMs: 900_000 } + ); +} + +export function runAiderTask(task, repoPath, indexElapsedMs) { + const result = runCommand( + pythonCmd, + [aiderScript, '--root', repoPath, '--query', task.query, '--limit', '5'], + { cwd: repoPath, timeoutMs: 300_000 } + ); + let topFiles = []; + let payload = result.stdout; + try { + const parsed = JSON.parse(result.stdout); + topFiles = Array.isArray(parsed.topFiles) ? parsed.topFiles : []; + payload = parsed.payload ?? result.stdout; + } catch { + topFiles = []; + } + const score = scoreTaskResult(task, { payload, topFiles }); + return { + status: result.status === 0 ? 'completed' : 'tool_error', + niche: task.niche, + ...score, + payloadBytes: countUtf8Bytes(payload), + estimatedTokens: estimateTokens(countUtf8Bytes(payload)), + queryElapsedMs: (() => { + try { + return JSON.parse(result.stdout || '{}').queryElapsedMs ?? result.durationMs; + } catch { + return result.durationMs; + } + })(), + indexElapsedMs, + topFiles + }; +} + +function rankRepomixPack(packText, query, limit) { + const tokens = tokenizeQuery(query); + const sections = packText.split(/\n={32}\n/); + const scores = new Map(); + for (const section of sections) { + const fileMatch = section.match(/^File:\s+(.+)$/m); + if (!fileMatch) continue; + const filePath = fileMatch[1].trim(); + const lower = section.toLowerCase(); + let score = 0; + for (const token of tokens) { + if (lower.includes(token)) score += 1; + } + if (score > 0) scores.set(filePath, score); + } + return [...scores.entries()] + .sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0])) + .slice(0, limit) + .map(([filePath]) => filePath.replace(/\\/g, '/')); +} + +export function warmRepomixRepo(repoPath) { + const started = Date.now(); + const result = runCommand( + process.platform === 'win32' ? 'npx.cmd' : 'npx', + ['--yes', 'repomix', '--stdout', '--quiet', '--style', 'plain'], + { cwd: repoPath, timeoutMs: 900_000 } + ); + return { + packText: result.stdout, + indexElapsedMs: Date.now() - started, + status: result.status === 0 ? 'completed' : 'setup_failed', + error: result.status === 0 ? null : result.stderr || result.error + }; +} + +export function runRepomixTask(task, packText, indexElapsedMs) { + const started = Date.now(); + const topFiles = rankRepomixPack(packText, task.query, 5); + const payload = topFiles.join('\n'); + const queryElapsedMs = Date.now() - started; + const score = scoreTaskResult(task, { payload, topFiles }); + return { + status: packText ? 'completed' : 'setup_failed', + niche: task.niche, + ...score, + payloadBytes: countUtf8Bytes(packText.slice(0, 50_000)), + estimatedTokens: estimateTokens(countUtf8Bytes(packText)), + queryElapsedMs, + indexElapsedMs, + topFiles + }; +} diff --git a/scripts/lib/niche-benchmark/scoring.mjs b/scripts/lib/niche-benchmark/scoring.mjs new file mode 100644 index 0000000..c3ad84c --- /dev/null +++ b/scripts/lib/niche-benchmark/scoring.mjs @@ -0,0 +1,134 @@ +export function normalizeText(value) { + return String(value ?? '') + .toLowerCase() + .replace(/\\/g, '/'); +} + +export function normalizeRelativePath(candidate) { + if (typeof candidate !== 'string') return null; + const trimmed = candidate.trim().replace(/^["']|["']$/g, ''); + if (!trimmed) return null; + const normalized = trimmed.replace(/\\/g, '/').replace(/^\.\//, ''); + if (/^[A-Za-z]:\//.test(normalized)) { + return normalized.replace(/^[A-Za-z]:\//, ''); + } + return normalized.split(':')[0]; +} + +export function matchesPatterns(candidate, patterns) { + if (!patterns?.length) return false; + const normalizedCandidate = normalizeText(candidate); + return patterns.some((pattern) => normalizedCandidate.includes(normalizeText(pattern))); +} + +export function firstRelevantHit(topFiles, patterns) { + if (!patterns?.length) return null; + for (let index = 0; index < topFiles.length; index++) { + if (matchesPatterns(topFiles[index], patterns)) return index + 1; + } + return null; +} + +export function scoreTaskResult(task, { payload = '', topFiles = [] }) { + const hit = firstRelevantHit(topFiles, task.expectedFilePatterns); + const top3Hit = hit !== null && hit <= 3; + const normalizedPayload = normalizeText(payload); + const matchedSignals = (task.expectedSignals ?? []).filter((signal) => + normalizedPayload.includes(normalizeText(signal)) + ); + const usefulnessScore = + task.expectedSignals?.length > 0 + ? matchedSignals.length / Math.max(task.expectedSignals.length, 1) + : top3Hit + ? 1 + : 0; + + return { + top3Hit, + firstRelevantHit: hit, + usefulnessScore, + matchedSignals + }; +} + +export function countUtf8Bytes(str) { + return Buffer.byteLength(String(str ?? ''), 'utf8'); +} + +export function estimateTokens(bytes) { + return Math.ceil(bytes / 4); +} + +export function aggregateLaneResults(taskResults) { + const completed = taskResults.filter((result) => result.status === 'completed'); + const n = taskResults.length; + const top3Hits = completed.filter((result) => result.top3Hit).length; + const avgUsefulness = + completed.length > 0 + ? completed.reduce((sum, result) => sum + result.usefulnessScore, 0) / completed.length + : null; + const avgTokens = + completed.length > 0 + ? completed.reduce((sum, result) => sum + result.estimatedTokens, 0) / completed.length + : null; + const avgQueryMs = + completed.length > 0 + ? completed.reduce((sum, result) => sum + result.queryElapsedMs, 0) / completed.length + : null; + const avgIndexMs = + completed.length > 0 + ? completed.reduce((sum, result) => sum + (result.indexElapsedMs ?? 0), 0) / completed.length + : null; + + const byNiche = {}; + for (const niche of ['team-convention', 'edit-preflight', 'symbol-retrieval']) { + const nicheRows = completed.filter((result) => result.niche === niche); + byNiche[niche] = { + tasks: nicheRows.length, + top3HitRate: nicheRows.length > 0 ? nicheRows.filter((row) => row.top3Hit).length / nicheRows.length : null + }; + } + + const failed = taskResults.filter((result) => result.status !== 'completed'); + return { + status: completed.length === 0 ? 'setup_failed' : failed.length > 0 ? 'partial' : 'completed', + tasksTotal: n, + tasksCompleted: completed.length, + tasksFailed: failed.length, + top3HitRate: completed.length > 0 ? top3Hits / completed.length : null, + averageUsefulness: avgUsefulness, + averageEstimatedTokens: avgTokens, + averageQueryElapsedMs: avgQueryMs, + averageIndexElapsedMs: avgIndexMs, + byNiche, + taskResults + }; +} + +export function aggregateEvidence(laneEvidence, manifest, protocol, runMeta) { + return { + artifactType: 'niche-prior-art-benchmark-evidence', + generatedAt: new Date().toISOString(), + protocol: { + name: protocol.name, + frozenDate: protocol.frozenDate, + manifestFrozenDate: manifest.frozenDate + }, + runMeta, + comparators: protocol.comparators.map((comparator) => ({ + id: comparator.id, + displayName: comparator.displayName, + credit: comparator.credit, + projectUrl: comparator.projectUrl, + ...(laneEvidence[comparator.id] ?? { status: 'not_run' }) + })), + summaryTable: Object.entries(laneEvidence).map(([laneId, lane]) => ({ + laneId, + status: lane.status, + top3HitRate: lane.top3HitRate, + averageEstimatedTokens: lane.averageEstimatedTokens, + averageQueryElapsedMs: lane.averageQueryElapsedMs, + averageIndexElapsedMs: lane.averageIndexElapsedMs + })) + }; +} diff --git a/scripts/run-niche-benchmark.mjs b/scripts/run-niche-benchmark.mjs new file mode 100644 index 0000000..06200d3 --- /dev/null +++ b/scripts/run-niche-benchmark.mjs @@ -0,0 +1,283 @@ +#!/usr/bin/env node +/** + * One-command niche prior-art benchmark runner. + * + * Usage: + * node scripts/run-niche-benchmark.mjs + * node scripts/run-niche-benchmark.mjs --lanes codebase-context,ripgrep --output results/niche-benchmark-evidence.json + * node scripts/run-niche-benchmark.mjs --validate-fixtures + */ + +import { createHash } from 'node:crypto'; +import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { parseArgs } from 'node:util'; +import { spawnSync } from 'node:child_process'; +import { + ensureRepoCheckout, + runAiderTask, + runCodebaseContextTask, + runJCodeMunchTask, + runRepomixTask, + runRipgrepTask, + warmAiderRepo, + warmCodebaseContextIndex, + warmJCodeMunchRepo, + warmRepomixRepo +} from './lib/niche-benchmark/lanes.mjs'; +import { aggregateEvidence, aggregateLaneResults } from './lib/niche-benchmark/scoring.mjs'; + +const __dirname = path.dirname(fileURLToPath(import.meta.url)); +const projectRoot = path.join(__dirname, '..'); +const defaultManifest = path.join(projectRoot, 'tests', 'fixtures', 'niche-benchmark-manifest.json'); +const defaultProtocol = path.join(projectRoot, 'tests', 'fixtures', 'niche-benchmark-protocol.json'); +const defaultOutput = path.join(projectRoot, 'results', 'niche-benchmark-evidence.json'); + +const ALL_LANES = ['codebase-context', 'jCodeMunch', 'aider-repo-map', 'repomix', 'ripgrep']; + +function sha256File(filePath) { + return createHash('sha256').update(readFileSync(filePath)).digest('hex'); +} + +function loadJson(filePath) { + return JSON.parse(readFileSync(filePath, 'utf8')); +} + +function ensureBuilt() { + const distIndex = path.join(projectRoot, 'dist', 'index.js'); + if (existsSync(distIndex)) return; + console.log('Building codebase-context (dist missing)...'); + const build = spawnSync('pnpm', ['run', 'build'], { + cwd: projectRoot, + stdio: 'inherit', + encoding: 'utf8' + }); + if (build.status !== 0) { + throw new Error('pnpm run build failed; dist/index.js is required for the codebase-context lane'); + } +} + +function validateFixtures(manifestPath, protocolPath) { + const manifest = loadJson(manifestPath); + const protocol = loadJson(protocolPath); + const errors = []; + if (manifest.tasks.length !== 18) errors.push(`expected 18 tasks, found ${manifest.tasks.length}`); + if (manifest.repos.length !== 3) errors.push(`expected 3 repos, found ${manifest.repos.length}`); + const nicheCounts = {}; + for (const task of manifest.tasks) { + nicheCounts[task.niche] = (nicheCounts[task.niche] ?? 0) + 1; + } + for (const niche of protocol.niches) { + if (nicheCounts[niche] !== 6) { + errors.push(`expected 6 ${niche} tasks, found ${nicheCounts[niche] ?? 0}`); + } + } + const repoIds = new Set(manifest.repos.map((repo) => repo.id)); + for (const task of manifest.tasks) { + if (!repoIds.has(task.repoId)) errors.push(`task ${task.id} references unknown repo ${task.repoId}`); + } + if (errors.length > 0) { + console.error(errors.join('\n')); + process.exit(1); + } + console.log('Fixture validation passed.'); + console.log(` manifest: ${manifestPath}`); + console.log(` protocol: ${protocolPath}`); + console.log(` manifest sha256: ${sha256File(manifestPath)}`); + console.log(` protocol sha256: ${sha256File(protocolPath)}`); +} + +async function runLane(laneId, manifest, repoCheckouts) { + const tasksByRepo = manifest.repos.map((repo) => ({ + repo, + checkout: repoCheckouts[repo.id], + tasks: manifest.tasks.filter((task) => task.repoId === repo.id) + })); + + const taskResults = []; + + if (laneId === 'ripgrep') { + for (const group of tasksByRepo) { + for (const task of group.tasks) { + taskResults.push({ + taskId: task.id, + ...runRipgrepTask(task, group.checkout, 0) + }); + } + } + return aggregateLaneResults(taskResults.map((row) => ({ ...row, status: row.status ?? 'completed' }))); + } + + for (const group of tasksByRepo) { + let indexElapsedMs = 0; + let repomixPack = ''; + let jcodemunchRepo = null; + + if (laneId === 'codebase-context') { + const index = await warmCodebaseContextIndex(group.checkout, projectRoot); + indexElapsedMs = index.durationMs; + if (index.status !== 0) { + return { + status: 'setup_failed', + reason: `codebase-context reindex failed on ${group.repo.id}`, + taskResults: [] + }; + } + } else if (laneId === 'jCodeMunch') { + try { + const warmed = await warmJCodeMunchRepo(group.checkout); + jcodemunchRepo = warmed.repoName; + indexElapsedMs = warmed.indexElapsedMs; + } catch (error) { + return { + status: 'setup_failed', + reason: `jCodeMunch setup failed on ${group.repo.id}: ${error.message}`, + taskResults: [] + }; + } + } else if (laneId === 'aider-repo-map') { + const warm = warmAiderRepo(group.checkout); + indexElapsedMs = warm.durationMs; + if (warm.status !== 0) { + return { + status: 'setup_failed', + reason: `Aider repo map warm index failed on ${group.repo.id}`, + taskResults: [] + }; + } + } else if (laneId === 'repomix') { + const warm = warmRepomixRepo(group.checkout); + indexElapsedMs = warm.indexElapsedMs; + if (warm.status !== 'completed') { + return { + status: 'setup_failed', + reason: warm.error ?? `Repomix pack failed on ${group.repo.id}`, + taskResults: [] + }; + } + repomixPack = warm.packText; + } + + for (const task of group.tasks) { + let row; + if (laneId === 'codebase-context') { + row = await runCodebaseContextTask(task, group.checkout, projectRoot, indexElapsedMs / group.tasks.length); + } else if (laneId === 'jCodeMunch') { + row = await runJCodeMunchTask(task, group.checkout, jcodemunchRepo, indexElapsedMs / group.tasks.length); + } else if (laneId === 'aider-repo-map') { + row = runAiderTask(task, group.checkout, indexElapsedMs / group.tasks.length); + } else if (laneId === 'repomix') { + row = runRepomixTask(task, repomixPack, indexElapsedMs / group.tasks.length); + } + taskResults.push({ taskId: task.id, ...row }); + } + } + + return aggregateLaneResults( + taskResults.map((row) => ({ + ...row, + status: row.status ?? 'completed' + })) + ); +} + +async function main() { + const { values } = parseArgs({ + options: { + help: { type: 'boolean', default: false }, + 'validate-fixtures': { type: 'boolean', default: false }, + manifest: { type: 'string' }, + protocol: { type: 'string' }, + lanes: { type: 'string' }, + output: { type: 'string' }, + 'skip-build': { type: 'boolean', default: false } + } + }); + + const manifestPath = path.resolve(values.manifest ?? defaultManifest); + const protocolPath = path.resolve(values.protocol ?? defaultProtocol); + const outputPath = path.resolve(values.output ?? defaultOutput); + + if (values.help) { + console.log(`Usage: node scripts/run-niche-benchmark.mjs [options] + + --validate-fixtures Validate frozen manifest/protocol and exit + --lanes Default: ${ALL_LANES.join(',')} + --output Default: ${defaultOutput} + --skip-build Do not auto-build dist before running +`); + process.exit(0); + } + + if (values['validate-fixtures']) { + validateFixtures(manifestPath, protocolPath); + process.exit(0); + } + + const manifest = loadJson(manifestPath); + const protocol = loadJson(protocolPath); + validateFixtures(manifestPath, protocolPath); + + const lanes = (values.lanes ?? ALL_LANES.join(',')) + .split(',') + .map((lane) => lane.trim()) + .filter(Boolean); + for (const lane of lanes) { + if (!ALL_LANES.includes(lane)) { + throw new Error(`Unknown lane ${lane}. Supported: ${ALL_LANES.join(', ')}`); + } + } + + if (!values['skip-build'] && lanes.includes('codebase-context')) { + ensureBuilt(); + } + + console.log(`\nNiche prior-art benchmark (${manifest.tasks.length} tasks, ${lanes.length} lanes)`); + const repoCheckouts = {}; + for (const repo of manifest.repos) { + console.log(`Checking out ${repo.id} @ ${repo.repositoryRef.slice(0, 12)}...`); + repoCheckouts[repo.id] = ensureRepoCheckout(repo, projectRoot); + } + + const laneEvidence = {}; + for (const laneId of lanes) { + console.log(`\n=== Lane: ${laneId} ===`); + try { + laneEvidence[laneId] = await runLane(laneId, manifest, repoCheckouts); + console.log( + ` top-3 hit rate: ${laneEvidence[laneId].top3HitRate === null ? 'n/a' : (laneEvidence[laneId].top3HitRate * 100).toFixed(1) + '%'}` + ); + console.log( + ` avg tokens: ${laneEvidence[laneId].averageEstimatedTokens === null ? 'n/a' : Math.round(laneEvidence[laneId].averageEstimatedTokens)}` + ); + console.log( + ` avg query ms: ${laneEvidence[laneId].averageQueryElapsedMs === null ? 'n/a' : Math.round(laneEvidence[laneId].averageQueryElapsedMs)}` + ); + } catch (error) { + laneEvidence[laneId] = { status: 'setup_failed', reason: error.message, taskResults: [] }; + console.error(` SETUP FAILED: ${error.message}`); + } + } + + const evidence = aggregateEvidence(laneEvidence, manifest, protocol, { + manifestPath, + protocolPath, + manifestSha256: sha256File(manifestPath), + protocolSha256: sha256File(protocolPath), + nodeVersion: process.version, + lanes, + repoCheckouts: Object.fromEntries( + manifest.repos.map((repo) => [repo.id, { ref: repo.repositoryRef, path: repoCheckouts[repo.id] }]) + ) + }); + + mkdirSync(path.dirname(outputPath), { recursive: true }); + writeFileSync(outputPath, `${JSON.stringify(evidence, null, 2)}\n`); + console.log(`\nEvidence written to ${outputPath}`); +} + +main().catch((error) => { + console.error('Fatal:', error); + process.exit(2); +}); diff --git a/src/analyzers/generic/index.ts b/src/analyzers/generic/index.ts index 78ed320..6fb0dd2 100644 --- a/src/analyzers/generic/index.ts +++ b/src/analyzers/generic/index.ts @@ -32,6 +32,10 @@ import { categorizeDependency } from '../../utils/dependency-detection.js'; import type { WorkspacePackageJson } from '../../utils/workspace-detection.js'; +import { + detectFrontendFrameworkFromDeps, + extractSingleFileComponentScript +} from '../../utils/frontend-framework-metadata.js'; export class GenericAnalyzer implements FrameworkAnalyzer { readonly name = 'generic'; @@ -91,7 +95,9 @@ export class GenericAnalyzer implements FrameworkAnalyzer { '.css', '.scss', '.sass', - '.less' + '.less', + '.vue', + '.svelte' ]; readonly priority = 10; // Low priority - fallback analyzer @@ -126,8 +132,23 @@ export class GenericAnalyzer implements FrameworkAnalyzer { usesTreeSitterSymbols = true; } - if (language === 'typescript' || language === 'javascript') { - const parsed = await this.parseJSTSFile(filePath, content, language); + let scriptLanguage: 'typescript' | 'javascript' | null = null; + let scriptBody: string | null = null; + if (language === 'vue' || language === 'svelte') { + const extracted = extractSingleFileComponentScript(content, language); + if (extracted) { + scriptBody = extracted.body; + scriptLanguage = extracted.language; + } + } + + const jstsLanguage = + scriptLanguage ?? + (language === 'typescript' || language === 'javascript' ? language : null); + const jstsContent = scriptBody ?? content; + + if (jstsLanguage) { + const parsed = await this.parseJSTSFile(filePath, jstsContent, jstsLanguage); imports = parsed.imports; exports = parsed.exports; @@ -247,6 +268,7 @@ export class GenericAnalyzer implements FrameworkAnalyzer { let workspaceType: string = 'single'; let workspacePackages: WorkspacePackageJson[] = []; + let frameworkClaim: CodebaseMetadata['framework']; try { workspaceType = await detectWorkspaceType(rootPath); @@ -276,6 +298,11 @@ export class GenericAnalyzer implements FrameworkAnalyzer { version: version as string, category: categorizeDependency(name) })); + + const framework = detectFrontendFrameworkFromDeps(rawDeps); + if (framework) { + frameworkClaim = framework; + } } catch (_error) { // skip } @@ -285,6 +312,7 @@ export class GenericAnalyzer implements FrameworkAnalyzer { rootPath, languages: [], dependencies, + framework: frameworkClaim, architecture: { type: 'mixed', layers: { diff --git a/src/analyzers/nextjs/index.ts b/src/analyzers/nextjs/index.ts index ea99dab..cf9e8ca 100644 --- a/src/analyzers/nextjs/index.ts +++ b/src/analyzers/nextjs/index.ts @@ -71,7 +71,19 @@ export class NextJsAnalyzer implements FrameworkAnalyzer { return false; } - if (isInAppRouter(filePath) || isInPagesRouter(filePath) || isProxyFile(filePath)) { + if (isSvelteKitRouteFile(filePath)) { + return false; + } + + if (isProxyFile(filePath)) { + return true; + } + + if (matchesNextAppRouterConvention(filePath)) { + return true; + } + + if (isInPagesRouter(filePath)) { return true; } @@ -342,6 +354,18 @@ function analyzeRouting(filePath: string, content: string): NextRoutingInfo { }; } +function isSvelteKitRouteFile(filePath: string): boolean { + const baseName = path.basename(filePath, path.extname(filePath)); + return baseName.startsWith('+'); +} + +function matchesNextAppRouterConvention(filePath: string): boolean { + if (!isInAppRouter(filePath)) { + return false; + } + return getAppRouteFileKind(path.basename(filePath)) !== null; +} + function isInAppRouter(filePath: string): boolean { const normalizedPath = filePath.replace(/\\/g, '/'); return normalizedPath.includes('/app/') || normalizedPath.includes('/src/app/'); diff --git a/src/cli-init.ts b/src/cli-init.ts index 4641e84..1888986 100644 --- a/src/cli-init.ts +++ b/src/cli-init.ts @@ -4,7 +4,11 @@ import { loadProjectConfig } from './server/config.js'; /** * Interactive setup wizard for codebase-context. * Handles `codebase-context init` — detects clients and merges project MCP configs - * for the four first-wave AI clients: Claude Code, Cursor, Codex, OpenCode. + * for Claude Code, Cursor, Codex, VS Code (Copilot), Windsurf, and OpenCode. + * Config merge shapes follow each client's official docs (VS Code uses a top-level + * `servers` object per the MCP filesystem server workspace examples; Windsurf uses + * `~/.codeium/windsurf/mcp_config.json`). Use `--dry-run` to preview writes first, + * matching the preview-before-apply pattern from MCP filesystem `edit_file`. * * Config writes run only after all selected registrations pass preflight. * File previews share the indexer's selection rules and project configuration. @@ -19,12 +23,12 @@ import { checkbox } from '@inquirer/prompts'; import { homedir } from 'node:os'; import { constants } from 'node:fs'; -export type Client = (typeof CLIENTS)[number]['id']; - export type McpConfigResult = { kind: 'file'; path: string; content: string }; export type McpConfigOptions = { rootPath?: string; + /** Override home for user-scoped configs (tests). Defaults to os.homedir(). */ + homeDirectory?: string; /** Node binary written into the client config. Defaults to the current process. */ nodeExecutable?: string; /** @@ -37,19 +41,79 @@ export type McpConfigOptions = { type JsonObject = Record; export const CLIENTS = [ - { id: 'claude-code', name: 'Claude Code', executable: 'claude', directory: '.claude' }, - { id: 'codex', name: 'Codex', executable: 'codex', directory: '.codex' }, - { id: 'cursor', name: 'Cursor', executable: 'cursor', directory: '.cursor' }, - { id: 'opencode', name: 'OpenCode', executable: 'opencode', directory: '.opencode' } + { + id: 'claude-code', + name: 'Claude Code', + executable: 'claude', + directory: '.claude', + scope: 'project' + }, + { id: 'codex', name: 'Codex', executable: 'codex', directory: '.codex', scope: 'project' }, + { id: 'cursor', name: 'Cursor', executable: 'cursor', directory: '.cursor', scope: 'project' }, + { + id: 'vscode', + name: 'VS Code (Copilot)', + executable: 'code', + directory: '.vscode', + scope: 'project' + }, + { + id: 'windsurf', + name: 'Windsurf', + executable: 'windsurf', + directory: '.codeium/windsurf', + scope: 'user' + }, + { + id: 'opencode', + name: 'OpenCode', + executable: 'opencode', + directory: '.opencode', + scope: 'project' + } ] as const; +export type Client = (typeof CLIENTS)[number]['id']; + +function resolveHome( + env: NodeJS.ProcessEnv = process.env, + platform: NodeJS.Platform = process.platform +): string { + const paths = platform === 'win32' ? path.win32 : path.posix; + return env.HOME || env.USERPROFILE || homedir(); +} + +function jsonMcpServerKey( + client: Extract +): string { + if (client === 'opencode') return 'mcp'; + if (client === 'vscode') return 'servers'; + return 'mcpServers'; +} + +export function resolveClientConfigPath( + client: Client, + rootPath: string, + env: NodeJS.ProcessEnv = process.env, + platform: NodeJS.Platform = process.platform +): string { + const paths = platform === 'win32' ? path.win32 : path.posix; + const home = resolveHome(env, platform); + if (client === 'codex') return path.join(rootPath, '.codex', 'config.toml'); + if (client === 'windsurf') return path.join(home, '.codeium', 'windsurf', 'mcp_config.json'); + if (client === 'claude-code') return path.join(rootPath, '.mcp.json'); + if (client === 'cursor') return path.join(rootPath, '.cursor', 'mcp.json'); + if (client === 'vscode') return path.join(rootPath, '.vscode', 'mcp.json'); + return path.join(rootPath, 'opencode.json'); +} + export async function detectClients( rootPath: string, env: NodeJS.ProcessEnv = process.env, platform: NodeJS.Platform = process.platform ): Promise { const paths = platform === 'win32' ? path.win32 : path.posix; - const home = env.HOME || env.USERPROFILE || homedir(); + const home = resolveHome(env, platform); const configHome = env.XDG_CONFIG_HOME || paths.join(home, '.config'); const searchPath = (env.PATH || env.Path || '').split(platform === 'win32' ? ';' : ':'); const extensions = platform === 'win32' ? ['', '.exe', '.cmd', '.bat', '.ps1'] : ['']; @@ -62,8 +126,15 @@ export async function detectClients( ? env.CLAUDE_CONFIG_DIR || paths.join(home, '.claude') : client.id === 'opencode' ? paths.join(configHome, 'opencode') - : paths.join(home, client.directory); - const directories = [paths.join(rootPath, client.directory), userDirectory]; + : client.id === 'windsurf' + ? paths.join(home, '.codeium', 'windsurf') + : paths.join(home, client.directory.split('/').at(-1) ?? client.directory); + const projectDirectory = + client.id === 'windsurf' + ? paths.join(home, '.codeium', 'windsurf') + : paths.join(rootPath, client.directory); + const directories = + client.scope === 'user' ? [userDirectory] : [projectDirectory, userDirectory]; const executables = searchPath .filter(Boolean) .flatMap((directory) => @@ -116,47 +187,45 @@ function resolveServerEntrypoint(explicit?: string): string { */ export function generateMcpConfig(client: Client, options: McpConfigOptions = {}): McpConfigResult { const rootPath = path.resolve(options.rootPath ?? process.cwd()); + const homeDirectory = options.homeDirectory ?? resolveHome(); const nodeExecutable = path.resolve(options.nodeExecutable ?? process.execPath); const serverEntrypoint = resolveServerEntrypoint(options.serverEntrypoint); const args = [serverEntrypoint, rootPath]; if (client === 'codex') { return { kind: 'file', - path: path.join(rootPath, '.codex', 'config.toml'), + path: resolveClientConfigPath('codex', rootPath, { HOME: homeDirectory }), content: `[mcp_servers.codebase-context]\ncommand = ${JSON.stringify(nodeExecutable)}\n` + `args = [${args.map((value) => JSON.stringify(value)).join(', ')}]\nstartup_timeout_sec = 120\n` }; } - const configPath = - client === 'claude-code' - ? '.mcp.json' + const configPath = resolveClientConfigPath(client, rootPath, { HOME: homeDirectory }); + const serverPayload = + client === 'opencode' + ? { type: 'local', command: [nodeExecutable, ...args] } : client === 'cursor' - ? '.cursor/mcp.json' - : 'opencode.json'; + ? { type: 'stdio', command: nodeExecutable, args } + : { command: nodeExecutable, args }; + const body = + client === 'opencode' + ? { + $schema: 'https://opencode.ai/config.json', + [jsonMcpServerKey('opencode')]: { 'codebase-context': serverPayload } + } + : client === 'vscode' + ? { [jsonMcpServerKey('vscode')]: { 'codebase-context': serverPayload } } + : client === 'windsurf' + ? { [jsonMcpServerKey('windsurf')]: { 'codebase-context': serverPayload } } + : { + mcpServers: { + 'codebase-context': serverPayload + } + }; return { kind: 'file', - path: path.join(rootPath, configPath), - content: JSON.stringify( - client === 'opencode' - ? { - $schema: 'https://opencode.ai/config.json', - mcp: { - 'codebase-context': { type: 'local', command: [nodeExecutable, ...args] } - } - } - : { - mcpServers: { - 'codebase-context': { - ...(client === 'cursor' ? { type: 'stdio' } : {}), - command: nodeExecutable, - args - } - } - }, - null, - 2 - ) + path: configPath, + content: JSON.stringify(body, null, 2) }; } @@ -281,13 +350,12 @@ function mergeGeneratedServer(existing: unknown, generated: unknown, client: Cli if (jsonValuesEqual(existing, generated)) return existing; if (!isJsonObject(existing) || !isJsonObject(generated) || 'url' in existing) return null; const local = client === 'opencode'; + const stdio = client === 'cursor'; const command = local && Array.isArray(existing.command) ? existing.command[0] : existing.command; const args = local && Array.isArray(existing.command) ? existing.command.slice(1) : existing.args; - if ( - (existing.type !== undefined && existing.type !== (local ? 'local' : 'stdio')) || - !isGeneratedLaunch(command, args, generatedEntrypoint(generated, client)) - ) - return null; + if (local && existing.type !== undefined && existing.type !== 'local') return null; + if (stdio && existing.type !== undefined && existing.type !== 'stdio') return null; + if (!isGeneratedLaunch(command, args, generatedEntrypoint(generated, client))) return null; return { ...existing, command: generated.command, @@ -303,7 +371,7 @@ function mergeGeneratedServer(existing: unknown, generated: unknown, client: Cli export async function _buildMergedMcpContent( filePath: string, generatedContent: string, - client: Extract + client: Extract ): Promise<{ content: string; mergedFromExisting: boolean }> { const existingText = await readExistingConfig(filePath); if (existingText === null) { @@ -324,7 +392,7 @@ export async function _buildMergedMcpContent( throw new Error(`Cannot merge ${client} config at ${filePath}: expected a JSON object.`); } - const serverKey = client === 'opencode' ? 'mcp' : 'mcpServers'; + const serverKey = jsonMcpServerKey(client); const existingServersValue = existing[serverKey]; if (existingServersValue !== undefined && !isJsonObject(existingServersValue)) { throw new Error( @@ -461,8 +529,40 @@ type InitOptions = { yes: boolean; help: boolean; preview: boolean; + dryRun: boolean; }; +function formatDisplayPath(configPath: string, rootPath: string): string { + const relative = path.relative(rootPath, configPath); + if (relative && !relative.startsWith('..') && !path.isAbsolute(relative)) return relative; + return configPath; +} + +function classifyInitChange( + existingText: string | null, + mergedContent: string +): 'create' | 'update' | 'unchanged' { + if (existingText === null) return 'create'; + if (existingText === mergedContent) return 'unchanged'; + return 'update'; +} + +function describeInitChange( + dryRun: boolean, + action: 'create' | 'update' | 'unchanged', + displayPath: string +): string { + if (action === 'unchanged') return `Unchanged: ${displayPath}`; + const prefix = dryRun + ? action === 'create' + ? 'Would create' + : 'Would update' + : action === 'create' + ? 'Created' + : 'Updated'; + return `${prefix}: ${displayPath}`; +} + function parseClient(value: string): Client { const client = CLIENTS.find((client) => client.id === value); if (client) return client.id; @@ -473,13 +573,22 @@ function parseClient(value: string): Client { function parseInitOptions(args: string[]): InitOptions { if (args.includes('--help') || args.includes('-h')) - return { clients: [], rootPath: process.cwd(), yes: false, help: true, preview: false }; + return { + clients: [], + rootPath: process.cwd(), + yes: false, + help: true, + preview: false, + dryRun: false + }; const clients: Client[] = []; let yes = false; let preview = false; + let dryRun = false; for (let index = 0; index < args.length; index++) { const arg = args[index]; if (arg === '--preview') preview = true; + else if (arg === '--dry-run') dryRun = true; else if (arg === '--yes' || arg === '-y') yes = true; else if (arg === '--client') { const value = args[++index]; @@ -495,7 +604,8 @@ function parseInitOptions(args: string[]): InitOptions { rootPath: resolveInitRoot(args, process.env.CODEBASE_ROOT), yes, help: false, - preview + preview, + dryRun }; } @@ -514,6 +624,7 @@ function printInitUsage(): void { ); console.log(' --root Project root (default: CODEBASE_ROOT or cwd)'); console.log(' --preview Show selected files and exclusions without writing config'); + console.log(' --dry-run Show MCP config changes without writing files'); console.log(' --yes, -y Configure every detected tool without prompts'); console.log(' --help, -h Show this help'); } @@ -554,30 +665,35 @@ export async function handleInitCli(args: string[]): Promise { } if (!clients.length) throw new Error( - 'No coding tools selected or detected. Choose with --client claude-code, --client codex, --client cursor, or --client opencode.' + `No coding tools selected or detected. Choose with --client (${CLIENTS.map((client) => client.id).join(', ')}).` ); const prepared = []; for (const client of clients) { const config = generateMcpConfig(client, { rootPath: options.rootPath }); + const existingText = await readExistingConfig(config.path); const merged = client === 'codex' ? await _buildMergedCodexContent(config.path, config.content) : await _buildMergedMcpContent(config.path, config.content, client); + const action = classifyInitChange(existingText, merged.content); prepared.push({ ...config, content: merged.content, - unchanged: (await readExistingConfig(config.path)) === merged.content + action, + displayPath: formatDisplayPath(config.path, options.rootPath) }); } for (const config of prepared) { - if (!config.unchanged) { + if (config.action !== 'unchanged' && !options.dryRun) { await fs.mkdir(path.dirname(config.path), { recursive: true }); await atomicWrite(config.path, config.content); } - console.log( - `${config.unchanged ? 'Unchanged' : 'Configured'}: ${path.relative(options.rootPath, config.path)}` - ); + console.log(describeInitChange(options.dryRun, config.action, config.displayPath)); + } + if (options.dryRun) { + console.log('Dry run complete; no files written.'); + return; } console.log('Reopen your tools; approve the server or project trust if asked.'); console.log("The first search builds this repository's index."); diff --git a/src/cli.ts b/src/cli.ts index 345eaec..08aa7bc 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -9,7 +9,7 @@ import path from 'path'; import { promises as fs } from 'fs'; import { KEYWORD_INDEX_FILENAME } from './constants/codebase-context.js'; import { makePaths } from './project-state.js'; -import { CodebaseIndexer } from './core/indexer.js'; +import { CodebaseIndexer, formatIndexingCompleteLine } from './core/indexer.js'; import { dispatchTool } from './tools/index.js'; import type { ToolContext } from './tools/index.js'; import type { IndexState } from './tools/types.js'; @@ -83,7 +83,7 @@ function printUsage(): void { console.log(' patterns [--category all|di|state|testing|libraries] Team patterns'); console.log(' refs --symbol [--limit ] Symbol references'); console.log(' cycles [--scope ] Circular dependency detection'); - console.log(' init [--client ] [--yes] Detect and configure coding tools'); + console.log(' init [--client ] [--yes] [--dry-run] Configure detected coding tools'); console.log(' map [--json] [--pretty] Conventions map (indexes first if needed)'); console.log(''); console.log('Global flags:'); @@ -155,11 +155,7 @@ async function initToolContext(): Promise { indexState.lastIndexed = new Date(); indexState.stats = stats; - console.error( - `Complete: ${stats.indexedFiles} files, ${stats.totalChunks} chunks in ${( - stats.duration / 1000 - ).toFixed(2)}s` - ); + console.error(formatIndexingCompleteLine(stats)); } catch (error) { indexState.status = 'error'; indexState.error = error instanceof Error ? error.message : String(error); diff --git a/src/core/codebase-map.ts b/src/core/codebase-map.ts index 87b491f..8ae5b05 100644 --- a/src/core/codebase-map.ts +++ b/src/core/codebase-map.ts @@ -78,6 +78,9 @@ const MAP_EXCLUDED_PATH_PATTERNS = [ /(?:^|\/)(?:fixtures?|__fixtures__)(?:\/|$)/i, /(?:^|\/)(?:generated|__generated__)(?:\/|$)/i, /(?:^|\/)[^/]*\.(?:generated|gen|min)\.[^/]+$/i, + /(?:^|\/)raw(?:\/|$)/i, + /(?:^|\/)live-from-ai(?:\/|$)/i, + /[^/]+-[A-Za-z0-9]{6,}\.(?:js|css|mjs|cjs)$/i, /\.snap$/i ] as const; @@ -180,6 +183,7 @@ async function buildPinnedCodebaseMap( const entrypoints: string[] = []; for (const [file, imports] of Object.entries(graphImports)) { if (imports.length === 0) continue; // no imports — not an entrypoint + if (!isMapEligiblePath(file, mode)) continue; const importers = graphImportedBy[file]; if (!importers || importers.length === 0) { entrypoints.push(file); @@ -193,7 +197,7 @@ async function buildPinnedCodebaseMap( graphImportedBy ).map(([file, importers]) => ({ file, count: importers.length })); const hubFiles: string[] = sortByCountThenAlpha( - importedByCounts, + importedByCounts.filter(({ file }) => isMapEligiblePath(file, mode)), (x) => x.count, (x) => x.file ) diff --git a/src/core/indexer.ts b/src/core/indexer.ts index 6d9ed37..b9078dc 100644 --- a/src/core/indexer.ts +++ b/src/core/indexer.ts @@ -141,6 +141,8 @@ export interface IndexerOptions { interface PersistedIndexingStats { indexedFiles: number; totalChunks: number; + searchableChunks?: number; + chunkLimitApplied?: boolean; totalFiles: number; generatedAt: string; } @@ -266,7 +268,9 @@ export class CodebaseIndexer { const defaultModel = defaultEmbeddingProvider === 'openai' && !process.env.EMBEDDING_MODEL ? 'text-embedding-3-small' - : DEFAULT_MODEL; + : defaultEmbeddingProvider === 'ollama' && !process.env.EMBEDDING_MODEL + ? 'nomic-embed-text' + : DEFAULT_MODEL; const defaultConfig: CodebaseConfig = { analyzers: { @@ -279,6 +283,7 @@ export class CodebaseIndexer { }, include: [ '**/*.{ts,tsx,js,jsx,mjs,cjs,mts,cts}', + '**/*.{vue,svelte}', '**/*.{html,htm,css,scss,sass,less}', '**/*.{py,pyi,rb,php}', '**/*.{java,kt,kts,scala,swift,cs}', @@ -956,7 +961,9 @@ export class CodebaseIndexer { let chunksToEmbed = chunksForEmbedding; if (chunksForEmbedding.length > maxChunks) { console.warn( - `WARNING: ${chunksForEmbedding.length} chunks exceed limit. Indexing first ${maxChunks} chunks.` + `WARNING: ${chunksForEmbedding.length} chunks exceed the searchable limit (${maxChunks}). ` + + `Only the first ${maxChunks} chunks are indexed for search. ` + + `Raise projects[].parsing.maxChunks in ~/.codebase-context/config.json for this repo root.` ); chunksToEmbed = chunksForEmbedding.slice(0, maxChunks); } @@ -1098,8 +1105,10 @@ export class CodebaseIndexer { // Keyword index always uses ALL chunks (full regen) const indexPath = path.join(generationDirectory, KEYWORD_INDEX_FILENAME); // Memory safety: cap keyword index too - const keywordChunks = - allChunks.length > maxChunks ? allChunks.slice(0, maxChunks) : allChunks; + const chunkLimitApplied = allChunks.length > maxChunks; + const keywordChunks = chunkLimitApplied ? allChunks.slice(0, maxChunks) : allChunks; + stats.searchableChunks = keywordChunks.length; + stats.chunkLimitApplied = chunkLimitApplied; await fs.writeFile( indexPath, JSON.stringify({ @@ -1206,6 +1215,8 @@ export class CodebaseIndexer { const persistedStats: PersistedIndexingStats = { indexedFiles: stats.indexedFiles, totalChunks: stats.totalChunks, + searchableChunks: stats.searchableChunks, + chunkLimitApplied: stats.chunkLimitApplied, totalFiles: stats.totalFiles, generatedAt }; @@ -1264,7 +1275,7 @@ export class CodebaseIndexer { ); } else { console.error(`Indexing complete in ${stats.duration}ms`); - console.error(`Indexed ${stats.indexedFiles} files, ${stats.totalChunks} chunks`); + console.error(formatIndexingProgressSummary(stats)); } return stats; @@ -1589,3 +1600,23 @@ export class CodebaseIndexer { return { ...this.progress }; } } + +export function formatIndexingProgressSummary(stats: IndexingStats): string { + const searchable = stats.searchableChunks ?? stats.totalChunks; + if (stats.chunkLimitApplied && searchable < stats.totalChunks) { + return ( + `Indexed ${stats.indexedFiles} files, ${stats.totalChunks} chunks analyzed, ` + + `${searchable} chunks searchable (limit applied)` + ); + } + return `Indexed ${stats.indexedFiles} files, ${stats.totalChunks} chunks`; +} + +export function formatIndexingCompleteLine(stats: IndexingStats): string { + const searchable = stats.searchableChunks ?? stats.totalChunks; + const durationSec = (stats.duration / 1000).toFixed(2); + if (stats.chunkLimitApplied && searchable < stats.totalChunks) { + return `Complete: ${stats.indexedFiles} files, ${searchable}/${stats.totalChunks} chunks searchable in ${durationSec}s`; + } + return `Complete: ${stats.indexedFiles} files, ${stats.totalChunks} chunks in ${durationSec}s`; +} diff --git a/src/core/reranker.ts b/src/core/reranker.ts index 9dabeac..f579289 100644 --- a/src/core/reranker.ts +++ b/src/core/reranker.ts @@ -11,6 +11,7 @@ import type { SearchResult } from '../types/index.js'; import type { PreTrainedTokenizer } from '@huggingface/transformers'; import os from 'os'; +import { assertMinimumOnnxRuntimeNodeVersion } from '../utils/onnxruntime-version.js'; const DEFAULT_RERANKER_MODEL = 'Xenova/ms-marco-MiniLM-L-6-v2'; @@ -48,6 +49,8 @@ async function ensureModelLoaded(): Promise { if (initPromise) return initPromise; initPromise = (async () => { + assertMinimumOnnxRuntimeNodeVersion(); + const { AutoTokenizer, AutoModelForSequenceClassification, env } = await import('@huggingface/transformers'); diff --git a/src/core/search.ts b/src/core/search.ts index 743a3b2..ce1d72b 100644 --- a/src/core/search.ts +++ b/src/core/search.ts @@ -39,6 +39,24 @@ import { VECTOR_DB_DIRNAME } from '../constants/codebase-context.js'; +/** File/path-like tokens in a natural-language query (e.g. NOW.md, check-agent-instructions.mjs). */ +export function extractQueryPathTokens(query: string): string[] { + const tokens = new Set(); + for (const match of query.matchAll(/\b([A-Za-z0-9][A-Za-z0-9._-]*\.(?:md|mdx|txt))\b/g)) { + tokens.add(match[1].toLowerCase()); + } + for (const match of query.matchAll(/\b([a-z][a-z0-9-]*(?:-[a-z0-9]+)+)\.(?:mjs|cjs|js|ts)\b/gi)) { + tokens.add(match[0].toLowerCase()); + tokens.add(match[1].toLowerCase()); + } + for (const match of query.matchAll(/\b([a-z][a-z0-9-]*(?:-[a-z0-9]+){2,})\b/gi)) { + if (match[1].length >= 12) { + tokens.add(match[1].toLowerCase()); + } + } + return [...tokens]; +} + export interface SearchOptions { useSemanticSearch?: boolean; useKeywordSearch?: boolean; @@ -1941,11 +1959,23 @@ export class CodebaseSearcher { } if (fuseResults.length === 0 || (fuseResults[0].score ?? 0) > KEYWORD_FUSE_OPTIONS.threshold!) { - const rescue = await this.rescueKeywordMatches(query, limit, filters, validation); - const seen = new Set(rescue.map(({ item }) => item.id)); - fuseResults = [...rescue, ...fuseResults.filter(({ item }) => !seen.has(item.id))].sort( - (a, b) => (a.score ?? 0) - (b.score ?? 0) + const pathRescue: FuseResult[] = this.matchChunksByPathTokens(query, filters).map( + (chunk, refIndex) => ({ + item: chunk, + refIndex, + score: 0 + }) ); + const rescue = await this.rescueKeywordMatches(query, limit, filters, validation); + const seen = new Set([ + ...pathRescue.map(({ item }) => item.id), + ...rescue.map(({ item }) => item.id) + ]); + fuseResults = [ + ...pathRescue, + ...rescue, + ...fuseResults.filter(({ item }) => !seen.has(item.id)) + ].sort((a, b) => (a.score ?? 0) - (b.score ?? 0)); } const current = await this.selectCurrentCandidates( fuseResults, @@ -1984,6 +2014,20 @@ export class CodebaseSearcher { score = Math.min(1.0, score + 0.1); } + for (const token of extractQueryPathTokens(query)) { + const base = path.basename(relativePathLower); + const stem = base.replace(/\.[^.]+$/, ''); + if ( + base === token || + relativePathLower.endsWith(`/${token}`) || + stem === token.replace(/\.[^.]+$/, '') + ) { + score = Math.min(1.0, score + 0.45); + } else if (stem.includes(token) || relativePathLower.includes(token)) { + score = Math.min(1.0, score + 0.25); + } + } + return { chunk, score @@ -1996,6 +2040,29 @@ export class CodebaseSearcher { ); } + private matchChunksByPathTokens(query: string, filters?: SearchFilters): CodeChunk[] { + const tokens = extractQueryPathTokens(query); + if (tokens.length === 0) { + return []; + } + return this.chunks.filter((item) => { + if (!this.matchesSearchFilters(item, filters)) { + return false; + } + const relativePathLower = item.relativePath.toLowerCase(); + const base = path.basename(relativePathLower); + const stem = base.replace(/\.[^.]+$/, ''); + return tokens.some( + (token) => + base === token || + relativePathLower.endsWith(`/${token}`) || + stem === token.replace(/\.[^.]+$/, '') || + stem.includes(token) || + relativePathLower.includes(token) + ); + }); + } + private generateRelevanceReason(chunk: CodeChunk, query: string): string { const reasons: string[] = []; diff --git a/src/embeddings/index.ts b/src/embeddings/index.ts index f8c1718..9ca6d9b 100644 --- a/src/embeddings/index.ts +++ b/src/embeddings/index.ts @@ -6,14 +6,17 @@ import { EmbeddingConfig, DEFAULT_EMBEDDING_CONFIG, DEFAULT_MODEL, - parseEmbeddingProviderName + DEFAULT_OLLAMA_MODEL, + parseEmbeddingProviderName, + resolveOllamaHost } from './types.js'; import { TransformersEmbeddingProvider, MODEL_CONFIGS } from './transformers.js'; +import { getOllamaModelDimensions } from './ollama.js'; import { assertOptionalDependency } from '../utils/optional-dependency.js'; export function assertEmbeddingRuntimeAvailable(config: Partial = {}): void { const provider = config.provider ?? DEFAULT_EMBEDDING_CONFIG.provider; - if (provider !== 'openai' && provider !== 'custom') { + if (provider !== 'openai' && provider !== 'custom' && provider !== 'ollama') { assertOptionalDependency('@huggingface/transformers', '^4.2.0'); } } @@ -30,6 +33,7 @@ export function getConfiguredDimensions(config: Partial = {}): config.provider ?? parseEmbeddingProviderName(process.env.EMBEDDING_PROVIDER) ?? 'transformers'; const model = config.model ?? process.env.EMBEDDING_MODEL ?? DEFAULT_MODEL; if (provider === 'openai') return model.includes('large') ? 3072 : 1536; // text-embedding-3-large: 3072, all others: 1536 + if (provider === 'ollama') return getOllamaModelDimensions(model); // Look up from the same MODEL_CONFIGS the provider uses — avoids stale hardcoded guesses return MODEL_CONFIGS[model]?.dimensions ?? 384; } @@ -42,7 +46,12 @@ export async function getEmbeddingProvider( ): Promise { assertEmbeddingRuntimeAvailable(config); const mergedConfig = { ...DEFAULT_EMBEDDING_CONFIG, ...config }; - const providerKey = `${mergedConfig.provider}:${mergedConfig.model}`; + const providerKey = [ + mergedConfig.provider, + mergedConfig.model, + mergedConfig.apiEndpoint ?? '', + mergedConfig.apiKey ?? '' + ].join(':'); if (cachedProvider && cachedProviderType === providerKey) { return cachedProvider; @@ -62,11 +71,19 @@ export async function getEmbeddingProvider( } if (mergedConfig.provider === 'custom') { - throw new Error("Custom provider not implemented. Use 'openai' or 'transformers'."); + throw new Error("Custom provider not implemented. Use 'openai', 'ollama', or 'transformers'."); } if (mergedConfig.provider === 'ollama') { - console.warn('Ollama provider not yet implemented, falling back to Transformers.js'); + const { OllamaEmbeddingProvider } = await import('./ollama.js'); + const provider = new OllamaEmbeddingProvider( + mergedConfig.model || DEFAULT_OLLAMA_MODEL, + mergedConfig.apiEndpoint ?? resolveOllamaHost() + ); + await provider.initialize(); + cachedProvider = provider; + cachedProviderType = providerKey; + return provider; } const provider = new TransformersEmbeddingProvider(mergedConfig.model); diff --git a/src/embeddings/ollama.ts b/src/embeddings/ollama.ts new file mode 100644 index 0000000..1c5d4c3 --- /dev/null +++ b/src/embeddings/ollama.ts @@ -0,0 +1,155 @@ +import { EmbeddingProvider } from './types.js'; + +/** Common Ollama embedding models and their vector sizes (for LanceDB dimension checks). */ +export const OLLAMA_MODEL_DIMENSIONS: Record = { + 'nomic-embed-text': 768, + 'mxbai-embed-large': 1024, + 'all-minilm': 384, + 'snowflake-arctic-embed:137m': 768, + 'snowflake-arctic-embed:335m': 1024 +}; + +interface OllamaEmbedResponse { + embeddings: number[][]; +} + +interface OllamaShowResponse { + modelfile?: string; + parameters?: string; +} + +function normalizeOllamaHost(host: string): string { + return host.replace(/\/+$/, ''); +} + +export function getOllamaModelDimensions(modelName: string): number { + const direct = OLLAMA_MODEL_DIMENSIONS[modelName]; + if (direct !== undefined) { + return direct; + } + const baseName = modelName.split(':')[0] ?? modelName; + const fromBase = OLLAMA_MODEL_DIMENSIONS[baseName]; + if (fromBase !== undefined) { + return fromBase; + } + const fromEnv = process.env.OLLAMA_EMBEDDING_DIMENSIONS; + if (fromEnv) { + const parsed = parseInt(fromEnv, 10); + if (!Number.isNaN(parsed) && parsed > 0) { + return parsed; + } + } + return 768; +} + +/** + * Ollama embedding provider — HTTP client for a local Ollama server (`/api/embed`). + */ +export class OllamaEmbeddingProvider implements EmbeddingProvider { + readonly name = 'ollama'; + private resolvedDimensions: number; + + constructor( + readonly modelName: string, + private host: string = 'http://localhost:11434' + ) { + this.host = normalizeOllamaHost(host); + this.resolvedDimensions = getOllamaModelDimensions(modelName); + } + + get dimensions(): number { + return this.resolvedDimensions; + } + + async initialize(): Promise { + let response: Response; + try { + response = await fetch(`${this.host}/api/show`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ name: this.modelName }) + }); + } catch (error) { + throw new Error( + `Cannot reach Ollama at ${this.host}. Start the server (ollama serve) or set OLLAMA_HOST. ${String(error)}` + ); + } + + if (response.status === 404) { + throw new Error( + `Ollama model "${this.modelName}" is not available. Pull it first: ollama pull ${this.modelName}` + ); + } + + if (!response.ok) { + const error = await response.text(); + throw new Error(`Ollama API Error ${response.status}: ${error}`); + } + + const show = (await response.json()) as OllamaShowResponse; + const fromModelfile = parseEmbeddingLengthFromShow(show); + if (fromModelfile !== undefined) { + this.resolvedDimensions = fromModelfile; + } + } + + isReady(): boolean { + return true; + } + + async embed(text: string): Promise { + const batch = await this.embedBatch([text]); + return batch[0]; + } + + async embedBatch(texts: string[]): Promise { + if (!texts.length) return []; + + try { + const response = await fetch(`${this.host}/api/embed`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + model: this.modelName, + input: texts.length === 1 ? texts[0] : texts + }) + }); + + if (!response.ok) { + const error = await response.text(); + if (response.status === 404 && error.includes('not found')) { + throw new Error( + `Ollama model "${this.modelName}" is not available. Pull it first: ollama pull ${this.modelName}` + ); + } + throw new Error(`Ollama API Error ${response.status}: ${error}`); + } + + const data = (await response.json()) as OllamaEmbedResponse; + if (!Array.isArray(data.embeddings) || data.embeddings.length !== texts.length) { + throw new Error( + `Ollama returned ${data.embeddings?.length ?? 0} embeddings for ${texts.length} inputs` + ); + } + + if (data.embeddings[0]?.length && data.embeddings[0].length !== this.resolvedDimensions) { + this.resolvedDimensions = data.embeddings[0].length; + } + + return data.embeddings; + } catch (error) { + console.error('Ollama Embedding Failed:', error); + throw error; + } + } +} + +function parseEmbeddingLengthFromShow(show: OllamaShowResponse): number | undefined { + const haystack = `${show.modelfile ?? ''}\n${show.parameters ?? ''}`; + const match = haystack.match(/embedding[_\s-]?length[\s=:]+(\d+)/i); + if (!match) { + return undefined; + } + const parsed = parseInt(match[1] ?? '', 10); + return Number.isNaN(parsed) || parsed <= 0 ? undefined : parsed; +} diff --git a/src/embeddings/transformers.ts b/src/embeddings/transformers.ts index bb358d1..db0b2b4 100644 --- a/src/embeddings/transformers.ts +++ b/src/embeddings/transformers.ts @@ -1,6 +1,7 @@ import { EmbeddingProvider, DEFAULT_MODEL } from './types.js'; import type { FeatureExtractionPipeline } from '@huggingface/transformers'; import os from 'os'; +import { assertMinimumOnnxRuntimeNodeVersion } from '../utils/onnxruntime-version.js'; import { assertOptionalDependency } from '../utils/optional-dependency.js'; /** @@ -58,6 +59,7 @@ export class TransformersEmbeddingProvider implements EmbeddingProvider { if (this.initPromise) return this.initPromise; assertOptionalDependency('@huggingface/transformers', '^4.2.0'); + assertMinimumOnnxRuntimeNodeVersion(); this.initPromise = this._initialize(); return this.initPromise; diff --git a/src/embeddings/types.ts b/src/embeddings/types.ts index 66e5a7a..a78433a 100644 --- a/src/embeddings/types.ts +++ b/src/embeddings/types.ts @@ -27,15 +27,72 @@ export function parseEmbeddingProviderName( return undefined; } +export const DEFAULT_OLLAMA_MODEL = 'nomic-embed-text'; +export const DEFAULT_TRANSFORMERS_MODEL = 'Xenova/bge-small-en-v1.5'; +export const DEFAULT_OPENAI_MODEL = 'text-embedding-3-small'; + // Default: bge-small (fast, ~2min indexing, consumer-hardware safe) // Opt-in: set EMBEDDING_MODEL=onnx-community/granite-embedding-small-english-r2-ONNX for // better conceptual search at the cost of 5-10x slower indexing and higher RAM usage -export const DEFAULT_MODEL = process.env.EMBEDDING_MODEL || 'Xenova/bge-small-en-v1.5'; +export const DEFAULT_MODEL = process.env.EMBEDDING_MODEL || DEFAULT_TRANSFORMERS_MODEL; + +const DEFAULT_OPENAI_API_ENDPOINT = 'https://api.openai.com/v1'; +const DEFAULT_OLLAMA_HOST = 'http://localhost:11434'; + +function stripTrailingSlashes(value: string): string { + return value.replace(/\/+$/, ''); +} + +function resolveOpenAiApiEndpoint(): string { + const fromEnv = process.env.OPENAI_BASE_URL?.trim(); + if (!fromEnv) { + return DEFAULT_OPENAI_API_ENDPOINT; + } + return stripTrailingSlashes(fromEnv); +} + +export function resolveOllamaHost(): string { + const fromEnv = process.env.OLLAMA_HOST?.trim(); + if (!fromEnv) { + return DEFAULT_OLLAMA_HOST; + } + return stripTrailingSlashes(fromEnv); +} + +export function resolveEmbeddingModel( + provider: EmbeddingConfig['provider'], + envModel: string | undefined = process.env.EMBEDDING_MODEL +): string { + if (envModel) { + return envModel; + } + if (provider === 'openai') { + return DEFAULT_OPENAI_MODEL; + } + if (provider === 'ollama') { + return DEFAULT_OLLAMA_MODEL; + } + return DEFAULT_TRANSFORMERS_MODEL; +} + +function resolveDefaultApiEndpoint(provider: EmbeddingConfig['provider']): string | undefined { + if (provider === 'openai') { + return resolveOpenAiApiEndpoint(); + } + if (provider === 'ollama') { + return resolveOllamaHost(); + } + return undefined; +} + +const defaultProvider = + parseEmbeddingProviderName(process.env.EMBEDDING_PROVIDER) ?? 'transformers'; export const DEFAULT_EMBEDDING_CONFIG: EmbeddingConfig = { - provider: parseEmbeddingProviderName(process.env.EMBEDDING_PROVIDER) ?? 'transformers', - model: DEFAULT_MODEL, + provider: defaultProvider, + model: resolveEmbeddingModel(defaultProvider), batchSize: 32, maxRetries: 3, - apiKey: process.env.OPENAI_API_KEY + apiKey: process.env.OPENAI_API_KEY, + apiEndpoint: resolveDefaultApiEndpoint(defaultProvider) }; diff --git a/src/index.ts b/src/index.ts index a3bd53b..3373378 100644 --- a/src/index.ts +++ b/src/index.ts @@ -13,7 +13,7 @@ import { fileURLToPath } from 'url'; import type { Server } from '@modelcontextprotocol/sdk/server/index.js'; import type { ProjectConfig } from './server/config.js'; import type { Resource } from '@modelcontextprotocol/sdk/types.js'; -import { CodebaseIndexer } from './core/indexer.js'; +import { CodebaseIndexer, formatIndexingCompleteLine } from './core/indexer.js'; import { analyzerRegistry } from './core/analyzer-registry.js'; import { AngularAnalyzer } from './analyzers/angular/index.js'; @@ -1389,11 +1389,7 @@ async function performIndexingOnce( project.indexState.lastIndexed = new Date(); project.indexState.stats = stats; - console.error( - `Complete: ${stats.indexedFiles} files, ${stats.totalChunks} chunks in ${( - stats.duration / 1000 - ).toFixed(2)}s` - ); + console.error(formatIndexingCompleteLine(stats)); // Auto-extract memories from git history (non-blocking, best-effort) try { diff --git a/src/types/index.ts b/src/types/index.ts index 619b5e9..230e603 100644 --- a/src/types/index.ts +++ b/src/types/index.ts @@ -433,6 +433,10 @@ export interface IndexingStats { indexedFiles: number; skippedFiles: number; totalChunks: number; + /** Chunks written to the keyword index (may be lower than totalChunks when capped). */ + searchableChunks?: number; + /** True when parsing.maxChunks (default 5000) truncated the searchable index. */ + chunkLimitApplied?: boolean; totalLines: number; duration: number; // milliseconds avgChunkSize: number; diff --git a/src/utils/frontend-framework-metadata.ts b/src/utils/frontend-framework-metadata.ts new file mode 100644 index 0000000..69de53b --- /dev/null +++ b/src/utils/frontend-framework-metadata.ts @@ -0,0 +1,98 @@ +import type { FrameworkInfo } from '../types/index.js'; +import { normalizeAnalyzerVersion } from '../analyzers/shared/metadata.js'; + +function hasDep(deps: Record, name: string): boolean { + return Object.prototype.hasOwnProperty.call(deps, name); +} + +/** + * Detect Vue / Nuxt / Svelte / SvelteKit from merged package dependencies. + * Uses the same MIN_INDICATORS=3 gate as indexer.selectFramework. + */ +export function detectFrontendFrameworkFromDeps( + deps: Record +): FrameworkInfo | undefined { + const indicators: string[] = []; + + if (hasDep(deps, 'nuxt')) indicators.push('dep:nuxt'); + if (hasDep(deps, '@nuxt/kit')) indicators.push('dep:@nuxt/kit'); + if (hasDep(deps, '@nuxt/schema')) indicators.push('dep:@nuxt/schema'); + if (hasDep(deps, 'vue')) indicators.push('dep:vue'); + if (hasDep(deps, '@sveltejs/kit')) indicators.push('dep:@sveltejs/kit'); + if (hasDep(deps, '@sveltejs/vite-plugin-svelte')) { + indicators.push('dep:@sveltejs/vite-plugin-svelte'); + } + if (hasDep(deps, 'svelte')) indicators.push('dep:svelte'); + if (hasDep(deps, 'vite')) indicators.push('dep:vite'); + + if (indicators.length < 3) { + return undefined; + } + + if (indicators.includes('dep:nuxt') || indicators.includes('dep:@nuxt/kit')) { + return { + name: 'Nuxt', + version: normalizeAnalyzerVersion(deps.nuxt ?? deps['@nuxt/kit'] ?? 'unknown'), + type: 'vue', + variant: 'nuxt', + uiLibraries: hasDep(deps, 'tailwindcss') ? ['Tailwind'] : undefined, + indicators + }; + } + + if (indicators.includes('dep:@sveltejs/kit')) { + return { + name: 'SvelteKit', + version: normalizeAnalyzerVersion(deps['@sveltejs/kit']), + type: 'svelte', + indicators + }; + } + + if (indicators.includes('dep:vue')) { + return { + name: 'Vue', + version: normalizeAnalyzerVersion(deps.vue), + type: 'vue', + indicators + }; + } + + if (indicators.includes('dep:svelte')) { + return { + name: 'Svelte', + version: normalizeAnalyzerVersion(deps.svelte), + type: 'svelte', + indicators + }; + } + + return undefined; +} + +export function extractSingleFileComponentScript( + content: string, + kind: 'vue' | 'svelte' +): { body: string; language: 'typescript' | 'javascript' } | null { + const tag = 'script'; + const match = content.match(new RegExp(`<${tag}([^>]*)>([\\s\\S]*?)<\\/${tag}>`, 'i')); + if (!match?.[2]) { + return null; + } + + const attrs = match[1] ?? ''; + const body = match[2].trim(); + if (!body) { + return null; + } + + const isTypeScript = + /\blang\s*=\s*['"]ts['"]/i.test(attrs) || + /\blang\s*=\s*['"]typescript['"]/i.test(attrs) || + (kind === 'svelte' && attrs.includes('lang="ts"')); + + return { + body, + language: isTypeScript ? 'typescript' : 'javascript' + }; +} diff --git a/src/utils/language-detection.ts b/src/utils/language-detection.ts index 2cc8bac..5df0fae 100644 --- a/src/utils/language-detection.ts +++ b/src/utils/language-detection.ts @@ -16,6 +16,8 @@ const extensionToLanguage: Record = { '.tsx': 'typescriptreact', '.mts': 'typescript', '.cts': 'typescript', + '.vue': 'vue', + '.svelte': 'svelte', // Web '.html': 'html', @@ -115,6 +117,8 @@ const baseCodeExtensions = new Set([ '.tsx', '.mts', '.cts', + '.vue', + '.svelte', '.html', '.htm', '.css', diff --git a/src/utils/onnxruntime-version.ts b/src/utils/onnxruntime-version.ts new file mode 100644 index 0000000..2e0df77 --- /dev/null +++ b/src/utils/onnxruntime-version.ts @@ -0,0 +1,65 @@ +import { readFileSync } from 'node:fs'; +import { createRequire } from 'node:module'; +import path from 'node:path'; + +/** microsoft/onnxruntime#24579 — mutex crash on macOS in 1.21.x and earlier. */ +export const MIN_ONNXRUNTIME_NODE_VERSION = '1.24.0'; + +const resolve = createRequire(import.meta.url).resolve; + +export function parseSemverTriple(version: string): [number, number, number] { + const match = /^(\d+)\.(\d+)\.(\d+)/.exec(version.trim()); + if (!match) { + return [0, 0, 0]; + } + return [Number(match[1]), Number(match[2]), Number(match[3])]; +} + +export function isSemverAtLeast(actual: string, minimum: string): boolean { + const [aMajor, aMinor, aPatch] = parseSemverTriple(actual); + const [mMajor, mMinor, mPatch] = parseSemverTriple(minimum); + if (aMajor !== mMajor) { + return aMajor > mMajor; + } + if (aMinor !== mMinor) { + return aMinor > mMinor; + } + return aPatch >= mPatch; +} + +function readInstalledOnnxRuntimeVersion(): string | null { + try { + const transformersPkg = resolve('@huggingface/transformers/package.json'); + const onnxPkg = resolve('onnxruntime-node/package.json', { + paths: [path.dirname(transformersPkg)] + }); + const pkg = JSON.parse(readFileSync(onnxPkg, 'utf8')) as { version?: string }; + return typeof pkg.version === 'string' ? pkg.version : null; + } catch { + try { + const onnxPkg = resolve('onnxruntime-node/package.json'); + const pkg = JSON.parse(readFileSync(onnxPkg, 'utf8')) as { version?: string }; + return typeof pkg.version === 'string' ? pkg.version : null; + } catch { + return null; + } + } +} + +/** Fail fast before ONNX embedding/rerank work when the native runtime is too old. */ +export function assertMinimumOnnxRuntimeNodeVersion(minimum = MIN_ONNXRUNTIME_NODE_VERSION): void { + const installed = readInstalledOnnxRuntimeVersion(); + if (!installed) { + return; + } + + if (isSemverAtLeast(installed, minimum)) { + return; + } + + throw new Error( + `onnxruntime-node@${installed} can crash on macOS during embedding (mutex lock failed; see microsoft/onnxruntime#24579). ` + + `Install @huggingface/transformers@^4.2.0 so onnxruntime-node resolves to >=${minimum}, ` + + `or set EMBEDDING_PROVIDER=openai to bypass local Transformers.js.` + ); +} diff --git a/tests/cli-entrypoint-runtime.test.ts b/tests/cli-entrypoint-runtime.test.ts index 3413363..314ad8e 100644 --- a/tests/cli-entrypoint-runtime.test.ts +++ b/tests/cli-entrypoint-runtime.test.ts @@ -142,6 +142,36 @@ describe('CLI entrypoint runtime', () => { await rm(projectRoot, { recursive: true, force: true }); } }, 70_000); + it('dispatches reindex when invoked through an npm-style symlink entrypoint', async () => { + const projectRoot = await mkdtemp(join(tmpdir(), 'cbc-cli-reindex-entrypoint-')); + const tempDirectory = join(tmpdir(), `cbc-cli-bin-${Date.now()}`); + const linkedEntrypoint = join(tempDirectory, 'codebase-context'); + + await writeFile(join(projectRoot, 'example.ts'), 'export const value = 1;\n'); + await mkdir(tempDirectory, { recursive: true }); + symlinkSync(entrypoint, linkedEntrypoint, 'file'); + + try { + const result = spawnSync( + process.execPath, + ['--import', import.meta.resolve('tsx'), linkedEntrypoint, 'reindex', '--no-embeddings'], + { + cwd: projectRoot, + env: { ...process.env }, + encoding: 'utf8', + timeout: 120_000 + } + ); + + expect(result.status, result.stderr).toBe(0); + expect(result.stderr).toContain('Complete:'); + expect(result.stderr).not.toContain('@modelcontextprotocol/sdk/server/stdio.js'); + } finally { + rmSync(tempDirectory, { recursive: true, force: true }); + await rm(projectRoot, { recursive: true, force: true }); + } + }, 120_000); + it('dispatches CLI help when invoked through a symlink entrypoint', (context) => { const tempDirectory = mkdtempSync(join(tmpdir(), 'cbc-cli-entrypoint-')); const linkedEntrypoint = join(tempDirectory, 'codebase-context.ts'); diff --git a/tests/cli-init-dry-run-home.test.ts b/tests/cli-init-dry-run-home.test.ts new file mode 100644 index 0000000..ff95b43 --- /dev/null +++ b/tests/cli-init-dry-run-home.test.ts @@ -0,0 +1,91 @@ +import { describe, it, expect, afterEach, vi } from 'vitest'; +import { promises as fs } from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { handleInitCli } from '../src/cli-init.js'; + +describe('init dry-run and user-scoped configs', () => { + const previousHome = process.env.HOME; + const previousUserProfile = process.env.USERPROFILE; + let tempHome = ''; + let projectRoot = ''; + let logs: string[] = []; + + afterEach(async () => { + process.env.HOME = previousHome; + process.env.USERPROFILE = previousUserProfile; + if (tempHome) await fs.rm(tempHome, { recursive: true, force: true }); + tempHome = ''; + projectRoot = ''; + logs = []; + }); + + async function runInit(args: string[]) { + logs = []; + const logSpy = vi.spyOn(console, 'log').mockImplementation((...parts: unknown[]) => { + logs.push(parts.map(String).join(' ')); + }); + try { + await handleInitCli(args); + } finally { + logSpy.mockRestore(); + } + return logs.join('\n'); + } + + it('dry-run writes Windsurf user config under a temp HOME without touching disk', async () => { + tempHome = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-init-home-')); + projectRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-init-project-')); + process.env.HOME = tempHome; + process.env.USERPROFILE = tempHome; + await fs.mkdir(path.join(tempHome, '.codeium', 'windsurf'), { recursive: true }); + + const output = await runInit([ + '--client', + 'windsurf', + '--yes', + '--dry-run', + '--root', + projectRoot + ]); + + expect(output).toContain(`Would create: ${path.join(tempHome, '.codeium', 'windsurf', 'mcp_config.json')}`); + expect(output).toContain('Dry run complete; no files written.'); + await expect( + fs.access(path.join(tempHome, '.codeium', 'windsurf', 'mcp_config.json')) + ).rejects.toMatchObject({ code: 'ENOENT' }); + }); + + it('merges VS Code servers and reports an update on repeat dry-run', async () => { + tempHome = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-init-home-')); + projectRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'cbc-init-project-')); + process.env.HOME = tempHome; + process.env.USERPROFILE = tempHome; + const vscodeDir = path.join(projectRoot, '.vscode'); + await fs.mkdir(vscodeDir, { recursive: true }); + await fs.writeFile( + path.join(vscodeDir, 'mcp.json'), + JSON.stringify( + { + servers: { + other: { command: 'echo', args: ['hi'] } + } + }, + null, + 2 + ) + ); + + const first = await runInit(['--client', 'vscode', '--yes', '--root', projectRoot]); + expect(first).toContain('Updated: .vscode/mcp.json'); + + const second = await runInit(['--client', 'vscode', '--yes', '--dry-run', '--root', projectRoot]); + expect(second).toContain('Unchanged: .vscode/mcp.json'); + + const saved = JSON.parse(await fs.readFile(path.join(vscodeDir, 'mcp.json'), 'utf8')) as { + servers: Record; + }; + expect(saved.servers.other).toEqual({ command: 'echo', args: ['hi'] }); + expect(saved.servers['codebase-context']).toBeDefined(); + }); +}); diff --git a/tests/cli-init.test.ts b/tests/cli-init.test.ts index bfff804..b397c7f 100644 --- a/tests/cli-init.test.ts +++ b/tests/cli-init.test.ts @@ -120,6 +120,33 @@ describe('generateMcpConfig', () => { command: [nodeExecutable, cacheEntrypoint, projectRoot] }); }); + + it('VS Code writes workspace servers config with the saved launch', () => { + const result = generateMcpConfig('vscode', savedLaunch(projectRoot)); + expect(result.path).toBe(path.join(projectRoot, '.vscode', 'mcp.json')); + expect(JSON.parse(result.content)).toEqual({ + servers: { + 'codebase-context': { + command: nodeExecutable, + args: [cacheEntrypoint, projectRoot] + } + } + }); + }); + + it('Windsurf writes user mcp_config.json with the saved launch', () => { + const home = path.resolve('C:/Users/Test User'); + const result = generateMcpConfig('windsurf', { ...savedLaunch(projectRoot), homeDirectory: home }); + expect(result.path).toBe(path.join(home, '.codeium', 'windsurf', 'mcp_config.json')); + expect(JSON.parse(result.content)).toEqual({ + mcpServers: { + 'codebase-context': { + command: nodeExecutable, + args: [cacheEntrypoint, projectRoot] + } + } + }); + }); }); describe('resolveInitRoot', () => { @@ -382,7 +409,14 @@ describe('detectClients', () => { const clients = await detectClients('/project', { PATH: 'C:/tools' }, 'win32'); - expect(CLIENTS.map(({ id }) => id)).toEqual(['claude-code', 'codex', 'cursor', 'opencode']); + expect(CLIENTS.map(({ id }) => id)).toEqual([ + 'claude-code', + 'codex', + 'cursor', + 'vscode', + 'windsurf', + 'opencode' + ]); expect(clients).toContain('codex'); expect(clients).not.toContain('claude-code'); expect( diff --git a/tests/cli-version.test.ts b/tests/cli-version.test.ts index c4c56f6..be57d93 100644 --- a/tests/cli-version.test.ts +++ b/tests/cli-version.test.ts @@ -1,6 +1,7 @@ import { spawnSync } from 'node:child_process'; -import { readFileSync } from 'node:fs'; -import { resolve } from 'node:path'; +import { mkdtempSync, readFileSync, rmSync, symlinkSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join, resolve } from 'node:path'; import { describe, expect, it } from 'vitest'; const root = resolve(import.meta.dirname, '..'); @@ -23,4 +24,24 @@ describe('version flag', () => { expect(result.status).toBe(0); expect(result.stdout.trim()).toBe(packageJson.version); }); + + it('prints the package version when invoked through an npm-style symlink', () => { + const entrypoint = resolve(root, 'src', 'index.ts'); + const tempDirectory = mkdtempSync(join(tmpdir(), 'cbc-cli-version-bin-')); + const linkedEntrypoint = join(tempDirectory, 'codebase-context'); + + symlinkSync(entrypoint, linkedEntrypoint, 'file'); + try { + const result = spawnSync( + process.execPath, + ['--import', 'tsx', linkedEntrypoint, '--version'], + { cwd: root, encoding: 'utf8', timeout: 30_000 } + ); + + expect(result.status, result.stderr).toBe(0); + expect(result.stdout.trim()).toBe(packageJson.version); + } finally { + rmSync(tempDirectory, { recursive: true, force: true }); + } + }); }); diff --git a/tests/codebase-map.test.ts b/tests/codebase-map.test.ts index 10f5b49..e390c99 100644 --- a/tests/codebase-map.test.ts +++ b/tests/codebase-map.test.ts @@ -126,6 +126,33 @@ describe('buildCodebaseMap', () => { expect(map.architecture.entrypoints).toEqual(['src/cli.ts', 'src/index.ts']); }); + it('excludes vendored raw bundles from bounded map entrypoints and layers', async () => { + const rootPath = await createTempMapProject({ + graph: { + imports: { + 'live-from-ai/raw/index-BMtpUqT3.js': ['live-from-ai/raw/vendor.js'], + 'NOW.md': [], + 'scripts/context-route.mjs': ['NOW.md'] + }, + importedBy: { + 'live-from-ai/raw/index-BMtpUqT3.js': [], + 'NOW.md': ['scripts/context-route.mjs'], + 'scripts/context-route.mjs': [] + }, + exports: {} + } + }); + try { + const project = createProjectState(rootPath); + const map = await buildCodebaseMap(project); + expect(map.architecture.layers.map((layer) => layer.name)).not.toContain('live-from-ai'); + expect(map.architecture.entrypoints.some((ep) => ep.includes('index-BMtpUqT3'))).toBe(false); + expect(map.architecture.entrypoints).toContain('scripts/context-route.mjs'); + } finally { + await removeTempMapProject(rootPath); + } + }); + it('derives hub files: top 5 by importedBy count, sorted count-desc then alpha', async () => { const project = createProjectState(FIXTURE_ROOT); const map = await buildCodebaseMap(project, { mode: 'full' }); diff --git a/tests/embedding-config-env.test.ts b/tests/embedding-config-env.test.ts new file mode 100644 index 0000000..650e304 --- /dev/null +++ b/tests/embedding-config-env.test.ts @@ -0,0 +1,39 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; + +describe('DEFAULT_EMBEDDING_CONFIG environment', () => { + afterEach(() => { + vi.unstubAllEnvs(); + vi.resetModules(); + }); + + it('uses OPENAI_BASE_URL when set', async () => { + vi.stubEnv('EMBEDDING_PROVIDER', 'openai'); + vi.stubEnv('OPENAI_BASE_URL', 'http://127.0.0.1:11434/v1/'); + const { DEFAULT_EMBEDDING_CONFIG } = await import('../src/embeddings/types.js'); + expect(DEFAULT_EMBEDDING_CONFIG.apiEndpoint).toBe('http://127.0.0.1:11434/v1'); + }); + + it('defaults to the official OpenAI endpoint when OPENAI_BASE_URL is unset', async () => { + vi.stubEnv('EMBEDDING_PROVIDER', 'openai'); + vi.unstubAllEnvs(); + vi.stubEnv('EMBEDDING_PROVIDER', 'openai'); + const { DEFAULT_EMBEDDING_CONFIG } = await import('../src/embeddings/types.js'); + expect(DEFAULT_EMBEDDING_CONFIG.apiEndpoint).toBe('https://api.openai.com/v1'); + }); + + it('uses OLLAMA_HOST when provider is ollama', async () => { + vi.stubEnv('EMBEDDING_PROVIDER', 'ollama'); + vi.stubEnv('OLLAMA_HOST', 'http://127.0.0.1:11434/'); + const { DEFAULT_EMBEDDING_CONFIG } = await import('../src/embeddings/types.js'); + expect(DEFAULT_EMBEDDING_CONFIG.provider).toBe('ollama'); + expect(DEFAULT_EMBEDDING_CONFIG.apiEndpoint).toBe('http://127.0.0.1:11434'); + expect(DEFAULT_EMBEDDING_CONFIG.model).toBe('nomic-embed-text'); + }); + + it('resolveEmbeddingModel picks provider-specific defaults', async () => { + const { resolveEmbeddingModel } = await import('../src/embeddings/types.js'); + expect(resolveEmbeddingModel('ollama', undefined)).toBe('nomic-embed-text'); + expect(resolveEmbeddingModel('openai', undefined)).toBe('text-embedding-3-small'); + expect(resolveEmbeddingModel('transformers', undefined)).toBe('Xenova/bge-small-en-v1.5'); + }); +}); diff --git a/tests/embedding-mismatch.test.ts b/tests/embedding-mismatch.test.ts index 67e78b3..6b38ade 100644 --- a/tests/embedding-mismatch.test.ts +++ b/tests/embedding-mismatch.test.ts @@ -95,4 +95,8 @@ describe('getConfiguredDimensions', () => { it('returns 384 as fallback for unknown transformers model', () => { expect(getConfiguredDimensions({ provider: 'transformers', model: 'some/unknown-model' })).toBe(384); }); + + it('returns 768 for nomic-embed-text via ollama provider', () => { + expect(getConfiguredDimensions({ provider: 'ollama', model: 'nomic-embed-text' })).toBe(768); + }); }); diff --git a/tests/embedding-provider-cache.test.ts b/tests/embedding-provider-cache.test.ts new file mode 100644 index 0000000..f4e0228 --- /dev/null +++ b/tests/embedding-provider-cache.test.ts @@ -0,0 +1,39 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; + +describe('getEmbeddingProvider cache key', () => { + afterEach(() => { + vi.resetModules(); + vi.doUnmock('../src/embeddings/openai.js'); + }); + + it('rebuilds the OpenAI provider when apiEndpoint changes', async () => { + const endpoints: string[] = []; + vi.doMock('../src/embeddings/openai.js', () => ({ + OpenAIEmbeddingProvider: class { + constructor(_model: string, _apiKey?: string, apiEndpoint?: string) { + endpoints.push(apiEndpoint ?? ''); + } + + async initialize(): Promise { + return undefined; + } + } + })); + + const { getEmbeddingProvider } = await import('../src/embeddings/index.js'); + await getEmbeddingProvider({ + provider: 'openai', + model: 'text-embedding-3-small', + apiKey: 'sk-test', + apiEndpoint: 'http://127.0.0.1:11434/v1' + }); + await getEmbeddingProvider({ + provider: 'openai', + model: 'text-embedding-3-small', + apiKey: 'sk-test', + apiEndpoint: 'http://127.0.0.1:8080/v1' + }); + + expect(endpoints).toEqual(['http://127.0.0.1:11434/v1', 'http://127.0.0.1:8080/v1']); + }); +}); diff --git a/tests/fixtures/frontend-vue-mini/Button.vue b/tests/fixtures/frontend-vue-mini/Button.vue new file mode 100644 index 0000000..b14d667 --- /dev/null +++ b/tests/fixtures/frontend-vue-mini/Button.vue @@ -0,0 +1,12 @@ + + + diff --git a/tests/fixtures/niche-benchmark-manifest.json b/tests/fixtures/niche-benchmark-manifest.json new file mode 100644 index 0000000..fde60c6 --- /dev/null +++ b/tests/fixtures/niche-benchmark-manifest.json @@ -0,0 +1,204 @@ +{ + "description": "Frozen 18-task niche benchmark across three public repos, two tasks per niche per repo. Expected patterns are path substring checks, not full paths.", + "frozenDate": "2026-10-30", + "cloneRoot": "benchmark-repos", + "repos": [ + { + "id": "angular-spotify", + "repository": "trungk18/angular-spotify", + "repositoryUrl": "https://github.com/trungk18/angular-spotify", + "repositoryRef": "ff9efa765c53cfde78c9a172c62d515ae8ef9fe0", + "language": "typescript", + "sizeClass": "medium", + "sourceFixture": "tests/fixtures/discovery-angular-spotify.json" + }, + { + "id": "excalidraw", + "repository": "excalidraw/excalidraw", + "repositoryUrl": "https://github.com/excalidraw/excalidraw", + "repositoryRef": "e18c1dd213000dde0ae94ef7eb00aab537b39708", + "language": "typescript", + "sizeClass": "large", + "sourceFixture": "tests/fixtures/discovery-excalidraw.json" + }, + { + "id": "fastapi", + "repository": "fastapi/fastapi", + "repositoryUrl": "https://github.com/fastapi/fastapi", + "repositoryRef": "628c34e0cae200564d191c95d7edea78c88c4b5e", + "language": "python", + "sizeClass": "medium", + "sourceFixture": null + } + ], + "tasks": [ + { + "id": "as-team-01", + "repoId": "angular-spotify", + "niche": "team-convention", + "sourceTaskId": "as-find-01", + "query": "What DI pattern dominates here?", + "expectedFilePatterns": ["service", "inject"], + "expectedSignals": ["dependencyInjection"], + "codebaseContext": { "command": "patterns", "args": ["--category", "di", "--json"] } + }, + { + "id": "as-team-02", + "repoId": "angular-spotify", + "niche": "team-convention", + "sourceTaskId": "as-find-02", + "query": "What state-management pattern should an agent imitate?", + "expectedFilePatterns": ["store", "state"], + "expectedSignals": ["stateManagement"], + "codebaseContext": { "command": "patterns", "args": ["--category", "state", "--json"] } + }, + { + "id": "as-ep-01", + "repoId": "angular-spotify", + "niche": "edit-preflight", + "sourceTaskId": "as-ep-01", + "query": "update how authorization token headers are attached to API requests", + "expectedFilePatterns": ["auth", "interceptor"], + "codebaseContext": { "command": "search", "args": ["--intent", "edit", "--limit", "5", "--json"] } + }, + { + "id": "as-ep-02", + "repoId": "angular-spotify", + "niche": "edit-preflight", + "sourceTaskId": "as-ep-02", + "query": "change the logic that skips to the next song", + "expectedFilePatterns": ["player-api", "player/api", "player"], + "codebaseContext": { "command": "search", "args": ["--intent", "edit", "--limit", "5", "--json"] } + }, + { + "id": "as-sym-01", + "repoId": "angular-spotify", + "niche": "symbol-retrieval", + "sourceTaskId": "as-search-01", + "query": "skip to next song", + "expectedFilePatterns": ["player-api", "player/api", "player"], + "codebaseContext": { "command": "search", "args": ["--intent", "explore", "--limit", "5", "--json"] } + }, + { + "id": "as-sym-02", + "repoId": "angular-spotify", + "niche": "symbol-retrieval", + "sourceTaskId": "as-search-03", + "query": "add authorization token to API requests", + "expectedFilePatterns": ["auth", "interceptor"], + "codebaseContext": { "command": "search", "args": ["--intent", "explore", "--limit", "5", "--json"] } + }, + { + "id": "ex-team-01", + "repoId": "excalidraw", + "niche": "team-convention", + "sourceTaskId": "ex-find-01", + "query": "What state-management pattern dominates here?", + "expectedFilePatterns": ["appstate", "state", "store"], + "expectedSignals": ["stateManagement"], + "codebaseContext": { "command": "patterns", "args": ["--category", "state", "--json"] } + }, + { + "id": "ex-team-02", + "repoId": "excalidraw", + "niche": "team-convention", + "sourceTaskId": "ex-find-04", + "query": "What dependency injection or dependency pattern dominates here?", + "expectedFilePatterns": ["component", "app", "src"], + "expectedSignals": ["dependencyInjection"], + "codebaseContext": { "command": "patterns", "args": ["--category", "di", "--json"] } + }, + { + "id": "ex-ep-01", + "repoId": "excalidraw", + "niche": "edit-preflight", + "sourceTaskId": "ex-ep-01", + "query": "change how scene updates are applied", + "expectedFilePatterns": ["scene"], + "codebaseContext": { "command": "search", "args": ["--intent", "edit", "--limit", "5", "--json"] } + }, + { + "id": "ex-ep-02", + "repoId": "excalidraw", + "niche": "edit-preflight", + "sourceTaskId": "ex-ep-03", + "query": "modify scene serialization to json export", + "expectedFilePatterns": ["scene", "json", "data"], + "codebaseContext": { "command": "search", "args": ["--intent", "edit", "--limit", "5", "--json"] } + }, + { + "id": "ex-sym-01", + "repoId": "excalidraw", + "niche": "symbol-retrieval", + "sourceTaskId": "ex-search-01", + "query": "element type definitions", + "expectedFilePatterns": ["element", "type"], + "codebaseContext": { "command": "search", "args": ["--intent", "explore", "--limit", "5", "--json"] } + }, + { + "id": "ex-sym-02", + "repoId": "excalidraw", + "niche": "symbol-retrieval", + "sourceTaskId": "ex-search-02", + "query": "scene serialization export json", + "expectedFilePatterns": ["scene", "json", "data"], + "codebaseContext": { "command": "search", "args": ["--intent", "explore", "--limit", "5", "--json"] } + }, + { + "id": "fa-team-01", + "repoId": "fastapi", + "niche": "team-convention", + "sourceTaskId": null, + "query": "How does FastAPI wire dependency injection for route handlers?", + "expectedFilePatterns": ["param_functions", "depends"], + "expectedSignals": ["Depends", "dependency"], + "codebaseContext": { "command": "patterns", "args": ["--category", "di", "--json"] } + }, + { + "id": "fa-team-02", + "repoId": "fastapi", + "niche": "team-convention", + "sourceTaskId": null, + "query": "Where are API routers defined and included in the application?", + "expectedFilePatterns": ["routing", "applications"], + "expectedSignals": ["APIRouter", "router"], + "codebaseContext": { "command": "patterns", "args": ["--category", "all", "--json"] } + }, + { + "id": "fa-ep-01", + "repoId": "fastapi", + "niche": "edit-preflight", + "sourceTaskId": null, + "query": "change how OAuth2 bearer tokens are extracted from requests", + "expectedFilePatterns": ["oauth2", "security"], + "codebaseContext": { "command": "search", "args": ["--intent", "edit", "--limit", "5", "--json"] } + }, + { + "id": "fa-ep-02", + "repoId": "fastapi", + "niche": "edit-preflight", + "sourceTaskId": null, + "query": "modify OpenAPI schema generation for response models", + "expectedFilePatterns": ["openapi"], + "codebaseContext": { "command": "search", "args": ["--intent", "edit", "--limit", "5", "--json"] } + }, + { + "id": "fa-sym-01", + "repoId": "fastapi", + "niche": "symbol-retrieval", + "sourceTaskId": null, + "query": "validate request body with pydantic field constraints", + "expectedFilePatterns": ["routing", "param"], + "codebaseContext": { "command": "search", "args": ["--intent", "explore", "--limit", "5", "--json"] } + }, + { + "id": "fa-sym-02", + "repoId": "fastapi", + "niche": "symbol-retrieval", + "sourceTaskId": null, + "query": "HTTP exception handler returns JSON error detail", + "expectedFilePatterns": ["exception", "routing"], + "codebaseContext": { "command": "search", "args": ["--intent", "explore", "--limit", "5", "--json"] } + } + ] +} diff --git a/tests/fixtures/niche-benchmark-protocol.json b/tests/fixtures/niche-benchmark-protocol.json new file mode 100644 index 0000000..2490015 --- /dev/null +++ b/tests/fixtures/niche-benchmark-protocol.json @@ -0,0 +1,73 @@ +{ + "name": "niche-prior-art-benchmark", + "frozenDate": "2026-10-30", + "scope": "local-context-retrieval", + "claimAllowed": false, + "purpose": "Reproducible comparison of codebase-context against credited prior art on team-convention lookup, edit-preflight navigation, and symbol retrieval. Not patch quality or agent productivity.", + "niches": ["team-convention", "edit-preflight", "symbol-retrieval"], + "comparators": [ + { + "id": "codebase-context", + "displayName": "Codebase Context", + "kind": "product", + "credit": "This repository (local dist CLI / MCP surfaces).", + "projectUrl": "https://github.com/PatrickSys/codebase-context" + }, + { + "id": "jCodeMunch", + "displayName": "jCodeMunch", + "kind": "mcp-comparator", + "credit": "jCodeMunch MCP symbol search and repo indexing (jcodemunch-mcp on PyPI).", + "projectUrl": "https://github.com/jgravelle/jcodemunch-mcp" + }, + { + "id": "aider-repo-map", + "displayName": "Aider repo map", + "kind": "cli-comparator", + "credit": "Paul Gauthier's Aider RepoMap ranking (aider-chat on PyPI). Query tokens are passed as mentioned identifiers; there is no separate natural-language search API.", + "projectUrl": "https://github.com/Aider-AI/aider" + }, + { + "id": "repomix", + "displayName": "Repomix", + "kind": "cli-comparator", + "credit": "Repomix repository packing (repomix on npm). Repomix does not expose query-specific retrieval; the lane charges pack generation to index time and ranks files by scanning pack sections for query tokens.", + "projectUrl": "https://github.com/yamadashy/repomix" + }, + { + "id": "ripgrep", + "displayName": "ripgrep (lexical baseline)", + "kind": "baseline", + "credit": "BurntSushi ripgrep JSON search over the repo checkout. No index; measures lexical retrieval only.", + "projectUrl": "https://github.com/BurntSushi/ripgrep" + } + ], + "metrics": { + "accuracy": { + "primary": "top3HitRate", + "definition": "Share of tasks where any expectedFilePattern appears in the first three ranked paths returned by the lane." + }, + "tokens": { + "primary": "estimatedTokens", + "definition": "UTF-8 payload bytes divided by four (same heuristic as discovery benchmark)." + }, + "latency": { + "primary": "queryElapsedMs", + "secondary": ["indexElapsedMs", "setupElapsedMs"], + "definition": "Wall time for the lane query step; index and setup are reported separately and amortized per repo where applicable." + } + }, + "fairnessRules": [ + "Task queries and expected patterns are frozen in tests/fixtures/niche-benchmark-manifest.json before benchmark code ships.", + "All lanes receive the same natural-language query string per task.", + "Do not adjust expected patterns after observing lane outputs.", + "Do not add task-specific ranking heuristics to codebase-context for this benchmark.", + "Report setup_failed lanes honestly; do not treat missing competitors as wins.", + "Credit prior art by display name and project URL in public docs." + ], + "blockedClaims": [ + "codebase_context_beats_all_competitors", + "universal_token_savings", + "patch_or_productivity_improvement" + ] +} diff --git a/tests/frontend-spa-indexing.test.ts b/tests/frontend-spa-indexing.test.ts new file mode 100644 index 0000000..ff7c63c --- /dev/null +++ b/tests/frontend-spa-indexing.test.ts @@ -0,0 +1,115 @@ +import { mkdir, rm, writeFile } from 'node:fs/promises'; +import path from 'node:path'; +import { randomUUID } from 'node:crypto'; +import { describe, expect, it } from 'vitest'; +import { isCodeFile, detectLanguage } from '../src/utils/language-detection.js'; +import { GenericAnalyzer } from '../src/analyzers/generic/index.js'; +import { NextJsAnalyzer } from '../src/analyzers/nextjs/index.js'; +import { + detectFrontendFrameworkFromDeps, + extractSingleFileComponentScript +} from '../src/utils/frontend-framework-metadata.js'; + +describe('frontend single-file component support', () => { + it('treats .vue and .svelte as code files', () => { + expect(isCodeFile('src/Button.vue')).toBe(true); + expect(isCodeFile('routes/+page.svelte')).toBe(true); + expect(detectLanguage('App.vue')).toBe('vue'); + expect(detectLanguage('Page.svelte')).toBe('svelte'); + }); + + it('extracts script blocks from vue and svelte files', () => { + const vue = extractSingleFileComponentScript( + '', + 'vue' + ); + expect(vue).toEqual({ body: 'export const x = 1', language: 'typescript' }); + + const svelte = extractSingleFileComponentScript( + '', + 'svelte' + ); + expect(svelte?.language).toBe('typescript'); + }); + + it('indexes vue components through the generic analyzer', async () => { + const fixture = path.join(process.cwd(), 'tests/fixtures/frontend-vue-mini/Button.vue'); + const analyzer = new GenericAnalyzer(); + expect(analyzer.canAnalyze(fixture)).toBe(true); + + const content = await import('node:fs/promises').then((fs) => fs.readFile(fixture, 'utf-8')); + const result = await analyzer.analyze(fixture, content); + expect(result.chunks.length).toBeGreaterThan(0); + const serialized = JSON.stringify(result); + expect(serialized.includes('ButtonProps')).toBe(true); + }); + + it('does not classify SvelteKit +page files or Nuxt app/pages vue as Next.js', () => { + const analyzer = new NextJsAnalyzer(); + expect( + analyzer.canAnalyze( + '/repo/packages/kit/test/+page.server.js', + 'export async function load() { return {}; }' + ) + ).toBe(false); + expect( + analyzer.canAnalyze( + '/repo/playgrounds/nuxt/app/pages/typography.vue', + '' + ) + ).toBe(false); + expect( + analyzer.canAnalyze( + path.join(process.cwd(), 'app', 'dashboard', 'page.tsx'), + 'export default function Page() { return null; }' + ) + ).toBe(true); + }); + + it('detects Nuxt and SvelteKit framework metadata from dependencies', () => { + const nuxt = detectFrontendFrameworkFromDeps({ + nuxt: '^4.0.0', + '@nuxt/kit': 'catalog:', + vue: '^3.5.0' + }); + expect(nuxt?.type).toBe('vue'); + expect(nuxt?.name).toBe('Nuxt'); + + const kit = detectFrontendFrameworkFromDeps({ + '@sveltejs/kit': '^2.0.0', + svelte: '^5.0.0', + vite: '^6.0.0' + }); + expect(kit?.type).toBe('svelte'); + expect(kit?.name).toBe('SvelteKit'); + }); +}); + +describe('frontend fixture indexing smoke', () => { + it('writes vue files into a temp project manifest via glob include patterns', async () => { + const root = path.join(process.cwd(), 'tests/fixtures/frontend-vue-mini'); + const analyzer = new GenericAnalyzer(); + const filePath = path.join(root, 'Button.vue'); + const content = await import('node:fs/promises').then((fs) => fs.readFile(filePath, 'utf-8')); + const result = await analyzer.analyze(filePath, content); + expect(result.language).toBe('vue'); + expect(result.chunks[0]?.metadata.analyzer).toBe('generic'); + }); + + it('creates an isolated vue file for incremental tests', async () => { + const dir = path.join(process.cwd(), 'tests/fixtures/.tmp', randomUUID()); + const filePath = path.join(dir, 'Widget.vue'); + const source = + ''; + await mkdir(dir, { recursive: true }); + try { + await writeFile(filePath, source); + const analyzer = new GenericAnalyzer(); + expect(analyzer.canAnalyze(filePath)).toBe(true); + const result = await analyzer.analyze(filePath, source); + expect(result.chunks.length).toBeGreaterThan(0); + } finally { + await rm(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/tests/indexer-chunk-limit-reporting.test.ts b/tests/indexer-chunk-limit-reporting.test.ts new file mode 100644 index 0000000..0156b0a --- /dev/null +++ b/tests/indexer-chunk-limit-reporting.test.ts @@ -0,0 +1,51 @@ +import { describe, expect, it } from 'vitest'; +import { + formatIndexingCompleteLine, + formatIndexingProgressSummary +} from '../src/core/indexer.js'; +import type { IndexingStats } from '../src/types/index.js'; + +function baseStats(overrides: Partial = {}): IndexingStats { + return { + totalFiles: 100, + indexedFiles: 90, + skippedFiles: 10, + totalChunks: 88424, + searchableChunks: 5000, + chunkLimitApplied: true, + totalLines: 1, + duration: 2500, + avgChunkSize: 100, + componentsByType: {}, + componentsByLayer: { + core: 0, + feature: 0, + shared: 0, + infrastructure: 0, + unknown: 0 + }, + errors: [], + startedAt: new Date(), + ...overrides + }; +} + +describe('indexer chunk limit reporting', () => { + it('formatIndexingProgressSummary distinguishes analyzed vs searchable counts', () => { + const line = formatIndexingProgressSummary(baseStats()); + expect(line).toContain('88424 chunks analyzed'); + expect(line).toContain('5000 chunks searchable'); + }); + + it('formatIndexingCompleteLine shows searchable fraction when capped', () => { + const line = formatIndexingCompleteLine(baseStats()); + expect(line).toBe('Complete: 90 files, 5000/88424 chunks searchable in 2.50s'); + }); + + it('formatIndexingCompleteLine stays compact when no cap applied', () => { + const line = formatIndexingCompleteLine( + baseStats({ totalChunks: 120, searchableChunks: 120, chunkLimitApplied: false }) + ); + expect(line).toBe('Complete: 90 files, 120 chunks in 2.50s'); + }); +}); diff --git a/tests/nextjs-analyzer.test.ts b/tests/nextjs-analyzer.test.ts index 4f6d9e6..1fbb508 100644 --- a/tests/nextjs-analyzer.test.ts +++ b/tests/nextjs-analyzer.test.ts @@ -15,6 +15,20 @@ describe('NextJsAnalyzer', () => { expect(analyzer.canAnalyze('/tmp/file.tsx', 'import React from "react";')).toBe(false); }); + it('ignores SvelteKit route files and Nuxt app/pages vue files', () => { + const analyzer = new NextJsAnalyzer(); + + expect( + analyzer.canAnalyze( + '/repo/routes/+page.server.js', + 'export async function load() { return {}; }' + ) + ).toBe(false); + expect( + analyzer.canAnalyze('/repo/app/pages/home.vue', '') + ).toBe(false); + }); + it('detects app router pages, client components, and metadata exports', async () => { const analyzer = new NextJsAnalyzer(); const filePath = path.join(process.cwd(), 'app', 'settings', 'page.tsx'); diff --git a/tests/niche-benchmark.test.ts b/tests/niche-benchmark.test.ts new file mode 100644 index 0000000..0d92dcf --- /dev/null +++ b/tests/niche-benchmark.test.ts @@ -0,0 +1,86 @@ +import { readFileSync } from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { describe, expect, it } from 'vitest'; +import { aggregateLaneResults, scoreTaskResult } from '../scripts/lib/niche-benchmark/scoring.mjs'; + +const __dirname = path.dirname(fileURLToPath(import.meta.url)); +const manifest = JSON.parse( + readFileSync(path.join(__dirname, 'fixtures', 'niche-benchmark-manifest.json'), 'utf8') +); +const protocol = JSON.parse( + readFileSync(path.join(__dirname, 'fixtures', 'niche-benchmark-protocol.json'), 'utf8') +); + +describe('niche benchmark fixtures', () => { + it('freezes 18 tasks across three repos and three niches', () => { + expect(manifest.tasks).toHaveLength(18); + expect(manifest.repos).toHaveLength(3); + const languages = new Set(manifest.repos.map((repo) => repo.language)); + expect(languages.size).toBeGreaterThanOrEqual(2); + for (const niche of protocol.niches) { + expect(manifest.tasks.filter((task) => task.niche === niche)).toHaveLength(6); + } + }); + + it('lists credited prior-art comparators including ripgrep baseline', () => { + const ids = protocol.comparators.map((comparator) => comparator.id); + expect(ids).toEqual([ + 'codebase-context', + 'jCodeMunch', + 'aider-repo-map', + 'repomix', + 'ripgrep' + ]); + for (const comparator of protocol.comparators) { + expect(comparator.projectUrl).toMatch(/^https:\/\//); + expect(comparator.credit.length).toBeGreaterThan(10); + } + expect(protocol.claimAllowed).toBe(false); + }); +}); + +describe('niche benchmark scoring', () => { + it('marks top-3 pattern hits without tuning scores upward', () => { + const task = manifest.tasks.find((entry) => entry.id === 'as-sym-01'); + expect(task).toBeDefined(); + const hit = scoreTaskResult(task, { + payload: 'player-api next track', + topFiles: ['libs/shared/player-api.ts', 'other.ts'] + }); + expect(hit.top3Hit).toBe(true); + const miss = scoreTaskResult(task, { + payload: 'nothing useful', + topFiles: ['README.md'] + }); + expect(miss.top3Hit).toBe(false); + }); + + it('aggregates lane metrics with failures visible', () => { + const aggregated = aggregateLaneResults([ + { + taskId: 't1', + niche: 'symbol-retrieval', + status: 'completed', + top3Hit: true, + usefulnessScore: 1, + estimatedTokens: 100, + queryElapsedMs: 10, + indexElapsedMs: 50 + }, + { + taskId: 't2', + niche: 'edit-preflight', + status: 'tool_error', + top3Hit: false, + usefulnessScore: 0, + estimatedTokens: 0, + queryElapsedMs: 0, + indexElapsedMs: 0 + } + ]); + expect(aggregated.tasksCompleted).toBe(1); + expect(aggregated.tasksFailed).toBe(1); + expect(aggregated.top3HitRate).toBe(1); + }); +}); diff --git a/tests/ollama-embeddings.test.ts b/tests/ollama-embeddings.test.ts new file mode 100644 index 0000000..c73c889 --- /dev/null +++ b/tests/ollama-embeddings.test.ts @@ -0,0 +1,106 @@ +import { describe, it, expect, vi, afterEach } from 'vitest'; +import { OllamaEmbeddingProvider } from '../src/embeddings/ollama.js'; + +function makeFetchResponse(body: unknown, ok = true, status = 200) { + return Promise.resolve({ + ok, + status, + json: () => Promise.resolve(body), + text: () => Promise.resolve(JSON.stringify(body)) + } as Response); +} + +describe('OllamaEmbeddingProvider', () => { + afterEach(() => { + vi.unstubAllGlobals(); + vi.unstubAllEnvs(); + }); + + it('uses known dimensions for nomic-embed-text', () => { + const provider = new OllamaEmbeddingProvider('nomic-embed-text'); + expect(provider.dimensions).toBe(768); + }); + + it('initialize() throws when the model is missing on the server', async () => { + vi.stubGlobal( + 'fetch', + vi.fn().mockReturnValue(makeFetchResponse({ error: 'not found' }, false, 404)) + ); + + const provider = new OllamaEmbeddingProvider('nomic-embed-text', 'http://127.0.0.1:11434'); + await expect(provider.initialize()).rejects.toThrow('not available'); + }); + + it('initialize() resolves when /api/show succeeds', async () => { + vi.stubGlobal( + 'fetch', + vi.fn().mockReturnValue(makeFetchResponse({ modelfile: 'FROM nomic-embed-text' })) + ); + + const provider = new OllamaEmbeddingProvider('nomic-embed-text'); + await expect(provider.initialize()).resolves.toBeUndefined(); + }); + + it('embedBatch() posts to /api/embed with model and input', async () => { + const mockFetch = vi.fn().mockReturnValue( + makeFetchResponse({ + embeddings: [[0.1, 0.2, 0.3]] + }) + ); + vi.stubGlobal('fetch', mockFetch); + + const provider = new OllamaEmbeddingProvider('nomic-embed-text', 'http://localhost:11434'); + const result = await provider.embedBatch(['hello world']); + + expect(result).toEqual([[0.1, 0.2, 0.3]]); + expect(mockFetch).toHaveBeenCalledOnce(); + const [url, init] = mockFetch.mock.calls[0] as [string, RequestInit]; + expect(url).toBe('http://localhost:11434/api/embed'); + const body = JSON.parse(init.body as string); + expect(body.model).toBe('nomic-embed-text'); + expect(body.input).toBe('hello world'); + }); + + it('embedBatch() sends array input for multiple texts', async () => { + const mockFetch = vi.fn().mockReturnValue( + makeFetchResponse({ + embeddings: [ + [0.1, 0.2], + [0.3, 0.4] + ] + }) + ); + vi.stubGlobal('fetch', mockFetch); + + const provider = new OllamaEmbeddingProvider('nomic-embed-text'); + await provider.embedBatch(['a', 'b']); + + const body = JSON.parse((mockFetch.mock.calls[0] as [string, RequestInit])[1].body as string); + expect(body.input).toEqual(['a', 'b']); + }); + + it('embedBatch() returns [] for empty input', async () => { + const provider = new OllamaEmbeddingProvider('nomic-embed-text'); + await expect(provider.embedBatch([])).resolves.toEqual([]); + }); + + it('embedBatch() throws on non-ok API response', async () => { + vi.stubGlobal( + 'fetch', + vi.fn().mockReturnValue(makeFetchResponse({ error: 'server error' }, false, 500)) + ); + + const provider = new OllamaEmbeddingProvider('nomic-embed-text'); + await expect(provider.embedBatch(['test'])).rejects.toThrow('500'); + }); + + it('embed() delegates to embedBatch', async () => { + vi.stubGlobal( + 'fetch', + vi.fn().mockReturnValue(makeFetchResponse({ embeddings: [[0.5, 0.6]] })) + ); + + const provider = new OllamaEmbeddingProvider('nomic-embed-text'); + await expect(provider.embed('hello')).resolves.toEqual([0.5, 0.6]); + }); +}); diff --git a/tests/onnxruntime-version.test.ts b/tests/onnxruntime-version.test.ts new file mode 100644 index 0000000..f3ae407 --- /dev/null +++ b/tests/onnxruntime-version.test.ts @@ -0,0 +1,47 @@ +import { readFileSync } from 'node:fs'; +import { resolve } from 'node:path'; +import { describe, expect, it } from 'vitest'; + +import { + MIN_ONNXRUNTIME_NODE_VERSION, + assertMinimumOnnxRuntimeNodeVersion, + isSemverAtLeast, + parseSemverTriple +} from '../src/utils/onnxruntime-version.js'; + +const root = resolve(import.meta.dirname, '..'); + +type PackageJson = { + peerDependencies?: Record; + overrides?: Record; + pnpm?: { overrides?: Record }; +}; + +describe('onnxruntime-node macOS mutex regression (#68)', () => { + const packageJson = JSON.parse(readFileSync(resolve(root, 'package.json'), 'utf8')) as PackageJson; + const lockfile = readFileSync(resolve(root, 'pnpm-lock.yaml'), 'utf8'); + + it('requires Transformers.js v4 peers (ships onnxruntime-node 1.24.x)', () => { + expect(packageJson.peerDependencies?.['@huggingface/transformers']).toBe('^4.2.0'); + }); + + it('pins onnxruntime-node 1.24.3 for npm and pnpm installs', () => { + expect(packageJson.overrides?.['@huggingface/transformers>onnxruntime-node']).toBe('1.24.3'); + expect(packageJson.pnpm?.overrides?.['@huggingface/transformers>onnxruntime-node']).toBe('1.24.3'); + }); + + it('locks onnxruntime-node 1.24.3 in pnpm-lock.yaml', () => { + expect(lockfile).toContain("'@huggingface/transformers>onnxruntime-node': 1.24.3"); + expect(lockfile).toMatch(/onnxruntime-node@1\.24\.3/); + }); + + it('compares semver triples for the minimum safe runtime', () => { + expect(parseSemverTriple('1.21.0')).toEqual([1, 21, 0]); + expect(isSemverAtLeast('1.24.3', MIN_ONNXRUNTIME_NODE_VERSION)).toBe(true); + expect(isSemverAtLeast('1.21.0', MIN_ONNXRUNTIME_NODE_VERSION)).toBe(false); + }); + + it('accepts onnxruntime-node 1.24.x when installed with Transformers.js', () => { + expect(() => assertMinimumOnnxRuntimeNodeVersion()).not.toThrow(); + }); +}); diff --git a/tests/openai-embeddings.test.ts b/tests/openai-embeddings.test.ts index 8763efd..4e722a2 100644 --- a/tests/openai-embeddings.test.ts +++ b/tests/openai-embeddings.test.ts @@ -46,6 +46,25 @@ describe('OpenAIEmbeddingProvider', () => { expect(result).toEqual([]); }); + it('embedBatch() uses custom apiEndpoint base URL', async () => { + const mockFetch = vi.fn().mockReturnValue( + makeFetchResponse({ + data: [{ embedding: [0.1] }] + }) + ); + vi.stubGlobal('fetch', mockFetch); + + const provider = new OpenAIEmbeddingProvider( + 'text-embedding-3-small', + 'sk-test', + 'http://127.0.0.1:11434/v1' + ); + await provider.embedBatch(['hello']); + + const [url] = mockFetch.mock.calls[0] as [string, RequestInit]; + expect(url).toBe('http://127.0.0.1:11434/v1/embeddings'); + }); + it('embedBatch() sends correct Authorization header, model, and encoding_format', async () => { const mockFetch = vi.fn().mockReturnValue( makeFetchResponse({ diff --git a/tests/search-path-token-boost.test.ts b/tests/search-path-token-boost.test.ts new file mode 100644 index 0000000..74a0ff2 --- /dev/null +++ b/tests/search-path-token-boost.test.ts @@ -0,0 +1,23 @@ +import { describe, expect, it } from 'vitest'; +import { extractQueryPathTokens } from '../src/core/search.js'; + +describe('extractQueryPathTokens', () => { + it('extracts markdown filenames from natural-language queries', () => { + const tokens = extractQueryPathTokens( + 'NOW.md live root entrypoint current state for routing' + ); + expect(tokens).toContain('now.md'); + }); + + it('extracts script paths and hyphenated stems', () => { + const tokens = extractQueryPathTokens( + 'what must I check before editing AGENTS.md protected agent instructions' + ); + expect(tokens).toContain('agents.md'); + }); + + it('extracts explicit script module names', () => { + const tokens = extractQueryPathTokens('run check-agent-instructions before commit'); + expect(tokens).toContain('check-agent-instructions'); + }); +});