Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions docs/benchmark.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,38 @@
# What our local code-context experiments show

## Front-end repo matrix (plan)

### Now

- Ship `.vue` / `.svelte` indexing, Nuxt/SvelteKit framework claims, and tighter Next.js route detection (SvelteKit `+` routes, Nuxt `app/pages/*.vue`).
- Re-run the five-repo smoke matrix after each fix; keep this section honest.

### Next

- Dedicated Vue/Nuxt and Svelte analyzers (patterns, route conventions) instead of generic-only SPA handling.
- Path-alias coverage for Vite/Nuxt `~/` and monorepo package exports beyond TypeScript `paths`.
- Markdown-heavy repos (docs sites) as a separate benchmark series.

### Done

- 2026-10-30: Cloned and exercised five OSS front-end repos (see table below); fixed SPA file gaps and Next.js false positives; added fixture tests in `tests/frontend-spa-indexing.test.ts`.

### Five-repo smoke (2026-10-30, `reindex --no-embeddings`, Node 22, this branch)

Clones: [shadcn-ui/taxonomy](https://github.com/shadcn-ui/taxonomy) (Next.js), [nuxt/ui](https://github.com/nuxt/ui), [angular/components](https://github.com/angular/components), [sveltejs/kit](https://github.com/sveltejs/kit), [vercel/turbo](https://github.com/vercel/turbo) (pnpm/turbo monorepo with UI apps).

| Repo | Init / reindex / status | Framework metadata | SPA sources in index | Symbol refs | Edit preflight (realistic query) |
| --- | --- | --- | --- | --- | --- |
| taxonomy (Next.js) | pass / pass / ready | Next.js detected | 94 `.tsx` indexed | `cn` useful (241 hits) | not ready — low confidence; top hit was relevant layout |
| nuxt/ui | pass / pass / ready | Nuxt detected (was missing) | **816** `.vue` indexed (was **0**) | `UButton` finds `Button.vue` imports | not ready — low confidence; stale-source hint on some queries |
| angular/components | pass / pass / ready | Angular patterns | TS/Angular only | `MatDialog` static refs OK | not ready — low confidence; dialog hits sensible |
| sveltejs/kit | pass / pass / ready | SvelteKit detected (monorepo deps) | **1011** `.svelte` indexed (was **0**) | `load` very noisy (587 lexical) | not ready — no longer mislabels SvelteKit tests as Next.js pages |
| vercel/turbo | pass / pass / ready; chunk cap warning | monorepo packages listed | UI via Next apps in `examples/` | `Button` finds UI components | not ready — low confidence; 5k chunk cap drops most of repo |

Known limits surfaced here (not fixed in this pass): default 5k chunk cap on very large repos, lexical `load`/`Button` ambiguity, edit preflight still conservative on broad natural-language queries, one Rust-generated TS parse warning in turbo.

---

This report describes local evidence from five ways of giving an AI coding agent context: Raw-native, Codebase Context, jCodeMunch, Repowise, and context-mode. The evidence is split into separate families because retrieval, repeated runs, full-agent observations, and a metered replay do not answer the same question. The families are not pooled into one score.

The clearest current finding is narrow: in this fixed retrieval setup, Codebase Context recovered 25.7% of expected gold files at 11.5% file precision, while fresh indexing had material local cost. These measurements do not show better completed coding tasks.
Expand Down
34 changes: 31 additions & 3 deletions src/analyzers/generic/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,10 @@ import {
categorizeDependency
} from '../../utils/dependency-detection.js';
import type { WorkspacePackageJson } from '../../utils/workspace-detection.js';
import {
detectFrontendFrameworkFromDeps,
extractSingleFileComponentScript
} from '../../utils/frontend-framework-metadata.js';

export class GenericAnalyzer implements FrameworkAnalyzer {
readonly name = 'generic';
Expand Down Expand Up @@ -91,7 +95,9 @@ export class GenericAnalyzer implements FrameworkAnalyzer {
'.css',
'.scss',
'.sass',
'.less'
'.less',
'.vue',
'.svelte'
];
readonly priority = 10; // Low priority - fallback analyzer

Expand Down Expand Up @@ -126,8 +132,23 @@ export class GenericAnalyzer implements FrameworkAnalyzer {
usesTreeSitterSymbols = true;
}

if (language === 'typescript' || language === 'javascript') {
const parsed = await this.parseJSTSFile(filePath, content, language);
let scriptLanguage: 'typescript' | 'javascript' | null = null;
let scriptBody: string | null = null;
if (language === 'vue' || language === 'svelte') {
const extracted = extractSingleFileComponentScript(content, language);
if (extracted) {
scriptBody = extracted.body;
scriptLanguage = extracted.language;
}
}

const jstsLanguage =
scriptLanguage ??
(language === 'typescript' || language === 'javascript' ? language : null);
const jstsContent = scriptBody ?? content;

if (jstsLanguage) {
const parsed = await this.parseJSTSFile(filePath, jstsContent, jstsLanguage);
Comment on lines +148 to +151

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve source line offsets when parsing extracted scripts

When a Vue or Svelte script follows a template or other leading content, parsing the extracted, trimmed body makes all component and import locations relative to the script rather than the original file. Those components are then passed to createChunksFromCode alongside the full source, so its component-boundary slicing selects unrelated lines. For example, a function on source line 24 becomes a component on line 1, producing a chunk of template lines that omits the function entirely. Preserve the original line positions during extraction or translate the parsed locations before using them for chunking.

Useful? React with 👍 / 👎.

Comment on lines +148 to +151

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Chunks show the wrong code

Parsing the trimmed script resets source line numbers, but createChunksFromCode still slices the full component file. In a Vue file with a long template before its script, a function near the start of the script gets a chunk containing template lines instead of the function. Import locations are shifted too.

Preserve the original line positions when extracting scripts, or map parsed positions back before using them.

imports = parsed.imports;
exports = parsed.exports;

Expand Down Expand Up @@ -247,6 +268,7 @@ export class GenericAnalyzer implements FrameworkAnalyzer {

let workspaceType: string = 'single';
let workspacePackages: WorkspacePackageJson[] = [];
let frameworkClaim: CodebaseMetadata['framework'];

try {
workspaceType = await detectWorkspaceType(rootPath);
Expand Down Expand Up @@ -276,6 +298,11 @@ export class GenericAnalyzer implements FrameworkAnalyzer {
version: version as string,
category: categorizeDependency(name)
}));

const framework = detectFrontendFrameworkFromDeps(rawDeps);
if (framework) {
frameworkClaim = framework;
}
} catch (_error) {
// skip
}
Expand All @@ -285,6 +312,7 @@ export class GenericAnalyzer implements FrameworkAnalyzer {
rootPath,
languages: [],
dependencies,
framework: frameworkClaim,
architecture: {
type: 'mixed',
layers: {
Expand Down
26 changes: 25 additions & 1 deletion src/analyzers/nextjs/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,19 @@ export class NextJsAnalyzer implements FrameworkAnalyzer {
return false;
}

if (isInAppRouter(filePath) || isInPagesRouter(filePath) || isProxyFile(filePath)) {
if (isSvelteKitRouteFile(filePath)) {
return false;
}

if (isProxyFile(filePath)) {
return true;
}

if (matchesNextAppRouterConvention(filePath)) {
return true;
}

if (isInPagesRouter(filePath)) {
return true;
}

Expand Down Expand Up @@ -342,6 +354,18 @@ function analyzeRouting(filePath: string, content: string): NextRoutingInfo {
};
}

function isSvelteKitRouteFile(filePath: string): boolean {
const baseName = path.basename(filePath, path.extname(filePath));
return baseName.startsWith('+');
}

function matchesNextAppRouterConvention(filePath: string): boolean {
if (!isInAppRouter(filePath)) {
return false;
}
return getAppRouteFileKind(path.basename(filePath)) !== null;
}
Comment on lines +362 to +367

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Metadata routes lose Next.js labels

The narrower filename check excludes real Next.js files such as app/robots.ts and app/sitemap.ts when they have no next import. Neither name appears in APP_ROUTE_FILE_KINDS, so these files lose their Next.js label and disappear from searches filtered to nextjs.

Include Next.js metadata filenames in the convention check while keeping the SvelteKit exclusion.


function isInAppRouter(filePath: string): boolean {
const normalizedPath = filePath.replace(/\\/g, '/');
return normalizedPath.includes('/app/') || normalizedPath.includes('/src/app/');
Expand Down
1 change: 1 addition & 0 deletions src/core/indexer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -279,6 +279,7 @@ export class CodebaseIndexer {
},
include: [
'**/*.{ts,tsx,js,jsx,mjs,cjs,mts,cts}',
'**/*.{vue,svelte}',
'**/*.{html,htm,css,scss,sass,less}',
'**/*.{py,pyi,rb,php}',
'**/*.{java,kt,kts,scala,swift,cs}',
Expand Down
98 changes: 98 additions & 0 deletions src/utils/frontend-framework-metadata.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
import type { FrameworkInfo } from '../types/index.js';
import { normalizeAnalyzerVersion } from '../analyzers/shared/metadata.js';

function hasDep(deps: Record<string, string>, name: string): boolean {
return Object.prototype.hasOwnProperty.call(deps, name);
}

/**
* Detect Vue / Nuxt / Svelte / SvelteKit from merged package dependencies.
* Uses the same MIN_INDICATORS=3 gate as indexer.selectFramework.
*/
export function detectFrontendFrameworkFromDeps(
deps: Record<string, string>
): FrameworkInfo | undefined {
const indicators: string[] = [];

if (hasDep(deps, 'nuxt')) indicators.push('dep:nuxt');
if (hasDep(deps, '@nuxt/kit')) indicators.push('dep:@nuxt/kit');
if (hasDep(deps, '@nuxt/schema')) indicators.push('dep:@nuxt/schema');
if (hasDep(deps, 'vue')) indicators.push('dep:vue');
if (hasDep(deps, '@sveltejs/kit')) indicators.push('dep:@sveltejs/kit');
if (hasDep(deps, '@sveltejs/vite-plugin-svelte')) {
indicators.push('dep:@sveltejs/vite-plugin-svelte');
}
if (hasDep(deps, 'svelte')) indicators.push('dep:svelte');
if (hasDep(deps, 'vite')) indicators.push('dep:vite');

if (indicators.length < 3) {
return undefined;
}

if (indicators.includes('dep:nuxt') || indicators.includes('dep:@nuxt/kit')) {
return {
name: 'Nuxt',
version: normalizeAnalyzerVersion(deps.nuxt ?? deps['@nuxt/kit'] ?? 'unknown'),
type: 'vue',
variant: 'nuxt',
uiLibraries: hasDep(deps, 'tailwindcss') ? ['Tailwind'] : undefined,
indicators
};
}

if (indicators.includes('dep:@sveltejs/kit')) {
return {
name: 'SvelteKit',
version: normalizeAnalyzerVersion(deps['@sveltejs/kit']),
type: 'svelte',
indicators
};
}

if (indicators.includes('dep:vue')) {
return {
name: 'Vue',
version: normalizeAnalyzerVersion(deps.vue),
type: 'vue',
indicators
};
}

if (indicators.includes('dep:svelte')) {
return {
name: 'Svelte',
version: normalizeAnalyzerVersion(deps.svelte),
type: 'svelte',
indicators
};
}

return undefined;
}

export function extractSingleFileComponentScript(
content: string,
kind: 'vue' | 'svelte'
): { body: string; language: 'typescript' | 'javascript' } | null {
const tag = 'script';
const match = content.match(new RegExp(`<${tag}([^>]*)>([\\s\\S]*?)<\\/${tag}>`, 'i'));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Extract both script blocks from single-file components

For components containing two script blocks, content.match extracts only the first and silently ignores the second. This affects Vue components combining a normal script with <script setup> and Svelte components combining module and instance scripts. If the first block contains module exports and the second imports child components, GenericAnalyzer.analyze never records those imports or the second block's declarations, leaving dependency metadata incomplete. Process all supported script blocks, preserving each block's source locations.

Useful? React with 👍 / 👎.

if (!match?.[2]) {
return null;
Comment on lines +78 to +80

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Second scripts disappear

The single regex match reads only the first script block. Vue files can contain both <script> and <script setup>, and Svelte files can contain module and instance scripts. Imports, exports, and declarations in the second block are never parsed, so they are missing from the analysis.

Extract and analyze every script block, keeping each block's language and original positions.

}

const attrs = match[1] ?? '';
const body = match[2].trim();
if (!body) {
return null;
}

const isTypeScript =
/\blang\s*=\s*['"]ts['"]/i.test(attrs) ||
/\blang\s*=\s*['"]typescript['"]/i.test(attrs) ||
(kind === 'svelte' && attrs.includes('lang="ts"'));

return {
body,
language: isTypeScript ? 'typescript' : 'javascript'
};
}
4 changes: 4 additions & 0 deletions src/utils/language-detection.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ const extensionToLanguage: Record<string, string> = {
'.tsx': 'typescriptreact',
'.mts': 'typescript',
'.cts': 'typescript',
'.vue': 'vue',
'.svelte': 'svelte',

// Web
'.html': 'html',
Expand Down Expand Up @@ -115,6 +117,8 @@ const baseCodeExtensions = new Set([
'.tsx',
'.mts',
'.cts',
'.vue',
'.svelte',
'.html',
'.htm',
'.css',
Expand Down
12 changes: 12 additions & 0 deletions tests/fixtures/frontend-vue-mini/Button.vue
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<script setup lang="ts">
export interface ButtonProps {
label: string;
variant?: 'primary' | 'ghost';
}

const props = defineProps<ButtonProps>();
</script>

<template>
<button type="button" :data-variant="props.variant">{{ props.label }}</button>
</template>
Loading
Loading