Repository navigation
fix: issue triage quick wins (chunk cap, map, path search) #154
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -39,6 +39,24 @@ import { | |
| VECTOR_DB_DIRNAME | ||
| } from '../constants/codebase-context.js'; | ||
|
|
||
| /** File/path-like tokens in a natural-language query (e.g. NOW.md, check-agent-instructions.mjs). */ | ||
| export function extractQueryPathTokens(query: string): string[] { | ||
| const tokens = new Set<string>(); | ||
| for (const match of query.matchAll(/\b([A-Za-z0-9][A-Za-z0-9._-]*\.(?:md|mdx|txt))\b/g)) { | ||
| tokens.add(match[1].toLowerCase()); | ||
| } | ||
| for (const match of query.matchAll(/\b([a-z][a-z0-9-]*(?:-[a-z0-9]+)+)\.(?:mjs|cjs|js|ts)\b/gi)) { | ||
| tokens.add(match[0].toLowerCase()); | ||
| tokens.add(match[1].toLowerCase()); | ||
| } | ||
| for (const match of query.matchAll(/\b([a-z][a-z0-9-]*(?:-[a-z0-9]+){2,})\b/gi)) { | ||
| if (match[1].length >= 12) { | ||
| tokens.add(match[1].toLowerCase()); | ||
| } | ||
| } | ||
| return [...tokens]; | ||
| } | ||
|
|
||
| export interface SearchOptions { | ||
| useSemanticSearch?: boolean; | ||
| useKeywordSearch?: boolean; | ||
|
|
@@ -1941,11 +1959,23 @@ export class CodebaseSearcher { | |
| } | ||
|
|
||
| if (fuseResults.length === 0 || (fuseResults[0].score ?? 0) > KEYWORD_FUSE_OPTIONS.threshold!) { | ||
| const pathRescue: FuseResult<CodeChunk>[] = this.matchChunksByPathTokens(query, filters).map( | ||
| (chunk, refIndex) => ({ | ||
| item: chunk, | ||
| refIndex, | ||
| score: 0 | ||
| }) | ||
|
Comment on lines
+1962
to
+1967
Contributor
There was a problem hiding this comment.
Every path match gets |
||
| ); | ||
| const rescue = await this.rescueKeywordMatches(query, limit, filters, validation); | ||
| const seen = new Set([ | ||
| ...pathRescue.map(({ item }) => item.id), | ||
| ...rescue.map(({ item }) => item.id) | ||
| ]); | ||
| fuseResults = [ | ||
| ...pathRescue, | ||
| ...rescue, | ||
| ...fuseResults.filter(({ item }) => !seen.has(item.id)) | ||
|
Comment on lines
+1974
to
+1977
There was a problem hiding this comment.
When a filename-bearing query triggers rescue and a chunk matches both its path token and the lexical rescue criteria, that chunk appears in both Useful? React with 👍 / 👎. |
||
| ].sort((a, b) => (a.score ?? 0) - (b.score ?? 0)); | ||
|
Comment on lines
+1974
to
+1978
Contributor
There was a problem hiding this comment.
|
||
| } | ||
| const current = await this.selectCurrentCandidates( | ||
| fuseResults, | ||
|
|
@@ -1984,6 +2014,20 @@ export class CodebaseSearcher { | |
| score = Math.min(1.0, score + 0.1); | ||
| } | ||
|
|
||
| for (const token of extractQueryPathTokens(query)) { | ||
| const base = path.basename(relativePathLower); | ||
| const stem = base.replace(/\.[^.]+$/, ''); | ||
| if ( | ||
| base === token || | ||
| relativePathLower.endsWith(`/${token}`) || | ||
| stem === token.replace(/\.[^.]+$/, '') | ||
| ) { | ||
| score = Math.min(1.0, score + 0.45); | ||
| } else if (stem.includes(token) || relativePathLower.includes(token)) { | ||
| score = Math.min(1.0, score + 0.25); | ||
|
Comment on lines
+2017
to
+2027
Contributor
There was a problem hiding this comment.
The new ranking adds Context Used: AGENTS.md (source) Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time! |
||
| } | ||
| } | ||
|
|
||
| return { | ||
| chunk, | ||
| score | ||
|
|
@@ -1996,6 +2040,29 @@ export class CodebaseSearcher { | |
| ); | ||
| } | ||
|
|
||
| private matchChunksByPathTokens(query: string, filters?: SearchFilters): CodeChunk[] { | ||
| const tokens = extractQueryPathTokens(query); | ||
| if (tokens.length === 0) { | ||
| return []; | ||
| } | ||
| return this.chunks.filter((item) => { | ||
| if (!this.matchesSearchFilters(item, filters)) { | ||
| return false; | ||
| } | ||
| const relativePathLower = item.relativePath.toLowerCase(); | ||
| const base = path.basename(relativePathLower); | ||
| const stem = base.replace(/\.[^.]+$/, ''); | ||
| return tokens.some( | ||
| (token) => | ||
| base === token || | ||
| relativePathLower.endsWith(`/${token}`) || | ||
| stem === token.replace(/\.[^.]+$/, '') || | ||
| stem.includes(token) || | ||
| relativePathLower.includes(token) | ||
| ); | ||
| }); | ||
| } | ||
|
|
||
| private generateRelevanceReason(chunk: CodeChunk, query: string): string { | ||
| const reasons: string[] = []; | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| import { describe, expect, it } from 'vitest'; | ||
| import { | ||
| formatIndexingCompleteLine, | ||
| formatIndexingProgressSummary | ||
| } from '../src/core/indexer.js'; | ||
| import type { IndexingStats } from '../src/types/index.js'; | ||
|
|
||
| function baseStats(overrides: Partial<IndexingStats> = {}): IndexingStats { | ||
| return { | ||
| totalFiles: 100, | ||
| indexedFiles: 90, | ||
| skippedFiles: 10, | ||
| totalChunks: 88424, | ||
| searchableChunks: 5000, | ||
| chunkLimitApplied: true, | ||
| totalLines: 1, | ||
| duration: 2500, | ||
| avgChunkSize: 100, | ||
| componentsByType: {}, | ||
| componentsByLayer: { | ||
| core: 0, | ||
| feature: 0, | ||
| shared: 0, | ||
| infrastructure: 0, | ||
| unknown: 0 | ||
| }, | ||
| errors: [], | ||
| startedAt: new Date(), | ||
| ...overrides | ||
| }; | ||
| } | ||
|
|
||
| describe('indexer chunk limit reporting', () => { | ||
| it('formatIndexingProgressSummary distinguishes analyzed vs searchable counts', () => { | ||
| const line = formatIndexingProgressSummary(baseStats()); | ||
| expect(line).toContain('88424 chunks analyzed'); | ||
| expect(line).toContain('5000 chunks searchable'); | ||
| }); | ||
|
|
||
| it('formatIndexingCompleteLine shows searchable fraction when capped', () => { | ||
| const line = formatIndexingCompleteLine(baseStats()); | ||
| expect(line).toBe('Complete: 90 files, 5000/88424 chunks searchable in 2.50s'); | ||
| }); | ||
|
|
||
| it('formatIndexingCompleteLine stays compact when no cap applied', () => { | ||
| const line = formatIndexingCompleteLine( | ||
| baseStats({ totalChunks: 120, searchableChunks: 120, chunkLimitApplied: false }) | ||
| ); | ||
| expect(line).toBe('Complete: 90 files, 120 chunks in 2.50s'); | ||
| }); | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| import { describe, expect, it } from 'vitest'; | ||
| import { extractQueryPathTokens } from '../src/core/search.js'; | ||
|
|
||
| describe('extractQueryPathTokens', () => { | ||
| it('extracts markdown filenames from natural-language queries', () => { | ||
| const tokens = extractQueryPathTokens( | ||
| 'NOW.md live root entrypoint current state for routing' | ||
| ); | ||
| expect(tokens).toContain('now.md'); | ||
| }); | ||
|
|
||
| it('extracts script paths and hyphenated stems', () => { | ||
| const tokens = extractQueryPathTokens( | ||
| 'what must I check before editing AGENTS.md protected agent instructions' | ||
| ); | ||
| expect(tokens).toContain('agents.md'); | ||
| }); | ||
|
|
||
| it('extracts explicit script module names', () => { | ||
| const tokens = extractQueryPathTokens('run check-agent-instructions before commit'); | ||
| expect(tokens).toContain('check-agent-instructions'); | ||
| }); | ||
| }); |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removing the unreleased label presents
projects[].parsing.maxChunksas available to npm users. The guide says unmarked features apply to the published package, but thev2.3.0config reader does not readparsing; this option was added later. Keep an unreleased or built-checkout qualifier here and inREADME.md. The repository requires public docs not to claim capabilities that are not shipped, so this must be corrected before merging.Context Used: AGENTS.md (source)