Reproducible niche prior-art benchmark (Codebase Context vs jCodeMunch, Aider, Repomix, ripgrep) by PatrickSys · Pull Request #156 · PatrickSys/codebase-context

PatrickSys · 2026-10-30T22:41:59Z

Adds a frozen 18-task benchmark across angular-spotify, Excalidraw, and FastAPI covering team-convention lookup, edit-preflight navigation, and symbol retrieval.

Runner: npm run benchmark:niche (scripts/run-niche-benchmark.mjs) clones pinned repos, runs five credited lanes, writes results/niche-benchmark-evidence.json.
Fixtures: tests/fixtures/niche-benchmark-manifest.json and niche-benchmark-protocol.json (frozen before this run; no post-hoc expected-pattern edits).
Docs: docs/benchmark.md updated with aggregate numbers, rerun instructions, and explicit places Codebase Context loses (team-convention file ranking, latency vs ripgrep, payload size vs Aider/jCodeMunch).
Tests: tests/niche-benchmark.test.ts guards fixture shape and scoring aggregation.

Headline results from the checked-in evidence (2026-10-30): Codebase Context 50% top-3 hit rate vs Aider 22%, Repomix 11%, jCodeMunch/ripgrep 5.6%; avg query latency 928 ms vs ripgrep 76 ms; avg tokens 642 vs Aider 95.

Does not publish to npm; extends existing eval/discovery harness patterns rather than replacing ContextBench families below in the doc.

Extend the existing eval fixtures with a frozen 18-task manifest across three public repos, a one-command runner comparing Codebase Context to jCodeMunch, Aider repo map, Repomix, and ripgrep, and checked-in evidence plus benchmark.md reporting including known losses.

chatgpt-codex-connector · 2026-10-30T22:42:07Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-30T22:43:47.028092Z	`453d271`	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 453d2712ec

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-30T22:43:43Z

+      const columns = line.split(',');
+      const fileColumn = columns[3] ?? columns[4];
+      const normalized = normalizeRelativePath(fileColumn ?? '');


Extract file paths rather than symbol kinds from compact results

For the compact jCodeMunch responses used by this run, this column selection extracts symbol kinds instead of file paths. The checked-in evidence confirms that jCodeMunch's topFiles contain values such as method, function, and type throughout all 18 tasks. Consequently, actual retrieved files cannot score hits, while type incorrectly satisfies the expected type substring for ex-sym-01. Parse the compact response's actual file field and regenerate the evidence and comparison table before publishing the 5.6% result.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-10-30T22:43:43Z

+    ['--yes', 'repomix', '--stdout', '--quiet', '--style', 'plain'],
+    { cwd: repoPath, timeoutMs: 900_000 }


Exclude generated benchmark indexes from the Repomix corpus

In the default lane order, Codebase Context creates .codebase-context indexes before Repomix packs each checkout, but this pack invocation does not exclude those generated artifacts. The checked-in evidence shows that every Repomix task returns only .codebase-context/.generations/.../index/index-build.json, rather than a source file from the pinned repository. This also creates false hits for Excalidraw queries accepting json, and charges the generated indexes to Repomix's token cost. Exclude benchmark-generated directories from the comparator corpus and regenerate the results.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-10-30T22:43:43Z

+  const manifest = loadJson(manifestPath);
+  const protocol = loadJson(protocolPath);
+  const errors = [];
+  if (manifest.tasks.length !== 18) errors.push(`expected 18 tasks, found ${manifest.tasks.length}`);


Meet the required minimum evaluation-set size

The runner explicitly rejects any task count other than 18, and the new fixture test enforces the same count. Root AGENTS.md requires at least 20 queries for an evaluation set and explicitly makes evaluation-integrity violations a shipping blocker. Expand the benchmark to meet that minimum and update validation and tests before regenerating the published results; the current validator instead locks in the noncompliant size.

AGENTS.md reference: AGENTS.md:L138-L143

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-10-30T22:43:43Z

+  "description": "Frozen 18-task niche benchmark across three public repos, two tasks per niche per repo. Expected patterns are path substring checks, not full paths.",
+  "frozenDate": "2026-10-30",


Establish a committed fixture baseline before implementation

This fixture first appears in the same commit as the benchmark implementation and completed evidence; it is absent from the parent commit. A frozenDate field therefore does not establish the independently committed preimplementation baseline required by root AGENTS.md. Establish that baseline for the next benchmark iteration before implementation and measurement, and do not describe the current results as satisfying the repository's frozen-fixture requirement.

AGENTS.md reference: AGENTS.md:L131-L135

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-10-30T22:43:43Z

+} from './lib/niche-benchmark/lanes.mjs';
+import { aggregateEvidence, aggregateLaneResults } from './lib/niche-benchmark/scoring.mjs';


Put the evaluation implementation under tests

The full benchmark implementation is introduced under scripts/, including the runner, lane adapters, scorer, and Python helper; tests/ contains only fixtures and a small test importing that implementation. Root AGENTS.md explicitly requires the full evaluation harness code to live in tests/. Move the implementation there and retain a thin npm entry point if needed so this benchmark meets the repository's public-evaluation layout requirement.

AGENTS.md reference: AGENTS.md:L145-L150

Useful? React with 👍 / 👎.

greptile-apps · 2026-10-30T22:46:14Z

Confidence Score: 3/5

[Low impact] The changes affect a local benchmark workflow and its report.

Do not merge the published comparison until file parsing and shared-checkout contamination are fixed and the repository's evaluation requirements are met.

Findings

Symbol kinds count as files ▶
Lanes search each other's files ▶
Late failures erase completed results ▶
Comparator versions are not frozen ▶
Evidence uses a different protocol ▶
Suite falls below required minimum ▶
Harness lives outside required folder ▶
Language results are reported incorrectly ▶
Query words earn usefulness credit ▶

Summary

Adds an 18-task benchmark across three pinned repositories, with five tool lanes, scoring tests, saved evidence, and rerun instructions.

Operators can rerun a fixed five-tool retrieval benchmark.
The benchmark report shows a recorded run and the limits of its results.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Frozen tasks and protocol] --> B[Pinned repository checkouts]
  B --> C[Warm each tool]
  C --> D[Run task queries]
  D --> E[Extract ranked file paths]
  E --> F[Score hits and costs]
  F --> G[Aggregate saved evidence]
  G --> H[Benchmark report]

_{Reviews (1) · Last reviewed commit: "Ignore Aider tag cache created by repo-m..." · Reviewed by Greptile}

greptile-apps · 2026-10-30T22:46:17Z

+      const columns = line.split(',');
+      const fileColumn = columns[3] ?? columns[4];
+      const normalized = normalizeRelativePath(fileColumn ?? '');


Symbol kinds count as files

The compact jCodeMunch parser reads symbol kinds instead of file paths. The saved ex-sym-01 result has topFiles: ["type"] and earns a hit because "type" matches an expected pattern. Other tasks contain "method" or "constant". This makes the published accuracy score invalid. Parse the actual file column, add a compact-response fixture, and regenerate the results.

greptile-apps · 2026-10-30T22:46:17Z

+  for (const laneId of lanes) {
+    console.log(`\n=== Lane: ${laneId} ===`);
+    try {
+      laneEvidence[laneId] = await runLane(laneId, manifest, repoCheckouts);


Lanes search each other's files

Every lane reuses the checkout after Codebase Context writes its index there. Neither ripgrep nor Repomix excludes .codebase-context, and the saved results show both returning those generated files. Their scores and token counts therefore measure more than the pinned repository and depend on earlier lanes. Use separate clean checkouts, or consistently exclude every lane's generated files, then regenerate the evidence.

Knowledge Base Used: Evaluation and ContextBench

greptile-apps · 2026-10-30T22:46:17Z

+      if (index.status !== 0) {
+        return {
+          status: 'setup_failed',
+          reason: `codebase-context reindex failed on ${group.repo.id}`,
+          taskResults: []
+        };


Late failures erase completed results

If setup fails on a later repository, this return discards rows already collected from earlier repositories. A thrown query error also reaches the outer catch and clears the whole lane. Keep completed rows and record failed or unrun tasks separately so the saved evidence still shows what finished.

greptile-apps · 2026-10-30T22:46:17Z

+  const started = Date.now();
+  const result = runCommand(
+    process.platform === 'win32' ? 'npx.cmd' : 'npx',
+    ['--yes', 'repomix', '--stdout', '--quiet', '--style', 'plain'],


Comparator versions are not frozen

The rerun uses unversioned Repomix, and the Python prerequisites have no recorded versions either. The evidence records Node's version but not the comparator versions. Pin these tools and save their versions with each run so users can recreate the behavior behind this frozen comparison.

greptile-apps · 2026-10-30T22:46:18Z

+      "id": "jCodeMunch",
+      "displayName": "jCodeMunch",
+      "credit": "jCodeMunch MCP symbol search and repo indexing (jcodemunch-mcp on PyPI).",
+      "projectUrl": "https://github.com/jgrahamc/jcodemunch",


Evidence uses a different protocol

The saved jCodeMunch URL differs from the committed protocol, although aggregateEvidence copies that URL directly from the protocol. The checked-in run therefore does not match the supplied inputs. Regenerate the evidence from the committed files and add a test checking the recorded fixture hashes against them.

Knowledge Base Used: Evaluation and ContextBench

greptile-apps · 2026-10-30T22:46:18Z

+  const manifest = loadJson(manifestPath);
+  const protocol = loadJson(protocolPath);
+  const errors = [];
+  if (manifest.tasks.length !== 18) errors.push(`expected 18 tasks, found ${manifest.tasks.length}`);


Suite falls below required minimum

The validator requires exactly 18 tasks, but the repository's evaluation rules require “Minimum 20 queries.” The fixture test also locks in 18. Expand the suite, update the count checks, and rerun the benchmark. This repository requirement must be satisfied before merging.

Context Used: AGENTS.md (source)

greptile-apps · 2026-10-30T22:46:18Z

+  console.log(`  protocol sha256: ${sha256File(protocolPath)}`);
+}
+
+async function runLane(laneId, manifest, repoCheckouts) {


Harness lives outside required folder

The new runner, lanes, and scoring code live under scripts/, but the repository's public evaluation rule says “Full eval harness code must be in tests/.” Move the implementation under tests/ and keep only a thin command entry point in scripts/. This repository requirement must be satisfied before merging.

Context Used: AGENTS.md (source)

greptile-apps · 2026-10-30T22:46:18Z

+- **Payload size vs Aider / jCodeMunch:** median-style averages **642** estimated tokens vs **95** (Aider) and **202** (jCodeMunch) on the same queries.
+- **Query latency vs ripgrep:** **928 ms** average query time vs **76 ms** for the ripgrep baseline (no index).
+- **Edit-preflight slice:** **67%** top-3 here, but still below perfect; broad unsafe tasks are out of scope for this table (see edit-preflight fixtures for abstention tests).
+- **Cross-repo variance:** symbol-retrieval top-3 was **83%** on the two TypeScript repos but **50%** on FastAPI in this run (see `byNiche` in the evidence JSON).


Language results are reported incorrectly

The saved results show three symbol-retrieval hits out of four for the TypeScript repositories, or 75%, and two out of two for FastAPI, or 100%. The 83% in byNiche covers all repositories, not just TypeScript. This bullet reverses the comparison; update it to match the saved task results.

greptile-apps · 2026-10-30T22:46:18Z

+        if len(top_files) >= limit:
+            break
+    payload = json.dumps(
+        {"mentionedIdents": mentioned, "rankedTags": len(tags), "topFiles": top_files},


Query words earn usefulness credit

Aider's scored payload includes mentionedIdents, which echoes the query rather than retrieved information. scoreTaskResult searches the whole payload for expected signals, so words such as "dependency" and "router" earn usefulness credit merely because the user asked about them. Keep echoed query fields out of the payload used for usefulness scoring.

cursoragent added 2 commits October 30, 2026 22:40

Ignore Aider tag cache created by repo-map lane

453d271

chatgpt-codex-connector Bot reviewed Oct 30, 2026

View reviewed changes

cursor Bot mentioned this pull request Oct 30, 2026

Team-convention queries return no ranked exemplar files (0/6 in niche benchmark) #157

Open

greptile-apps Bot reviewed Oct 30, 2026

View reviewed changes

cursor Bot mentioned this pull request Oct 30, 2026

docs: prepare 2.3.0 npm release (honest positioning, tarball smoke) #148

Open

6 tasks

PatrickSys mentioned this pull request Oct 30, 2026

Integration preview: 2.3.0 relaunch (do not merge) #158

Draft

		['--yes', 'repomix', '--stdout', '--quiet', '--style', 'plain'],
		{ cwd: repoPath, timeoutMs: 900_000 }

		"description": "Frozen 18-task niche benchmark across three public repos, two tasks per niche per repo. Expected patterns are path substring checks, not full paths.",
		"frozenDate": "2026-10-30",

		} from './lib/niche-benchmark/lanes.mjs';
		import { aggregateEvidence, aggregateLaneResults } from './lib/niche-benchmark/scoring.mjs';

Conversation

PatrickSys commented Oct 30, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 3/5

Findings

Summary

Diagram

Uh oh!

greptile-apps Bot Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chatgpt-codex-connector Bot commented Oct 30, 2026 •

edited

Loading

greptile-apps Bot commented Oct 30, 2026 •

edited

Loading