Skip to content

chore: delete benchmark harnesses that target the removed API - #536

Merged
JohnnyWilson16 merged 1 commit into
mainfrom
chore/purge-dead-benchmarks
Nov 18, 2026
Merged

JohnnyWilson16 merged 1 commit into
mainfrom
chore/purge-dead-benchmarks

Conversation

@JohnnyWilson16

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #535. That PR kept these benchmark harnesses because live code still imported them. Every one of them targets the removed v1/v2 API, so this PR deletes them together with the workflows, Make targets and docs that ran or described them.

133 files changed, 11 insertions(+), 17908 deletions(-)

Evidence they are dead

I ran each workflow's command on main (86c83e8):

Command Result
python -m benchmarks.cleanbench --tracks T1 --report site TypeError: clean() got an unexpected keyword argument 'verbose'
python -m benchmarks.cleanbench --verify-results ... and reproducibility audit-readme fail on README claims the v3 README no longer contains
python -m benchmarks.gauntlet run --rows 300 --check ImportError: cannot import name 'FieldSpec'
python benchmarks/bench.py ModuleNotFoundError: freshdata._sentinels
python benchmarks/public_benchmark.py runs bench_report.py, which imports freshdata.benchmarks._harness (removed)

A static import graph shows that every truthbench module fails at import through its surfaces package. benchmarks/corpus had no users except adapters for gauntlet and truthbench fixtures.

Deleted

  • Benchmarks: cleanbench/, gauntlet/, truthbench/, corpus/, bench.py, harness_metrics.py, bench_report.py, public_benchmark.py, baselines/great_expectations_baseline.py and baselines/llm_agent_baseline.py (both import cleanbench), and benchmarks/README.md (it documented only bench.py).
  • Examples: cleaning_memory_replay.py, freshdata_ai_copilot_demo.py, integrations/airflow_task.py, interactive_html_reports.py and notebooks/02_profiling_deep_dive.ipynb, with their entries in examples/README.md and notebook 01.
  • Workflows: cleanbench-full, gauntlet, perf-regression, reproducibility, public-benchmark, nightly-real-model and benchmark, plus the "Validation Gauntlet" step in release.yml.
  • Make targets: cleanbench-full, truthbench-release, truthbench-pr, benchmark, benchmark-ci, benchmark-report and benchmark-fixtures.
  • Docs: docs/validation-gauntlet.md, and the sections and lines that point at the deleted code, in benchmarks.md, ai-copilot.md, comparison.md, limitations.md, trust-claims.md and others.

None of the deleted workflows run on pull requests. The three PR workflows (benchmark-gate, ci-verification-gates, security-gate) reference none of the deleted paths.

Not in this PR

  • benchmarks/fixtures/ now has no code importers (its consumer was bench.py), but it does not use the removed API.
  • Many docs (trust-claims.md, limitations.md, comparison.md, semantic-models.md, ai-copilot.md) still describe v2 features. This PR removes only pointers to deleted code.
  • The docs build already fails on main: api-reference.md asks mkdocstrings for freshdata.clean_csv. docs.yml also installs a docs extra that pyproject.toml does not define.

Test plan

  • uv sync --frozen, uv run pytest -q: 66 passed
  • ruff check .: 113 errors, all present on main, none new
  • mypy src/freshdata: 1 error, present on main
  • import freshdata works in a clean venv without polars
  • All 13 remaining workflows parse with yaml.safe_load, and every needs: resolves
  • make -n help parses
  • Relative doc links: no new broken links compared with main (1 pre-existing)
  • PR checks

🤖 Generated with Claude Code

Every entry point here fails on the v3 API: cleanbench runs call
clean(verbose=...), gauntlet and bench.py fail at import, truthbench imports
dead surfaces, and public_benchmark.py runs the dead bench_report.py.

- Delete benchmarks/cleanbench, gauntlet, truthbench and corpus (corpus only
  adapted gauntlet and truthbench fixtures), bench.py, harness_metrics.py,
  bench_report.py, public_benchmark.py, the two baselines that import
  cleanbench, benchmarks/README.md (it documented only bench.py), and
  docs/validation-gauntlet.md.
- Delete 4 examples and notebook 02 that use the removed API, with their
  entries in examples/README.md and notebook 01.
- Delete the cleanbench-full, gauntlet, perf-regression, reproducibility,
  public-benchmark, nightly-real-model and benchmark workflows, the
  Validation Gauntlet release step, and the matching Makefile targets.
- Remove pointers to the deleted code from docs, templates, CODEOWNERS and
  .gitignore.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: fb0c2ca6-4371-47f0-a4c5-65f8b05cf2ce
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@JohnnyWilson16
JohnnyWilson16 merged commit cb5840c into main Nov 18, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant