Repository navigation
chore: delete benchmark harnesses that target the removed API - #536
Merged
Merged
Conversation
Every entry point here fails on the v3 API: cleanbench runs call clean(verbose=...), gauntlet and bench.py fail at import, truthbench imports dead surfaces, and public_benchmark.py runs the dead bench_report.py. - Delete benchmarks/cleanbench, gauntlet, truthbench and corpus (corpus only adapted gauntlet and truthbench fixtures), bench.py, harness_metrics.py, bench_report.py, public_benchmark.py, the two baselines that import cleanbench, benchmarks/README.md (it documented only bench.py), and docs/validation-gauntlet.md. - Delete 4 examples and notebook 02 that use the removed API, with their entries in examples/README.md and notebook 01. - Delete the cleanbench-full, gauntlet, perf-regression, reproducibility, public-benchmark, nightly-real-model and benchmark workflows, the Validation Gauntlet release step, and the matching Makefile targets. - Remove pointers to the deleted code from docs, templates, CODEOWNERS and .gitignore. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #535. That PR kept these benchmark harnesses because live code still imported them. Every one of them targets the removed v1/v2 API, so this PR deletes them together with the workflows, Make targets and docs that ran or described them.
133 files changed, 11 insertions(+), 17908 deletions(-)Evidence they are dead
I ran each workflow's command on
main(86c83e8):python -m benchmarks.cleanbench --tracks T1 --report siteTypeError: clean() got an unexpected keyword argument 'verbose'python -m benchmarks.cleanbench --verify-results ...andreproducibility audit-readmepython -m benchmarks.gauntlet run --rows 300 --checkImportError: cannot import name 'FieldSpec'python benchmarks/bench.pyModuleNotFoundError: freshdata._sentinelspython benchmarks/public_benchmark.pybench_report.py, which importsfreshdata.benchmarks._harness(removed)A static import graph shows that every
truthbenchmodule fails at import through itssurfacespackage.benchmarks/corpushad no users except adapters forgauntletandtruthbenchfixtures.Deleted
cleanbench/,gauntlet/,truthbench/,corpus/,bench.py,harness_metrics.py,bench_report.py,public_benchmark.py,baselines/great_expectations_baseline.pyandbaselines/llm_agent_baseline.py(both importcleanbench), andbenchmarks/README.md(it documented onlybench.py).cleaning_memory_replay.py,freshdata_ai_copilot_demo.py,integrations/airflow_task.py,interactive_html_reports.pyandnotebooks/02_profiling_deep_dive.ipynb, with their entries inexamples/README.mdand notebook 01.cleanbench-full,gauntlet,perf-regression,reproducibility,public-benchmark,nightly-real-modelandbenchmark, plus the "Validation Gauntlet" step inrelease.yml.cleanbench-full,truthbench-release,truthbench-pr,benchmark,benchmark-ci,benchmark-reportandbenchmark-fixtures.docs/validation-gauntlet.md, and the sections and lines that point at the deleted code, inbenchmarks.md,ai-copilot.md,comparison.md,limitations.md,trust-claims.mdand others.None of the deleted workflows run on pull requests. The three PR workflows (
benchmark-gate,ci-verification-gates,security-gate) reference none of the deleted paths.Not in this PR
benchmarks/fixtures/now has no code importers (its consumer wasbench.py), but it does not use the removed API.trust-claims.md,limitations.md,comparison.md,semantic-models.md,ai-copilot.md) still describe v2 features. This PR removes only pointers to deleted code.main:api-reference.mdasks mkdocstrings forfreshdata.clean_csv.docs.ymlalso installs adocsextra thatpyproject.tomldoes not define.Test plan
uv sync --frozen,uv run pytest -q: 66 passedruff check .: 113 errors, all present onmain, none newmypy src/freshdata: 1 error, present onmainimport freshdataworks in a clean venv without polarsyaml.safe_load, and everyneeds:resolvesmake -n helpparsesmain(1 pre-existing)🤖 Generated with Claude Code