Skip to content

chore: cut freshdata to its v3 core - #535

Merged
JohnnyWilson16 merged 11 commits into
mainfrom
chore/v3-purge
Nov 18, 2026
Merged

JohnnyWilson16 merged 11 commits into
mainfrom
chore/v3-purge

Conversation

@JohnnyWilson16

Copy link
Copy Markdown
Contributor

Summary

The v3 rewrite cut src/ to about 1,240 lines, but about 21k lines around it still targeted the deleted v1/v2 API. This PR deletes that code, fixes two core bugs, and closes the gaps between the Polars and pandas backends so that freshdata.polars_engine can go. Each phase is one commit. Merge with a merge commit or rebase to keep them.

189 files changed, 482 insertions(+), 21624 deletions(-)

Commit Change
fix(core) 1252483 import freshdata works without polars. fd.profile on a pl.DataFrame returns real null and unique counts.
refactor(core) ca20b17 Simplify stages, backends and the package init. __version__ now comes from package metadata. Delete ActionRecord.params and ActionRecord.affected_rows.
chore ec069cb Delete 30 dead scripts plus training/, freshdata-benchmarks/, crates/freshcore/, marketing/, growth-audit.md and admin-actions.md. Remove references to them from docs and config.
ci 39dc545 Delete CI jobs and steps that run deleted code.
build 5de24f5 Install CI from uv.lock with uv sync --frozen. Relock (the old lock failed uv lock --check and resolved numpy 2.5.1 against numpy<2.4.0). Delete constraints/ci.txt.
fix(core) b8b449a The Polars backend renames columns, protects id and target columns, and clips with real IQR bounds, matching pandas.
refactor! 950d200 Delete freshdata.polars_engine. Move its lazy-frame tests onto the freshdata API.
docs(readme) dd7e0e4 Rewrite the quickstart and plan-diff examples with output captured from the current API.
refactor(core)! 23ba87b Delete ActionRecord.operation.

Breaking changes

  • freshdata.polars_engine is removed. Use fd.clean, fd.suggest_plan and fd.apply_plan, which accept polars.DataFrame and polars.LazyFrame. Its flag_iqr, drop_column, string mode imputation and baked fill values have no core equivalent.
  • ActionRecord.operation, .params and .affected_rows are removed. Use .action, .parameters and .rows_affected.
  • from freshdata._core import <name> no longer works. Import from freshdata.
  • freshdata.__version__ reports the installed version (0.4.0), not 3.0.0.
  • apply_plan fills with value for any impute_* action in a custom plan.

CI jobs removed

Only benchmark-gate, ci-verification-gates and security-gate run on pull requests, and none of their jobs was removed. ci.yml and plugin-contract.yml run only on manual dispatch, so the removals below should not affect required checks. Confirm against branch protection:

  • ci.yml: truthbench, freshcore-native
  • plugin-contract.yml: plugin-contract (the whole workflow)
  • release.yml, job test: the steps "CleanBench release gates" and "TruthBench release verification"
  • ci.yml step "uv.lock and constraints/ci.txt are up to date" is renamed "uv.lock is up to date"

Kept on purpose

These files still use the removed API, but live code imports, runs or links to them, so they stay for a follow-up decision:

  • benchmarks/cleanbench, benchmarks/gauntlet, and the benchmarks/truthbench runner and surfaces
  • benchmarks/bench_report.py and benchmarks/harness_metrics.py
  • four examples listed in examples/README.md, and notebooks/02_profiling_deep_dive.ipynb

Workflows that still run that code: cleanbench-full, gauntlet, perf-regression, reproducibility, public-benchmark, nightly-real-model, ci-verification-gates, and the "Validation Gauntlet" step in release.yml.

Test plan

  • uv sync --frozen
  • uv run pytest -q: 66 passed, coverage above 90%
  • ruff check .: 218 errors, all present on main (362 there), none new
  • mypy src/freshdata: 1 error, present on main
  • import freshdata in a clean venv without polars
  • New tests fail on the old code and pass now (polars-free import, profile counts, pandas/Polars plan parity, lazy apply without collect)
  • Simulated the new uv sync --frozen --inexact --extra dev CI install locally with uv 0.12.10
  • Every path that the three PR workflows reference still exists
  • PR checks: benchmark-gate, ci-verification-gates, security-gate

Not covered: the manual ci.yml. Its last run on main (Oct 22) failed in every job except large-data-status. Its quality-fast job also installs only the dev extra, which lacks libcst, so tests/test_migrate_codemod.py cannot be collected there.

🤖 Generated with Claude Code

JohnnyWilson16 and others added 9 commits November 18, 2026 12:47
…counts

- Import polars only in the Polars code paths. A polars-module frame without
  an importable polars raises ImportError that names the polars extra.
- PolarsBackend.profile wrote literal zeros into null and unique counts.
  It now reads them from the eager DataFrame. LazyFrame profiling still
  does not collect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Replace the 7 hand-built row filters with out[mask].reset_index(drop=True).
- Use dataclasses.replace in normalize_config.
- Merge the impute_* branches in PandasBackend.apply_plan.
- Compute IQR bounds once in stage_outliers. Merge the two drop branches
  in stage_missing (empty frames still record drop_nulls).
- Run the six suggest_plan stages in a loop.
- Delete unused ActionRecord.params and ActionRecord.affected_rows.
- Import from _core submodules and empty _core/__init__.py.
- Read __version__ from package metadata. It now reports 0.4.0, not 3.0.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Delete 30 benchmark, example, notebook, script and training files that use
  freshdata names that no longer exist.
- Delete training/, freshdata-benchmarks/ (with committed .asv results),
  crates/freshcore/, marketing/, growth-audit.md and admin-actions.md.
- Delete docs/freshcore.md and docs/developer-training-pipeline.md, which
  only documented deleted paths.
- Remove references to the deleted paths from docs, mkdocs.yml, .gitignore,
  MANIFEST.in and Makefile.

Files that live code imports or runs stay, even when they use the removed
API (for example benchmarks/bench_report.py and the truthbench surfaces).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Delete plugin-contract.yml. It ran a missing tests/test_plugins.py and the
  deleted example plugins.
- ci.yml: delete the truthbench and freshcore-native jobs.
- release.yml: delete the CleanBench and TruthBench release-gate steps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Replace each `pip install -c constraints/ci.txt -e ".[...]"` with
  `uv sync --frozen --inexact` into the setup-python interpreter. Extras
  match what pip installed: ml and bench are not defined in pyproject.toml,
  so pip skipped them.
- The lockfile job now only runs `uv lock --check`. Delete the Makefile
  constraints target, which only refreshed the deleted file.
- Relock uv.lock with uv 0.12.10. The committed lock was stale: it resolved
  numpy 2.5.1 against the numpy<2.4.0 pin and lacked the migrate extra.
- constraints/ci.txt held only exact versions from that stale lock. No pin
  existed only there, so pyproject.toml is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- PolarsBackend.suggest_plan emits normalize_name actions and checks id,
  target and rules against normalized names. Before, a target such as
  "Churn" with target_column="churn" was imputed.
- clip_iqr actions now carry lower/upper bounds from one 1-row aggregate
  query (linear quartiles on the imputed column, as in the pandas stages).
  Before, they had no bounds and apply_plan skipped them.
- apply_plan chains expressions per column, so impute and clip on one
  column compose instead of colliding in with_columns.
- Move the polars_engine lazy-frame tests onto the freshdata API.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The core PolarsBackend now covers lazy renaming, target protection and IQR
clipping, so the second Polars implementation is redundant.

- Delete src/freshdata/polars_engine.py and tests/test_polars_engine.py.
  Its lazy-frame tests now live in tests/test_backend_honesty.py.
- Keep benchmarks/bench_v3_lazy_vs_pandas.py, because tests and
  benchmarks/perf_gate.py import it. It now uses the freshdata API.
- Point benchmark-gate.yml and the README at the moved tests.

BREAKING CHANGE: freshdata.polars_engine is removed. Use freshdata.clean,
freshdata.suggest_plan and freshdata.apply_plan, which accept
polars.DataFrame and polars.LazyFrame. The plan uses the core
ActionRecord vocabulary (impute_median, clip_iqr, normalize_name), and
CleanPlan.diff returns {"added", "removed", "modified"}. The polars_engine
features flag_iqr, drop_column, string mode imputation and baked fill
values have no core equivalent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…amples

- Print ActionRecord.action and parameters. Core actions carry no rationale.
- Replace the polars_engine output (strip_whitespace, fill_null, preserve)
  with output captured from the current API.
- Drop the outliers="flag" rule. The core backends ignore it.
- Diff on default_outliers, not strategy. strategy does not change the plan.
  Use the {"added", "removed", "modified"} shape that CleanPlan.diff returns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing in the repository reads it since the README examples moved to
ActionRecord.action.

BREAKING CHANGE: ActionRecord.operation is removed. Use ActionRecord.action.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 275b45d7-08d5-4124-9c81-a4c0e6be83df
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

JohnnyWilson16 and others added 2 commits November 18, 2026 13:44
The job failed before running any test: pyproject addopts pass --cov flags,
and the job did not install pytest-cov. With pytest-cov installed, the
3-file subset reaches 46% coverage, below the 90% repo-wide floor. Use the
40% floor that the property-invariant lane in ci-verification-gates.yml
uses, so coverage stays on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The base step copies the PR's benchmarks/ over the base tree to time the
same harness on both sides. That leaves tracked files modified, so the plain
`git checkout <head>` in the compare step aborts for every PR that changes a
file under benchmarks/. --force discards only that overlay; the head
commit holds the same files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JohnnyWilson16

Copy link
Copy Markdown
Contributor Author

Why "Polars LazyFrame Runtime & Memory Gate" fails

The gate reports total_ms +42.5% (limit +15%) at 1b6acef:

Metric Base PR Change
total_ms 378.0 ms 538.8 ms +42.5%
plan_ms 2.4 ms 179.1 ms +7333%
compile_ms 0.19 ms 0.20 ms +4.7%
peak_rss_mb 596.8 MB 601.5 MB +0.8%

Cause

The gate runs this PR's benchmarks/bench_v3_lazy_vs_pandas.py against the base code. On base, that script goes through the core Polars backend, which emitted clip_iqr without bounds, so apply_plan never clipped. This PR (b8b449a) makes the core backend clip with real IQR bounds, the same as the pandas backend. The extra time is the cost of computing those bounds.

No correct clipping fits the 15% budget

Timings of the cleaning step alone on the gate's 2M-row parquet file (best of 5, local):

Variant Time
Impute only, no clipping (what base does) 86 ms
This PR: bounds from one summary query at plan time 216 ms
Bounds computed lazily inside the query, best expression form 209 ms

Exact quartiles over 2M rows cost about 120 ms however they are expressed. Base +15% is about 99 ms. I tried lazy bounds and did not push them, because they gave no speedup and they remove the bounds from the plan.

Against what main actually benchmarked

On main, the same script used freshdata.polars_engine, which also computed bounds at plan time. Locally, this PR compares to that as follows:

Metric main (polars_engine) This PR Change
total_ms 351.8 ms 247.8 ms -29.6%
plan_ms 203.8 ms 46.5 ms -77.2%
peak_rss_mb 643.1 MB 632.7 MB -1.6%

Also fixed in this PR

The gate previously could not reach its comparison:

  • 762f1c5: the job did not install pytest-cov, so pytest rejected the --cov options in pyproject.toml.
  • 1b6acef: the return checkout aborted for any PR that changes benchmarks/, because the base step copies the PR's benchmarks/ over the base tree. It now uses git checkout --force.

Merging needs a maintainer to accept this regression against a base that did not clip. After merge, base and PR both clip, so later PRs compare like with like.

🤖 Generated with Claude Code

@JohnnyWilson16
JohnnyWilson16 merged commit 86c83e8 into main Nov 18, 2026
13 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant