Repository navigation
chore: cut freshdata to its v3 core - #535
Conversation
…counts - Import polars only in the Polars code paths. A polars-module frame without an importable polars raises ImportError that names the polars extra. - PolarsBackend.profile wrote literal zeros into null and unique counts. It now reads them from the eager DataFrame. LazyFrame profiling still does not collect. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Replace the 7 hand-built row filters with out[mask].reset_index(drop=True). - Use dataclasses.replace in normalize_config. - Merge the impute_* branches in PandasBackend.apply_plan. - Compute IQR bounds once in stage_outliers. Merge the two drop branches in stage_missing (empty frames still record drop_nulls). - Run the six suggest_plan stages in a loop. - Delete unused ActionRecord.params and ActionRecord.affected_rows. - Import from _core submodules and empty _core/__init__.py. - Read __version__ from package metadata. It now reports 0.4.0, not 3.0.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Delete 30 benchmark, example, notebook, script and training files that use freshdata names that no longer exist. - Delete training/, freshdata-benchmarks/ (with committed .asv results), crates/freshcore/, marketing/, growth-audit.md and admin-actions.md. - Delete docs/freshcore.md and docs/developer-training-pipeline.md, which only documented deleted paths. - Remove references to the deleted paths from docs, mkdocs.yml, .gitignore, MANIFEST.in and Makefile. Files that live code imports or runs stay, even when they use the removed API (for example benchmarks/bench_report.py and the truthbench surfaces). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Delete plugin-contract.yml. It ran a missing tests/test_plugins.py and the deleted example plugins. - ci.yml: delete the truthbench and freshcore-native jobs. - release.yml: delete the CleanBench and TruthBench release-gate steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Replace each `pip install -c constraints/ci.txt -e ".[...]"` with `uv sync --frozen --inexact` into the setup-python interpreter. Extras match what pip installed: ml and bench are not defined in pyproject.toml, so pip skipped them. - The lockfile job now only runs `uv lock --check`. Delete the Makefile constraints target, which only refreshed the deleted file. - Relock uv.lock with uv 0.12.10. The committed lock was stale: it resolved numpy 2.5.1 against the numpy<2.4.0 pin and lacked the migrate extra. - constraints/ci.txt held only exact versions from that stale lock. No pin existed only there, so pyproject.toml is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- PolarsBackend.suggest_plan emits normalize_name actions and checks id, target and rules against normalized names. Before, a target such as "Churn" with target_column="churn" was imputed. - clip_iqr actions now carry lower/upper bounds from one 1-row aggregate query (linear quartiles on the imputed column, as in the pandas stages). Before, they had no bounds and apply_plan skipped them. - apply_plan chains expressions per column, so impute and clip on one column compose instead of colliding in with_columns. - Move the polars_engine lazy-frame tests onto the freshdata API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The core PolarsBackend now covers lazy renaming, target protection and IQR
clipping, so the second Polars implementation is redundant.
- Delete src/freshdata/polars_engine.py and tests/test_polars_engine.py.
Its lazy-frame tests now live in tests/test_backend_honesty.py.
- Keep benchmarks/bench_v3_lazy_vs_pandas.py, because tests and
benchmarks/perf_gate.py import it. It now uses the freshdata API.
- Point benchmark-gate.yml and the README at the moved tests.
BREAKING CHANGE: freshdata.polars_engine is removed. Use freshdata.clean,
freshdata.suggest_plan and freshdata.apply_plan, which accept
polars.DataFrame and polars.LazyFrame. The plan uses the core
ActionRecord vocabulary (impute_median, clip_iqr, normalize_name), and
CleanPlan.diff returns {"added", "removed", "modified"}. The polars_engine
features flag_iqr, drop_column, string mode imputation and baked fill
values have no core equivalent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…amples
- Print ActionRecord.action and parameters. Core actions carry no rationale.
- Replace the polars_engine output (strip_whitespace, fill_null, preserve)
with output captured from the current API.
- Drop the outliers="flag" rule. The core backends ignore it.
- Diff on default_outliers, not strategy. strategy does not change the plan.
Use the {"added", "removed", "modified"} shape that CleanPlan.diff returns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing in the repository reads it since the README examples moved to ActionRecord.action. BREAKING CHANGE: ActionRecord.operation is removed. Use ActionRecord.action. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
The job failed before running any test: pyproject addopts pass --cov flags, and the job did not install pytest-cov. With pytest-cov installed, the 3-file subset reaches 46% coverage, below the 90% repo-wide floor. Use the 40% floor that the property-invariant lane in ci-verification-gates.yml uses, so coverage stays on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The base step copies the PR's benchmarks/ over the base tree to time the same harness on both sides. That leaves tracked files modified, so the plain `git checkout <head>` in the compare step aborts for every PR that changes a file under benchmarks/. --force discards only that overlay; the head commit holds the same files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why "Polars LazyFrame Runtime & Memory Gate" failsThe gate reports
CauseThe gate runs this PR's No correct clipping fits the 15% budgetTimings of the cleaning step alone on the gate's 2M-row parquet file (best of 5, local):
Exact quartiles over 2M rows cost about 120 ms however they are expressed. Base +15% is about 99 ms. I tried lazy bounds and did not push them, because they gave no speedup and they remove the bounds from the plan. Against what
|
| Metric | main (polars_engine) |
This PR | Change |
|---|---|---|---|
total_ms |
351.8 ms | 247.8 ms | -29.6% |
plan_ms |
203.8 ms | 46.5 ms | -77.2% |
peak_rss_mb |
643.1 MB | 632.7 MB | -1.6% |
Also fixed in this PR
The gate previously could not reach its comparison:
762f1c5: the job did not installpytest-cov, so pytest rejected the--covoptions inpyproject.toml.1b6acef: the return checkout aborted for any PR that changesbenchmarks/, because the base step copies the PR'sbenchmarks/over the base tree. It now usesgit checkout --force.
Merging needs a maintainer to accept this regression against a base that did not clip. After merge, base and PR both clip, so later PRs compare like with like.
🤖 Generated with Claude Code
Summary
The v3 rewrite cut
src/to about 1,240 lines, but about 21k lines around it still targeted the deleted v1/v2 API. This PR deletes that code, fixes two core bugs, and closes the gaps between the Polars and pandas backends so thatfreshdata.polars_enginecan go. Each phase is one commit. Merge with a merge commit or rebase to keep them.189 files changed, 482 insertions(+), 21624 deletions(-)fix(core)1252483import freshdataworks without polars.fd.profileon apl.DataFramereturns real null and unique counts.refactor(core)ca20b17__version__now comes from package metadata. DeleteActionRecord.paramsandActionRecord.affected_rows.choreec069cbtraining/,freshdata-benchmarks/,crates/freshcore/,marketing/,growth-audit.mdandadmin-actions.md. Remove references to them from docs and config.ci39dc545build5de24f5uv.lockwithuv sync --frozen. Relock (the old lock faileduv lock --checkand resolved numpy 2.5.1 againstnumpy<2.4.0). Deleteconstraints/ci.txt.fix(core)b8b449arefactor!950d200freshdata.polars_engine. Move its lazy-frame tests onto thefreshdataAPI.docs(readme)dd7e0e4refactor(core)!23ba87bActionRecord.operation.Breaking changes
freshdata.polars_engineis removed. Usefd.clean,fd.suggest_planandfd.apply_plan, which acceptpolars.DataFrameandpolars.LazyFrame. Itsflag_iqr,drop_column, string mode imputation and baked fill values have no core equivalent.ActionRecord.operation,.paramsand.affected_rowsare removed. Use.action,.parametersand.rows_affected.from freshdata._core import <name>no longer works. Import fromfreshdata.freshdata.__version__reports the installed version (0.4.0), not3.0.0.apply_planfills withvaluefor anyimpute_*action in a custom plan.CI jobs removed
Only
benchmark-gate,ci-verification-gatesandsecurity-gaterun on pull requests, and none of their jobs was removed.ci.ymlandplugin-contract.ymlrun only on manual dispatch, so the removals below should not affect required checks. Confirm against branch protection:ci.yml:truthbench,freshcore-nativeplugin-contract.yml:plugin-contract(the whole workflow)release.yml, jobtest: the steps "CleanBench release gates" and "TruthBench release verification"ci.ymlstep "uv.lock and constraints/ci.txt are up to date" is renamed "uv.lock is up to date"Kept on purpose
These files still use the removed API, but live code imports, runs or links to them, so they stay for a follow-up decision:
benchmarks/cleanbench,benchmarks/gauntlet, and thebenchmarks/truthbenchrunner and surfacesbenchmarks/bench_report.pyandbenchmarks/harness_metrics.pyexamples/README.md, andnotebooks/02_profiling_deep_dive.ipynbWorkflows that still run that code:
cleanbench-full,gauntlet,perf-regression,reproducibility,public-benchmark,nightly-real-model,ci-verification-gates, and the "Validation Gauntlet" step inrelease.yml.Test plan
uv sync --frozenuv run pytest -q: 66 passed, coverage above 90%ruff check .: 218 errors, all present onmain(362 there), none newmypy src/freshdata: 1 error, present onmainimport freshdatain a clean venv without polarsuv sync --frozen --inexact --extra devCI install locally with uv 0.12.10benchmark-gate,ci-verification-gates,security-gateNot covered: the manual
ci.yml. Its last run onmain(Oct 22) failed in every job exceptlarge-data-status. Itsquality-fastjob also installs only thedevextra, which lacks libcst, sotests/test_migrate_codemod.pycannot be collected there.🤖 Generated with Claude Code