Repository navigation
fix: backend parity and fail-loud execution for pandas and Polars - #537
Merged
Merged
Conversation
- Add stages._is_protected so pandas and Polars use one ID/target/protected predicate instead of three copies. - Move the per-cell string cleaner to module level, check for str first, and replace re.sub with str.split/join. Output is identical (checked against every Unicode whitespace character and 1,500 fuzzed configs). - Rewrite ARCHITECTURE.md to describe the v3 code that exists. pandas clean() is about 33% faster on 300k rows x 11 columns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
suggest_plan collapsed inner whitespace and turned "nan"/"none"/"null" text
into missing values. apply_plan only stripped. Plan statistics (mode values,
medians, rows_affected) therefore described data that apply_plan never
produced. apply_plan also replaced every non-string cell in an object column
with None.
Both paths now call stages._strip_str: strip surrounding whitespace, turn
empty strings into None, keep non-string cells. This matches the README
(".str.strip_chars()") and the Polars executor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The plan stored str(mode). apply_plan then wrote "1.0" into a float column and turned it into an object column. Store the plain Python scalar instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ID columns, the target column, *_id columns and protected=True rules were stripped and currency-parsed on pandas. The Polars planner already skipped them. Both stages now use stages._is_protected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Polars used a column's rule for both impute and outliers whenever the rule existed, so ColumnRule(outliers="clip") silently turned off default imputation (and the reverse). Both backends now resolve each field through stages._impute_strategy and stages._outlier_strategy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Polars apply_plan ignored drop_nulls, drop_outliers, parse_currency and impute_mode, and both backends skipped unknown actions and malformed parameters without a word. pandas also skipped impute actions that carry no precomputed value, which is every impute action a Polars plan contains. - Polars: native executors for the four missing actions. Row filters run after the fused with_columns. No collect and no Python UDFs. - pandas: compute median/mean/mode when the plan has no value. - Polars planner: honor impute="mode", impute="drop" and outliers="drop". - Both: an action that cannot run raises ValueError when strict_backend=True (the default, as the README promises). With strict_backend=False it is skipped with a UserWarning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both options were accepted by ColumnRule/CleanConfig and did nothing. - impute="ffill": forward-fill the column (pandas ffill, Polars forward_fill). - outliers="flag": add a boolean <col>_outlier column marking values outside the IQR fences, the meaning the v2 docs gave it. Values are not changed. If <col>_outlier already exists, planning and apply raise ValueError instead of overwriting it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hing ColumnRule and CleanConfig accepted any value, so a typo such as impute="medain" silently did nothing, and id_columns="customer_id" matched column names by substring. - Reject unknown role/impute/outliers/strategy/default values, a bare string for id_columns, non-ColumnRule rules, and a negative iqr_factor. - Widen the Literal hints to every value the stages already handle (default_outliers="drop" is what freshdata-migrate emits). - ColumnRule(role="id"|"target") now protects the column, like id_columns and target_column. - Warn when an option has no effect: strategy other than "balanced", other roles, and default_outliers="auto". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
freshdata-migrate rewrites v2 calls into fd.clean(df, default_impute=...) and fd.Cleaner(id_columns=...), but both raised TypeError on v3. Keyword arguments now override fields of the given (or default) CleanConfig and go through the same validation. Unknown keywords still raise TypeError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
profile() returned 0 rows, 0 nulls and 0 unique values for every LazyFrame column, which reads as real data. Both DataFrame and LazyFrame input now run one select of len/null_count/n_unique, which Polars evaluates in parallel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pandas reported the frame's row count for normalize_name, strip_strings, parse_currency and clip_iqr. Polars reported 0 for every action. - pandas: renames change 0 cells. strip_strings counts changed cells, parse_currency counts parsed cells, clip_iqr counts clipped values. - Polars: the planner computes the same counts in its one aggregate query (null counts, strip changes, values outside the IQR fences). drop_duplicates stays uncounted on Polars, and its rationale says why: counting needs a full-row hash, which would break streaming-scale planning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inlining the quantiles in the outside-fence count made Polars evaluate each one three times (79ms vs 26ms for 6 columns x 300k rows). The planner now adds the quantiles with with_columns and counts against those columns. It is still one query. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
apply_plan(..., return_report=True) called len(lf.columns), which makes Polars emit a PerformanceWarning in the README quickstart. Use collect_schema() for Polars frames. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ARCHITECTURE.md now states the shared-helper and strict-execution rules and replaces the fixed "Known issues" with the backend differences that remain. CHANGELOG.md lists the user-visible changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Type runtime-checked config values as object, take pandas column labels as Any, import Container from collections.abc, and drop a bool check on iqr_factor that mypy proves unreachable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
… free The Polars LazyFrame perf gate failed: total_ms +17.1% (limit +15%), driven by plan_ms 111 -> 174 ms. Counting strip_strings changes needs a per-cell string pass, and counting values outside the IQR fences needs the quantiles broadcast as columns. Together they added about 60% to planning on the 2M-row gate workload. The planner keeps the free counts (null counts for imputation, drop_nulls and ffill) and goes back to main's single-query shape. strip_strings and outlier actions keep rows_affected=0, with a rationale that says why, like drop_duplicates. pandas still counts every action. Local gate reproduction (5 iterations, 2M rows) against origin/main: plan_ms -1.4%, total_ms -15.1%. The previous head gave plan_ms +155%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed?
pandas and Polars now plan and execute the same actions, and the core library fails loudly where it used to fail silently. 17 commits. Each fix is its own commit and comes with a test that fails on
main.Refactor (no output change):
_is_protected,_impute_strategy,_outlier_strategy,_strip_str. Previously 3 copies of the protection check had drifted apart.Fixes:
apply_planturned every non-string cell in an object column intoNone([1, " a "]→[None, "a"]). Planning and applying also used different string logic, so plan statistics described data thatapply_plannever produced.protected=Truecolumns.impute_modestored the mode as a string, so a float column becameobject.ColumnRule(outliers="clip")turned off default imputation on Polars only.drop_nulls,drop_outliers,parse_currency, andimpute_mode. Both backends skipped unknown actions and malformed parameters without a word.ffill,flag,strict_backend, androlewere accepted and did nothing. Typos such asimpute="medain"were accepted silently too.profile()on a LazyFrame reported 0 rows, 0 nulls, and 0 unique values.rows_affected. pandas reported the frame's row count. Polars reported 0 for every action. pandas now counts every action exactly. Polars counts the nulls that imputation fills ordrop_nullsremoves, and states a reason wherever it does not count.freshdata-migrateemitsfd.clean(df, default_impute=...), which raisedTypeErroron v3.apply_plan(lf, ..., return_report=True)raised a PolarsPerformanceWarningin the README quickstart.Why?
The README promises a dry-run plan "without silent mutations", backend honesty, and
strict_backend=Truerefusing anything that cannot run natively. Each item above breaks one of those promises. I confirmed every bug with a repro before fixing it.How was it tested?
tests/(66 → 94 tests). Each fix's test fails onmain. A parity test runs the same plan on pandas, Polars DataFrame, and LazyFrame and compares rows.pytest -m "not online and not large". Result: 94 passed, coverage 95.96%.ruff check .: 112 findings, against 113 onmain. All 112 exist onmaintoo.mypy src/freshdata: clean, against 1 error onmain.collectand no Python UDF inapply_plan.Any performance impact?
clean(): 2.57s → 1.90s (−26%). The refactor alone gave −33%. Exactrows_affectedcounting costs some of that back.profile()on 200 columns: 0.24s → 0.10s. It now runs one parallel query.suggest_plan: unchanged frommain. On the CI gate workload (2M rows), counting strip changes and outliers added about 60% to planning and failed the perf gate (total_ms+17.1%). 55ee1d5 keeps only the null counts, which cost nothing. Local gate rerun againstmain:plan_ms−1.4%,total_ms−15.1%.apply_plan: unchanged.Any compatibility concerns?
[Unreleased]in CHANGELOG.md:strip_stringsstrips surrounding whitespace and turns empty strings into missing values, on both planning and apply. pandas planning no longer collapses inner whitespace or treats the text"nan"/"none"/"null"as missing.strict_backend=True(the default) now raisesValueErrorfor an action a backend cannot run.strict_backend=Falsewarns and skips it.CleanConfigandColumnRuleraise on unknown values, a bare-stringid_columns, and a negativeiqr_factor.strategy,rolevalues other thanid/target, anddefault_outliers="auto"now emit aUserWarning, because they still have no effect.ColumnRule(role="id"|"target")now protects the column.outliers="flag"adds a<col>_outliercolumn. It raises rather than overwrite an existing column with that name.rows_affectedvalues change to real counts on pandas, and to null counts on Polars.fd.clean(df, **config_fields)andfd.Cleaner(**config_fields).Reviewer notes:
strip_strings, outlier actions, anddrop_duplicateskeeprows_affected=0there, and each action'srationalesays why. Counting them needs an extra string pass, broadcast quantiles, or a full-row hash. That breaks the perf gate or streaming-scale planning."".Documentation updated?
docs/. Not applicable:docs/still describes the v2 API and needs its own pass.examples/still use v2 arguments.[Unreleased]inCHANGELOG.md. ARCHITECTURE.md is also rewritten.🤖 Generated with Claude Code