Skip to content

fix: backend parity and fail-loud execution for pandas and Polars - #537

Merged
JohnnyWilson16 merged 17 commits into
mainfrom
fix/backend-parity-and-safety
Nov 18, 2026
Merged

JohnnyWilson16 merged 17 commits into
mainfrom
fix/backend-parity-and-safety

Conversation

@JohnnyWilson16

@JohnnyWilson16 JohnnyWilson16 commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

What changed?

pandas and Polars now plan and execute the same actions, and the core library fails loudly where it used to fail silently. 17 commits. Each fix is its own commit and comes with a test that fails on main.

Refactor (no output change):

  • One shared helper per decision: _is_protected, _impute_strategy, _outlier_strategy, _strip_str. Previously 3 copies of the protection check had drifted apart.
  • Rewrote ARCHITECTURE.md. The old version described about 20 v2 modules that no longer exist.

Fixes:

  1. pandas strings. pandas apply_plan turned every non-string cell in an object column into None ([1, " a "] → [None, "a"]). Planning and applying also used different string logic, so plan statistics described data that apply_plan never produced.
  2. pandas protection. pandas stripped and currency-parsed ID, target, and protected=True columns.
  3. pandas mode type. impute_mode stored the mode as a string, so a float column became object.
  4. Rule fallback. Polars used a column's rule for both fields whenever the rule existed, so ColumnRule(outliers="clip") turned off default imputation on Polars only.
  5. Skipped actions. Polars ignored drop_nulls, drop_outliers, parse_currency, and impute_mode. Both backends skipped unknown actions and malformed parameters without a word.
  6. Value-less plans. pandas skipped impute actions without a precomputed value, so Polars plans lost their imputations when applied to a pandas frame.
  7. Unused options. ffill, flag, strict_backend, and role were accepted and did nothing. Typos such as impute="medain" were accepted silently too.
  8. Lazy profile. profile() on a LazyFrame reported 0 rows, 0 nulls, and 0 unique values.
  9. rows_affected. pandas reported the frame's row count. Polars reported 0 for every action. pandas now counts every action exactly. Polars counts the nulls that imputation fills or drop_nulls removes, and states a reason wherever it does not count.
  10. Codemod output. freshdata-migrate emits fd.clean(df, default_impute=...), which raised TypeError on v3.
  11. Report warning. apply_plan(lf, ..., return_report=True) raised a Polars PerformanceWarning in the README quickstart.

Why?

The README promises a dry-run plan "without silent mutations", backend honesty, and strict_backend=True refusing anything that cannot run natively. Each item above breaks one of those promises. I confirmed every bug with a repro before fixing it.

How was it tested?

  • New unit tests added in tests/ (66 → 94 tests). Each fix's test fails on main. A parity test runs the same plan on pandas, Polars DataFrame, and LazyFrame and compares rows.
  • Ran fast CI lane locally: pytest -m "not online and not large". Result: 94 passed, coverage 95.96%.
  • Ran linting and type checks:
    • ruff check .: 112 findings, against 113 on main. All 112 exist on main too.
    • mypy src/freshdata: clean, against 1 error on main.
  • The refactor commit (8b222ea) was checked against a frozen copy of the original package. 1,500 fuzzed configs across pandas, Polars, and LazyFrame gave identical plans, outputs, dtypes, reports, and errors.
  • The lazy poison test still passes. A LazyFrame stays lazy, with no collect and no Python UDF in apply_plan.

Any performance impact?

  • None / negligible
  • Measured on 300k rows × 11 columns:
    • pandas clean(): 2.57s → 1.90s (−26%). The refactor alone gave −33%. Exact rows_affected counting costs some of that back.
    • Polars profile() on 200 columns: 0.24s → 0.10s. It now runs one parallel query.
    • Polars lazy suggest_plan: unchanged from main. On the CI gate workload (2M rows), counting strip changes and outliers added about 60% to planning and failed the perf gate (total_ms +17.1%). 55ee1d5 keeps only the null counts, which cost nothing. Local gate rerun against main: plan_ms −1.4%, total_ms −15.1%.
    • Polars apply_plan: unchanged.

Any compatibility concerns?

  • None / fully backward-compatible
  • Behavior changes. All are listed under [Unreleased] in CHANGELOG.md:
    • strip_strings strips surrounding whitespace and turns empty strings into missing values, on both planning and apply. pandas planning no longer collapses inner whitespace or treats the text "nan"/"none"/"null" as missing.
    • strict_backend=True (the default) now raises ValueError for an action a backend cannot run. strict_backend=False warns and skips it.
    • CleanConfig and ColumnRule raise on unknown values, a bare-string id_columns, and a negative iqr_factor.
    • strategy, role values other than id/target, and default_outliers="auto" now emit a UserWarning, because they still have no effect.
    • ColumnRule(role="id"|"target") now protects the column.
    • outliers="flag" adds a <col>_outlier column. It raises rather than overwrite an existing column with that name.
    • rows_affected values change to real counts on pandas, and to null counts on Polars.
    • New: fd.clean(df, **config_fields) and fd.Cleaner(**config_fields).

Reviewer notes:

  • Some actions are not counted on Polars. strip_strings, outlier actions, and drop_duplicates keep rows_affected=0 there, and each action's rationale says why. Counting them needs an extra string pass, broadcast quantiles, or a full-row hash. That breaks the perf gate or streaming-scale planning.
  • Remaining backend differences are recorded, not fixed. ARCHITECTURE.md lists them under "Known backend differences". Examples: the order of row operations, and Polars keeping "".

Documentation updated?

  • Documentation updated in docs/. Not applicable: docs/ still describes the v2 API and needs its own pass.
  • Examples verified or updated. Not done: several examples/ still use v2 arguments.
  • Note added under [Unreleased] in CHANGELOG.md. ARCHITECTURE.md is also rewritten.

🤖 Generated with Claude Code

JohnnyWilson16 and others added 16 commits November 18, 2026 17:30
- Add stages._is_protected so pandas and Polars use one ID/target/protected
  predicate instead of three copies.
- Move the per-cell string cleaner to module level, check for str first, and
  replace re.sub with str.split/join. Output is identical (checked against
  every Unicode whitespace character and 1,500 fuzzed configs).
- Rewrite ARCHITECTURE.md to describe the v3 code that exists.

pandas clean() is about 33% faster on 300k rows x 11 columns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
suggest_plan collapsed inner whitespace and turned "nan"/"none"/"null" text
into missing values. apply_plan only stripped. Plan statistics (mode values,
medians, rows_affected) therefore described data that apply_plan never
produced. apply_plan also replaced every non-string cell in an object column
with None.

Both paths now call stages._strip_str: strip surrounding whitespace, turn
empty strings into None, keep non-string cells. This matches the README
(".str.strip_chars()") and the Polars executor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The plan stored str(mode). apply_plan then wrote "1.0" into a float column
and turned it into an object column. Store the plain Python scalar instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ID columns, the target column, *_id columns and protected=True rules were
stripped and currency-parsed on pandas. The Polars planner already skipped
them. Both stages now use stages._is_protected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Polars used a column's rule for both impute and outliers whenever the rule
existed, so ColumnRule(outliers="clip") silently turned off default
imputation (and the reverse). Both backends now resolve each field through
stages._impute_strategy and stages._outlier_strategy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Polars apply_plan ignored drop_nulls, drop_outliers, parse_currency and
impute_mode, and both backends skipped unknown actions and malformed
parameters without a word. pandas also skipped impute actions that carry no
precomputed value, which is every impute action a Polars plan contains.

- Polars: native executors for the four missing actions. Row filters run
  after the fused with_columns. No collect and no Python UDFs.
- pandas: compute median/mean/mode when the plan has no value.
- Polars planner: honor impute="mode", impute="drop" and outliers="drop".
- Both: an action that cannot run raises ValueError when strict_backend=True
  (the default, as the README promises). With strict_backend=False it is
  skipped with a UserWarning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both options were accepted by ColumnRule/CleanConfig and did nothing.

- impute="ffill": forward-fill the column (pandas ffill, Polars forward_fill).
- outliers="flag": add a boolean <col>_outlier column marking values outside
  the IQR fences, the meaning the v2 docs gave it. Values are not changed.
  If <col>_outlier already exists, planning and apply raise ValueError
  instead of overwriting it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hing

ColumnRule and CleanConfig accepted any value, so a typo such as
impute="medain" silently did nothing, and id_columns="customer_id" matched
column names by substring.

- Reject unknown role/impute/outliers/strategy/default values, a bare string
  for id_columns, non-ColumnRule rules, and a negative iqr_factor.
- Widen the Literal hints to every value the stages already handle
  (default_outliers="drop" is what freshdata-migrate emits).
- ColumnRule(role="id"|"target") now protects the column, like id_columns
  and target_column.
- Warn when an option has no effect: strategy other than "balanced", other
  roles, and default_outliers="auto".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
freshdata-migrate rewrites v2 calls into fd.clean(df, default_impute=...)
and fd.Cleaner(id_columns=...), but both raised TypeError on v3. Keyword
arguments now override fields of the given (or default) CleanConfig and go
through the same validation. Unknown keywords still raise TypeError.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
profile() returned 0 rows, 0 nulls and 0 unique values for every LazyFrame
column, which reads as real data. Both DataFrame and LazyFrame input now run
one select of len/null_count/n_unique, which Polars evaluates in parallel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pandas reported the frame's row count for normalize_name, strip_strings,
parse_currency and clip_iqr. Polars reported 0 for every action.

- pandas: renames change 0 cells. strip_strings counts changed cells,
  parse_currency counts parsed cells, clip_iqr counts clipped values.
- Polars: the planner computes the same counts in its one aggregate query
  (null counts, strip changes, values outside the IQR fences).
  drop_duplicates stays uncounted on Polars, and its rationale says why:
  counting needs a full-row hash, which would break streaming-scale planning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inlining the quantiles in the outside-fence count made Polars evaluate each
one three times (79ms vs 26ms for 6 columns x 300k rows). The planner now
adds the quantiles with with_columns and counts against those columns. It is
still one query.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
apply_plan(..., return_report=True) called len(lf.columns), which makes
Polars emit a PerformanceWarning in the README quickstart. Use
collect_schema() for Polars frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ARCHITECTURE.md now states the shared-helper and strict-execution rules and
replaces the fixed "Known issues" with the backend differences that remain.
CHANGELOG.md lists the user-visible changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Type runtime-checked config values as object, take pandas column labels as
Any, import Container from collections.abc, and drop a bool check on
iqr_factor that mypy proves unreachable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 6b0c4fb3-9bd1-47a3-b760-5199e8b84e6a
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

… free

The Polars LazyFrame perf gate failed: total_ms +17.1% (limit +15%), driven
by plan_ms 111 -> 174 ms. Counting strip_strings changes needs a per-cell
string pass, and counting values outside the IQR fences needs the
quantiles broadcast as columns. Together they added about 60% to planning
on the 2M-row gate workload.

The planner keeps the free counts (null counts for imputation, drop_nulls
and ffill) and goes back to main's single-query shape. strip_strings and
outlier actions keep rows_affected=0, with a rationale that says why, like
drop_duplicates. pandas still counts every action.

Local gate reproduction (5 iterations, 2M rows) against origin/main:
plan_ms -1.4%, total_ms -15.1%. The previous head gave plan_ms +155%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JohnnyWilson16
JohnnyWilson16 merged commit 0e89722 into main Nov 18, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant