A map for contributors: what lives where in src/freshdata, how a call flows
through the code, and where to add behavior. Read it once before your first
change. Module docstrings and tests hold the detail.
For user-facing docs, see the documentation site.
fd.suggest_plan(df) builds an immutable, diffable CleanPlan of
ActionRecords. fd.apply_plan(df, plan) executes that plan with the backend
that matches the frame type. fd.clean(df) runs both in sequence.
src/freshdata/
├── __init__.py public API re-exports
├── _core/
│ ├── models.py frozen dataclasses: ColumnRule, CleanConfig, ActionRecord,
│ │ CleanPlan (+ diff), ColumnProfile, ProfileReport, CleanReport
│ ├── engine.py public functions + Cleaner; config defaults; report assembly
│ ├── backends.py PandasBackend, PolarsBackend, resolve_backend()
│ └── stages.py pandas planning stages + shared helpers
│ (_normalize_name, normalize_config, _is_protected)
└── cli/migrate.py freshdata-migrate: libcst codemod for 2.x -> 3.x scripts
Dependency direction is strictly one way:
engine -> backends -> stages -> models. Inside _core, no module imports engine.
fd.clean(df, cfg)
├─ suggest_plan(df, cfg)
│ └─ resolve_backend(df) dispatch on frame type
│ ├─ PandasBackend.suggest_plan
│ │ normalize_config(cfg)
│ │ for stage in (column_names, strings, dtypes,
│ │ duplicates, missing, outliers):
│ │ df, actions = stage(df, cfg) eager; each stage sees
│ │ the previous stage's output
│ └─ PolarsBackend.suggest_plan
│ schema only + at most ONE 1-row aggregate query
│ (IQR quantiles for clip columns)
│ -> CleanPlan(actions)
└─ apply_plan(df, plan, cfg)
├─ PandasBackend.apply_plan renames, then actions in plan order (eager)
└─ PolarsBackend.apply_plan renames -> one fused with_columns(...) ->
optional unique(); never collects
- Plans are data.
CleanPlanandActionRecordare frozen. A plan can be serialized (to_dict), diffed (diff), and edited before it is applied. - Lazy stays lazy.
PolarsBackend.apply_planmust not call.collect(),.to_pandas(), or Python UDFs.tests/test_backend_honesty.pyenforces this. - One definition per decision.
stages._is_protecteddecides which columns no stage may touch.stages._impute_strategyandstages._outlier_strategyresolve rules against defaults.stages._strip_stris the only pandas string executor. Do not copy these. - Every action runs everywhere or fails loudly. Both executors handle the
same action set. An action a backend cannot run raises
ValueErrorwhenstrict_backend=True(the default), and warns and skips otherwise. - pandas is required. Polars is optional.
resolve_backendimports Polars only for Polars frames.
| Change | Edit |
|---|---|
| New pandas planning rule | a stage_* function in stages.py, then add it to the tuple in PandasBackend.suggest_plan |
| Execute a new action | both PandasBackend.apply_plan and PolarsBackend.apply_plan, plus a case in test_every_pandas_action_runs_identically_on_polars |
| New config option | models.py (including __post_init__ validation), then every backend that must honor it |
| New frame type | a backend class with profile / suggest_plan / apply_plan, plus a branch in resolve_backend |
Both backends run the same actions. These differences remain by design or because closing them costs streaming-scale performance.
- Order of row operations. pandas runs actions in plan order, so it
deduplicates before it imputes, and a row dropped for one column changes the
statistics of the next. Polars computes each column's statistics on the input
frame, runs row filters after the fused
with_columns, and deduplicates last. - Statistics in plans. pandas plans carry imputation values. Polars plans compute them at apply time, inside the lazy query.
- Empty strings. pandas
strip_stringsturns""into a missing value. Polars keeps"". - Currency detection. Only the pandas planner detects currency columns
(
fix_dtypes). Both executors run aparse_currencyaction. Polars always returnsFloat64, and pandas may returnint64. - Counts. Polars plans count only what the planning query gets for free:
null counts for imputation,
drop_nulls, andffill.strip_strings, outlier actions, anddrop_duplicateskeeprows_affected=0, and theirrationalesays why. Counting them added about 60% to plan time on the CI performance gate. pandas counts every action.CleanReportrow counts are 0 for aLazyFrame, becauseapply_plannever collects. - Not detected: two raw column names that normalize to the same name.
- The pandas path is eager.
clean()transforms the frame twice (once to plan, once to apply), and every stage copies the frame. For large data, use a PolarsLazyFrame. - The pandas string stages run one Python call per cell.
_strip_stris the hottest function inclean(). Keep it allocation-light. - The Polars planner runs at most one 1-row aggregate query.
profile()also runs one query, so Polars evaluates all columns in parallel.