Dry-run data repair plans for Polars and pandas—compiled into native expressions without silent mutations.
Most automated data-cleaning libraries fail in production for three reasons:
- Silent corruption: One-call black boxes impute missing target labels or clip primary keys without warning.
- Broken lazy execution: Wrappers claim Polars support but silently convert
LazyFrameinputs to pandas or call.unique(maintain_order=True), blowing up memory on multi-gigabyte scans. - Zero diffability: You cannot review or version-control what a heuristic cleaner is about to do before it mutates your data.
freshdata separates planning from execution:
fd.suggest_plan(lf)runs a single streaming aggregation pass over a PolarsLazyFrame(or pandasDataFrame), protects ID and target columns automatically, and outputs an inspectable, diffableCleanPlancontaining precomputed scalar thresholds.fd.apply_plan(lf, plan)compiles those approved actions into pure, fused Polars expressions (pl.Expr)—requiring zero.collect()calls and zero Python UDFs. ALazyFramein stays aLazyFrameout.
pip install freshdata-cleanerTo install the AST codemod tool for upgrading legacy 2.x scripts:
pip install "freshdata-cleaner[migrate]"import polars as pl
import freshdata as fd
# 1. Point to a lazy Parquet scan (no rows loaded into memory)
lf = pl.scan_parquet("s3://bucket/raw_customer_events.parquet")
# 2. Define explicit column protections and rules
config = fd.CleanConfig(
id_columns=("customer_id",),
target_column="churned",
default_impute="auto", # median for numeric columns
default_outliers="preserve",
drop_duplicates=True, # streaming-safe unique(maintain_order=False)
rules={
"signup_age": fd.ColumnRule(impute="median", outliers="clip"),
},
)
# 3. Generate a dry-run repair plan (IQR bounds come from one 1-row summary query)
plan = fd.suggest_plan(lf, config=config)
for action in plan.actions:
print(f"[{action.stage}] {action.column}: {action.action} {action.parameters}")
# Example output (abridged) for raw columns "Customer ID", "Signup Age", "Segment", ...:
# [column_names] Signup Age: normalize_name {'to': 'signup_age'}
# [missing] signup_age: impute_median {}
# [strings] segment: strip_strings {}
# [outliers] signup_age: clip_iqr {'lower': 27.125, 'upper': 48.125}
# [duplicates] *: drop_duplicates {}
# customer_id (id) and churned (target) get no repair actions.
# 4. Compile the approved plan onto the LazyFrame (zero materialization)
cleaned_lf, report = fd.apply_plan(lf, plan, return_report=True)
assert isinstance(cleaned_lf, pl.LazyFrame)
assert "PYTHON_UDF" not in cleaned_lf.explain()
# 5. Materialize only when your downstream pipeline is ready
df = cleaned_lf.collect(engine="streaming")Never guess how a configuration change alters your dataset. Compare two CleanPlan artifacts directly in code or CI:
import dataclasses
clipped = fd.suggest_plan(lf, dataclasses.replace(config, default_outliers="clip"))
diff = plan.diff(clipped)
for action in diff["added"]:
print(f"+ [{action.stage}] {action.column}: {action.action} {action.parameters}")
for action in diff["removed"]:
print(f"- [{action.stage}] {action.column}: {action.action}")
for change in diff["modified"]:
print(f"~ [{change['stage']}] {change['column']}: {change['before']['parameters']} -> {change['after']['parameters']}")
# + [outliers] invoice_total: clip_iqr {'lower': 7.125, 'upper': 15.125}freshdata v3.0 exposes exactly 12 public symbols. There are no hidden submodules, natural-language rule compilers, or domain-specific bloat.
| Export | Type | Purpose |
|---|---|---|
fd.profile |
Function | Read-only column null, cardinality, and quantile summary |
fd.suggest_plan |
Function | Single-pass streaming planner returning a dry-run CleanPlan |
fd.apply_plan |
Function | Compiles a CleanPlan into native expressions without materializing |
fd.clean |
Function | Shorthand for apply_plan(df, suggest_plan(df, config)) |
fd.Cleaner |
Class | Reusable stateful wrapper around CleanConfig for batch pipelines |
fd.CleanConfig |
Dataclass | Frozen pipeline specification with strict backend enforcement |
fd.ColumnRule |
Dataclass | Explicit per-column role, imputation, and outlier override |
fd.CleanPlan |
Dataclass | Serializable, diffable tuple of planned ActionRecord items |
fd.ActionRecord |
Dataclass | Single column operation with baked scalar parameters and rationale |
fd.CleanReport |
Dataclass | Post-application audit record of renamed/dropped columns and actions |
fd.ProfileReport |
Dataclass | Dataset-level read-only diagnostic container |
fd.ColumnProfile |
Dataclass | Per-column statistical summary emitted by fd.profile() |
freshdata is a planning and repair engine, not a schema validation gate or orchestration framework. Use it alongside focused tools in your stack:
| Tool | Primary Job | Mutates Data? | Lazy Polars Native? |
|---|---|---|---|
| Pandera / Great Expectations | Assert schema contracts and fail bad batches | No | Partial |
| pyjanitor | Imperative method-chaining helpers | Yes (manual) | No (pandas-first) |
| freshdata | Propose, diff, and compile automated repair plans | Yes (via CleanPlan) |
Yes (pl.Expr) |
By default, CleanConfig(strict_backend=True) refuses to run any operation that cannot be expressed natively on your input frame:
- No silent
.to_pandas()conversions: Passing apl.LazyFrameorpl.DataFrameexecutes 100% inside Polars. - No Python row loops: String normalization, null imputation, and IQR clipping compile down to
.str.strip_chars(),.fill_null(), and.clip(). - Streaming-compatible deduplication: When
drop_duplicates=True,freshdataemits.unique(maintain_order=False)so Polars streaming pipelines never stall holding row order in RAM.
If you are upgrading from freshdata 2.x (freshdata-cleaner), run the included libcst codemod to rewrite legacy wrappers (clean_csv, fill_missing, compare_plans, etc.) into the v3.0 core API:
# Preview unified diff without modifying files
freshdata-migrate src/ tests/ --check --diff
# Apply deterministic AST rewrites in-place
freshdata-migrate src/ tests/Bug reports, benchmark reproductions, and pull requests are welcome.
- Open an issue with a minimal reproducible Polars or pandas snippet.
- Run the test suite locally via
pytest -q. - Ensure any changes to
_core/backends.pypass the lazy materialization poison test (tests/test_backend_honesty.py).
MIT License. See LICENSE for details.