Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
8b222ea
refactor(core): share the protection check and speed up string cleaning
JohnnyWilson16 Nov 18, 2026
50de81f
fix(pandas): use one strip_strings executor for planning and apply
JohnnyWilson16 Nov 18, 2026
bfe0bc7
fix(pandas): keep the mode value's type in impute_mode plans
JohnnyWilson16 Nov 18, 2026
c13e96a
fix(pandas): skip protected columns in the strings and dtypes stages
JohnnyWilson16 Nov 18, 2026
f443da7
fix(polars): fall back to config defaults per rule field, like pandas
JohnnyWilson16 Nov 18, 2026
07b4982
fix: run every plan action on both backends and enforce strict_backend
JohnnyWilson16 Nov 18, 2026
354b90f
feat: implement impute="ffill" and outliers="flag" on both backends
JohnnyWilson16 Nov 18, 2026
5720830
fix(config): validate options and stop accepting settings that do not…
JohnnyWilson16 Nov 18, 2026
03ff405
fix(api): let clean() and Cleaner() take CleanConfig fields as keywords
JohnnyWilson16 Nov 18, 2026
f3efa2c
fix(polars): profile LazyFrames with real counts in one aggregate query
JohnnyWilson16 Nov 18, 2026
397a493
fix: make rows_affected count the cells an action changes
JohnnyWilson16 Nov 18, 2026
416871f
perf(polars): compute each IQR quantile once when counting outliers
JohnnyWilson16 Nov 18, 2026
def2908
fix(api): count LazyFrame report columns from the schema
JohnnyWilson16 Nov 18, 2026
0305b52
docs: record the backend contract and the remaining differences
JohnnyWilson16 Nov 18, 2026
f44a037
chore(types): keep mypy clean for the new validation and helpers
JohnnyWilson16 Nov 18, 2026
8f04493
test: drop a duplicate pytest import
JohnnyWilson16 Nov 18, 2026
55ee1d5
perf(polars): keep plan-time counting to what the planning query gets…
JohnnyWilson16 Nov 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
194 changes: 98 additions & 96 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -1,111 +1,113 @@
# Architecture

This is a map for contributors: what lives where in `src/freshdata`, how a
`clean()` call flows through the code, and where to add new behavior. It is
deliberately high-level — read it once before your first code change, then let
the module docstrings and tests fill in the detail.
A map for contributors: what lives where in `src/freshdata`, how a call flows
through the code, and where to add behavior. Read it once before your first
change. Module docstrings and tests hold the detail.

For *user*-facing explanations of features, see the
[documentation site](https://freshcode-org.github.io/freshdata/). This file is
about the *code*.
For user-facing docs, see the
[documentation site](https://freshcode-org.github.io/freshdata/).

## The one-sentence version

`fd.clean(df)` profiles every column, asks a rule-based **decision engine** which
action each column needs, applies those actions as small vectorized **steps**,
and returns the cleaned frame plus a **report** that explains every change.
`fd.suggest_plan(df)` builds an immutable, diffable `CleanPlan` of
`ActionRecord`s. `fd.apply_plan(df, plan)` executes that plan with the backend
that matches the frame type. `fd.clean(df)` runs both in sequence.

## Core flow
## Layout

```
fd.clean(df) # api.py — the public entry point
│
▼
Cleaner # cleaner.py — orchestrates the pipeline
│
├─ profile the frame # profile.py — per-column stats & inferred role
│
├─ decision engine # engine/ — pick an action per column
│ ├─ missing.py # imputation strategy
│ ├─ outliers.py # outlier handling
│ ├─ context.py # column-role / context inference
│ └─ model_select.py # when to use model-based imputation
│
├─ apply cleaning steps # steps/ — pure vectorized transforms
│ ├─ missing.py dtypes.py duplicates.py
│ ├─ outliers.py strings.py columns.py
│ └─ prune.py memory.py
│
└─ build the report # report.py — every Action, with rationale,
# risk, confidence, affected-cell count
src/freshdata/
├── __init__.py public API re-exports
├── _core/
│ ├── models.py frozen dataclasses: ColumnRule, CleanConfig, ActionRecord,
│ │ CleanPlan (+ diff), ColumnProfile, ProfileReport, CleanReport
│ ├── engine.py public functions + Cleaner; config defaults; report assembly
│ ├── backends.py PandasBackend, PolarsBackend, resolve_backend()
│ └── stages.py pandas planning stages + shared helpers
│ (_normalize_name, normalize_config, _is_protected)
└── cli/migrate.py freshdata-migrate: libcst codemod for 2.x -> 3.x scripts
```

Two invariants hold everywhere in this path and are enforced in review (see
[CONTRIBUTING.md](CONTRIBUTING.md)):
Dependency direction is strictly one way:
`engine -> backends -> stages -> models`. Inside `_core`, no module imports `engine`.

1. **Everything is reported.** A transformation that changes data must record an
`Action` (see `report.py`) with an affected-cell count.
2. **Vectorized only.** No row-wise `apply` or Python loops over rows in the
cleaning path.
## Data flow

## Top-level modules
```
fd.clean(df, cfg)
├─ suggest_plan(df, cfg)
│ └─ resolve_backend(df) dispatch on frame type
│ ├─ PandasBackend.suggest_plan
│ │ normalize_config(cfg)
│ │ for stage in (column_names, strings, dtypes,
│ │ duplicates, missing, outliers):
│ │ df, actions = stage(df, cfg) eager; each stage sees
│ │ the previous stage's output
│ └─ PolarsBackend.suggest_plan
│ schema only + at most ONE 1-row aggregate query
│ (IQR quantiles for clip columns)
│ -> CleanPlan(actions)
└─ apply_plan(df, plan, cfg)
├─ PandasBackend.apply_plan renames, then actions in plan order (eager)
└─ PolarsBackend.apply_plan renames -> one fused with_columns(...) ->
optional unique(); never collects
```

| Module | Responsibility |
|---|---|
| `api.py` | The public convenience functions: `clean`, `clean_csv`, `profile`, `plan`, `learn`, `mask_dataframe`, `link`. Start here to trace any public call. |
| `cleaner.py` | The `Cleaner` front-end and the pipeline that wires profiling → engine → steps → report. |
| `profile.py` | Read-only profiling: what is in a frame and what `clean()` *would* do, with no mutation. |
| `plan.py` / `repairplan.py` | Dry-run plans (`plan.py`) and executable, reviewable repair plans (`repairplan.py`). |
| `report.py`, `result.py`, `_reportframe.py` | The report objects returned to users — `Action` records, summaries, `to_dict()`. |
| `provenance.py` | Provenance-aware cleaning for OCR / PDF / document-derived tables. |
| `guard.py` | The hard safety gate that physically protects columns from modification. |
| `findings.py`, `fieldcheck.py`, `textclean.py`, `textlint.py` | Per-cell validation (`fieldcheck`), field-aware text cleaning (`textclean`), and the normalized finding model (`findings`). |
| `insight.py`, `stakeholder.py`, `explain.py` | Presentation layers: action-oriented insight output, business-language stakeholder summaries, and per-decision explanations. |
| `config.py`, `_util.py`, `_sentinels.py` | Configuration and shared internals. |

## Subpackages

Each subpackage has a one-line docstring at the top of its `__init__.py` —
these summaries come from that code.

| Package | What it does |
## Invariants

1. **Plans are data.** `CleanPlan` and `ActionRecord` are frozen. A plan can be
serialized (`to_dict`), diffed (`diff`), and edited before it is applied.
2. **Lazy stays lazy.** `PolarsBackend.apply_plan` must not call `.collect()`,
`.to_pandas()`, or Python UDFs. `tests/test_backend_honesty.py` enforces this.
3. **One definition per decision.** `stages._is_protected` decides which
columns no stage may touch. `stages._impute_strategy` and
`stages._outlier_strategy` resolve rules against defaults. `stages._strip_str`
is the only pandas string executor. Do not copy these.
4. **Every action runs everywhere or fails loudly.** Both executors handle the
same action set. An action a backend cannot run raises `ValueError` when
`strict_backend=True` (the default), and warns and skips otherwise.
5. **pandas is required. Polars is optional.** `resolve_backend` imports Polars
only for Polars frames.

## Where to add behavior

| Change | Edit |
|---|---|
| `engine/` | The rule-based decision engine behind `strategy=` — picks an action per column. |
| `steps/` | Individual cleaning steps, each a pure vectorized function. |
| `imputation/` | Internal imputation engines (mean/median/mode → KNN/model-based with the `ml` extra). |
| `domains/` | Domain-specific validator packs (finance, healthcare, energy, retail, transport, education, agriculture, media) plus reference data. **The most self-contained place to start** — see [CONTRIBUTING_DOMAINS.md](CONTRIBUTING_DOMAINS.md). |
| `semantic/` | The offline Semantic Cleaning Layer — experts, scoring, routing. Off by default. |
| `context/` | The deterministic natural-language context compiler (plain-English rules → policy). |
| `enterprise/` | Clustering / entity resolution, trust scoring, lineage, and PII masking. |
| `compliance/` | Maps a `CleanReport` to named control frameworks (21 CFR Part 11, GDPR, HIPAA, SOX, ALCOA+). |
| `execution/` | Pluggable out-of-core backends (Polars / DuckDB / Spark) behind a shared logical plan. |
| `streaming/` | Micro-batch and time-series-aware cleaning in bounded memory. |
| `parsers/` | Structural format readers (HL7 v2, GPX, SDMX, EDIFACT, FHIR). |
| `render/` | Interactive HTML rendering of report objects (the `viz` extra). |
| `models/` | Optional local model registry and runtime (`fd.models`). |
| `learning/` | Paired-data learning: `fd.learn` and reusable `.fdprofile` profiles. |
| `integrations/` | Run `clean` + a trust gate inside orchestrators (Dagster / Airflow / dbt). |
| `adapters/` | Optional framework adapters (Polars, etc.). |
| `experimental/` | Features whose APIs may change between minor releases (e.g. the AI Copilot). |
| `plugins.py` + `testing.py` | The entry-point plugin system and `fd.testing` contract helpers for third-party experts/backends/validators. |

## Where do I add…?

- **A new cleaning behavior** → a step in `steps/`, wired through the engine in
`engine/`, with an `Action` recorded in the report. Add the "does not fire
when it shouldn't" test.
- **A new domain validator** → a pack in `domains/`. Follow
[CONTRIBUTING_DOMAINS.md](CONTRIBUTING_DOMAINS.md) — this path is designed to
be self-contained and is the friendliest first code contribution.
- **A new imputation method** → `imputation/`, selected from `engine/model_select.py`.
- **A new execution backend** → `execution/` + a plugin registration; see
[`docs/plugins.md`](https://freshcode-org.github.io/freshdata/plugins/).
- **A new output/report surface** → `render/`, `insight.py`, or `stakeholder.py`.
- **A format parser** → `parsers/`.

## What to read next

- [CONTRIBUTING.md](CONTRIBUTING.md) — setup, the checks CI runs, and the PR workflow.
- [CONTRIBUTING_DOMAINS.md](CONTRIBUTING_DOMAINS.md) — the deep dive for domain packs.
- [Contributor roadmap](docs/community/contributor-roadmap.md) — tasks grouped by difficulty.
- The module docstring and the tests next to any file you plan to change.
| New pandas planning rule | a `stage_*` function in `stages.py`, then add it to the tuple in `PandasBackend.suggest_plan` |
| Execute a new action | both `PandasBackend.apply_plan` and `PolarsBackend.apply_plan`, plus a case in `test_every_pandas_action_runs_identically_on_polars` |
| New config option | `models.py` (including `__post_init__` validation), then every backend that must honor it |
| New frame type | a backend class with `profile` / `suggest_plan` / `apply_plan`, plus a branch in `resolve_backend` |

## Known backend differences

Both backends run the same actions. These differences remain by design or
because closing them costs streaming-scale performance.

- **Order of row operations.** pandas runs actions in plan order, so it
deduplicates before it imputes, and a row dropped for one column changes the
statistics of the next. Polars computes each column's statistics on the input
frame, runs row filters after the fused `with_columns`, and deduplicates last.
- **Statistics in plans.** pandas plans carry imputation values. Polars plans
compute them at apply time, inside the lazy query.
- **Empty strings.** pandas `strip_strings` turns `""` into a missing value.
Polars keeps `""`.
- **Currency detection.** Only the pandas planner detects currency columns
(`fix_dtypes`). Both executors run a `parse_currency` action. Polars always
returns `Float64`, and pandas may return `int64`.
- **Counts.** Polars plans count only what the planning query gets for free:
null counts for imputation, `drop_nulls`, and `ffill`. `strip_strings`,
outlier actions, and `drop_duplicates` keep `rows_affected=0`, and their
`rationale` says why. Counting them added about 60% to plan time on the CI
performance gate. pandas counts every action. `CleanReport` row counts are 0 for a
`LazyFrame`, because `apply_plan` never collects.
- **Not detected:** two raw column names that normalize to the same name.

## Performance notes

- The pandas path is eager. `clean()` transforms the frame twice (once to plan,
once to apply), and every stage copies the frame. For large data, use a
Polars `LazyFrame`.
- The pandas string stages run one Python call per cell. `_strip_str` is the
hottest function in `clean()`. Keep it allocation-light.
- The Polars planner runs at most one 1-row aggregate query. `profile()` also
runs one query, so Polars evaluates all columns in parallel.
35 changes: 35 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,43 @@ adheres to [Semantic Versioning](https://semver.org/).

## [Unreleased]

### Added

- `impute="ffill"` forward-fills a column, and `outliers="flag"` adds a boolean
`<col>_outlier` column. Both were accepted before but did nothing.
- `fd.clean()` and `fd.Cleaner()` take `CleanConfig` fields as keyword
arguments, so the code that `freshdata-migrate` writes now runs.

### Changed

- `strip_strings` now means one thing everywhere: strip surrounding whitespace
and turn empty strings into missing values. pandas planning no longer
collapses inner whitespace or treats `"nan"`/`"none"`/`"null"` text as missing.
- `strict_backend=True` (the default) now raises `ValueError` for any plan
action a backend cannot run, as documented. `strict_backend=False` warns and
skips it.
- `CleanConfig` and `ColumnRule` reject unknown option values, a bare string for
`id_columns`, and a negative `iqr_factor`. Options with no effect
(`strategy`, most `role` values, `default_outliers="auto"`) warn.
- pandas `clean()` is about 33% faster on string-heavy frames.

### Fixed

- pandas `apply_plan` no longer replaces non-string cells in object columns
with `None`.
- pandas no longer strips or currency-parses ID, target, or protected columns.
`ColumnRule(role="id"|"target")` now protects a column.
- pandas `impute_mode` keeps the value's type instead of writing a string into
numeric columns.
- Polars falls back to the config defaults per rule field, like pandas.
- Polars runs `drop_nulls`, `drop_outliers`, `parse_currency` and `impute_mode`
actions, and its planner honors `mode`, `drop` and outlier `drop` rules.
pandas runs value-less impute actions from Polars plans.
- Polars `profile()` reports real row, null and unique counts for `LazyFrame`.
- pandas `rows_affected` counts the cells an action changes. Polars counts the
nulls that imputation fills or `drop_nulls` removes. Where Polars does not
count (to keep planning to one cheap query), `rationale` says so.

- Polars and DuckDB now disclose a pandas fallback for categorical and Arrow
dictionary-backed pandas inputs. This preserves pandas text-cleaning and
category metadata behavior rather than silently skipping or changing it.
Expand Down
Loading
Loading