[ChrisW09]

Fixes #76.

With scikit-learn's global output set to pandas or polars (sklearn.set_config(transform_output="pandas")), a Preprocessor that one-hot encodes a column (categorical_method="one-hot", a "one-hot" entry in feature_preprocessing, or preset="expanded") failed in fit with "Pandas output does not support sparse data". A one-hot Preprocessor fitted before the setting was switched on failed the same way in transform.

Root cause

Preprocessor.fit built and fitted its internal ColumnTransformer without an output configuration of its own, so the global transform_output applied to it and to every step inside it. The one-hot steps use a OneHotEncoder with the default sparse_output=True, and scikit-learn refuses pandas and polars output for sparse data. sparse_output is not exposed through Preprocessor, so users had no way around it other than switching the global setting off.

Fix

• Preprocessor.fit calls self.column_transformer_.set_output(transform="default") before fitting the internal ColumnTransformer. This also pins its steps, so they always produce plain arrays and the global setting no longer reaches them. The configuration is stored on the fitted ColumnTransformer, so transform of a Preprocessor fitted before the global setting was switched on works as well.
• Preprocessor.transform is unchanged: it still reads the requested container from its own output configuration (set_output or the global setting) and builds the pandas or polars frame from the stacked array itself, so it keeps returning a DataFrame under the global setting.

Behaviour changes

• Under a global transform_output="pandas" or "polars", one-hot encoding and preset="expanded" now fit, and transform returns a float64 DataFrame with the get_feature_names_out() columns and the same values as with the default configuration.
• fingerprint_ of a newly fitted Preprocessor changes, because the fitted internal ColumnTransformer now records its output configuration. The transformed output is unchanged, and no golden regression baseline pins a fingerprint.

Documentation

• docs/core_concepts/outputs_and_inspection.md ("DataFrame output"): a global sklearn.set_config(transform_output=...) is honoured like set_output and applies to the Preprocessor's own output only; the internal per-column steps always produce arrays, so every method, including the sparse one-hot encoding, works under it.
• Preprocessor.transform docstring (Returns): the pandas / polars DataFrame is also returned under the global setting, which applies to this output only.
• The README and the docs homepage do not describe the global output setting, so they are unchanged.

Tests

• test_one_hot_works_under_a_global_pandas_output_config[options0, options1] (categorical_method="one-hot", preset="expanded"): inside sklearn.config_context(transform_output="pandas"), fit and transform return a DataFrame with the get_feature_names_out() columns, all float64, and values equal to the default-configuration output. Fails on main (raises), passes here.
• test_global_polars_output_config_is_honoured: the same under transform_output="polars": a polars frame with the right column names and values (skipped when polars is not installed). Fails on main (raises), passes here.

Reproduction

Output of the script from #76 on this branch. The loop also calls transform and prints the output type, shape and dtypes, and the script adds the two other cases from the description: transform of a one-hot Preprocessor fitted before the global setting was switched on, and transform_output="polars".

(40, 8)
{'categorical_method': 'one-hot'} -> DataFrame (40, 10) ['float64']
{'preset': 'expanded'} -> DataFrame (40, 13) ['float64']
fitted before, transform -> DataFrame (40, 10)
polars -> polars DataFrame (40, 10)

On main, the last four lines are each "Pandas output does not support sparse data. Set sparse_output=False ..." ("Polars output ..." for the last).

Checks

• just test: 1441 passed, 59 skipped, 7 xfailed; coverage 93.65% (gate 90%).
• Minimum dependencies (CI job: Python 3.10, numpy 1.24.4, pandas 2.0.3, scipy 1.10.1, scikit-learn 1.6.0): 1427 passed, 73 skipped, 7 xfailed; scripts/quickstart.py passes.
• just docs: strict build succeeds.
• just check: all pre-commit and pre-push hooks pass (ruff format, ruff lint, prettier, file hygiene, pyright with 0 errors).
• Import direction between subpackages is unchanged.

Merge notes

• Base: main.
• The PRs for Preprocessor.fit crashes on any bool column under default settings #54 (fix(preprocessor): encode boolean columns as binary categoricals #99), Preprocessor routes integer column labels positionally, silently swapping columns #55 (fix(preprocessor): route columns to the ColumnTransformer by label #100), output_format="auto" chooses sparse vs dense per transform batch, breaking dense-only estimators at predict #73 (fix(preprocessor): resolve output_format='auto' once per fit #113) and Preprocessor crashes on an all-missing column at fit (numerical, one-hot) or silently drops it (int) #95 (fix(preprocessor): keep a block for columns with no observed value at fit #130) change the same self.column_transformer_.fit(X, y) line in Preprocessor.fit: Preprocessor.fit crashes on any bool column under default settings #54 and Preprocessor routes integer column labels positionally, silently swapping columns #55 pass self._column_transformer_input(X), output_format="auto" chooses sparse vs dense per transform batch, breaking dense-only estimators at predict #73 replaces it with fit_transform plus resolve_output_format, and Preprocessor crashes on an all-missing column at fit (numerical, one-hot) or silently drops it (int) #95 wraps it in a try/except. Whichever lands later gets a conflict there: keep this PR's comment and set_output(transform="default") line directly before whatever fit call results. output_format="auto" chooses sparse vs dense per transform batch, breaking dense-only estimators at predict #73 also appends tests at the end of tests/integration/test_output_format.py; keep both blocks. Once this PR is in, output_format="auto" chooses sparse vs dense per transform batch, breaking dense-only estimators at predict #73's np.asarray conversion of a dense training output is redundant but harmless.
• The PRs for Preprocessor.transform does not check n_features_in_ for ndarray input; extra columns are silently misaligned #72 (fix(preprocessor): reject NumPy input of the wrong width at transform #112) and Preprocessor fitted on a DataFrame rejects a same-width NumPy array at transform (and vice versa) #78 (fix(preprocessor): match array input by position after a named fit #116) add a Raises section to the Preprocessor.transform docstring directly after the Returns line this PR extends, and fix(output): preserve input index in set_output pandas container #68 (the PR for Preprocessor pandas output (set_output) drops the input index, breaking ColumnTransformer/FeatureUnion #60 (fix(output): preserve input index in set_output pandas container #68)) extends the same line with "(a pandas DataFrame keeps the index of X)". Keep both texts.
• The PR for add_missing_indicator / impute_with_indicator feeds the indicator through the scaler and basis; lineage crashes #62 (fix(preprocessor): keep the missing indicator out of the representation #106) (missing indicator as a separate MissingIndicator branch) merges cleanly with this one. With add_missing_indicator / impute_with_indicator feeds the indicator through the scaler and basis; lineage crashes #62 alone, the float representation blocks and the boolean indicator blocks are stacked into an object-dtype DataFrame under a global transform_output="pandas"; with this pin as well, the output is float64. Whichever of the two PRs lands second adds test_missing_indicator_stays_numeric_under_a_global_pandas_output_config to tests/integration/test_output_format.py: under that setting, Preprocessor(numerical_method="minmax", add_missing_indicator=True) returns only float columns and num_a__missingindicator_a equals X["a"].isna(). It is not part of this PR because on main the indicator comes from SimpleImputer(add_indicator=True), is already float, and the test passes without the fix.
• Preprocessor rejects polars DataFrames, so pipelines with set_output(transform="polars") crash #94 (polars input) touches no code changed here and merges cleanly. Under a global transform_output="polars", a pipeline passes polars frames between steps, so a Preprocessor after another step also needs Preprocessor rejects polars DataFrames, so pipelines with set_output(transform="polars") crash #94.

This change was previously part of #86, which is closed in favour of one pull request per issue.