[ChrisW09]

Fixes #90.

CrossFittedTransformer.fit_transform wrote the out-of-fold rows into a preallocated dense float64 array whatever the wrapped transformer returned, so it disagreed with transform: string columns left unchanged (Preprocessor(categorical_method=None)) and sparse output crashed, float32 output came back as float64, and a DataFrame-returning transformer gave an ndarray.

Root cause

fit_transform allocated out = np.empty((n_rows, width), dtype=float), converted each fold with np.asarray(fold.transform(...)) and assigned it into out. np.asarray turns a sparse matrix into a 0-d object array, so reading fold_out.shape[1] raised IndexError, and the assignment cast every fold to float64, which fails for strings.

Fix

• fit_transform collects each fold's output with its held-out row indices, and the new _stack_folds in pretab/core/supervised.py stacks them in the row order of X, in the container the wrapped transformer returns:
  • sparse blocks: scipy.sparse.vstack, rows reordered with the inverse permutation, returned in the first block's format;
  • pandas DataFrames: pd.concat and reordering; the index of X is kept when the blocks carry it (scikit-learn's set_output copies the input index), otherwise the rows are numbered from 0, as transform numbers the rows of a Preprocessor;
  • polars DataFrames: pl.concat and reordering (polars is imported only when the output already is a polars frame);
  • dense blocks: a buffer of their common dtype (np.result_type, object for strings).
• The width check still runs on every fold. A fold whose DataFrame columns differ from the all-data fit's at the same width now raises, because frames are concatenated by column label and would otherwise be widened into NaN-padded columns. A dict of blocks raises a clear error.

Behaviour changes

• fit_transform returns the container and dtype transform returns: a sparse matrix in the same format, a pandas or polars DataFrame, or a dense array of the folds' common dtype. For example, cross-fitting a standalone PLETransformer now gives float32, its native dtype, where fit_transform used to give float64.
• A DataFrame result is numbered from 0 like transform's output, since on main the wrapped transformer always receives an ndarray. With the PR for CrossFittedTransformer converts DataFrames to ndarray; a wrapped Preprocessor treats every column as categorical #58 (fix(cross-fitting): pass DataFrames to the wrapped transformer unchanged #103), the index of a DataFrame X is kept when the fold frames carry it.
• A fold whose DataFrame output has other column labels than the all-data fit raises IncompatibleParamsError ("Cross-fitting requires the same output columns across folds; a fold's columns differ from the all-data fit in [...]").
• A wrapped transformer that returns a dict of blocks (Preprocessor(output_structure="blocks")) raises IncompatibleParamsError ("... the wrapped transformer returned a dict of blocks. Use output_structure='matrix' on a wrapped Preprocessor.") instead of an IndexError.

Documentation

• CrossFittedTransformer class docstring: fit_transform and transform both return the wrapped transformer's kind of output (a dense array of the same dtype, a sparse matrix in the same format, or a DataFrame).
• fit_transform docstring: new Returns and Raises sections.
• docs/core_concepts/target_awareness.md: one sentence in the cross-fitting section saying both methods return the same kind of output, the one the wrapped transformer produces.
• Docstring of the new private _stack_folds.

Tests

All in tests/core/test_cross_fitted.py:

• test_fit_transform_keeps_strings_left_unchanged: Preprocessor(categorical_method=None) on an object array returns an object array equal to X, like transform. Fails on main ("could not convert string to float"), passes here.
• test_fit_transform_stacks_out_of_fold_values_with_strings: a ColumnTransformer with a PLE column and a passthrough string column gives an object array whose PLE part equals the out-of-fold PLE of the numeric column alone and whose strings are unchanged. Fails on main, passes here.
• test_fit_transform_keeps_sparse_output: output_format="sparse" gives CSR like transform, with the values of the dense run. Fails on main (IndexError), passes here.
• test_fit_transform_keeps_the_configured_dtype: dtype=np.float32 gives float32 like transform, with the values of the default run (rtol 1e-6). Fails on main, passes here.
• test_fit_transform_returns_the_dataframe_transform_returns: with set_output(transform="pandas"), a DataFrame whose columns and index equal transform's and whose values equal the array run. Fails on main, passes here.
• test_fit_transform_returns_polars_like_transform: the same with set_output(transform="polars"). Fails on main, passes here; skipped without polars (the minimum-dependencies job).
• test_stacked_dataframe_folds_keep_the_input_index_they_carry: unit test of _stack_folds. Frames that carry X's index keep it, in X's row order; frames that do not are renumbered. Fails on main (no _stack_folds), passes here.
• test_fit_transform_rejects_folds_with_other_dataframe_columns: with unshuffled folds and a one-hot encoder with max_categories=2, fold 0 keeps category "b" where the all-data fit keeps "a"; the error names x1_a. Fails on main, passes here.
• test_cross_fitting_rejects_blocks: output_structure="blocks" raises the "dict of blocks" error. Fails on main (IndexError), passes here.

Reproduction

Output of the script from #90 on this branch:

categorical_method=None          transform: ndarray object | fit_transform: ndarray object
one-hot, output_format='sparse'  transform: csr_matrix float64 | fit_transform: IncompatibleParamsError: Cross-fitting requires a fixed output width across folds; expected 203, got 136. Disable adaptive sizing on the wrapped transformer.
dtype=float32                    transform: ndarray float32 | fit_transform: ndarray float32

The literal sparse case also needs the PRs for #58 (#103) and #79 (#117). It no longer fails on the container but on the width: on main the mixed frame becomes an object array (#58), so the float column a is detected as categorical and one-hot encoded too (200 + 3 = 203 columns), and the per-fold refit relearns those categories on 133 training rows (#79), giving 136. With #58 and #79 added, the literal script returns csr_matrix float64 for this case. The sparse container itself works on this branch, as does the pandas case from the issue description; both on the same data with city coded as integers, so the frame stays numeric:

one-hot, output_format='sparse'  transform: csr_matrix float32 | fit_transform: csr_matrix float32
set_output(transform='pandas')   transform: DataFrame ['float32'] | fit_transform: DataFrame ['float32']

Checks

• just test: 1447 passed, 59 skipped, 7 xfailed; coverage 93.67% (gate 90%).
• Minimum dependencies (CI job: Python 3.10, numpy 1.24.4, pandas 2.0.3, scipy 1.10.1, scikit-learn 1.6.0): 1433 passed, 73 skipped, 7 xfailed; scripts/quickstart.py passes.
• just docs: strict build succeeds.
• just check: all pre-commit and pre-push hooks pass (ruff format, ruff lint, prettier, file hygiene, pyright with 0 errors).
• Import direction between subpackages is unchanged.

Merge notes

• Base: main. Not stacked: every symptom in the issue reproduces on main with NumPy input.
• The PR for CrossFittedTransformer(Preprocessor) relearns column types and category codes per fold, unlike transform #79 (fix(cross-fitting): refit only target-aware Preprocessor blocks per fold #117) rewrites the same fold loop. The merged loop gets fold = self._fit_fold(...) from CrossFittedTransformer(Preprocessor) relearns column types and category codes per fold, unlike transform #79, then fold_out = fold.transform(...), this PR's dict / shape / width / column checks and blocks.append(...). Take CrossFittedTransformer(Preprocessor) relearns column types and category codes per fold, unlike transform #79's width-mismatch message, or CrossFittedTransformer(Preprocessor(add_missing_indicator=True)) crashes when a fold's training rows contain no missing value #89's once that lands ("... configure it to produce the same columns on every fold."). The output-container handling (sparse output, the dict-of-blocks check) is only in this PR.
• The PR for CrossFittedTransformer converts DataFrames to ndarray; a wrapped Preprocessor treats every column as categorical #58 (fix(cross-fitting): pass DataFrames to the wrapped transformer unchanged #103) conflicts in the fold loop: take the fold rows with _take_rows(X_arr, ...). _stack_folds(blocks, test_indices, X_arr) then receives the DataFrame, so keeping its index becomes reachable through the public API; whichever of the two PRs lands second can add a test that the index of scikit-learn set_output frames is kept.
• With CrossFittedTransformer converts DataFrames to ndarray; a wrapped Preprocessor treats every column as categorical #58, CrossFittedTransformer(Preprocessor) relearns column types and category codes per fold, unlike transform #79, CrossFittedTransformer(Preprocessor(add_missing_indicator=True)) crashes when a fold's training rows contain no missing value #89 and CrossFittedTransformer.fit_transform forces a dense float64 array: string and sparse outputs crash #90 combined, the full test suite passes (1473 passed, 59 skipped, 8 xfailed), and the script from CrossFittedTransformer.fit_transform forces a dense float64 array: string and sparse outputs crash #90 shows every case fixed.
• _stack_folds follows whatever container the wrapped transformer returns, so the other output-container fixes (Preprocessor pandas output (set_output) drops the input index, breaking ColumnTransformer/FeatureUnion #60, Preprocessor with one-hot encoding or preset="expanded" fails under set_config(transform_output="pandas") #76) need no change here. Polars input (Preprocessor rejects polars DataFrames, so pipelines with set_output(transform="polars") crash #94, which needs a follow-up in CrossFittedTransformer once CrossFittedTransformer converts DataFrames to ndarray; a wrapped Preprocessor treats every column as categorical #58 lands) is independent of the polars output stacking here.

This change was previously part of #97, which is closed in favour of one pull request per issue.