[ChrisW09]

Fixes #80.

Every spec that to_spec(path) wrote for an imputing Preprocessor, which is the default, contained bare NaN tokens for the imputers' missing_values=np.nan. RFC 8259 does not allow these, so strict JSON parsers (JavaScript's JSON.parse, Go, Rust, Jackson with its default settings) rejected the file. Only Python's own reader accepted it.

Root cause

_encode passed Python and numpy floats, float arrays and object-array elements through unchanged, including NaN and ±inf. to_spec(path) and fingerprint_ then dumped them with the default allow_nan=True, which writes NaN, Infinity and -Infinity tokens.

Fix

• The new _encode_float (pretab/compose/serialize.py) tags a non-finite float as {"__float__": "nan"} (or "inf", "-inf"). _encode applies it to Python floats and np.floating scalars, and, through the new _map_elements helper, to the elements of a float array that holds a non-finite value. _json_safe, which builds the informational params block, tags floats as well.
• On main, object arrays are written with a raw tolist(), and a NaN category of an un-imputed one-hot encoder sits in such an array. _encode_object_element tags the non-finite float elements of object arrays and leaves every other element as main writes it.
• _decode turns the tag back into a float, and _decode_ndarray restores tagged elements of float and object arrays (_decode_float_element).
• Preprocessor.to_spec(path) and Preprocessor.fingerprint_ dump with allow_nan=False.

Behaviour changes

• Spec files and to_spec() dicts contain {"__float__": "nan"} (or "inf", "-inf") where they had NaN, Infinity or -Infinity, including in the params block. The files parse with any RFC 8259 parser.
• fingerprint_ changes for every fitted preprocessor whose state holds a non-finite float, which is every preprocessor with an imputer. Preprocessors without imputers keep their fingerprint.
• Specs written by 1.0.0 still load: Python's reader accepts their NaN tokens, and untagged floats decode as before. Checked with specs written by main for four configurations (default, one-hot with a NaN category, add_missing_indicator=True with bspline, ple; 5 to 9 NaN tokens each): on this branch each loads, reproduces transform byte for byte and re-saves as strict JSON. Its fingerprint is the new one, equal to that of the same fit made on this branch.

Documentation

• docs/core_concepts/reproducibility.md ("Portable serialization"): the spec is strict JSON, non-finite floats are stored as a tagged value, and specs written by PreTab 1.0.0 with bare NaN tokens still load.
• pretab/compose/serialize.py module docstring: strict JSON and the tag.
• Preprocessor.to_spec docstring, path parameter: written as strict UTF-8 JSON, with the tag format.
• Preprocessor.from_spec docstring, source parameter: specs written by PreTab 1.0.0 still load.
• Docstrings of the new helpers.

Tests

• test_default_spec_file_is_strict_json: a default Preprocessor fitted on data with missing values writes a file without NaN / Infinity that parses with a strict parser and reloads with identical output and fingerprint. Fails on main, passes here.
• test_nan_category_in_an_object_array_is_strict_json: one-hot encoding without categorical imputation keeps NaN in the object array categories_; the spec is strict JSON, contains the tag, and reloads with identical output and fingerprint. Fails on main, passes here.
• test_non_finite_floats_round_trip: the array [[1, nan], [inf, -inf]] and the scalars nan and -inf survive a strict-JSON round trip, with the array's dtype. Fails on main, passes here.
• test_specs_with_bare_nan_tokens_still_load: a spec whose tags are rewritten to bare NaN, as 1.0.0 wrote them, loads and reproduces the output. Guard test: passes on both by design.

Reproduction

Output of the script from #80 on this branch:

NaN tokens in spec: 0
strict parse: ok
tagged values in spec: 4
node JSON.parse: ok

The script from the issue, plus a count of the tagged values and the node -e 'JSON.parse(...)' check that the issue describes.

Checks

• just test: 1442 passed, 59 skipped, 7 xfailed; coverage 93.69% (gate 90%).
• Minimum dependencies (CI job: Python 3.10, numpy 1.24.4, pandas 2.0.3, scipy 1.10.1, scikit-learn 1.6.0): 1429 passed, 72 skipped, 7 xfailed; scripts/quickstart.py passes.
• just docs: strict build succeeds.
• just check: all pre-commit and pre-push hooks pass (ruff format, ruff lint, prettier, file hygiene, pyright with 0 errors).
• Import direction between subpackages is unchanged.

Merge notes

• Base: main. The element-wise encoding of object arrays and the _spec_json helper come with the PR for to_spec / fingerprint_ fail or write unloadable specs for object arrays of numpy scalars and builtin dtypes #66 (fix(serialization): encode object arrays and builtin dtypes in specs #132), which is not on main. Without them, allow_nan=False would make to_spec raise for a NaN category in an object array. This branch therefore adds _encode_object_element and _decode_float_element and passes allow_nan=False directly to the two dumps.

• The PR for to_spec / fingerprint_ fail or write unloadable specs for object arrays of numpy scalars and builtin dtypes #66 (fix(serialization): encode object arrays and builtin dtypes in specs #132) changes the same code. Both add the same _map_elements helper, which git merges once, and they conflict in the ndarray branch of _encode, in _decode_ndarray, in to_spec and fingerprint_ in pretab/preprocessor.py, and at the end of tests/integration/test_serialization.py, where both append tests. Resolution:

  • _encode: keep to_spec / fingerprint_ fail or write unloadable specs for object arrays of numpy scalars and builtin dtypes #66's object branch, data = _map_elements(data, obj.ndim, _encode) (_encode then tags non-finite floats itself), followed by this PR's elif obj.dtype.kind == "f" and not np.isfinite(obj).all(): data = _map_elements(data, obj.ndim, _encode_float).
  • _decode_ndarray: keep to_spec / fingerprint_ fail or write unloadable specs for object arrays of numpy scalars and builtin dtypes #66's element-by-element object branch (its _decode knows the __float__ tag), then decode float arrays with data = _map_elements(data, len(payload["shape"]), _decode) when dtype.kind == "f".
  • Drop this PR's _encode_object_element and _decode_float_element.
  • pretab/preprocessor.py: keep to_spec / fingerprint_ fail or write unloadable specs for object arrays of numpy scalars and builtin dtypes #66's to_spec and fingerprint_ and pass allow_nan=False in its _spec_json (json.dumps(spec, allow_nan=False, **kwargs)) instead of this PR's two direct dumps.
  • Tests: keep both blocks.

  Resolved this way, the full suite passes (1449 passed, 59 skipped, 7 xfailed), and the scripts from to_spec / fingerprint_ fail or write unloadable specs for object arrays of numpy scalars and builtin dtypes #66 and Default to_spec() files contain bare NaN tokens and are not valid JSON for strict parsers #80 both show the fixed behaviour.

• The PR for Standalone transformers drop DataFrame feature names, so reordered columns are silently misaligned #85 (fix(transformers): record and check DataFrame feature names #135) changes _representation_summary and the ..exceptions import in pretab/compose/serialize.py. Git merges it cleanly with this branch.

This change was previously part of #86, which is closed in favour of one pull request per issue.