Repository navigation
Store pickled extension data in zarr SortingAnalyzers as byte arrays instead of object arrays - #4882
Merged
Merged
Conversation
…instead of object arrays
alejoe91
reviewed
Oct 20, 2026
| """ | ||
| encoding = array.attrs.get("encoding", None) | ||
| if encoding is None: | ||
| # With love for Alessio |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the last peel-off from #4260, the second half of the format change (4 in my review there), and it builds on #4881, so please review that one first. After #4881 the only object arrays left in a zarr
SortingAnalyzerare the extension data that is not JSON-serializable, which in spikeinterface means the scikit-learn PCA models ofprincipal_components. I now store their pickle as bytes in auint8array, the same way #4881 stores JSON. The content is the same as before and the same as the.pklfiles ofbinary_folder; only the container changes, so a zarr analyzer contains no object arrays at all and #4260 no longer needs a way to write scikit-learn models to zarr. Old analyzers still load.This is a different approach from #4260, which writes the models with a new helper that saves the estimator's attributes as zarr arrays and JSON. I chose to do what
binary_folderalready does, which is to pickle them, so both formats store the extension data the same way and we don't add a zarr-only serializer to maintain. I think the models should be stored as plain arrays instead of pickles, but that should change both formats at once, so I will propose it in a separate issue.I also route the legacy
dictandobjectextension branches throughread_encoded_array, so the reader of old object arrays under zarr-python 3 in #4260 has a single place to hook into.