Repository navigation
mosaic.from_list: read only the window of each file for an unweighted mosaic with bounds - #65
Merged
Merged
Conversation
…le's window from_list with bounds dropped the inputs outside them, then read every remaining input WHOLE (setup_fields' prototype and replace()) and cropped after. ATL11_to_ATL15 reads its reference ATL14 this way, once per tile: for one 60 km tile of the GL ATL14 on S3 that was 113-225 s and 6.5 GiB of memory for a 641 x 641 window; reading the window is 15.6 s and 0.21 GiB, and the mosaic is identical (h, h_sigma, x, y). The readers' bounds select cells with the same >= / <= test as crop(). A weighted mosaic (pad or feather) still reads whole inputs: its weights are measured from each input's own edges. Also: setup_bounds_from_list rebound an in-memory grid to its cropped copy and then could not remove it from the list (ValueError) when it fell outside bounds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
grid.mosaic.from_list(..., bounds=...)already dropped the inputs that fall outsidebounds, but then read every remaining input whole (thesetup_fieldsprototype andreplace()) and cropped afterwards. With this change, an unweighted (replace) mosaic passesboundsto the readers, so only each file's window is read. The readers select cells with the same>=/<=test ascrop(), so the mosaic is unchanged.A weighted mosaic (
padorfeather) still reads its inputs whole, because its weights are measured from each input's own edges and a window would move them.Why
ATL1415's
ATL11_to_ATL15reads its--ATL14_reference_filethis way once per monthly tile, and Antarctic monthly will read up to four sector ATL14s per tile from S3. Measured on the ADE for one 60 km tile ofATL14_GL_0332_100m_006_02.ncon S3 (641 x 641 window):h, h_sigma, x and y are identical (
np.array_equal, NaNs equal).Also fixed
setup_bounds_from_listrebound an in-memory grid to its cropped copy and then could not remove it from the list (ValueError: list.remove(x)) when it fell outsidebounds.Tests
tests/test_mosaic_list.py: bounded replace mosaics equal the crop of whole tiles (2-D and 3-D groups; serial,workers,block_size; local and an in-memory remote), a spy that every data read is inside the window, and a netCDF root-group case like the ATL14 reference read. Full suite: 345 passed, 3 skipped.🤖 Generated with Claude Code