Repository navigation
Conversation
- finalizeObjects: sort small z6 tiles in parallel (one thread each, largest first, at most 1/8 of all objects in flight); z6 tiles with more than 1/threads of the objects keep the all-threads sort - populateTilesAtZoom(threadNum): workers mark tiles per z6 column in bitmaps of their own, merged at the end (PreciseTileCoordinatesSet::merge) - addCoveredTilesToOutput: one setBox per zoom instead of setting each z14 tile and its parents; setBox fills rows word by word Planet: collecting tiles 25.4 s -> 1.1 s (-96%), z6 finalizing 26.3 s -> 7.3 s (-72%), wall -1.0%. Europe: collecting tiles 3.4 s -> 0.6 s (-82%), z6 finalizing 7.3 s -> 4.8 s (-34%), wall -2.0%. Output unchanged (Germany compare_area).
This was referenced Oct 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gain (planet, PMTiles, 32 threads): the two serial steps between reading and tile writing drop from 70.5 s to 24.2 s (−66%; −46 s, about 2.7% of the run): collecting tiles 25.4 → 1.1 s (−96%), z6 finalizing 26.3 → 7.3 s (−72%). Total wall 1703 → 1686 s (−1.0%, one run each; the untouched tile phase varies by ±25 s between runs). Output unchanged.
Between reading the .pbf and writing tiles, two steps run on few threads. On the planet (32 threads) they take about 50 s, with most cores idle:
finalizeObjectssorts the objects of the 4096 z6 tiles one after another. Each sort uses all threads, which suits small extracts (one z6 tile may hold most objects) but leaves the cores idle for the many small tiles of a large extract. A comment insortOutputObjectsalready names this as a future improvement.populateTilesAtZoomwalks every output object to mark the tiles that hold objects, then marks every z14 tile in the bounding box of each large object (e.g. ocean polygons) together with its 14 parents, one tile at a time.This PR changes three things.
finalizeObjects: z6 tiles holding more than 1/threads of all objects are still sorted one at a time with all threads. The others are sorted in parallel, one thread each, largest first. Each sort copies its z6 tile into a buffer, so the number of objects being sorted at once is capped at 1/8 of all objects (a single tile may always run). The counting sort is stable whatever its chunk count, so the order of objects, and the output, are unchanged.populateTilesAtZoomtakesthreadNum. Workers take whole columns of z6 tiles, mark tiles in bitmaps of their own (allocated lazily, so each touches only its part), and the bitmaps are merged at the end with the newPreciseTileCoordinatesSet::merge.addCoveredTilesToOutputnow calls the newTileCoordinatesSet::setBoxonce per zoom.PreciseTileCoordinatesSet::setBoxfills each row word by word. It ignores the part of a box outside the grid;set()wrapped a y beyond the last row into the next row.New unit tests check
setBoxagainstset()for boxes that start and end mid-word, span several words or touch the edge, andmergeagainst setting both sets' tiles.Europe, PMTiles, 32 threads,
--store --fastPlanet, PMTiles, 32 threads,
--store --fastThe two steps save 46 s. The total gains less because the tile phase of this run took 24 s longer (1046 s vs 1022 s). This PR doesn't touch the tile phase, and identical planet runs have varied by tens of seconds before, so I take that as run-to-run noise (one run each).
Output
Germany, compared with compare_area (per-tile area/length per layer and tag group): all layers identical. Only
poidiffers, which already differs between identical runs.Baseline for the measurements: master + #961 + #966 + #968 + the precheck and malloc follow-ups.
🤖 Generated with Claude Code