Repository navigation
perf: wider validity precheck, validate shapefile polygons once (planet 27% faster, 33% less CPU) - #961
Merged
Conversation
This was referenced Oct 29, 2026
The cheap validity precheck gave up on any multipolygon whose parts have overlapping bounding boxes, and on polygons with more than 256 holes. Both are common after clipping, so is_valid ran anyway. It now checks that rings of different parts never touch and that no part lies inside another (unless in one of its holes), and replaces the hole cap with a work limit. Shapefile polygons were stored without a validity flag, so every tile they touched took the slow path: is_valid, then Boost intersection on the original. They are now validated once when stored. Planet: 27% less wall time, 33% less CPU. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dktue
force-pushed
the
perf/precheck-shapefile
branch
from
October 31, 2026 22:41
1896774 to
9e5b8e1
Compare
Owner
|
Very neat - Europe is down from 48m30 to 41m32 on my machine, output looks the same and RAM usage is unaffected. Thank you. (Mostly looking at performance for now and I'll move onto correctness later!) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gain (planet,
--store, 32 threads): wall 49.9 → 36.4 min (−27%), CPU 1,320 → 885 min (−33%). Germany 52.5 → 43.6 s (−17%), CPU −10%.Needs the
scaleRingfix from #933 (or #941): without it, any change that alters where a clipped ring starts can drop polygons through thescaleRingbug. (#949, which this was built on, is merged now.)What changes
Validity precheck (
multiPolygonDefinitelyValid). It gave up, sois_validran anyway, on:
clipping a concave polygon). It now checks that no ring of one part touches a
ring of another and that no part lies inside another, unless inside one of its
holes.
hole-against-hole tests use a sweep.
The precheck still answers "certainly valid" only when
is_validagrees. AGermany run that compared every positive answer with
is_validfound nodisagreement, and neither did the randomised test.
Shapefile polygons were stored without a validity flag, so every tile they
touched took the slow path:
is_valid, then Boostintersectionon the originalpolygon. On the planet that was 21% of all CPU. They are now validated once when
stored, so tiles take the
rect_clippath.Results
Planet, 32 threads,
--store, compared with #949 + #941. This run also includedthe #949 fix above and an unrelated hash-map change, both speed-neutral on Germany.
Germany, 32 threads: 52.5 s -> 43.6 s wall (-17%), 1167 s -> 1045 s CPU (-10%).
The higher peak heap on the planet is not explained yet (Germany's peak goes
down slightly). I'm looking into it.
Output
Planet: the same 345.9M tiles. Of a 200k-tile sample, 89.4% are byte-identical,
10.0% have the same features in a different order, and 0.64% differ in content.
Feature counts match except
poi(which already differs between identical runs)and one landcover feature.
Germany, compared per tile, layer and tags: with simplification off, nothing is
lost. With simplification on, a few small polygons at z6–z12 differ by a few
percent, both ways, because rings now start at a different vertex.
Tests
New unit tests for the precheck: overlapping bounding boxes, a polygon in
another's hole, nested and crossing polygons, touching at a vertex, and 400
holes.
🤖 Generated with Claude Code