Skip to content

perf: wider validity precheck, validate shapefile polygons once (planet 27% faster, 33% less CPU) - #961

Merged
systemed merged 1 commit into
systemed:masterfrom
dktue:perf/precheck-shapefile
Nov 1, 2026
Merged

systemed merged 1 commit into
systemed:masterfrom
dktue:perf/precheck-shapefile

Conversation

@dktue

@dktue dktue commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Gain (planet, --store, 32 threads): wall 49.9 → 36.4 min (−27%), CPU 1,320 → 885 min (−33%). Germany 52.5 → 43.6 s (−17%), CPU −10%.

Needs the scaleRing fix from #933 (or #941): without it, any change that alters where a clipped ring starts can drop polygons through the scaleRing bug. (#949, which this was built on, is merged now.)

What changes

Validity precheck (multiPolygonDefinitelyValid). It gave up, so is_valid
ran anyway, on:

  • multipolygons whose parts have overlapping bounding boxes (common after
    clipping a concave polygon). It now checks that no ring of one part touches a
    ring of another and that no part lies inside another, unless inside one of its
    holes.
  • polygons with more than 256 holes. The cap is replaced by a work limit, and
    hole-against-hole tests use a sweep.

The precheck still answers "certainly valid" only when is_valid agrees. A
Germany run that compared every positive answer with is_valid found no
disagreement, and neither did the randomised test.

Shapefile polygons were stored without a validity flag, so every tile they
touched took the slow path: is_valid, then Boost intersection on the original
polygon. On the planet that was 21% of all CPU. They are now validated once when
stored, so tiles take the rect_clip path.

Results

Planet, 32 threads, --store, compared with #949 + #941. This run also included
the #949 fix above and an unrelated hash-map change, both speed-neutral on Germany.

before after
wall 49.9 min 36.4 min -27%
CPU (user+sys) 1320 min 885 min -33%
peak heap (anon) 25.3 GB 27.3 GB +2 GB

Germany, 32 threads: 52.5 s -> 43.6 s wall (-17%), 1167 s -> 1045 s CPU (-10%).

The higher peak heap on the planet is not explained yet (Germany's peak goes
down slightly). I'm looking into it.

Output

Planet: the same 345.9M tiles. Of a 200k-tile sample, 89.4% are byte-identical,
10.0% have the same features in a different order, and 0.64% differ in content.
Feature counts match except poi (which already differs between identical runs)
and one landcover feature.

Germany, compared per tile, layer and tags: with simplification off, nothing is
lost. With simplification on, a few small polygons at z6–z12 differ by a few
percent, both ways, because rings now start at a different vertex.

Tests

New unit tests for the precheck: overlapping bounding boxes, a polygon in
another's hole, nested and crossing polygons, touching at a vertex, and 400
holes.

🤖 Generated with Claude Code

The cheap validity precheck gave up on any multipolygon whose parts
have overlapping bounding boxes, and on polygons with more than 256
holes. Both are common after clipping, so is_valid ran anyway. It now
checks that rings of different parts never touch and that no part lies
inside another (unless in one of its holes), and replaces the hole cap
with a work limit.

Shapefile polygons were stored without a validity flag, so every tile
they touched took the slow path: is_valid, then Boost intersection on
the original. They are now validated once when stored.

Planet: 27% less wall time, 33% less CPU.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@dktue
dktue force-pushed the perf/precheck-shapefile branch from 1896774 to 9e5b8e1 Compare October 31, 2026 22:41
@systemed
systemed merged commit d0ab14b into systemed:master Nov 1, 2026
10 of 11 checks passed
@systemed

systemed commented Nov 1, 2026

Copy link
Copy Markdown
Owner

Very neat - Europe is down from 48m30 to 41m32 on my machine, output looks the same and RAM usage is unaffected. Thank you.

(Mostly looking at performance for now and I'll move onto correctness later!)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants