[dktue]

perf: 3.4x faster on the full planet, 3-4.5x on extracts, much less RAM, deterministic output

Full planet: 2 h 10 min -> 38 min 38 s (3.4x faster), 4x less CPU time, 96 GB less peak RAM. On extracts: 3-4.5x faster and 11-20x less peak memory. The tiles are the same apart from small differences in simplified landcover/landuse/water polygons (see Output).

Full planet

Planet (planet-261005, 89 GB PBF)	Before (parent commit)	After	Improvement
Wall time	2 h 10 min 08 s	38 min 38 s	3.4x faster
CPU time (user + system)	190 h	47.6 h	4.0x less
Peak RSS	270 GB	174 GB	-96 GB (-36%)
Output size	75.470 GB	75.454 GB	-0.02%

Setup: OpenMapTiles profile (resources/config-openmaptiles.json + process-openmaptiles.lua) with coastline and landcover shapefiles, PMTiles output, in memory (no --store), default thread count. Server: AMD EPYC 9454P (48 cores / 96 threads), 503 GB RAM, Ubuntu 22.04, GCC 11.4. Measured with /usr/bin/time -v, one run per version, back to back on an otherwise idle machine. Both runs wrote all 345,907,442 tiles and exited cleanly.

Part of the planet RAM saving comes from closed ways that are only wrongly oriented. These are now kept as reversed references into the way store, instead of being repaired and stored as separate generated polygons: "Generated polygons" in the log falls from 300M to 7.6M.

Extracts

Extract	Wall time before -> after	Speedup	Peak RAM before -> after	RAM reduction
Luxembourg	7.8 s -> 1.8 s	4.4x faster	5.22 GB -> 0.25 GB	20x less
Stuttgart (Regierungsbezirk)	21.3 s -> 6.7 s	3.2x faster	6.19 GB -> 0.54 GB	11x less

Total CPU time also falls 1.5-1.8x, so the speedup is not just better use of cores; the work itself shrank.

Measurement: interleaved A/B runs (6 pairs per extract) against the parent commit, after a discarded warm-up run. Setup: OpenMapTiles profile, gzip MVT to .mbtiles, 22-thread laptop (Core Ultra 7 165H). Times and RAM are medians.

Output

• Tile output no longer depends on thread count or scheduling. Previously, runs with different --threads values differed in up to ~200 Luxembourg tiles, because what the ClipCache held depended on scheduling. Tiles are now split into fixed batches, each with its own clip cache.
• Because tiles are now always clipped from the full geometry instead of from whichever ancestor clip happened to be cached, simplified landcover/landuse/water polygons at z9-12 (rarely z14) differ slightly from before: on Luxembourg 69 of 2081 tiles (3.3%), typically 2-20 units of 4096 (max 110). The original already differed from itself between runs in the same way (about 4 tiles per run, up to 45 units); the new output is identical on every run.
• Every other change gives identical output. This was checked with a semantic tile diff on Monaco and Luxembourg, plus Monaco with: maxzoom 16, minzoom 13, --materialize-geometries, --store, --shard-stores and --threads 1.

Main changes

• Memory
  • mmap_allocator: per-thread regions use anonymous mmap instead of zero-filled vectors. This is the largest RSS win.
  • UsedWays is a sparse sorted vector, with a dense-bitmap fallback for planet-scale inputs.
  • On glibc, M_TRIM_THRESHOLD is set unless the user tunes malloc through the environment.
  • Subdivided relation blocks are decompressed once and shared between chunks.
  • Per-batch cache resets.
• Tiles
  • The batch partition and per-batch ClipCache (with per-zoom levels) described under Output.
  • A conservative sweep-based "definitely valid" precheck runs before boost::geometry::is_valid. Boost still runs whenever the precheck is unsure.
  • Visvalingam: rings that are wrong only in orientation are reversed instead of dissolved, the heap is indexed, and MVT attribute writing is memoized.
  • Tile coordinate filtering scans words instead of calling test() per tile.
• Ingest
  • Lua: Find/Holds/Attribute*/MinZoom/ZOrder are native lua_CFunctions that keep kaguya's argument-type rules. Each lua_State gets a size-class pool allocator.
  • Ways whose only fault is wrong orientation are reversed instead of being sent through make_valid.
  • SortedNodeStore: a decoded-chunk cache and batched lookups.
  • TagMap uses flat shards, and AttributePairStore has a thread-local lookup table.
  • Shapefiles: records outside the bbox are skipped using the bbox in each record header.
• polylabel: provably no-op cells are pruned, and an SSE2 distance kernel is used on x86-64. Both give bit-identical results.

Fast paths were cross-checked against the original code paths during development (that verification code has been removed from this PR).

Not tested

• MSVC/Windows and macOS builds. Portable fallbacks exist for the SIMD and popcount code.
• A tile-by-tile comparison of the planet output; only extracts were diffed. Planet --store mode (only in-memory planet runs were timed).

🤖 Generated with Claude Code

[systemed]

Very neat set of changes, thank you.

I observed less of a speedup but then my machine is rather less powerful and fairly IO-constrained: the UK (RAM-resident) went from 4m33 to 3m14, Europe (RAM-resident) went from 1hr11 to 58m, the planet (with --store) went from 5hr28 to 5hr11. All very much worth having. Better still, RAM usage fell by 40% for Europe and 63% (!) for the UK.

Could you take out the code guarded by TMOPT_VERIFY? I don't think we need this in the codebase and the commit message suggests it shouldn't have been included. I've merged the MLT PR so you can rebase on top of that.

@laem @etienneJr If you have time to check out this PR I'd be really interested to hear your experiences - it looks like it could be a handy speedup for cartes.app.

[laem]

These two PR are great ! We get slightly faster tiles to transfer with MLT, faster decoding, and faster generation with less RAM.

This is especially cool because RAM is on shortage. Our cloud provider (Scaleway) has a shortage of 384 Gb or RAM compute machines.

Will test as soon as possible, thanks for the mention.

[dktue]

Thanks for testing, and for the numbers!

• Removed all TMOPT_VERIFY code (separate commit, -796 lines). Rebuilt and re-checked: tile output is unchanged on Monaco and Luxembourg, also with --store, --shard-stores, --materialize-geometries, --threads 1, minzoom 13 and maxzoom 16.
• Rebased onto master now that Add MapLibre Tile (MLT) output #938 is merged, so the PR only contains the perf changes.

On the smaller speedup: my numbers were in-memory runs on a 96-thread EPYC, where the work is CPU-bound. Most of these changes cut CPU work, so on an IO-constrained machine, and especially with --store, the disk becomes the limit and less of the gain shows. I didn't look at --store performance specifically; that would be a good follow-up.

I also corrected the output section of the description: the geometry differences from the deterministic clipping are a bit larger than I first wrote (details there). They are the same kind of difference the original already produced between runs.

[systemed]

Thank you - this is an amazing speedup. I'm still looking through the changes in detail, but I'm confident enough in what I've seen in outline that I'm happy to merge now and we can push any further adjustments if needed. The ability to run tilemaker on the planet in less than an hour is pretty impressive!

[laem]

The ability to run tilemaker on the planet in less than an hour is pretty impressive!

Yes, that's great. If only we had daily osm.pbf releases :/ To my knownledge, they're only weekly, even with torrents.

[laem]

Just ran the planet build with the latest master version, compiled with boost 1.90 and MLT.

Interesting, after using all the cores, it's been using only one (2% of the overall) at 168 Gb of RAM.

Machine : EPYC 32 cores 380 Gb of ram.

Looks like there is a single-core process that takes a significant portion of the process's timeline.

Edit : 40 minutes after, it comes back at 100 % CPUS used.

[dktue]

@laem, sorry that this went unanswered. Thanks for the report and the screenshot.

Since then we went looking for single-threaded stretches in the planet build and fixed the ones we found (planet, PMTiles, 32 threads):

Stretch on one or a few cores	Before	After	PR
Shapefile loading (one giant ocean polygon)	95 s	21 s	#970
End of the relations phase (Lake Huron, Guri Reservoir, Lake Nasser…)	~46 s at 1–3 cores	~15 s	#974, #975, #976
Collecting tiles	25 s on one core	1 s	#972
Finalizing z6 tiles	26 s	7 s	#972
Process exit with --store	30 s	8 s	#973

At 168 GB, an in-memory planet run has usually finished reading, which is where most of these sit: the end of the relations phase (Lake Huron alone took 103 s on one core), then collecting tiles and finalizing. So there's a good chance this is what you saw, and that it's much shorter now. To be sure, could you share:

1. The last progress line tilemaker printed during the single-core stretch. It names the phase (e.g. Block …/…, finalizing z6 tile, writing tile …).
2. The output format (.pmtiles or .mbtiles), and whether you used --store.
3. The profile and config (the bundled OpenMapTiles one, or cartes.app's own?).
4. The command line, and the Boost version you built with (we use 1.74; 1.90 is newer than anything we tested).

If it turns out to be something else, we're happy to reproduce it on our 96-thread EPYC with your setup. If you'd like to try the current state in the meantime, dktue/tilemaker → perf/all-open-prs has all open performance PRs together (numbers in #977).