[dktue]

--store runs that hung for over an hour now finish in about 5 minutes.

Gains at a glance

• Over 12x faster when RAM is tight. Germany with an 8 GB memory limit goes from 69+ min (stopped, unfinished) to 5 min 36 s.
• 67x less disk writing. 1.3 TB drops to 20 GB for the same run. That matters most on slow or IO-bound disks, and it is gentler on SSD wear.
• No more random 20+ minute hangs at 12 GB. Before, the same run took 5:46 one time and hung for over 20 minutes (271 GB written) another. Now it takes 5:25 every time.
• 3.2% less CPU work in every --store run (callgrind, Luxembourg). Node lookups now use the batched fast path from perf: 3.4x faster planet build (2h10 -> 39 min), less RAM, deterministic output #939.
• No more crash on many-threaded machines. --store used to abort with Failed to open mmap file once there were more than ~1000 store files.
• --store output now matches in-memory output exactly. Before, --store silently dropped multipolygon relations that cross the extract boundary. On Germany that meant 119 missing tiles and ~5,700 missing features: the Szczecin Lagoon, Usedom, and border nature reserves such as Pommersche Bucht, Gelderse Poort and Dinkelland. Now all 273,199 tiles match the in-memory run. This bug predates perf: 3.4x faster planet build (2h10 -> 39 min), less RAM, deterministic output #939.
• Otherwise the same output. Apart from the restored relations, tile content is unchanged, and so is memory use.

Germany, --store	Before	After
8 GB RAM	69+ min, stopped (1,348 GB written)	5:36 (20 GB written)
12 GB RAM	5:46 or 20+ min hang (up to 271 GB written)	5:25, reliably (20 GB written)

Follow-up to #939, aimed at --store on machines where the store doesn't fit comfortably in RAM. This is the IO-bound case where #939 gave only 5h28 → 5h11 on the planet.

Problem

#939 sorts the output objects of each z6 tile with a counting sort. With --store, those objects live in the mmap'd store files, and the sort's final pass scattered them directly into their target slots: tens of thousands of interleaved write streams across the store. Once the page cache is under pressure, the kernel has to write the same dirty pages back to disk over and over.

Measured on Germany (4.9 GB PBF, 7.1 GB store, 22 threads, --store, memory capped with a cgroup limit):

• 8 GB limit: the "finalizing z6 tile" step had not finished after 65 minutes and had written 1.3 TB to disk.
• 12 GB limit: sometimes fine (5:46). One run hung for over 20 minutes and wrote 271 GB.

Fix

1. Sort without scattered writes into the store (tile_sorting.cpp). The sort now counts keys straight from the objects, scatters copies into the temporary heap buffer it already allocated, then copies the buffer back in order. Each store page is written once, sequentially. Memory use and the resulting order are unchanged.
2. Release store file handles after mapping (mmap_allocator.cpp). The mapped region stays valid without the handle. Before this, a large --store run with many threads, which uses one ~1 GB file per thread and more as they fill, could exceed the default open-files limit of 1024 and abort with Failed to open mmap file.
3. Batched node lookups in ShardedNodeStore (sharded_node_store.cpp), which --store uses. It didn't override the batched at(ids, n, out) from perf: 3.4x faster planet build (2h10 -> 39 min), less RAM, deterministic output #939, so every node of every way was looked up on its own, plus an extra contains() check. That made SortedNodeStore::contains the top entry in a --store profile (3.6%). Now the whole way is looked up as one batch in the shard of its first node. Only ways that cross a shard boundary fall back to per-node lookups.
4. Don't drop relations crossing the extract boundary (pbf_processor.cpp). With shards, a relation was read only in the pass for the shard of its first member way. If that way wasn't in the extract, every pass skipped the relation. Now the shard is decided by the first member way that is in the store. A relation with no stored member ways is read once, in the first pass.

Verification

Setup: laptop (Core Ultra 7 165H, 22 threads, NVMe under WSL2), Germany 4.9 GB PBF, store 7.1 GB, memory capped with a cgroup limit.

Output: all 273,080 Germany tiles are semantically identical between before and after, and between the 8 GB and 12 GB runs. Monaco and Luxembourg match the reference in all config variants tested.

Border relations: Germany with --store vs in memory. Before: 119 tiles missing and 4,421 tiles with different content. After: 0 missing and 0 differing out of 273,199.

Batched lookups: callgrind on Luxembourg with --store drops from 125.7 to 121.7 billion instructions (−3.2%). Germany output is again identical (0 of 273,080 tiles differ).

The open-files fix was checked by running with ulimit -n 48: before, tilemaker aborts with "Too many open files"; after, it completes.

Notes

• In --store mode, output is not byte-identical between runs. Tile content is identical, but the feature order within a tile can differ. That was already the case before this PR. The determinism claim in perf: 3.4x faster planet build (2h10 -> 39 min), less RAM, deterministic output #939 holds for tile content in this mode, not for the bytes.
• Below roughly 5 GB, Germany --store still thrashes. At that point the node and way stores themselves (4.8 GB) no longer fit, and tile generation reads them in random order. Fixing that would need a spatially ordered store, which is out of scope here.
• I haven't re-run the planet with this change. Feedback from an IO-bound --store setup would be very welcome.

🤖 Generated with Claude Code

[systemed]

Thank you! I’ll run a full planet this weekend and report back.

[dktue]

I re-ran the benchmarks on bare-metal Linux (96 threads, 503 GB RAM, NVMe RAID1, Ubuntu 22.04), Germany extract, with config-openmaptiles.json / process-openmaptiles.lua from current master for both builds. Note this branch is behind master (missing #934/#940), so running it with its own Lua profile gives different output that has nothing to do with the changes here. Memory caps via cgroup v2 (systemd-run --user --scope -p MemoryMax=… -p MemorySwapMax=0).

Correctness: confirmed. Master --store vs in-memory: 119 tiles missing, plus 3,536 water / 2,117 park / 30 landcover / 5 place / 4 landuse features missing. This PR's --store vs in-memory: no missing tiles, none of those losses. Remaining differences are poi-only and show up equally between two identical master runs, so that's pre-existing nondeterminism.

CPU: confirmed, slightly smaller than I measured. 3 interleaved pairs, --store, no cap: user CPU −2.4% (2,330 s → 2,275 s), wall −1.5% (62.2 s → 61.3 s).

Memory-pressure speed-up: does not reproduce here.

Germany --store	master	PR
22 threads, 8 GB	93 s, 15 GB written	90 s, 15 GB written
22 threads, 6 GB	117 s	102 s
96 threads, 6 GB	97 s	112 s

No thrashing or write blow-up on master at 8 GB (single runs, so the 6 GB rows are within noise). So the 69+ min / 1.3 TB numbers in the description look specific to my WSL2 setup (possibly its page cache / VHDX writeback) rather than something master does on Linux in general. Didn't hit the open-files crash on Germany at 96 threads with ulimit -n 1024 either (it does hit on the planet, see below).

Planet run in progress, will follow up.

[dktue]

Planet follow-up (same machine and setup as above: 96 threads, --store, no memory cap, master's config-openmaptiles.json / process-openmaptiles.lua for both builds, planet-latest from 2026-10-22).

Open-files crash: confirmed, and it hits the planet at default settings. With the default ulimit -n 1024, master aborts after 6:42 during way processing:

terminate called after throwing an instance of 'std::runtime_error'
  what():  Failed to open mmap file

The store reaches 1,639 files on the planet, so anyone running a planet build with --store on a many-core machine without raising the limit hits this. With this PR it completes at 1024.

Speed: no difference. With the same limit (ulimit -n 65536) for both:

Planet --store	wall	user CPU	sys CPU	peak RSS	written
master	43:48	155,745 s	23,806 s	180 GB	298 GB
PR	43:13	155,556 s	25,015 s	180 GB	298 GB

(The PR's first planet run, at ulimit -n 1024, took 48:09; a rerun with the same limit as master gave the 43:13 above, so I'm treating the 48:09 as an outlier. Didn't get a second master run.)

Output: equivalent. Both have exactly 346,003,119 tiles with identical per-zoom counts. In a random sample of 200,000 tiles: 183,749 byte-identical, 15,978 with the same features in a different order, 273 differing only in poi with equal feature counts on both sides, the same pre-existing nondeterminism seen between two master runs. As expected, the relation fix doesn't change the planet output, since there's no extract boundary to cross.

Summary: the correctness fix (extracts) and the open-files fix (planet on many cores) both check out; planet speed is unchanged; the Germany CPU saving is ~2.4%; the low-memory speedup I measured under WSL2 didn't reproduce on bare-metal Linux.

[systemed]

Thank you! Together with #956 this takes the planet down from 5hr11 to 4hr54 on my setup and it looks like the gain will be really noticeable on more constrained machines.