[dktue]

Gain: a huge relation (Tongass National Forest, 1.9M points) is built 2.9× faster again (677 → 232 s), planet wall another −2.6% (1,572.5 → 1,532.0 s) for +0.85 GB peak heap, on top of the top pad's planet −3.6% and Germany −5% (page faults −42%). A planet on 4 CPUs with 16 GB is unaffected (same heap peak, time within run-to-run noise). Output unchanged.

Update: mmap threshold 8 MB

Setting any malloc parameter also freezes glibc's mmap threshold at its 128 KB default. Then every buffer above 128 KB is a fresh mmap whose pages the kernel has to zero on first touch. For a huge polygon, Boost's is_valid (in sectionalize) grows such vectors over and over: in @cldellow's report in #977, the Tongass National Forest relation (r6535292, 29,059 ways) took 22.8 minutes on a GitHub runner, about 3× as long as before #939. A profile showed most of that time in vector reallocation. With the mmap threshold raised, the relation is back to its pre-#939 time. So this PR now also sets M_MMAP_THRESHOLD to 8 MB.

Tongass alone (osmium getid -r from North America), relation phase, 4 threads:

	time
before #939	235.8 s
master (trim threshold 4 MB)	676.8 s
top pad 16 MB + mmap threshold 4 MB	316.0 s
top pad 16 MB + mmap threshold 8 MB	231.9 s
top pad 16 MB + mmap threshold 16 / 32 / 64 MB	231.8 / 232.9 / 234.0 s

Planet, PMTiles, 32 threads, --store --fast, master + open performance PRs, one run each:

	top pad 16 MB	+ mmap threshold 8 MB	+ 32 MB	+ 64 MB
wall	1,572.5 s	1,532.0 s (−2.6%)	1,532.9 s (−2.5%)	1,539.1 s (−2.1%)
sys CPU	7,756 s	7,272 s (−6%)	7,213 s	7,292 s
peak heap	13.68 GB	14.53 GB (+0.85)	15.00 GB (+1.32)	15.51 GB (+1.83)

Planet on a small machine (4 CPUs pinned with taskset, 16 GB cgroup limit without swap, --store), to check the extra memory doesn't hurt there:

	wall	peak heap
top pad 16 MB, run 1	18,927 s	12.02 GB
top pad 16 MB, run 2	19,524 s	12.04 GB
+ mmap threshold 8 MB	19,866 s	12.05 GB

Two identical runs differ by 3.2% there, so the 8 MB run (+1.8% against run 2) is within noise, with the same heap peak. Germany (in memory, 32 threads, 2 rounds) runs at the same speed with all thresholds. 8 MB is the smallest value that fixes Tongass, is as fast as larger ones on the planet, and costs the least memory. Setting the threshold with mallopt behaves the same as the tunable (checked with 32 MB: Tongass 230.8 s, Germany peak RSS identical).

#939 calls mallopt(M_TRIM_THRESHOLD, 4 MB) at startup. Setting any malloc parameter explicitly stops glibc from raising its mmap threshold on the fly, which had inflated peak RSS. This PR sets M_TOP_PAD to 16 MB instead. That also stops the dynamic adjustment, and it makes the heap grow and shrink in 16 MB steps rather than 128 KB ones. With 32 threads allocating, the small steps mean many brk() calls and page faults, which shows up as sys time.

As before, nothing is set when the user tunes malloc through GLIBC_TUNABLES or the MALLOC_*_ variables (now including MALLOC_TOP_PAD_), and nothing changes on non-glibc systems.

Planet, PMTiles, 32 threads, --store --fast

	trim 4 MB (master)	top pad 16 MB	change
wall	1784 s	1719 s	−65 s (−3.6%)
sys CPU	8,509 s	8,055 s	−5.3%
user CPU	39,165 s	39,039 s	−0.3%
anon memory peak	13.48 GB	13.74 GB	+0.26 GB

One run each; the planet runs used GLIBC_TUNABLES=glibc.malloc.top_pad=16777216 on the same binary. The Germany runs below show that the mallopt call in this PR behaves the same as the tunable.

Germany, PMTiles, 32 threads, two rounds

	trim 4 MB (master)	tunable top pad 16 MB	this PR
wall	40.3 / 39.4 s	38.3 / 38.2 s	37.8 / 37.9 s
sys CPU	45.6 / 45.5 s	34.5 / 33.6 s	35.8 / 33.9 s
page faults	6.11 M	3.56 M	3.53 M
max RSS	8.51 GB	8.59 GB	8.57 GB

An earlier series on the MBTiles writer (planet, same machine) found the same effect at larger scale: top pad 16 MB −8.4% wall, 64 MB −10% wall for +2.6 GB. 16 MB is the middle ground between speed and memory.

The output is unchanged (only memory management differs).

🤖 Generated with Claude Code