perf: reset scratch from what was written on restore by danbugs · Pull Request #1901 · hyperlight-dev/hyperlight

danbugs · 2026-10-07T23:57:32Z

A restore must leave scratch as it was at the snapshot: all zero. Today each hypervisor does that by discarding everything:

KVM: drops scratch with MADV_DONTNEED, or zeroes all of it with fill(0) in builds with MSHV support.
MSHV: zeroes all of it.
Windows: maps fresh memory (fix(restore): replace scratch zero with fresh allocation on Windows #1765).

So a restore costs time in the size of scratch, or a fault on every page the next run touches, however little the guest wrote. This PR makes a restore reset only what changed:

KVM:
- The kernel reports which scratch pages hold memory, through /proc/self/pagemap.
- On Linux 6.7+ it's read with PAGEMAP_SCAN, which skips untouched regions, so a restore costs time in how much scratch is in use, not in its size. Older kernels read one entry per page (about 500 µs for 256 MiB).
- Pages holding data are zeroed in place and stay mapped, so the next run takes no faults on them. Pages holding only zeros stay too, and every 16th restore gives them back, so memory still shrinks after a heavy run.
WHP (x86_64 and ARM64):
- Scratch is mapped with dirty-page tracking, and each restore zeroes only the pages in the log. Written pages stay resident between restores.
- Where Windows can't track dirty pages, scratch up to 16 MiB is zeroed in place, which is 2–6x faster than fresh memory. Larger scratch is still replaced, so it doesn't all stay resident.
MSHV: the same, with MSHV's dirty log. Tracking costs the guest a fault on its first write to each page, so it's used only where it pays off. Scratch under 2 MiB isn't tracked. After a run writes more than a tenth of scratch (a quarter from 32 MiB), where zeroing everything is cheaper, tracking pauses for 16 restores.
Host writes: the host also writes into scratch, for example virtqueue buffers, which a hypervisor's log doesn't see. So HostSharedMemory records every page its writes touch, and resets zero those too. A write through exclusive access counts as touching all of scratch.

Every threshold comes from measurements. I measured scratch from 1 to 256 MiB, with guests writing from nothing up to a third of it:

KVM: dropping scratch, zeroing all of it, and zeroing from pagemap.
MSHV and Windows: zeroing all of it, mapping fresh memory, and zeroing from the dirty log.

On MSHV, tracking made write-heavy guests slower, as #683 found, which is why tracking pauses past those fractions.

Larger sandboxes (simpleguest restore + call; KVM and MSHV on nested Azure VMs, Windows on bare metal). The default sandbox is unchanged.

scratch / written	KVM main → PR	MSHV main → PR	Windows main → PR
16 MiB / 0	713 → 74 µs	803 → 364 µs	1,062 → 144 µs
64 MiB / 1 MiB	6,018 → 372 µs	5,820 → 1,153 µs	7,278 → 720 µs
256 MiB / 4 MiB	24,497 → 1,339 µs	22,969 → 3,831 µs	19,752 → 1,998 µs
256 MiB / 16 MiB	28,912 → 7,530 µs	29.6 → 12.8 ms	–

hyperlight-unikraft restore + call, stock 0.17.0 → this PR:

runtime	KVM	MSHV	Windows
dummy (64 MiB scratch)	890 → 210 µs	6.1 → 1.1 ms	3.7 → 0.62 ms
quickjs (64 MiB)	1,070 → 265 µs	6.1 → 1.18 ms	4.1 → 0.68 ms
python (256 MiB)	1,860 → 450 µs	22.9 → 2.5 ms	9.6 → 1.8 ms

Copilot

🔵 Needs a closer look

The pagemap-driven reset policy affects sandbox isolation and needs fixes plus human validation.

3 open findings

Forked parallel test can fail and hang unsafely · New
Stale pages remain resident after working-set shrink · New
Merged spans can exceed the total scan budget · New

What changed in this PR

This experiment targets faster KVM snapshot restores by retaining and zeroing selected scratch pages in Hyperlight’s host memory manager.

Changes:

Adaptive pagemap-based scratch resets with platform fallbacks.
Per-sandbox reset state, memory-behavior tests, and changelog documentation.

File	Description
src/hyperlight_host/src/mem/shared_mem.rs	Selects adaptive KVM resets with zeroing fallbacks.
src/hyperlight_host/src/mem/scratch_reset.rs	Implements residency tracking, reset policies, and tests.
src/hyperlight_host/src/mem/mod.rs	Provides platform-gated reset modules.
src/hyperlight_host/src/mem/mgr.rs	Maintains reset state across scratch restores.
CHANGELOG.md	Documents retained KVM scratch memory.

🧠 Review effort: Balanced

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Patch hyperlight-host and hyperlight-common to danbugs/hyperlight c854bb43 (0.17.0 plus the scratch reset of hyperlight-dev/hyperlight#1901), to measure it in CI. Signed-off-by: danbugs <danilochiarlone@gmail.com>

A restore zeroed scratch by dropping it with MADV_DONTNEED on KVM (fill(0) in builds with mshv3), by fill(0) on MSHV, and by mapping fresh memory on Windows (#1765). Each costs time in the size of scratch, or a fault on every page the next run touches. On KVM, every reset now reads /proc/self/pagemap for all of scratch. A page with memory of its own that holds data is zeroed and kept, so the next run takes no faults on it. One that holds only zeros was not written since the last reset and is dropped, so what stays resident follows what runs write. Swapped pages and shared pages holding data are dropped. The rule is the same from the first restore on. Scratch is backed 4 KiB at a time on KVM, so a reset zeroes only what was touched. On Windows, scratch of up to 16 MiB is now zeroed in place, which is about 2x faster than a fresh mapping (over 5x when the guest writes a MiB or more); larger scratch is still replaced, so it does not all become resident. HostSharedMemory now notes the pages its writes touch, and zero_written zeroes only a given set of pages plus those, for resets that know what the guest wrote (next commits). Signed-off-by: danbugs <danilochiarlone@gmail.com>

Add a DirtyLog supertrait of VirtualMachine: how a VM tracks the pages the guest writes, and reading and clearing that log for a range. On MSHV (x86_64), tracking is a partition property switched on and off. Reads use get_dirty_log with clear; stopping sets the bits back first, which MSHV requires. On WHP (x86_64 and ARM64), scratch is mapped with WHvMapGpaRangeFlagTrackDirtyPages and read with WHvQueryGpaRangeDirtyBitmap, which clears what it reads. If the tracked mapping fails, scratch is mapped as before and tracking is not tried again for the VM. Other backends do not track. A backend that claims to but does not read reports every page, so a reset zeroes too much, never too little. Signed-off-by: danbugs <danilochiarlone@gmail.com>

Where the hypervisor logs the guest's writes, a restore now zeroes only the pages written since the last restore: those in the log, the pages host writes touched, and the page-table copy (#1766). Tracking starts when scratch is mapped, before the guest runs, so the first restore uses it too (hyperlight_vm/dirty_log.rs): - WHP: always. Tracking costs the guest nothing measurable there. The log is cleared when scratch is mapped, so only written pages are ever touched and the rest of scratch never becomes resident. - MSHV: tracking makes the guest's first write to each page after a read fault to the hypervisor (about 1 us nested), and reading the log costs time in the size of scratch. Measured against zeroing all of scratch, that loses below 2 MiB of scratch, and once a run writes more than about a tenth of scratch (a quarter from 32 MiB, where zeroing gets about 3x slower per MiB). So scratch under 2 MiB is not tracked, and tracking stops after such a run until scratch is mapped again. The first log after mapping reports every page, so the guest sets up without faults. - KVM keeps its in-place reset; no log is read. Signed-off-by: danbugs <danilochiarlone@gmail.com>

The restore benchmarks use 352 KiB to 1 MiB of scratch, where resetting it costs little however it is done. Add call_with_restore with 64 and 256 MiB of scratch, with a guest that writes a few pages (Echo) and one that writes 1 MiB a call, and report them, with the default size, in pull request comments. Signed-off-by: danbugs <danilochiarlone@gmail.com>

ludfjig · 2026-10-13T23:36:10Z

Hi this looks good! I reviewed this with help of ai. A couple of points:

Guest call performance imediatley after a restore is regressed, but not visible on benchmarks because all exisitng benchmarks time the call and restore together. Can you add a benchmark that times these separate?
It seems most benchmarks only writes small amount of scratch. Could we also add a variant that writes some more to get better insight into perofrmance for these types of workloads?
Have you compared to how wasmtime implements pagemap? see https://github.com/bytecodealliance/wasmtime/blob/main/crates/wasmtime/src/runtime/vm/sys/unix/pagemap.rs. Since it's a lot of unsafe code, it's easy to be unsound here. One differnce is that it's one fd per process instead of per sandbox.

Everythign under here is ai generated, so please take it with a grain of salt. I think most the points are valid though:

-- slop line --

KVM

The default sandbox restores slower. CI shows call plus restore 1.44x slower on AMD and 2.14x on Intel, and I measured 1.66x on an AMD VM. The call costs 33 µs more, likely from faulting back in pages the restore dropped, and the restore costs 12 µs more for the pagemap scan.
pagemap_scan is a safe function, but the kernel writes to the address in arg.vec, and nothing ties that address to a live buffer. Taking &mut [PageRegion] and filling in vec inside the function would fix that.

MSHV

Every call after a restore is slower by about 1.1 µs per page it writes. On a nested host, the first write to each tracked page exits to the outer hypervisor. I measured the same cost on kernel-mshv 6.6.100 and 6.18.
Tracking stays on long after it stops paying off. Writing 16 MiB of 256 MiB scratch (6%) makes call plus restore about 20% slower than before on both kernels. The break-even in my runs is around 4% of scratch, but the code keeps tracking up to 25%, or 10% below 32 MiB.
One run that trips writes_much turns tracking off for the rest of the sandbox's life, because scratch is only remapped when its size changes. Every later restore then fills all of scratch. A benchmark with one heavy call followed by many light calls would show this cost.
"Which MSHV holds resident anyway" is true on kernel-mshv 6.6.100, which pins guest memory. From 6.6.121 and on 6.18, regions are movable, and the first full fill(0) is what makes scratch resident.

WHP

WHP uses the same hypervisor tracking as MSHV, so nested hosts pay the same per-page exit, but WHP has no cutoff. The large-scratch CI numbers suggest 4 to 17 µs extra per page written.
Written scratch pages are never given back to the OS, so a sandbox holds as much memory as its largest run ever wrote. In a pool of long-lived sandboxes, one large request raises memory for good.
map_gpa_range moved out of unsafe fn map_memory but is safe itself, so its callers no longer promise that the memory outlives the mapping. It should be an unsafe fn with a # Safety section, and its unsafe blocks need // SAFETY: comments. The ARM64 mapping closure has the same gap.

All platforms

Only debug builds check that the dirty log covered every written page. A future host write to scratch that bypasses the logged write methods would silently leak one call's data into the next in release builds. Routing every host write to scratch through one logged path, or adding an opt-in release check, would guard against that.
The debug_assert! in zero_written scans all of scratch on every tracked restore in any debug build, and on WHP it also brings untouched pages into memory.
The integration test passes even when tracking is unused or has failed, so it does not show that the dirty-log path ran.

Nits

Reset policy is split between HyperlightVm, SandboxMemoryManager, and HostSharedMemory, and the memory type queries the hypervisor type itself. One policy chosen at sandbox creation would be easier to follow.
HostWrites lives on every HostSharedMemory and runs on every virtqueue write, but only scratch with a dirty log reads it.
The default DirtyLog::read_dirty_log, which reports every page as written, has no user and no test.
The fake VM never fails enable or disable, so cleanup after those failures is untested.
On WHP, the dirty-log read in ScratchDirtyLog::mapped has no effect, because WHP already clears the bits when it maps a tracked range.
On WHP, a failed tracked mapping logs at debug!, while the matching failure in ScratchDirtyLog logs at warn!.
On WHP, pages the guest writes stay resident, not committed. The whole section is charged to commit when it is created.
The new_scratch doc blames GPA alignment, but scratch starts on a 2 MiB boundary whenever its size is a multiple of 2 MiB. KVM uses 4 KiB mappings because the guard page puts the host address 4 KiB off from the GPA.
Commit 1 says every KVM reset reads pagemap for all of scratch, which is not true with PAGEMAP_SCAN.

Measurements

The call after a restore got slower in these cases, measured against base ad05110e on the same VMs:

Case	Host	Call	Call plus restore
default, Echo	KVM 6.17	45 → 78 µs (1.72x slower)	67 → 111 µs (1.66x slower)
256 MiB scratch, Echo	MSHV 6.6.100 / 6.18	1.22x / 1.13x slower	faster
256 MiB scratch, write 1 MiB	MSHV 6.6.100 / 6.18	1.83x / 1.72x slower	faster
256 MiB scratch, write 16 MiB	MSHV 6.6.100 / 6.18	2.05x / 1.86x slower	1.2x slower on both

The restore itself got 2x to 44x faster in the 256 MiB cases, which is why call plus restore still improves everywhere else. Each iteration ran one call and then a restore, timed separately, on nested Azure VMs with 4 vCPUs, default features, and release builds.

perf(mem): reset scratch in place and log host writes to it A restore zeroed scratch by dropping it with MADV_DONTNEED on KVM (fill(0) in builds with mshv3), by fill(0) on MSHV, and by mapping fresh memory on Windows (#1765). Each costs time in the size of scratch, or a fault on every page the next run touches. On KVM, every reset now reads /proc/self/pagemap for scratch, as ranges of pages in use with PAGEMAP_SCAN on Linux 6.7 or newer. A page with memory of its own that holds data is zeroed and kept, so the next run takes no faults on it. One that holds only zeros is kept as it is: a guest may write zeros to it every run, and dropping it would make every run fault on it again. Every 16th reset gives those back instead, so what stays resident follows what recent runs wrote. Swapped pages and shared pages holding data are dropped. Scratch is backed 4 KiB at a time on KVM, so a reset zeroes only what was touched. Pagemap is opened once per process. On Windows, scratch of up to 16 MiB is now zeroed in place, which is about 2x faster than a fresh mapping (over 5x when the guest writes a MiB or more); larger scratch is still replaced, so it does not all become resident. HostSharedMemory now notes the pages its writes touch, and writes with exclusive access count as touching all of it. zero_written zeroes only a given set of pages plus those, for resets that know what the guest wrote (next commits). Signed-off-by: danbugs <danilochiarlone@gmail.com>

Signed-off-by: danbugs <danilochiarlone@gmail.com>

perf(sandbox): reset scratch from the dirty log on MSHV and WHP Where the hypervisor logs the guest's writes, a restore now zeroes only the pages written since the last restore: those in the log and the pages host writes touched, the page-table copy included (#1766). Tracking starts when scratch is mapped, before the guest runs, so the first restore uses it too (hyperlight_vm/dirty_log.rs): - WHP: always. Tracking costs the guest nothing measurable there. A range mapped with tracking starts clean, so only written pages are ever touched and the rest of scratch never becomes resident. - MSHV: tracking makes the guest's first write to each page after a read fault to the hypervisor (about 1 us nested), and reading the log costs time in the size of scratch. Measured against zeroing all of scratch, that loses below 2 MiB of scratch, and once a run writes more than about a tenth of scratch (a quarter from 32 MiB, where zeroing gets about 3x slower per MiB). So scratch under 2 MiB is not tracked, and after such a run tracking pauses for 16 restores, which zero all of scratch, then resumes. The first log after mapping reports every page, so the guest sets up without faults. - KVM keeps its in-place reset; no log is read. Signed-off-by: danbugs <danilochiarlone@gmail.com>

bench: restore with large scratch The restore benchmarks use 352 KiB to 1 MiB of scratch, where resetting it costs little however it is done. Add call_with_restore with 64 and 256 MiB of scratch, with a guest that writes a few pages (Echo), 1 MiB or 16 MiB a call, and 32 light calls after one that writes 32 MiB. Time the call after a restore and the restore after a call apart too, since a restore can leave the next call faults to take. Report them, with the default size, in pull request comments. Signed-off-by: danbugs <danilochiarlone@gmail.com>

Signed-off-by: danbugs <danilochiarlone@gmail.com>

- KVM keeps scratch pages holding only zeros, and every 16th restore gives them back (64 runs at most). Dropping them on every restore made a guest that writes zeros fault them back in on each call. - MSHV pauses dirty tracking for 16 restores after a run writes much of scratch, instead of stopping it for good. - One pagemap fd per process; a forked child opens its own. pagemap_scan fills its buffer from a borrowed slice. - Host writes through exclusive access note all of scratch, and the page-table copy goes through the logged write. - map_gpa_range is unsafe, with its safety contract documented. Signed-off-by: danbugs <danilochiarlone@gmail.com>

…ight [do not merge] build against the resident scratch reset hyperlight Patch hyperlight-host and hyperlight-common to danbugs/hyperlight ff80dda2 (0.17.0 plus the scratch reset of hyperlight-dev/hyperlight#1901), to measure it in CI. Signed-off-by: danbugs <danilochiarlone@gmail.com>

danbugs · 2026-10-14T04:45:09Z

Thanks for the review! I pushed fixups for most of it, to be squashed before merge.

Benchmarks

call_after_restore/* times the call alone and snapshots/restore/scratch_256mib* times the restore alone.
call_with_restore/scratch_256mib_write_16mib writes more of scratch.
call_with_restore/32_after_32mib_scratch_64mib runs one heavy call, then 32 light ones.

KVM

The default sandbox regression was real: the guest writes zeros to some pages on every call, and the reset dropped every all-zero page, so each call faulted them back in. Zero pages are now kept, and every 16th restore gives them back, so a heavy call doesn't pin memory for good. CI's default call plus restore is back to main's (1.02x/1.07x).
pagemap_scan takes &mut [PageRegion] and sets vec and vec_len itself.
After looking at wasmtime's pagemap.rs, pagemap is now opened once per process. A forked child opens its own, since the parent's fd reports the parent's pages.

MSHV

Good catch on tracking staying off. I reproduced it: after one heavy run, every restore took 23.7 ms instead of 1.7 ms. Tracking now pauses for 16 restores and then resumes.
I couldn't reproduce the 4% break-even. On kernel 6.6.57 (nested Azure VM) I measure about 0.8 µs per written page, and writing 16 MiB of 256 MiB scratch takes 12.8 ms per call plus restore against 29.6 ms on main. Break-even there is about 45% of scratch at 256 MiB and 25% at 64 MiB, so I kept the cutoffs. I don't have 6.6.100 or 6.18. Could you share how you ran yours (heap size, iterations)? If it's the kernel, lowering the cutoffs is a small change.
Removed the "resident anyway" claim.

WHP

map_gpa_range is an unsafe fn with a # Safety section, and its unsafe blocks and the ARM64 closure have // SAFETY: comments.
Removed the dirty-log read at map time. I confirmed that a newly tracked mapping starts with a clean log.
Per-page cost: on bare metal, tracking cost nothing I could measure, but the nested CI runners look expensive. call_after_restore/scratch_256mib_write_16mib takes 169 ms on AMD and 428 ms on Intel. That bench has no main baseline, and main also faults on every page it touches because it maps fresh scratch, so I'd like a main number from the same runners before picking a cutoff. I can open a PR with just the benches for that.
Memory: agreed. A sandbox keeps resident what its largest run wrote. The commit charge is unchanged, since the whole section is charged when it's created. I can add a periodic give-back like KVM's if you'd prefer that to relying on Windows trimming the working set.

All platforms

Host writes made through with_exclusivity now mark all of scratch as written, so a future write that skips the logged methods costs a full reset instead of leaking data. The page-table copy goes through the logged path, and a test covers it.
The zero check now runs only in unit tests.
The integration test asserts that the log was used on more than 500 restores on MSHV, and on WHP where tracking works.

Nits

Fixed:
- test for the default read_dirty_log
- fake VM failing enable and disable, with tests for the cleanup
- new_scratch doc (guard page)
- commit 1 message
- "resident" instead of "committed"
WHP debug! vs warn!: kept as is and commented. A failed tracked mapping is expected where Windows can't track, but a failed log read isn't.
Policy split and HostWrites on every HostSharedMemory: agreed it would be cleaner. I'd rather do it in a follow-up to keep this diff smaller, if that works for you.

hyperlight-gh-bot · 2026-10-14T05:18:37Z

Benchmark Results

Measured commit: 643911613bc4
Baseline commit: ad05110e17bf

kvm / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`794.28 ns` (➖ 1.07x slower)
`vec_bytes`	`593.80 ns` (➖ 1.01x slower)
		`373.40 µs` (➖ 1.00x slower)

guest_calls

	`call_with_restore`	`call_after_restore`
`scratch_256mib`	`133.60 µs` (---)	`75.07 µs` (---)
`scratch_256mib_write_1mib`	`401.76 µs` (---)	`305.14 µs` (---)
`scratch_256mib_write_16mib`	`21.30 ms` (---)	`12.66 ms` (---)
`32_after_32mib_scratch_64mib`	`55.57 ms` (---)
`default`	`82.86 µs` (➖ 1.05x slower)	`62.50 µs` (---)
`scratch_64mib_write_1mib`	`380.81 µs` (---)
`scratch_64mib`	`107.10 µs` (---)

payload_allocation

	`slot_pool_segmented`
`262144`	`521.70 ns` (➖ 1.02x faster)
`65536`	`141.66 ns` (➖ 1.05x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`80.52 ms` (➖ 1.01x slower)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`7.76 ns` (➖ 1.06x faster)	`7.72 ns` (➖ 1.00x faster)	`7.72 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`93.43 µs` (➖ 1.01x slower)

snapshots

	`restore`
`scratch_256mib_write_16mib`	`1.23 ms` (---)
`scratch_256mib_write_1mib`	`81.73 µs` (---)
`default`	`19.19 µs` (➖ 1.13x slower)
`scratch_256mib`	`57.46 µs` (---)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`7.04 µs` (➖ 1.05x faster)	`7.20 µs` (➖ 1.02x slower)
`65536`	`2.10 µs` (➖ 1.09x slower)	`1.94 µs` (➖ 1.08x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`6.29 µs` (➖ 1.01x faster)	`6.42 µs` (➖ 1.02x faster)
`8192`	`1.10 µs` (➖ 1.04x slower)	`1.08 µs` (➖ 1.01x faster)
`262144`		`27.22 µs` (➖ 1.01x faster)

kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`753.51 ns` (➖ 1.01x faster)
`vec_bytes`	`562.39 ns` (➖ 1.01x faster)
		`705.80 µs` (➖ 1.02x slower)

guest_calls

	`call_with_restore`	`call_after_restore`
`scratch_256mib`	`93.06 µs` (---)	`38.80 µs` (---)
`scratch_256mib_write_1mib`	`425.94 µs` (---)	`299.78 µs` (---)
`scratch_256mib_write_16mib`	`18.54 ms` (---)	`9.39 ms` (---)
`32_after_32mib_scratch_64mib`	`42.63 ms` (---)
`default`	`48.11 µs` (➖ 1.16x slower)	`33.25 µs` (---)
`scratch_64mib_write_1mib`	`353.94 µs` (---)
`scratch_64mib`	`56.97 µs` (---)

payload_allocation

	`slot_pool_segmented`
`262144`	`508.82 ns` (➖ 1.06x faster)
`65536`	`136.90 ns` (➖ 1.07x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`83.23 ms` (➖ 1.07x slower)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`7.65 ns` (➖ 1.00x slower)	`7.69 ns` (➖ 1.03x slower)	`7.66 ns` (➖ 1.02x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`49.38 µs` (➖ 1.02x faster)

snapshots

	`restore`
`scratch_256mib_write_16mib`	`1.29 ms` (---)
`scratch_256mib_write_1mib`	`111.45 µs` (---)
`default`	`13.34 µs` (➖ 1.16x faster)
`scratch_256mib`	`54.48 µs` (---)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`8.26 µs` (➖ 1.01x slower)	`8.39 µs` (➖ 1.02x slower)
`65536`	`2.35 µs` (➖ 1.00x slower)	`2.37 µs` (➖ 1.02x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.04 µs` (➖ 1.03x slower)	`7.97 µs` (➖ 1.00x faster)
`8192`	`878.51 ns` (➖ 1.04x slower)	`849.30 ns` (➖ 1.03x slower)
`262144`		`33.82 µs` (➖ 1.06x slower)

mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`949.31 ns` (➖ 1.01x faster)
`vec_bytes`	`709.38 ns` (➖ 1.02x faster)
		`345.57 µs` (➖ 1.25x slower)

guest_calls

	`call_with_restore`	`call_after_restore`
`scratch_256mib`	`1.68 ms` (---)	`138.32 µs` (---)
`scratch_256mib_write_1mib`	`2.46 ms` (---)	`696.18 µs` (---)
`scratch_256mib_write_16mib`	`14.01 ms` (---)	`10.45 ms` (---)
`32_after_32mib_scratch_64mib`	`18.33 ms` (---)
`default`	`159.83 µs` (➖ 1.09x slower)	`76.97 µs` (---)
`scratch_64mib_write_1mib`	`1.15 ms` (---)
`scratch_64mib`	`566.08 µs` (---)

payload_allocation

	`slot_pool_segmented`
`262144`	`712.06 ns` (➖ 1.01x faster)
`65536`	`193.53 ns` (➖ 1.02x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`60.52 ms` (➖ 1.03x slower)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`9.40 ns` (➖ 1.00x slower)	`9.70 ns` (➖ 1.02x faster)	`9.49 ns` (➖ 1.00x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`86.54 µs` (➖ 1.00x slower)

snapshots

	`restore`
`scratch_256mib_write_16mib`	`3.99 ms` (---)
`scratch_256mib_write_1mib`	`1.78 ms` (---)
`default`	`66.97 µs` (➖ 1.01x faster)
`scratch_256mib`	`1.53 ms` (---)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`10.05 µs` (➖ 1.09x faster)	`10.11 µs` (➖ 1.06x slower)
`65536`	`2.47 µs` (➖ 1.00x faster)	`2.24 µs` (➖ 1.01x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.41 µs` (➖ 1.16x faster)	`8.34 µs` (➖ 1.22x faster)
`8192`	`1.26 µs` (➖ 1.05x faster)	`1.24 µs` (➖ 1.01x faster)
`262144`		`36.50 µs` (➖ 1.03x faster)

mshv3 / intel (Linux) (❌ 1.77x)

Top regressions

function_call_codec/decode_vec_bytes_copy — ❌ 1.77x slower

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.01 µs` (➖ 1.13x slower)
`vec_bytes`	`702.65 ns` (➖ 1.06x slower)
		`1.22 ms` (❌ 1.77x slower)

guest_calls

	`call_with_restore`	`call_after_restore`
`scratch_256mib`	`1.77 ms` (---)	`151.74 µs` (---)
`scratch_256mib_write_1mib`	`2.94 ms` (---)	`703.66 µs` (---)
`scratch_256mib_write_16mib`	`18.26 ms` (---)	`9.87 ms` (---)
`32_after_32mib_scratch_64mib`	`18.29 ms` (---)
`default`	`196.87 µs` (➖ 1.01x faster)	`95.37 µs` (---)
`scratch_64mib_write_1mib`	`1.14 ms` (---)
`scratch_64mib`	`573.83 µs` (---)

payload_allocation

	`slot_pool_segmented`
`262144`	`647.09 ns` (➖ 1.01x faster)
`65536`	`173.81 ns` (➖ 1.02x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`75.12 ms` (➖ 1.02x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`9.62 ns` (➖ 1.00x slower)	`9.48 ns` (➖ 1.00x slower)	`10.20 ns` (➖ 1.12x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`54.03 µs` (➖ 1.06x slower)

snapshots

	`restore`
`scratch_256mib_write_16mib`	`10.20 ms` (---)
`scratch_256mib_write_1mib`	`2.65 ms` (---)
`default`	`103.27 µs` (➖ 1.06x slower)
`scratch_256mib`	`1.74 ms` (---)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`8.07 µs` (➖ 1.01x faster)	`8.01 µs` (➖ 1.01x slower)
`65536`	`2.31 µs` (➖ 1.01x slower)	`2.31 µs` (➖ 1.02x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`7.69 µs` (➖ 1.02x slower)	`7.67 µs` (➖ 1.01x faster)
`8192`	`924.50 ns` (➖ 1.04x faster)	`942.66 ns` (➖ 1.02x slower)
`262144`		`39.58 µs` (➖ 1.02x faster)

hyperv-ws2025 / amd (Windows) (🌟 5.75x)

Top improvements

guest_calls/call_with_restore/default — 🌟 5.75x faster
snapshots/restore/default — 🌟 2.75x faster

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.21 µs` (➖ 1.01x slower)
`vec_bytes`	`790.86 ns` (➖ 1.00x slower)
		`1.87 ms` (➖ 1.00x faster)

guest_calls

	`call_with_restore`	`call_after_restore`
`scratch_256mib`	`2.85 ms` (---)	`307.34 µs` (---)
`scratch_256mib_write_1mib`	`5.48 ms` (---)	`1.97 ms` (---)
`scratch_256mib_write_16mib`	`325.83 ms` (---)	`157.87 ms` (---)
`32_after_32mib_scratch_64mib`	`30.93 ms` (---)
`default`	`269.21 µs` (🌟 5.75x faster)	`152.08 µs` (---)
`scratch_64mib_write_1mib`	`1.92 ms` (---)
`scratch_64mib`	`871.31 µs` (---)

payload_allocation

	`slot_pool_segmented`
`262144`	`786.91 ns` (➖ 1.01x slower)
`65536`	`236.25 ns` (➖ 1.01x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`77.86 ms` (➖ 1.03x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`10.55 ns` (➖ 1.02x slower)	`10.41 ns` (➖ 1.04x faster)	`10.17 ns` (➖ 1.01x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`636.24 µs` (➖ 1.15x faster)

snapshots

	`restore`
`scratch_256mib_write_16mib`	`23.69 ms` (---)
`scratch_256mib_write_1mib`	`2.62 ms` (---)
`default`	`121.14 µs` (🌟 2.75x faster)
`scratch_256mib`	`2.84 ms` (---)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`8.93 µs` (➖ 1.05x faster)	`9.04 µs` (➖ 1.04x faster)
`65536`	`2.44 µs` (➖ 1.01x faster)	`2.39 µs` (➖ 1.02x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`9.00 µs` (➖ 1.04x faster)	`9.05 µs` (➖ 1.03x slower)
`8192`	`1.35 µs` (➖ 1.02x faster)	`1.30 µs` (➖ 1.01x slower)
`262144`		`40.06 µs` (➖ 1.00x faster)

hyperv-ws2025 / intel (Windows) (🌟 6.07x)

Top improvements

guest_calls/call_with_restore/default — 🌟 6.07x faster
snapshots/restore/default — 🌟 2.51x faster

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.20 µs` (➖ 1.03x faster)
`vec_bytes`	`783.26 ns` (➖ 1.01x faster)
		`3.03 ms` (➖ 1.04x faster)

guest_calls

	`call_with_restore`	`call_after_restore`
`scratch_256mib`	`3.83 ms` (---)	`502.59 µs` (---)
`scratch_256mib_write_1mib`	`9.45 ms` (---)	`2.48 ms` (---)
`scratch_256mib_write_16mib`	`493.25 ms` (---)	`460.28 ms` (---)
`32_after_32mib_scratch_64mib`	`36.23 ms` (---)
`default`	`331.45 µs` (🌟 6.07x faster)	`185.77 µs` (---)
`scratch_64mib_write_1mib`	`2.47 ms` (---)
`scratch_64mib`	`1.15 ms` (---)

payload_allocation

	`slot_pool_segmented`
`262144`	`734.87 ns` (➖ 1.00x slower)
`65536`	`210.91 ns` (➖ 1.01x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`110.06 ms` (➖ 1.03x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`10.57 ns` (➖ 1.06x slower)	`10.02 ns` (➖ 1.01x slower)	`10.38 ns` (➖ 1.04x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`691.49 µs` (➖ 1.06x faster)

snapshots

	`restore`
`scratch_256mib_write_16mib`	`31.47 ms` (---)
`scratch_256mib_write_1mib`	`2.95 ms` (---)
`default`	`161.63 µs` (🌟 2.51x faster)
`scratch_256mib`	`2.63 ms` (---)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`7.85 µs` (➖ 1.02x slower)	`7.82 µs` (➖ 1.01x slower)
`65536`	`2.40 µs` (➖ 1.03x slower)	`2.40 µs` (➖ 1.02x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.45 µs` (➖ 1.03x faster)	`8.54 µs` (➖ 1.02x faster)
`8192`	`1.13 µs` (➖ 1.13x faster)	`1.11 µs` (➖ 1.01x slower)
`262144`		`41.26 µs` (➖ 1.04x faster)

_{Reported by cargo ci bench-report --candidate run:38511685441 --config-file bench_report.toml.}

syntactically · 2026-10-14T13:17:42Z

I haven't reviewed this code yet (would like to do so later) but I had a couple of comments based on the PR description.

I agree with @ludfjig that the performance tradeoff with actual guest execution is significant and not well accounted for here. I recall that we did some benchmarking on a similar mechanism for tracking dirty pages long ago, before guest cow, and it was not a win in most cases. Since the performance impact is pretty opaque, I think it's important that if this is tuned transparently with heuristic thresholds we should require them to be calibrated for the actual deployment target---they will vary a lot per machine, HV/driver version (I think various people are testing/deploying on at least 3 versions of the mshv driver which have drastically different performance here---we should also detect when movable pages are on and do the pagemap thing in that case), etc. so setting static thresholds in the code is a good recipe for them to get out of date. I'm imagining here something like encouraging app authors who have a wide range of sandboxes to do hyperlight_host::calibrate_memory_restore_heuristics() once early on a machine to get a little serialisable blob that they can pass in to sandbox creation---something that makes it clear that just setting static thresholds is unlikely to last well). (I also think it is probably worth having an override for the heuristics, even if the heuristics are initialised based on benchmarking on the machine).

I also think that a big part of the motivation here is just the problem of having the scratch region be oversized drastically. One other thing that might be worth looking into instead is whether things like "allow changing the size of the scratch region after a snapshot" would allow workloads to tune their scratch sizes to be closer to the actual amount of data written, largely eliminating the gap. It might also be worth eventually revisiting the question of whether allowing sandboxes to request more memory from the host is worthwhile, since it would make sizing these things neatly easier. Shrinking the scratch region should have a better performance impact overall without the tradeoff in guest execution time.

danbugs requested review from andreiltd, dblnz, devigned, jprendes, jsturtevant, ludfjig, simongdavies, squillace and syntactically as code owners October 7, 2026 23:57

Copilot AI balanced review requested due to automatic review settings October 7, 2026 23:57

danbugs added kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. area/performance Addresses performance labels Oct 7, 2026

Copilot started reviewing on behalf of danbugs October 8, 2026 00:01 View session

Copilot AI reviewed Oct 8, 2026

View reviewed changes

Comment thread src/hyperlight_host/src/mem/scratch_reset.rs

Comment thread src/hyperlight_host/src/mem/scratch_reset.rs Outdated

Comment thread src/hyperlight_host/src/mem/scratch_reset.rs Outdated

danbugs force-pushed the resident-scratch-reset branch from 74aa786 to b907442 Compare October 8, 2026 00:11