Layered/incremental snapshots by ludfjig · Pull Request #1902 · hyperlight-dev/hyperlight

ludfjig · 2026-10-08T01:42:03Z

Moved to #1926, which stacks with #1927 and #1928.

Summary

Snapshots share immutable memory layers. A capture copies only the pages changed since its parent snapshot, and Sandbox::snapshot switches the sandbox to the capture so the next one is incremental too. This makes snapshot(), restore and sandbox creation much faster for larger sandboxes, and lets related snapshots share memory. See HIP 0003.

Deviations from the merged HIP 0003 draft (the HIP is updated):

Each snapshot stores its page tables in a separate block. A restore uses only the newest page tables, so copies in shared blobs would waste memory and disk. The guest and CPU write to page tables, so a restore must copy them into writable scratch, and keeping them in a read-only blob saves no copy.
Sandbox::snapshot installs the capture, because otherwise every later capture copies all pages changed since the last restore again. Installing removes map_region and map_file_cow regions, because the snapshot now holds their mapped pages and later blobs may reuse their addresses.
Snapshots from earlier versions are rejected instead of loaded as one layer. This is simpler, and there is little reason to keep backwards compatibility here.

Changes

Snapshot memory is a list of immutable shared blobs with per-snapshot live ranges, plus separate page tables copied into scratch on restore.
Capture reuses parent pages the guest did not change and copies the rest into one new blob.
Sandbox::snapshot installs the capture into the running sandbox. Scratch pages it frees are zeroed.
The OCI format stores ordered data layers, a page-table layer and a transport layer (config v4, ABI 6).
Snapshot performance: adjacent page-table walk mappings are merged, and page-table pages are cached during capture.
KVM: KVM_X86_QUIRK_SLOT_ZAP_ALL is disabled where supported, so deleting a memory slot flushes only that slot.
New benchmarks snapshots/call_and_create and snapshots/restore_alternate cover the cost moved into the next call and restores to a non-current snapshot.

Breaking changes

Sandbox::snapshot removes map_region and map_file_cow regions if guest has not mapped them in. Pages of these regions the guest has not mapped are lost.
PtRootFinder receives a SnapshotMemoryReader and returns Result<Vec<u64>>.
A snapshot fails if it needs more than 30 memory mappings. This bounds restore cost by to limiting the number of hypercalls to map memory.

Commits

Add call_and_create and restore_alternate snapshot benchmarks. Lands first so the benchmarks run before and after.
Implement layered and incremental snapshots.
Install captured snapshots into the sandbox.
vmem: merge adjacent mappings in walk_va_spaces.
Cache page-table pages in PageTableReader.
Disable KVM_X86_QUIRK_SLOT_ZAP_ALL.

Benchmarks

MSHV: AMD EPYC 7763, Linux 6.6 mshv.
KVM AMD: AMD EPYC 7763, Linux 6.17.
KVM Intel: Xeon Platinum 8370C, Linux 6.17.

Highlights

Benchmark	MSHV	KVM AMD	KVM Intel
`snapshots/create/default`	452 µs → 241 µs (−46%)	370 µs → 283 µs (−23%)	350 µs → 195 µs (−42%)
`snapshots/create/large`	165 ms → 1.92 ms (−99%)	200 ms → 1.82 ms (−99%)	155 ms → 1.99 ms (−99%)
`snapshots/call_and_create/medium`	38.9 ms → 755 µs (−98%)	35 ms → 817 µs (−98%)	37.4 ms → 665 µs (−98%)
`sandboxes/create_initialized/large`	38.6 ms → 10.8 ms (−72%)	26.1 ms → 3.34 ms (−87%)	58.9 ms → 2.58 ms (−96%)
`snapshot_files/cold_start_via_evolve/large`	38.9 ms → 10.9 ms (−72%)	27.1 ms → 3.54 ms (−87%)	63.2 ms → 2.55 ms (−96%)

Guest calls and restores to the current snapshot show no consistent change.

Regressions

Benchmark	MSHV	KVM AMD	KVM Intel	Cause
`snapshot_files/load_snapshot_unverified/default`	87.6 µs → 134 µs (+55%)	93.4 µs → 140 µs (+50%)	48.8 µs → 70.1 µs (+44%)	Maps 3 blob files instead of 1 (2 layers + page tables)
`snapshot_files/load_snapshot/default`	603 µs → 690 µs (+14%)	600 µs → 699 µs (+17%)	626 µs → 700 µs (+12%)	Same 3 files, plus hashing 44 KiB of overwritten pages kept in the base layer
`snapshot_files/save_snapshot/default`	811 µs → 884 µs (+9%)	28.5 ms → 33 ms (+16%)	27.7 ms → 35.2 ms (+20%)	Writes and hashes 3 blob files instead of 1
`sandboxes/sandbox_from_snapshot/large`	6.05 ms → 6.09 ms (~)	1.02 ms → 1.58 ms (+49%)	696 µs → 1.16 ms (+74%)	KVM: 16 memory slots, one per live range
`snapshots/restore_alternate/default`	184 µs → 138 µs (−24%)	235 µs → 389 µs (+66%)	146 µs → 276 µs (+89%)	KVM: one memory slot change per range that differs between the two snapshots
`snapshot_files/save_snapshot/small`	6.24 ms → 6.33 ms (~)	36.9 ms → 40.9 ms (+11%)	37.2 ms → 42.5 ms (+14%)	Writes and hashes 3 blob files instead of 1
`sandboxes/sandbox_from_snapshot/default`	601 µs → 596 µs (~)	765 µs → 916 µs (+15%)	520 µs → 662 µs (+23%)	KVM: one memory slot per live range
`snapshots/restore_alternate/small`	356 µs → 137 µs (−62%)	265 µs → 431 µs (+59%)	175 µs → 295 µs (+78%)	KVM: one memory slot change per range that differs between the two snapshots
`snapshot_files/cold_start_via_snapshot_unverified/small`	2.86 ms → 1.49 ms (−49%)	1.76 ms → 2.24 ms (+29%)	1.34 ms → 1.49 ms (+16%)	KVM: the load and `sandbox_from_snapshot` costs combined

KVM sandbox_from_snapshot is slower at every size (+0.14 to +0.56 ms) and restore_alternate up to medium. KVM numbers are from Linux 6.17. Kernels before 6.12 cannot disable KVM_X86_QUIRK_SLOT_ZAP_ALL, so each memory slot change flushes all guest mappings. With the quirk enabled, the first call after snapshot() measured about +650 µs instead of +60 µs. WHP and aarch64 were not benchmarked. A reused layer keeps the pages its descendants overwrote, so a single snapshot can be slightly larger on disk (44 KiB for default).

Most of the KVM sandbox_from_snapshot regression comes from the guest heap allocator. Each live range is a KVM memory slot, set with one KVM_SET_USER_MEMORY_REGION ioctl. buddy_system_allocator writes a free-list pointer at the start of every power-of-two heap block during guest init. Each written page leaves a hole in the base layer, so a snapshot needs 5 to 16 slots depending on heap size. Switching the guest allocator to talc keeps the heap one live range, and every size needs 4 slots. Compared to this PR with buddy_system_allocator, using talc instead results in 8 to 31% faster KVM sandbox_from_snapshot in one short round. AMD medium showed no significant change. Four slots remain against one on main, so a smaller regression remains.

Full results

All benchmarks on all three hosts.

Full results (click to expand)

Before → after (change). Negative is faster. Bold is a regression. ~ is not significant or under ±3%. KVM AMD compares against main for benchmarks that existed before this PR.

snapshots

Benchmark	Size	MSHV	KVM AMD	KVM Intel
`call_and_create`	default	576 µs → 367 µs (−35%)	453 µs → 397 µs (−12%)	407 µs → 296 µs (−26%)
`call_and_create`	small	1.81 ms → 415 µs (−77%)	2.37 ms → 453 µs (−81%)	2.44 ms → 321 µs (−86%)
`call_and_create`	medium	38.9 ms → 755 µs (−98%)	35 ms → 817 µs (−98%)	37.4 ms → 665 µs (−98%)
`call_and_create`	large	164 ms → 2.05 ms (−99%)	146 ms → 2.04 ms (−98%)	143 ms → 1.79 ms (−99%)
`create`	default	452 µs → 241 µs (−46%)	370 µs → 283 µs (−23%)	350 µs → 195 µs (−42%)
`create`	small	1.64 ms → 297 µs (−82%)	1.96 ms → 339 µs (−83%)	2.31 ms → 253 µs (−89%)
`create`	medium	38.7 ms → 659 µs (−98%)	34.5 ms → 685 µs (−98%)	36.9 ms → 554 µs (−98%)
`create`	large	165 ms → 1.92 ms (−99%)	200 ms → 1.82 ms (−99%)	155 ms → 1.99 ms (−99%)
`restore`	default	89.7 µs → 88.7 µs (~)	19.7 µs → 20.1 µs (~)	13.1 µs → 13.1 µs (~)
`restore`	small	92.9 µs → 89.4 µs (−7%)	20.9 µs → 20.9 µs (~)	13.6 µs → 13.7 µs (~)
`restore`	medium	201 µs → 95.4 µs (−69%)	52.8 µs → 24.7 µs (−69%)	55.8 µs → 17.2 µs (−82%)
`restore`	large	4.47 ms → 125 µs (−97%)	1.69 ms → 46.3 µs (−97%)	1.13 ms → 60 µs (−95%)
`restore_alternate`	default	184 µs → 138 µs (−24%)	235 µs → 389 µs (+66%)	146 µs → 276 µs (+89%)
`restore_alternate`	small	356 µs → 137 µs (−62%)	265 µs → 431 µs (+59%)	175 µs → 295 µs (+78%)
`restore_alternate`	medium	1.21 ms → 147 µs (−88%)	249 µs → 404 µs (+60%)	280 µs → 255 µs (~)
`restore_alternate`	large	4.38 ms → 182 µs (−96%)	1.02 ms → 423 µs (−59%)	1.15 ms → 365 µs (−69%)

guest_calls

Benchmark	Size	MSHV	KVM AMD	KVM Intel
`call`	default	44.8 µs → 41.5 µs (~)	17.8 µs → 17.3 µs (~)	17.2 µs → 18.2 µs (+7%)
`call`	small	45.2 µs → 44.2 µs (~)	17.9 µs → 17.9 µs (~)	18.2 µs → 17.8 µs (−4%)
`call`	medium	44.7 µs → 42.6 µs (−4%)	17.5 µs → 17.9 µs (~)	18 µs → 17.2 µs (~)
`call`	large	44.4 µs → 43.3 µs (~)	17.6 µs → 18.2 µs (~)	18.1 µs → 18.4 µs (~)
`call_with_host_function`	default	73.9 µs → 73.9 µs (~)	34.1 µs → 34.2 µs (~)	34.3 µs → 31.8 µs (−7%)
`call_with_host_function`	small	76.4 µs → 75 µs (~)	34 µs → 34 µs (~)	33.5 µs → 34.8 µs (+4%)
`call_with_host_function`	medium	76 µs → 72.5 µs (−5%)	33.8 µs → 34 µs (~)	34.4 µs → 34.8 µs (~)
`call_with_host_function`	large	74.6 µs → 74.9 µs (~)	34.2 µs → 34.4 µs (~)	34 µs → 34.6 µs (~)
`call_with_restore`	default	159 µs → 155 µs (~)	59.2 µs → 61.2 µs (~)	38.6 µs → 37.4 µs (~)
`call_with_restore`	small	162 µs → 161 µs (~)	62.2 µs → 63.3 µs (~)	39.1 µs → 38.8 µs (~)
`call_with_restore`	medium	166 µs → 162 µs (−4%)	67.4 µs → 67.9 µs (~)	43.5 µs → 43.2 µs (~)
`call_with_restore`	large	200 µs → 184 µs (−12%)	91.5 µs → 89.1 µs (−6%)	99.6 µs → 96 µs (−4%)
`different_thread`	-	65.6 µs → 69.7 µs (+4%)	41.2 µs → 41.3 µs (~)	72.9 µs → 75.5 µs (+3%)
`interrupt_latency`	-	45 µs → 44.2 µs (~)	22.9 µs → 23 µs (~)	39.5 µs → 17.4 µs (~)

guest_functions_with_large_parameters

Benchmark	Size	MSHV	KVM AMD	KVM Intel
`guest_call_with_large_parameters`	-	163 ms → 174 ms (+7%)	964 ms → 853 ms (−11%)	690 ms → 624 ms (−9%)

sample_workloads

Benchmark	Size	MSHV	KVM AMD	KVM Intel
`24K_in_8K_out_c`	-	54.4 µs → 54.2 µs (~)	25.9 µs → 27.5 µs (+4%)	24.3 µs → 24.5 µs (~)
`24K_in_8K_out_rust`	-	55 µs → 54.4 µs (~)	27.3 µs → 27.4 µs (~)	28.2 µs → 27.9 µs (~)

sandboxes

Benchmark	Size	MSHV	KVM AMD	KVM Intel
`create_initialized`	default	1.42 ms → 1.42 ms (~)	2.42 ms → 2.42 ms (~)	1.75 ms → 1.78 ms (~)
`create_initialized`	small	4.22 ms → 3.07 ms (−29%)	4.45 ms → 2.61 ms (−41%)	3.52 ms → 1.87 ms (−46%)
`create_initialized`	medium	11.3 ms → 5.47 ms (−52%)	9.05 ms → 2.83 ms (−68%)	16.4 ms → 2.04 ms (−87%)
`create_initialized`	large	38.6 ms → 10.8 ms (−72%)	26.1 ms → 3.34 ms (−87%)	58.9 ms → 2.58 ms (−96%)
`create_initialized_and_drop`	default	20.4 ms → 18.9 ms (~)	10 ms → 11.8 ms (~)	9.68 ms → 7.81 ms (−17%)
`create_initialized_and_drop`	small	28.7 ms → 25.1 ms (~)	15.1 ms → 12.3 ms (~)	11.5 ms → 8.27 ms (−34%)
`create_initialized_and_drop`	medium	30.3 ms → 30.4 ms (~)	18.9 ms → 11.2 ms (−38%)	22.5 ms → 8.72 ms (−61%)
`create_initialized_and_drop`	large	57 ms → 29.9 ms (−47%)	34.5 ms → 17.1 ms (−54%)	66.8 ms → 9.42 ms (−86%)
`sandbox_from_snapshot`	default	601 µs → 596 µs (~)	765 µs → 916 µs (+15%)	520 µs → 662 µs (+23%)
`sandbox_from_snapshot`	small	792 µs → 854 µs (+9%)	822 µs → 1.07 ms (+41%)	564 µs → 784 µs (+41%)
`sandbox_from_snapshot`	medium	2.01 ms → 2.03 ms (~)	874 µs → 1.29 ms (+40%)	572 µs → 834 µs (+49%)
`sandbox_from_snapshot`	large	6.05 ms → 6.09 ms (~)	1.02 ms → 1.58 ms (+49%)	696 µs → 1.16 ms (+74%)

snapshot_files

Benchmark	Size	MSHV	KVM AMD	KVM Intel
`cold_start_via_evolve`	default	1.5 ms → 1.43 ms (~)	2.84 ms → 2.61 ms (−10%)	2.03 ms → 1.84 ms (−8%)
`cold_start_via_evolve`	small	3.41 ms → 2.95 ms (−15%)	4.73 ms → 2.72 ms (−44%)	3.82 ms → 2.06 ms (−46%)
`cold_start_via_evolve`	medium	13 ms → 5.52 ms (−57%)	9.1 ms → 2.97 ms (−68%)	16.8 ms → 2.09 ms (−87%)
`cold_start_via_evolve`	large	38.9 ms → 10.9 ms (−72%)	27.1 ms → 3.54 ms (−87%)	63.2 ms → 2.55 ms (−96%)
`cold_start_via_snapshot`	default	1.9 ms → 1.8 ms (~)	2.45 ms → 2.53 ms (+7%)	1.83 ms → 1.79 ms (~)
`cold_start_via_snapshot`	small	9.35 ms → 7.84 ms (−16%)	8.33 ms → 8.58 ms (+4%)	8.46 ms → 8.83 ms (+4%)
`cold_start_via_snapshot`	medium	51.8 ms → 49.2 ms (−5%)	50 ms → 50.5 ms (~)	58.6 ms → 59.3 ms (~)
`cold_start_via_snapshot`	large	194 ms → 190 ms (~)	192 ms → 193 ms (~)	229 ms → 231 ms (~)
`cold_start_via_snapshot_unverified`	default	1.24 ms → 1.28 ms (~)	1.96 ms → 2.09 ms (+8%)	1.33 ms → 1.33 ms (~)
`cold_start_via_snapshot_unverified`	small	2.86 ms → 1.49 ms (−49%)	1.76 ms → 2.24 ms (+29%)	1.34 ms → 1.49 ms (+16%)
`cold_start_via_snapshot_unverified`	medium	4.21 ms → 2.65 ms (−37%)	1.91 ms → 2.34 ms (+26%)	1.44 ms → 1.36 ms (~)
`cold_start_via_snapshot_unverified`	large	8.27 ms → 7.05 ms (−16%)	2.15 ms → 2.55 ms (+21%)	1.54 ms → 1.79 ms (+18%)
`load_snapshot`	default	603 µs → 690 µs (+14%)	600 µs → 699 µs (+17%)	626 µs → 700 µs (+12%)
`load_snapshot`	small	6.34 ms → 6.4 ms (~)	6.28 ms → 6.43 ms (~)	7.22 ms → 7.37 ms (~)
`load_snapshot`	medium	46.2 ms → 46.4 ms (~)	47.8 ms → 48.7 ms (~)	57.2 ms → 65.9 ms (+15%)
`load_snapshot`	large	183 ms → 184 ms (~)	190 ms → 190 ms (~)	239 ms → 229 ms (−4%)
`load_snapshot_unverified`	default	87.6 µs → 134 µs (+55%)	93.4 µs → 140 µs (+50%)	48.8 µs → 70.1 µs (+44%)
`load_snapshot_unverified`	small	88 µs → 135 µs (+53%)	94.8 µs → 141 µs (+49%)	49.1 µs → 70.8 µs (+45%)
`load_snapshot_unverified`	medium	89.5 µs → 136 µs (+52%)	94.6 µs → 143 µs (+52%)	52.1 µs → 72.1 µs (+38%)
`load_snapshot_unverified`	large	93.7 µs → 166 µs (+66%)	97.7 µs → 147 µs (+51%)	53.3 µs → 73.8 µs (+38%)
`save_snapshot`	default	811 µs → 884 µs (+9%)	28.5 ms → 33 ms (+16%)	27.7 ms → 35.2 ms (+20%)
`save_snapshot`	small	6.24 ms → 6.33 ms (~)	36.9 ms → 40.9 ms (+11%)	37.2 ms → 42.5 ms (+14%)
`save_snapshot`	medium	45.4 ms → 44.9 ms (~)	82.5 ms → 84.4 ms (~)	84.9 ms → 93.1 ms (+10%)
`save_snapshot`	large	179 ms → 175 ms (~)	215 ms → 217 ms (~)	253 ms → 248 ms (~)

Copilot

🔵 Needs a closer look

It changes snapshot ABI and guest memory ownership across multiple hypervisor backends, including unbenchmarked paths.

2 open findings

Avoid flooding logs with per-sandbox fallback warnings · New
Document MT_PAGE_TABLES_V1 as a separate format marker · New

What changed in this PR

Adds incremental snapshots that share immutable memory layers, reducing capture and restore costs.

Changes:

Introduces layered snapshot memory and page-table storage.
Installs captures for incremental reuse across snapshots.
Updates OCI ABI, hypervisor mappings, tests, docs, and benchmarks.

File	Description
`src/tests/rust_guests/simpleguest/src/main.rs`	Adds zero-page mapping fixture.
`src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs`	Bumps goldens to v6.
`src/hyperlight_host/src/sandbox/uninitialized.rs`	Updates mapping size semantics and docs.
`src/hyperlight_host/src/sandbox/trace/mem_profile.rs`	Reads layered snapshot GPAs.
`src/hyperlight_host/src/sandbox/snapshot/tripwires.rs`	Pins ABI 6 media types.
`src/hyperlight_host/src/sandbox/snapshot/memory/mod.rs`	Defines layered snapshot memory.
`src/hyperlight_host/src/sandbox/snapshot/memory/backing.rs`	Maps and reads snapshot layers.
`src/hyperlight_host/src/sandbox/snapshot/file/mod.rs`	Persists and loads layered OCI snapshots.
`src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs`	Adds v4/v2/page-table media types.
`src/hyperlight_host/src/sandbox/snapshot/file/config.rs`	Defines layered config schema.
`src/hyperlight_host/src/sandbox/snapshot/file_tests.rs`	Expands layered-format tests.
`src/hyperlight_host/src/sandbox/mod.rs`	Exports snapshot memory reader.
`src/hyperlight_host/src/sandbox/initialized.rs`	Installs captured snapshots incrementally.
`src/hyperlight_host/src/mem/virtq/tests.rs`	Updates scratch restore expectations.
`src/hyperlight_host/src/mem/shared_mem.rs`	Adds immutable memory freezing and range mapping.
`src/hyperlight_host/src/mem/mgr.rs`	Resolves layered physical memory.
`src/hyperlight_host/src/mem/layout.rs`	Removes flat snapshot layout fields.
`src/hyperlight_host/src/hypervisor/virtual_machine/kvm/x86_64.rs`	Disables KVM slot-zap quirk.
`src/hyperlight_host/src/hypervisor/virtual_machine/hvf/mod.rs`	Adapts HVF auxiliary mapping.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs`	Uses layered mappings on x86-64.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/test_support.rs`	Updates mapping fault support.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/mod.rs`	Manages multiple snapshot slots.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/aarch64.rs`	Uses layered mappings on AArch64.
`src/hyperlight_host/src/hypervisor/gdb/mod.rs`	Resolves debugger access by backing.
`src/hyperlight_host/build.rs`	Removes obsolete memory cfg.
`src/hyperlight_host/benches/benchmarks.rs`	Adds incremental snapshot benchmarks.
`src/hyperlight_common/src/vmem.rs`	Coalesces adjacent walk mappings.
`src/hyperlight_common/src/arch/amd64/vmem.rs`	Applies mapping coalescing on AMD64.
`src/hyperlight_common/src/arch/aarch64/vmem.rs`	Applies mapping coalescing on AArch64.
`proposals/0003-incremental-snapshots/README.md`	Updates the incremental snapshot design.
`docs/snapshot-versioning.md`	Documents ABI 6 versioning.
`docs/snapshot-oci-format.md`	Documents layered OCI storage.
`docs/paging-development-notes.md`	Describes incremental page capture.
`CHANGELOG.md`	Records breaking snapshot changes.

🧠 Review effort: Balanced

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

call_and_create times a guest call plus snapshot creation, so the call pays for memory remapping done by the previous snapshot. restore_alternate restores two snapshots in turn, so each restore targets memory that is not current. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

Share immutable data blobs and store page tables separately. Persist layered snapshots as OCI image layouts. Capture copies only pages that changed since the snapshot the sandbox was created from or last restored to. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

Snapshot capture switches the running sandbox to the captured snapshot. The next capture reuses its layers and copies only pages changed after it. Scratch pages used before the capture become free again. They are zeroed so guest allocations return zeroed pages, as after restore. Capture removes map_region and map_file_cow regions. Pages of these regions that the guest has not mapped are lost. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

walk_va_spaces merges mappings that have contiguous physical and virtual addresses and the same kind. Snapshot capture classifies and rebuilds each merged mapping once. Sandbox::snapshot on KVM, release build: * large heap: 22% to 37% faster * 1 GiB heap: 26% to 43% faster Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

A page-table walk reads every entry through GuestPhysicalMemoryView, which resolves the backing and copies 8 bytes through a locked volatile copy. The reader keeps a copy of the last table page it read, so each table costs one copy. Snapshot capture and crash dumps use the reader. The walk of a default sandbox drops from 187 us to 44 us. Criterion snapshots/create on MSHV: * default: 425 us to 252 us (-36%) * medium: 977 us to 645 us (-34%) Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

By default, deleting a memslot flushes the guest mappings of every memslot. KVM keeps that flush as a workaround for VMs with assigned GPUs. Hyperlight assigns no devices and disables the quirk when the kernel supports it (Linux 6.12 and later). Deleting a memslot then flushes only that memslot's mappings. Snapshot capture deletes a memslot when a layer drops out of the snapshot. In a loop of one call and one snapshot on KVM (Linux 6.18, nested under WSL2): * call after snapshot: 914-993 us to 143-185 us * snapshot: 595-722 us to 414-640 us Restore to another snapshot and region unmapping also delete memslots. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

hyperlight-gh-bot · 2026-10-08T19:53:32Z

Benchmark Results

Measured commit: ca2ae172aefc
Baseline commit: 73f8f3ef9d3d

kvm / amd (Linux) (❌ 1.56x | 🚀 6.09x)

Top improvements

sandboxes/create_initialized_and_drop/medium — 🚀 6.09x faster

Top regressions

snapshot_files/load_snapshot_unverified/small — ❌ 1.56x slower

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`774.32 ns` (➖ 1.01x slower)
`vec_bytes`	`590.38 ns` (➖ 1.02x slower)
		`373.93 µs` (➖ 1.02x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`522.22 ns` (➖ 1.00x slower)
`65536`	`141.43 ns` (➖ 1.00x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`13.17 ms` (🚀 6.09x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`7.72 ns` (➖ 1.01x faster)	`7.74 ns` (➖ 1.05x faster)	`7.85 ns` (➖ 1.01x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`144.21 µs` (❌ 1.56x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`7.14 µs` (➖ 1.01x slower)	`7.28 µs` (➖ 1.03x slower)
`65536`	`2.05 µs` (➖ 1.06x slower)	`1.92 µs` (➖ 1.09x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`6.24 µs` (➖ 1.02x faster)	`6.23 µs` (➖ 1.01x faster)
`8192`	`1.09 µs` (➖ 1.05x slower)	`1.05 µs` (➖ 1.01x slower)
`262144`		`28.25 µs` (➖ 1.04x slower)

kvm / intel (Linux) (❌ 1.66x | 🚀 5.78x)

Top improvements

sandboxes/create_initialized_and_drop/medium — 🚀 5.78x faster

Top regressions

snapshot_files/load_snapshot_unverified/small — ❌ 1.66x slower

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`774.52 ns` (➖ 1.03x faster)
`vec_bytes`	`578.15 ns` (➖ 1.03x slower)
		`730.97 µs` (➖ 1.00x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`644.82 ns` (➖ 1.23x slower)
`65536`	`159.75 ns` (➖ 1.11x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`13.83 ms` (🚀 5.78x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`8.07 ns` (➖ 1.09x slower)	`8.08 ns` (➖ 1.11x slower)	`8.11 ns` (➖ 1.09x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`78.04 µs` (❌ 1.66x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`8.87 µs` (➖ 1.03x slower)	`9.07 µs` (➖ 1.06x slower)
`65536`	`2.55 µs` (➖ 1.05x slower)	`2.51 µs` (➖ 1.08x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.18 µs` (➖ 1.08x slower)	`8.19 µs` (➖ 1.07x slower)
`8192`	`848.31 ns` (➖ 1.06x slower)	`844.99 ns` (➖ 1.07x slower)
`262144`		`35.10 µs` (➖ 1.12x slower)

mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`947.07 ns` (➖ 1.03x faster)
`vec_bytes`	`708.11 ns` (➖ 1.00x faster)
		`269.82 µs` (➖ 1.01x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`726.11 ns` (➖ 1.00x faster)
`65536`	`192.47 ns` (➖ 1.04x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`60.68 ms` (➖ 1.08x slower)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`10.38 ns` (➖ 1.06x slower)	`9.73 ns` (➖ 1.00x slower)	`9.69 ns` (➖ 1.02x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`124.72 µs` (➖ 1.50x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`10.05 µs` (➖ 1.01x slower)	`9.49 µs` (➖ 1.01x faster)
`65536`	`2.46 µs` (➖ 1.02x faster)	`2.25 µs` (➖ 1.01x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.38 µs` (➖ 1.16x faster)	`8.11 µs` (➖ 1.18x faster)
`8192`	`1.27 µs` (➖ 1.03x faster)	`1.31 µs` (➖ 1.04x slower)
`262144`		`36.34 µs` (➖ 1.04x faster)

mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`918.95 ns` (➖ 1.00x faster)
`vec_bytes`	`664.73 ns` (➖ 1.01x slower)
		`737.00 µs` (➖ 1.03x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`619.67 ns` (➖ 1.09x faster)
`65536`	`164.56 ns` (➖ 1.01x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`76.32 ms` (➖ 1.13x slower)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`9.63 ns` (➖ 1.01x faster)	`8.86 ns` (➖ 1.01x slower)	`8.78 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`63.97 µs` (➖ 1.39x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`8.00 µs` (➖ 1.00x faster)	`8.02 µs` (➖ 1.01x slower)
`65536`	`2.27 µs` (➖ 1.07x faster)	`2.28 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`7.52 µs` (➖ 1.00x faster)	`7.49 µs` (➖ 1.02x faster)
`8192`	`910.17 ns` (➖ 1.03x faster)	`914.23 ns` (➖ 1.01x slower)
`262144`		`39.18 µs` (➖ 1.02x faster)

hyperv-ws2025 / amd (Windows) (🚀 8.27x)

Top improvements

sandboxes/create_initialized_and_drop/medium — 🚀 8.27x faster

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.17 µs` (➖ 1.08x faster)
`vec_bytes`	`795.77 ns` (➖ 1.02x slower)
		`2.04 ms` (➖ 1.11x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`797.76 ns` (➖ 1.00x faster)
`65536`	`230.93 ns` (➖ 1.01x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`12.03 ms` (🚀 8.27x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`9.94 ns` (➖ 1.12x faster)	`10.36 ns` (➖ 1.02x slower)	`10.13 ns` (➖ 1.05x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`1.17 ms` (➖ 1.32x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`9.37 µs` (➖ 1.04x slower)	`9.13 µs` (➖ 1.03x slower)
`65536`	`2.41 µs` (➖ 1.03x slower)	`2.38 µs` (➖ 1.01x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.92 µs` (➖ 1.01x slower)	`8.90 µs` (➖ 1.01x slower)
`8192`	`1.47 µs` (➖ 1.02x slower)	`1.31 µs` (➖ 1.03x slower)
`262144`		`40.04 µs` (➖ 1.01x slower)

hyperv-ws2025 / intel (Windows) (❌ 2.95x | 🚀 8.83x)

Top improvements

sandboxes/create_initialized_and_drop/medium — 🚀 8.83x faster

Top regressions

snapshot_files/load_snapshot_unverified/small — ❌ 2.95x slower

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.20 µs` (➖ 1.00x slower)
`vec_bytes`	`778.97 ns` (➖ 1.02x faster)
		`2.80 ms` (➖ 1.10x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`736.39 ns` (➖ 1.02x faster)
`65536`	`215.48 ns` (➖ 1.01x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`13.13 ms` (🚀 8.83x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`10.64 ns` (➖ 1.03x slower)	`10.18 ns` (➖ 1.02x slower)	`10.47 ns` (➖ 1.01x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`1.44 ms` (❌ 2.95x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`7.76 µs` (➖ 1.02x faster)	`7.81 µs` (➖ 1.01x faster)
`65536`	`2.33 µs` (➖ 1.02x faster)	`2.34 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.49 µs` (➖ 1.01x faster)	`9.11 µs` (➖ 1.04x slower)
`8192`	`1.40 µs` (➖ 1.00x slower)	`1.11 µs` (➖ 1.02x faster)
`262144`		`41.87 µs` (➖ 1.05x faster)

_{Reported by cargo ci bench-report --candidate run:37830255482 --config-file bench_report.toml.}

ludfjig · 2026-10-12T20:01:55Z

Closing in favor of #1926. The branch moved to upstream so follow-up PRs can stack on it: #1927 writes initial snapshot memory in place, and #1928 switches the guest allocator to talc.

ludfjig added regen-goldens Regenerate snapshot golden fixtures kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. labels Oct 8, 2026

ludfjig force-pushed the ls-v5 branch from 586ed6c to c57cef9 Compare October 8, 2026 02:04

ludfjig marked this pull request as ready for review October 8, 2026 02:06

ludfjig requested review from andreiltd, danbugs, dblnz, devigned, jprendes, jsturtevant, simongdavies, squillace and syntactically as code owners October 8, 2026 02:06

Copilot AI balanced review requested due to automatic review settings October 8, 2026 02:06

Copilot started reviewing on behalf of ludfjig October 8, 2026 02:09 View session

Copilot AI reviewed Oct 8, 2026

View reviewed changes

Comment thread src/hyperlight_host/src/hypervisor/virtual_machine/kvm/x86_64.rs Outdated

Comment thread docs/snapshot-versioning.md Outdated

This comment has been minimized.

Sign in to view

ludfjig force-pushed the ls-v5 branch from c57cef9 to 3d32254 Compare October 8, 2026 17:40

ludfjig force-pushed the ls-v5 branch from 3d32254 to 5c3bae9 Compare October 8, 2026 17:59

This comment has been minimized.

Sign in to view

ludfjig added the ready-for-review PR is ready for (re-)review label Oct 8, 2026

ludfjig added 5 commits October 8, 2026 12:12

ludfjig force-pushed the ls-v5 branch from 5c3bae9 to ca2ae17 Compare October 8, 2026 19:12

ludfjig mentioned this pull request Oct 12, 2026

Layered/incremental snapshots #1926

Open

ludfjig closed this Oct 12, 2026

Conversation

ludfjig commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Breaking changes

Commits

Benchmarks

Highlights

Regressions

Full results

snapshots

guest_calls

guest_functions_with_large_parameters

sample_workloads

sandboxes

snapshot_files

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

hyperlight-gh-bot Bot commented Oct 8, 2026

Benchmark Results

Top improvements

Top regressions

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Top improvements

Top regressions

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Top improvements

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Top improvements

Top regressions

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

ludfjig commented Oct 8, 2026 •

edited

Loading