Layered/incremental snapshots by ludfjig · Pull Request #1926 · hyperlight-dev/hyperlight

ludfjig · 2026-10-12T20:00:48Z

Replaces #1902, which moved to an upstream branch so follow-up PRs can stack on it. Earlier review is on #1902.

Summary

Snapshots share immutable memory layers. A capture copies only the pages changed since its parent snapshot, and Sandbox::snapshot switches the sandbox to the capture so the next one is incremental too. This makes snapshot() much faster for larger sandboxes. It also improves memory density: each snapshot holds only its changed pages and shares the rest with its parent, where main holds a full copy of guest memory per snapshot. See HIP 0003.

Deviations from the merged HIP 0003 draft (the HIP is updated):

Each snapshot stores its page tables in a separate block. A restore uses only the newest page tables, so copies in shared blobs would waste memory and disk. The guest and CPU write to page tables, so a restore must copy them into writable scratch, and keeping them in a read-only blob saves no copy.
Sandbox::snapshot installs the capture, because otherwise every later capture copies all pages changed since the last restore again. Installing removes map_region and map_file_cow regions, because the snapshot now holds their mapped pages and later blobs may reuse their addresses.
Snapshots from earlier versions are rejected instead of loaded as one layer. This is simpler, and there is little reason to keep backwards compatibility here.

Changes

Snapshot memory is a list of immutable shared blobs with per-snapshot live ranges, plus separate page tables copied into scratch on restore.
Capture reuses parent pages the guest did not change and copies the rest into one new blob.
Sandbox::snapshot installs the capture into the running sandbox. Scratch pages it frees are zeroed.
The OCI format stores ordered data layers, a page-table layer and a transport layer (config v4, ABI 6).
Snapshot performance: adjacent page-table walk mappings are merged, and page-table pages are cached during capture.
KVM: KVM_X86_QUIRK_SLOT_ZAP_ALL is disabled where supported, so deleting a memory slot flushes only that slot.
New benchmarks snapshots/call_and_create and snapshots/restore_alternate cover the cost moved into the next call and restores to a non-current snapshot.

Breaking changes

Sandbox::snapshot removes map_region and map_file_cow regions if guest has not mapped them in. Pages of these regions the guest has not mapped are lost.
PtRootFinder receives a SnapshotMemoryReader and returns Result<Vec<u64>>.
A snapshot fails if it needs more than 30 memory mappings. This bounds restore cost by to limiting the number of hypercalls to map memory.

Commits

Add call_and_create and restore_alternate snapshot benchmarks. Lands first so the benchmarks run before and after.
Implement layered and incremental snapshots.
Install captured snapshots into the sandbox.
vmem: merge adjacent mappings in walk_va_spaces.
Cache page-table pages in PageTableReader.
Disable KVM_X86_QUIRK_SLOT_ZAP_ALL.

Benchmarks

Compared to main.

MSHV: Linux 6.6 mshv.
KVM: Linux 6.17.

Highlights

Benchmark	MSHV	KVM
`snapshots/create/default`	454 µs → 247 µs (−46%)	373 µs → 283 µs (−24%)
`snapshots/create/large`	173 ms → 1.86 ms (−99%)	157 ms → 1.84 ms (−99%)
`snapshots/call_and_create/medium`	39.2 ms → 774 µs (−98%)	35.3 ms → 817 µs (−98%)
`snapshots/restore_alternate/large`	4.44 ms → 157 µs (−96%)	1.32 ms → 478 µs (−64%)

Guest calls show no consistent change.

Regressions

Benchmark	MSHV	KVM	Cause
`snapshot_files/load_snapshot_unverified/default`	88.8 µs → 135 µs (+52%)	93.6 µs → 141 µs (+51%)	Maps 3 blob files instead of 1 (2 layers + page tables)
`snapshot_files/load_snapshot/default`	617 µs → 707 µs (+15%)	608 µs → 702 µs (+15%)	Maps 3 blob files instead of 1 (2 layers + page tables), and hashes 44 KiB of overwritten pages kept in the base layer
`snapshot_files/save_snapshot/small`	8.93 ms → 9.07 ms (~)	58.2 ms → 98.8 ms (+70%)	Writes, hashes and fsyncs 3 blob files instead of 1. fsync is slow on the KVM host
`sandboxes/sandbox_from_snapshot/small`	822 µs → 914 µs (+11%)	843 µs → 1.18 ms (+40%)	One map hypercall (MSHV) or memory slot (KVM) per live range
`snapshots/restore_alternate/default`	162 µs → 121 µs (−26%)	253 µs → 442 µs (+75%)	KVM: one memory slot change per range that differs between the two snapshots
`snapshot_files/cold_start_via_snapshot_unverified/small`	2.24 ms → 1.61 ms (−28%)	1.96 ms → 2.31 ms (+18%)	KVM: loading maps 3 blob files instead of 1, and creating the sandbox sets one memory slot per live range
`sandboxes/create_initialized/medium`	11.7 ms → 12.2 ms (+5%)	9.15 ms → 9.67 ms (+6%)	On KVM, the host backs the new snapshot mapping with 2 fewer transparent huge pages, so copying it in takes about 260 more page faults. A later PR in this stack removes the copy

Kernels before 6.12 cannot disable KVM_X86_QUIRK_SLOT_ZAP_ALL, so each memory slot change flushes all guest mappings. With the quirk enabled, the first call after snapshot() measured about +650 µs instead of +60 µs. WHP and aarch64 were not benchmarked.

A saved snapshot stores each layer in full. Base layer pages that a newer layer overwrote stay in the file, which adds 44 KiB for default.

Most of the KVM sandbox_from_snapshot regression comes from the guest heap allocator. Each live range is a KVM memory slot, set with one KVM_SET_USER_MEMORY_REGION ioctl. buddy_system_allocator writes a free-list pointer at the start of every power-of-two heap block during guest init. Each written page leaves a hole in the base layer, so a snapshot needs 5 to 16 slots depending on heap size. A later PR in this stack switches the guest allocator to talc, which keeps the heap one live range, and every size needs 4 slots. That makes KVM sandbox_from_snapshot 20 to 26% faster for small to large heaps. Four slots remain against one on main, so a smaller regression remains.

Full results

All benchmarks on both hosts (click to expand)

Before → after (change). Negative is faster. Bold is a regression. ~ is no consistent change.

snapshots

Benchmark	Size	MSHV	KVM
`call_and_create`	default	564 µs → 381 µs (−32%)	453 µs → 417 µs (−8%)
`call_and_create`	small	2.74 ms → 426 µs (−84%)	2.29 ms → 468 µs (−80%)
`call_and_create`	medium	39.2 ms → 774 µs (−98%)	35.3 ms → 817 µs (−98%)
`call_and_create`	large	173 ms → 2.01 ms (−99%)	161 ms → 1.98 ms (−99%)
`create`	default	454 µs → 247 µs (−46%)	373 µs → 283 µs (−24%)
`create`	small	2.46 ms → 296 µs (−88%)	2.11 ms → 334 µs (−84%)
`create`	medium	41.2 ms → 656 µs (−98%)	34.6 ms → 686 µs (−98%)
`create`	large	173 ms → 1.86 ms (−99%)	157 ms → 1.84 ms (−99%)
`restore`	default	65 µs → 66.1 µs (~)	20.3 µs → 20.4 µs (~)
`restore`	small	66.4 µs → 66.7 µs (~)	20.8 µs → 21.3 µs (~)
`restore`	medium	74.6 µs → 70.1 µs (−6%)	25.8 µs → 26.2 µs (~)
`restore`	large	1.22 ms → 98.3 µs (−92%)	392 µs → 52.1 µs (−87%)
`restore_alternate`	default	162 µs → 121 µs (−26%)	253 µs → 442 µs (+75%)
`restore_alternate`	small	273 µs → 124 µs (−54%)	270 µs → 452 µs (+67%)
`restore_alternate`	medium	922 µs → 128 µs (−86%)	284 µs → 457 µs (+61%)
`restore_alternate`	large	4.44 ms → 157 µs (−96%)	1.32 ms → 478 µs (−64%)

guest_calls

Benchmark	Size	MSHV	KVM
`call`	default	44.9 µs → 43.4 µs (~)	18 µs → 18.2 µs (~)
`call`	small	44.6 µs → 44.5 µs (~)	18.3 µs → 18.1 µs (~)
`call`	medium	44.6 µs → 44 µs (~)	18.1 µs → 17.4 µs (~)
`call`	large	44.6 µs → 44.3 µs (~)	18.2 µs → 17.8 µs (~)
`call_with_host_function`	default	73.1 µs → 74.4 µs (~)	34.1 µs → 33.5 µs (~)
`call_with_host_function`	small	73.1 µs → 73.4 µs (~)	32.5 µs → 33.8 µs (~)
`call_with_host_function`	medium	73.8 µs → 75.4 µs (~)	33.8 µs → 36.4 µs (~)
`call_with_host_function`	large	73.1 µs → 73.7 µs (~)	33.8 µs → 31.7 µs (~)
`call_with_restore`	default	133 µs → 134 µs (~)	64.2 µs → 62.4 µs (~)
`call_with_restore`	small	134 µs → 135 µs (~)	64.1 µs → 65.6 µs (~)
`call_with_restore`	medium	144 µs → 140 µs (~)	69.3 µs → 68.7 µs (~)
`call_with_restore`	large	171 µs → 166 µs (~)	92.6 µs → 90.3 µs (~)
`different_thread`	-	67.6 µs → 66.1 µs (~)	41.1 µs → 41.3 µs (~)
`interrupt_latency`	-	47.4 µs → 47.2 µs (~)	23.6 µs → 25.5 µs (~)

guest_functions_with_large_parameters

Benchmark	Size	MSHV	KVM
`guest_call_with_large_parameters`	-	97.7 ms → 97.5 ms (~)	1.01 s → 874 ms (−14%)

sample_workloads

Benchmark	Size	MSHV	KVM
`24K_in_8K_out_c`	-	56 µs → 53.9 µs (~)	27.1 µs → 26.9 µs (~)
`24K_in_8K_out_rust`	-	54.8 µs → 54 µs (~)	28.1 µs → 27.7 µs (~)

sandboxes

Benchmark	Size	MSHV	KVM
`create_initialized`	default	1.54 ms → 1.52 ms (~)	2.5 ms → 2.61 ms (~)
`create_initialized`	small	4.32 ms → 4.32 ms (~)	4.54 ms → 4.52 ms (~)
`create_initialized`	medium	11.7 ms → 12.2 ms (+5%)	9.15 ms → 9.67 ms (+6%)
`create_initialized`	large	38.3 ms → 38.1 ms (~)	26.3 ms → 26.7 ms (~)
`create_initialized_and_drop`	default	20.3 ms → 21.2 ms (~)	13.4 ms → 11.8 ms (−12%)
`create_initialized_and_drop`	small	28.2 ms → 27.6 ms (~)	14.4 ms → 14.3 ms (~)
`create_initialized_and_drop`	medium	31.6 ms → 32.1 ms (~)	18.5 ms → 20.2 ms (~)
`create_initialized_and_drop`	large	57 ms → 60.9 ms (~)	37.6 ms → 38.8 ms (~)
`sandbox_from_snapshot`	default	619 µs → 656 µs (+6%)	785 µs → 992 µs (+26%)
`sandbox_from_snapshot`	small	822 µs → 914 µs (+11%)	843 µs → 1.18 ms (+40%)
`sandbox_from_snapshot`	medium	2.09 ms → 2.15 ms (~)	891 µs → 1.34 ms (+51%)
`sandbox_from_snapshot`	large	6.06 ms → 6.16 ms (~)	1.04 ms → 1.57 ms (+50%)

snapshot_files

Benchmark	Size	MSHV	KVM
`cold_start_via_evolve`	default	1.59 ms → 1.61 ms (~)	2.77 ms → 2.86 ms (~)
`cold_start_via_evolve`	small	4.51 ms → 4.47 ms (~)	4.63 ms → 4.77 ms (~)
`cold_start_via_evolve`	medium	13 ms → 12.4 ms (~)	9.44 ms → 9.94 ms (+5%)
`cold_start_via_evolve`	large	38.2 ms → 38.1 ms (~)	27.6 ms → 26.8 ms (~)
`cold_start_via_snapshot`	default	1.82 ms → 1.94 ms (+7%)	2.44 ms → 2.62 ms (+7%)
`cold_start_via_snapshot`	small	9.14 ms → 8.44 ms (−8%)	8.2 ms → 8.58 ms (+5%)
`cold_start_via_snapshot`	medium	53.9 ms → 53 ms (~)	50 ms → 50.5 ms (~)
`cold_start_via_snapshot`	large	204 ms → 204 ms (~)	192 ms → 193 ms (~)
`cold_start_via_snapshot_unverified`	default	1.26 ms → 1.38 ms (+10%)	1.95 ms → 2.11 ms (+8%)
`cold_start_via_snapshot_unverified`	small	2.24 ms → 1.61 ms (−28%)	1.96 ms → 2.31 ms (+18%)
`cold_start_via_snapshot_unverified`	medium	3.77 ms → 2.94 ms (−22%)	2.07 ms → 2.47 ms (+19%)
`cold_start_via_snapshot_unverified`	large	7.66 ms → 7.37 ms (−4%)	2.2 ms → 2.65 ms (+21%)
`load_snapshot`	default	617 µs → 707 µs (+15%)	608 µs → 702 µs (+15%)
`load_snapshot`	small	6.62 ms → 6.68 ms (~)	6.29 ms → 6.39 ms (~)
`load_snapshot`	medium	50.1 ms → 50.2 ms (~)	47.8 ms → 48 ms (~)
`load_snapshot`	large	197 ms → 196 ms (~)	190 ms → 190 ms (~)
`load_snapshot_unverified`	default	88.8 µs → 135 µs (+52%)	93.6 µs → 141 µs (+51%)
`load_snapshot_unverified`	small	90 µs → 139 µs (+55%)	95 µs → 142 µs (+50%)
`load_snapshot_unverified`	medium	91 µs → 141 µs (+55%)	96.3 µs → 144 µs (+50%)
`load_snapshot_unverified`	large	94.6 µs → 145 µs (+53%)	99.9 µs → 148 µs (+48%)
`save_snapshot`	default	845 µs → 934 µs (+10%)	31.5 ms → 36.7 ms (+16%)
`save_snapshot`	small	8.93 ms → 9.07 ms (~)	58.2 ms → 98.8 ms (+70%)
`save_snapshot`	medium	68.2 ms → 68.4 ms (~)	540 ms → 656 ms (~)
`save_snapshot`	large	269 ms → 269 ms (~)	2.54 s → 2.41 s (~)

call_and_create times a guest call plus snapshot creation, so the call pays for memory remapping done by the previous snapshot. restore_alternate restores two snapshots in turn, so each restore targets memory that is not current. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

Share immutable data blobs and store page tables separately. Persist layered snapshots as OCI image layouts. Capture copies only pages that changed since the snapshot the sandbox was created from or last restored to. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

Snapshot capture switches the running sandbox to the captured snapshot. The next capture reuses its layers and copies only pages changed after it. Scratch pages used before the capture become free again. They are zeroed so guest allocations return zeroed pages, as after restore. Capture removes map_region and map_file_cow regions. Pages of these regions that the guest has not mapped are lost. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

walk_va_spaces merges mappings that have contiguous physical and virtual addresses and the same kind. Snapshot capture classifies and rebuilds each merged mapping once. Sandbox::snapshot on KVM, release build: * large heap: 22% to 37% faster * 1 GiB heap: 26% to 43% faster Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

A page-table walk reads every entry through GuestPhysicalMemoryView, which resolves the backing and copies 8 bytes through a locked volatile copy. The reader keeps a copy of the last table page it read, so each table costs one copy. Snapshot capture and crash dumps use the reader. The walk of a default sandbox drops from 187 us to 44 us. Criterion snapshots/create on MSHV: * default: 425 us to 252 us (-36%) * medium: 977 us to 645 us (-34%) Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

By default, deleting a memslot flushes the guest mappings of every memslot. KVM keeps that flush as a workaround for VMs with assigned GPUs. Hyperlight assigns no devices and disables the quirk when the kernel supports it (Linux 6.12 and later). Deleting a memslot then flushes only that memslot's mappings. Snapshot capture deletes a memslot when a layer drops out of the snapshot. In a loop of one call and one snapshot on KVM (Linux 6.18, nested under WSL2): * call after snapshot: 914-993 us to 143-185 us * snapshot: 595-722 us to 414-640 us Restore to another snapshot and region unmapping also delete memslots. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

Copilot

🔵 Needs a closer look

The VMM memory boundary changes substantially, and OCI loading maps oversized blobs before enforcing resource limits.

1 open finding

Preflight descriptor sizes before mapping to prevent excessive reservations · New

What changed in this PR

Adds layered incremental snapshots that share immutable memory, persist OCI data and page-table layers, and optimize restore mappings.

Changes:

Adds shared snapshot layers with incremental capture and installation.
Introduces snapshot ABI 6 and OCI config v4.
Optimizes page-table walks, KVM invalidation, tests, and benchmarks.

File	Description
`src/tests/rust_guests/simpleguest/src/main.rs`	Adds zero-init mapping test support.
`src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs`	Bumps golden snapshots to v6.
`src/hyperlight_host/src/sandbox/uninitialized.rs`	Documents capture behavior and uses GPA spans.
`src/hyperlight_host/src/sandbox/trace/mem_profile.rs`	Reads layered memory by GPA.
`src/hyperlight_host/src/sandbox/snapshot/tripwires.rs`	Pins ABI 6 media types.
`src/hyperlight_host/src/sandbox/snapshot/memory/mod.rs`	Defines layered snapshot memory.
`src/hyperlight_host/src/sandbox/snapshot/memory/backing.rs`	Maps and accesses snapshot layers.
`src/hyperlight_host/src/sandbox/snapshot/file/mod.rs`	Saves and loads layered OCI snapshots.
`src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs`	Adds config v4 and new layer media types.
`src/hyperlight_host/src/sandbox/snapshot/file/config.rs`	Defines the layered snapshot schema.
`src/hyperlight_host/src/sandbox/snapshot/file_tests.rs`	Tests layered persistence and validation.
`src/hyperlight_host/src/sandbox/mod.rs`	Exports `SnapshotMemoryReader`.
`src/hyperlight_host/src/sandbox/initialized.rs`	Installs captures and updates snapshot APIs.
`src/hyperlight_host/src/mem/virtq/tests.rs`	Updates scratch transport tests.
`src/hyperlight_host/src/mem/shared_mem.rs`	Adds immutable memory and range mappings.
`src/hyperlight_host/src/mem/layout.rs`	Removes flat snapshot layout state.
`src/hyperlight_host/src/hypervisor/virtual_machine/kvm/x86_64.rs`	Disables the slot-wide KVM flush quirk.
`src/hyperlight_host/src/hypervisor/virtual_machine/hvf/mod.rs`	Uses range-based snapshot mappings.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs`	Adopts layered VM memory.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/test_support.rs`	Updates mapping-state fault tests.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/mod.rs`	Manages multiple snapshot mappings.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/aarch64.rs`	Adopts layered VM memory.
`src/hyperlight_host/src/hypervisor/gdb/mod.rs`	Resolves layered debug memory.
`src/hyperlight_host/build.rs`	Removes obsolete memory-read configuration.
`src/hyperlight_host/benches/benchmarks.rs`	Adds incremental snapshot benchmarks.
`src/hyperlight_common/src/vmem.rs`	Coalesces adjacent mappings.
`src/hyperlight_common/src/arch/amd64/vmem.rs`	Applies mapping coalescing on AMD64.
`src/hyperlight_common/src/arch/aarch64/vmem.rs`	Applies mapping coalescing on AArch64.
`proposals/0003-incremental-snapshots/README.md`	Updates the snapshot design.
`docs/snapshot-versioning.md`	Documents ABI 6 versioning.
`docs/snapshot-oci-format.md`	Documents layered OCI storage.
`docs/paging-development-notes.md`	Describes incremental capture behavior.
`CHANGELOG.md`	Records breaking API and format changes.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+        let mut snap_files = data_descs
+            .iter()
+            .chain([pt_desc])
+            .map(|desc| open_snapshot_blob(&blobs_dir, desc))
+            .collect::<crate::Result<Vec<_>>>()?;


hyperlight-gh-bot · 2026-10-12T20:46:32Z

Benchmark Results

Measured commit: 8c92fa0ae671
Baseline commit: dcb53c04a5a7

kvm / amd (Linux) (❌ 1.55x)

Top regressions

snapshot_files/load_snapshot_unverified/small — ❌ 1.55x slower

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`762.00 ns` (➖ 1.04x faster)
`vec_bytes`	`586.72 ns` (➖ 1.02x slower)
		`374.35 µs` (➖ 1.00x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`524.77 ns` (➖ 1.01x faster)
`65536`	`148.54 ns` (➖ 1.10x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`78.24 ms` (➖ 1.06x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`7.73 ns` (➖ 1.00x faster)	`8.05 ns` (➖ 1.04x slower)	`7.82 ns` (➖ 1.01x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`144.17 µs` (❌ 1.55x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`7.34 µs` (➖ 1.02x slower)	`6.97 µs` (➖ 1.01x faster)
`65536`	`2.05 µs` (➖ 1.06x slower)	`1.92 µs` (➖ 1.10x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`6.24 µs` (➖ 1.03x faster)	`6.20 µs` (➖ 1.10x faster)
`8192`	`1.08 µs` (➖ 1.02x slower)	`1.05 µs` (➖ 1.01x faster)
`262144`		`28.91 µs` (➖ 1.05x slower)

kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`698.24 ns` (➖ 1.04x slower)
`vec_bytes`	`503.89 ns` (➖ 1.02x faster)
		`653.51 µs` (➖ 1.02x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`509.66 ns` (➖ 1.01x slower)
`65536`	`135.19 ns` (➖ 1.00x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`77.01 ms` (➖ 1.04x slower)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`6.91 ns` (➖ 1.00x faster)	`6.90 ns` (➖ 1.00x faster)	`6.91 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`68.13 µs` (➖ 1.50x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`7.50 µs` (➖ 1.00x slower)	`7.56 µs` (➖ 1.01x slower)
`65536`	`2.12 µs` (➖ 1.01x slower)	`2.10 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`7.05 µs` (➖ 1.01x faster)	`6.96 µs` (➖ 1.02x faster)
`8192`	`726.41 ns` (➖ 1.03x faster)	`710.83 ns` (➖ 1.01x faster)
`262144`		`28.99 µs` (➖ 1.03x faster)

mshv3 / amd (Linux) (❌ 1.62x)

Top regressions

snapshot_files/load_snapshot_unverified/small — ❌ 1.62x slower

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`966.91 ns` (➖ 1.01x faster)
`vec_bytes`	`712.54 ns` (➖ 1.00x faster)
		`328.73 µs` (➖ 1.00x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`679.59 ns` (➖ 1.01x slower)
`65536`	`181.65 ns` (➖ 1.00x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`64.28 ms` (➖ 1.04x slower)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`10.02 ns` (➖ 1.01x slower)	`9.92 ns` (➖ 1.00x faster)	`10.15 ns` (➖ 1.02x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`140.65 µs` (❌ 1.62x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`9.06 µs` (➖ 1.17x faster)	`9.35 µs` (➖ 1.11x faster)
`65536`	`2.64 µs` (➖ 1.03x faster)	`2.48 µs` (➖ 1.01x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.02 µs` (➖ 1.01x faster)	`8.02 µs` (➖ 1.08x faster)
`8192`	`1.34 µs` (➖ 1.01x slower)	`1.34 µs` (➖ 1.04x faster)
`262144`		`35.18 µs` (➖ 1.00x slower)

mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`936.36 ns` (➖ 1.00x faster)
`vec_bytes`	`671.41 ns` (➖ 1.02x slower)
		`882.41 µs` (➖ 1.32x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`627.98 ns` (➖ 1.01x slower)
`65536`	`164.82 ns` (➖ 1.05x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`73.08 ms` (➖ 1.10x slower)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`9.54 ns` (➖ 1.02x faster)	`8.90 ns` (➖ 1.01x slower)	`8.77 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`67.88 µs` (➖ 1.38x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`8.01 µs` (➖ 1.00x slower)	`7.99 µs` (➖ 1.01x slower)
`65536`	`2.30 µs` (➖ 1.00x slower)	`2.27 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`7.58 µs` (➖ 1.01x slower)	`7.46 µs` (➖ 1.08x faster)
`8192`	`910.77 ns` (➖ 1.04x faster)	`912.24 ns` (➖ 1.00x slower)
`262144`		`38.40 µs` (➖ 1.07x faster)

hyperv-ws2025 / amd (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.19 µs` (➖ 1.00x faster)
`vec_bytes`	`806.11 ns` (➖ 1.02x slower)
		`2.00 ms` (➖ 1.10x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`786.70 ns` (➖ 1.05x faster)
`65536`	`239.47 ns` (➖ 1.13x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`68.57 ms` (➖ 1.24x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`10.02 ns` (➖ 1.10x faster)	`10.12 ns` (➖ 1.01x faster)	`10.13 ns` (➖ 1.03x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`852.99 µs` (➖ 1.16x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`9.38 µs` (➖ 1.07x faster)	`9.18 µs` (➖ 1.00x faster)
`65536`	`2.50 µs` (➖ 1.02x slower)	`2.35 µs` (➖ 1.00x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.79 µs` (➖ 1.00x faster)	`9.06 µs` (➖ 1.02x slower)
`8192`	`1.31 µs` (➖ 1.00x slower)	`1.30 µs` (➖ 1.15x faster)
`262144`		`47.18 µs` (➖ 1.22x slower)

hyperv-ws2025 / intel (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.23 µs` (➖ 1.04x slower)
`vec_bytes`	`796.37 ns` (➖ 1.01x slower)
		`2.89 ms` (➖ 1.06x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`759.81 ns` (➖ 1.02x faster)
`65536`	`214.00 ns` (➖ 1.01x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`88.84 ms` (➖ 1.33x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`10.38 ns` (➖ 1.01x faster)	`10.02 ns` (➖ 1.09x faster)	`10.55 ns` (➖ 1.02x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`742.80 µs` (➖ 1.05x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`7.76 µs` (➖ 1.01x slower)	`8.23 µs` (➖ 1.06x slower)
`65536`	`2.34 µs` (➖ 1.02x slower)	`2.33 µs` (➖ 1.00x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.45 µs` (➖ 1.02x faster)	`8.70 µs` (➖ 1.03x slower)
`8192`	`1.14 µs` (➖ 1.14x faster)	`1.10 µs` (➖ 1.01x faster)
`262144`		`43.80 µs` (➖ 1.06x slower)

_{Reported by cargo ci bench-report --candidate run:38301435985 --config-file bench_report.toml.}

ludfjig added 6 commits October 9, 2026 14:01

ludfjig added kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. regen-goldens Regenerate snapshot golden fixtures labels Oct 12, 2026

Copilot AI balanced review requested due to automatic review settings October 12, 2026 20:00

ludfjig added the ready-for-review PR is ready for (re-)review label Oct 12, 2026

ludfjig requested review from danbugs, dblnz and devigned as code owners October 12, 2026 20:00

ludfjig added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Oct 12, 2026

ludfjig requested review from jprendes, jsturtevant and syntactically as code owners October 12, 2026 20:00

ludfjig added the regen-goldens Regenerate snapshot golden fixtures label Oct 12, 2026

ludfjig requested review from andreiltd and simongdavies as code owners October 12, 2026 20:00

ludfjig added the ready-for-review PR is ready for (re-)review label Oct 12, 2026

ludfjig requested a review from squillace as a code owner October 12, 2026 20:00

ludfjig added this pull request to stack #1929 October 12, 2026 20:01

ludfjig mentioned this pull request Oct 12, 2026

Layered/incremental snapshots #1902

Closed

Copilot started reviewing on behalf of ludfjig October 12, 2026 20:05 View session

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Comment thread src/hyperlight_host/src/sandbox/snapshot/file/mod.rs

Comment on lines +901 to +905

let mut snap_files = data_descs

.iter()

.chain([pt_desc])

.map(|desc| open_snapshot_blob(&blobs_dir, desc))

.collect::<crate::Result<Vec<_>>>()?;

Conversation

ludfjig commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Breaking changes

Commits

Benchmarks

Highlights

Regressions

Full results

snapshots

guest_calls

guest_functions_with_large_parameters

sample_workloads

sandboxes

snapshot_files

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

hyperlight-gh-bot Bot commented Oct 12, 2026

Benchmark Results

Top regressions

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Top regressions

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

ludfjig commented Oct 12, 2026 •

edited

Loading