Write initial snapshot memory in place by ludfjig · Pull Request #1927 · hyperlight-dev/hyperlight

ludfjig · 2026-10-12T20:01:02Z

New sandboxes skip copying their initial memory. Snapshot::from_env writes the guest code, PEB and init data in place, and the host populates the rest, mostly the heap, on first guest touch. Sandbox creation gets 30 to 87% faster for small to large heaps.

Guest calls that touch much of the heap pay for that population. On MSHV, guest_call_with_large_parameters gets 4% slower. Copying the memory up front removes that cost and the creation gains.

Benchmarks

Compared to the PR below.

MSHV: Linux 6.6 mshv.
KVM: Linux 6.17.

Benchmark	MSHV	KVM
`sandboxes/create_initialized/small`	4.32 ms → 3.01 ms (−30%)	4.52 ms → 2.72 ms (−40%)
`sandboxes/create_initialized/medium`	12.2 ms → 5.6 ms (−54%)	9.67 ms → 2.91 ms (−70%)
`sandboxes/create_initialized/large`	38.1 ms → 11.1 ms (−71%)	26.7 ms → 3.42 ms (−87%)
`snapshot_files/cold_start_via_evolve/large`	38.1 ms → 11.3 ms (−70%)	26.8 ms → 3.57 ms (−87%)
`guest_functions_with_large_parameters`	97.5 ms → 101 ms (+4%)	874 ms → 867 ms (~)

Full results

All benchmarks on both hosts (click to expand)

Before → after (change). Negative is faster. Bold is a regression. ~ is no consistent change.

snapshots

Benchmark	Size	MSHV	KVM
`call_and_create`	default	381 µs → 373 µs (~)	417 µs → 413 µs (~)
`call_and_create`	small	426 µs → 419 µs (~)	468 µs → 474 µs (~)
`call_and_create`	medium	774 µs → 781 µs (~)	817 µs → 806 µs (~)
`call_and_create`	large	2.01 ms → 1.98 ms (~)	1.98 ms → 1.93 ms (~)
`create`	default	247 µs → 242 µs (~)	283 µs → 287 µs (~)
`create`	small	296 µs → 299 µs (~)	334 µs → 330 µs (~)
`create`	medium	656 µs → 643 µs (~)	686 µs → 671 µs (~)
`create`	large	1.86 ms → 1.84 ms (~)	1.84 ms → 1.84 ms (~)
`restore`	default	66.1 µs → 65.6 µs (~)	20.4 µs → 20.8 µs (~)
`restore`	small	66.7 µs → 66.6 µs (~)	21.3 µs → 21.4 µs (~)
`restore`	medium	70.1 µs → 71.6 µs (~)	26.2 µs → 25 µs (~)
`restore`	large	98.3 µs → 96.1 µs (~)	52.1 µs → 48.3 µs (~)
`restore_alternate`	default	121 µs → 121 µs (~)	442 µs → 452 µs (~)
`restore_alternate`	small	124 µs → 123 µs (~)	452 µs → 451 µs (~)
`restore_alternate`	medium	128 µs → 129 µs (~)	457 µs → 451 µs (~)
`restore_alternate`	large	157 µs → 156 µs (~)	478 µs → 478 µs (~)

guest_calls

Benchmark	Size	MSHV	KVM
`call`	default	43.3 µs → 44.4 µs (~)	18.2 µs → 18.2 µs (~)
`call`	small	44 µs → 44.1 µs (~)	18.1 µs → 18.1 µs (~)
`call`	medium	43.6 µs → 44.9 µs (~)	17.4 µs → 18.2 µs (~)
`call`	large	44.2 µs → 43.4 µs (~)	17.8 µs → 18.2 µs (~)
`call_with_host_function`	default	74.4 µs → 74.9 µs (~)	33.5 µs → 33.9 µs (~)
`call_with_host_function`	small	73.4 µs → 72.6 µs (~)	33.8 µs → 34.7 µs (~)
`call_with_host_function`	medium	75.4 µs → 73.3 µs (~)	36.4 µs → 34.6 µs (~)
`call_with_host_function`	large	73.7 µs → 73.4 µs (~)	31.7 µs → 32.9 µs (~)
`call_with_restore`	default	134 µs → 138 µs (~)	62.4 µs → 63.8 µs (~)
`call_with_restore`	small	135 µs → 136 µs (~)	65.6 µs → 64.8 µs (~)
`call_with_restore`	medium	140 µs → 140 µs (~)	68.7 µs → 69.2 µs (~)
`call_with_restore`	large	166 µs → 161 µs (~)	90.3 µs → 90.2 µs (~)
`different_thread`	-	66.1 µs → 66.6 µs (~)	41.3 µs → 41.6 µs (~)
`interrupt_latency`	-	47.2 µs → 45.5 µs (~)	25.5 µs → 23.9 µs (~)

guest_functions_with_large_parameters

Benchmark	Size	MSHV	KVM
`guest_call_with_large_parameters`	-	97.5 ms → 101 ms (+4%)	874 ms → 867 ms (~)

sample_workloads

Benchmark	Size	MSHV	KVM
`24K_in_8K_out_c`	-	53.9 µs → 55.4 µs (~)	26.9 µs → 27 µs (~)
`24K_in_8K_out_rust`	-	54 µs → 54.7 µs (~)	27.7 µs → 27.2 µs (~)

sandboxes

Benchmark	Size	MSHV	KVM
`create_initialized`	default	1.52 ms → 1.47 ms (~)	2.61 ms → 2.53 ms (−3%)
`create_initialized`	small	4.32 ms → 3.01 ms (−30%)	4.52 ms → 2.72 ms (−40%)
`create_initialized`	medium	12.2 ms → 5.6 ms (−54%)	9.67 ms → 2.91 ms (−70%)
`create_initialized`	large	38.1 ms → 11.1 ms (−71%)	26.7 ms → 3.42 ms (−87%)
`create_initialized_and_drop`	default	21.2 ms → 21.7 ms (~)	11.8 ms → 13.6 ms (~)
`create_initialized_and_drop`	small	27.6 ms → 25.3 ms (~)	14.3 ms → 12.7 ms (~)
`create_initialized_and_drop`	medium	32.1 ms → 27.3 ms (−15%)	20.2 ms → 12.3 ms (−39%)
`create_initialized_and_drop`	large	60.9 ms → 30.8 ms (−49%)	38.8 ms → 13.7 ms (−65%)
`sandbox_from_snapshot`	default	656 µs → 649 µs (~)	992 µs → 955 µs (~)
`sandbox_from_snapshot`	small	914 µs → 910 µs (~)	1.18 ms → 1.17 ms (~)
`sandbox_from_snapshot`	medium	2.15 ms → 2.13 ms (~)	1.34 ms → 1.34 ms (~)
`sandbox_from_snapshot`	large	6.16 ms → 6.13 ms (~)	1.57 ms → 1.58 ms (~)

snapshot_files

Benchmark	Size	MSHV	KVM
`cold_start_via_evolve`	default	1.61 ms → 1.55 ms (−3%)	2.86 ms → 2.75 ms (−4%)
`cold_start_via_evolve`	small	4.47 ms → 3.17 ms (−29%)	4.77 ms → 2.83 ms (−41%)
`cold_start_via_evolve`	medium	12.4 ms → 5.78 ms (−53%)	9.94 ms → 3.11 ms (−69%)
`cold_start_via_evolve`	large	38.1 ms → 11.3 ms (−70%)	26.8 ms → 3.57 ms (−87%)
`cold_start_via_snapshot`	default	1.94 ms → 1.95 ms (~)	2.62 ms → 2.52 ms (~)
`cold_start_via_snapshot`	small	8.44 ms → 8.59 ms (~)	8.58 ms → 8.6 ms (~)
`cold_start_via_snapshot`	medium	53 ms → 53.5 ms (~)	50.5 ms → 50.4 ms (~)
`cold_start_via_snapshot`	large	204 ms → 203 ms (~)	193 ms → 193 ms (~)
`cold_start_via_snapshot_unverified`	default	1.38 ms → 1.37 ms (~)	2.11 ms → 2.09 ms (~)
`cold_start_via_snapshot_unverified`	small	1.61 ms → 1.58 ms (~)	2.31 ms → 2.37 ms (~)
`cold_start_via_snapshot_unverified`	medium	2.94 ms → 2.85 ms (~)	2.47 ms → 2.42 ms (~)
`cold_start_via_snapshot_unverified`	large	7.37 ms → 7.27 ms (~)	2.65 ms → 2.58 ms (~)
`load_snapshot`	default	707 µs → 704 µs (~)	702 µs → 697 µs (~)
`load_snapshot`	small	6.68 ms → 6.74 ms (~)	6.39 ms → 6.37 ms (~)
`load_snapshot`	medium	50.2 ms → 50.6 ms (~)	48 ms → 48 ms (~)
`load_snapshot`	large	196 ms → 197 ms (~)	190 ms → 190 ms (~)
`load_snapshot_unverified`	default	135 µs → 134 µs (~)	141 µs → 139 µs (~)
`load_snapshot_unverified`	small	139 µs → 136 µs (~)	142 µs → 141 µs (~)
`load_snapshot_unverified`	medium	141 µs → 142 µs (~)	144 µs → 143 µs (~)
`load_snapshot_unverified`	large	145 µs → 140 µs (~)	148 µs → 146 µs (~)
`save_snapshot`	default	934 µs → 933 µs (~)	36.7 ms → 38.5 ms (~)
`save_snapshot`	small	9.07 ms → 9.09 ms (~)	98.8 ms → 89.7 ms (~)
`save_snapshot`	medium	68.4 ms → 67.2 ms (~)	656 ms → 642 ms (~)
`save_snapshot`	large	269 ms → 265 ms (~)	2.41 s → 2.51 s (~)

Snapshot::from_env writes the guest code, PEB and init data straight into shared memory and freezes it. The host populates the remaining pages on first guest touch, so sandbox creation cost stays independent of heap size. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

hyperlight-gh-bot · 2026-10-12T20:54:12Z

Benchmark Results

Measured commit: 96fab649f47c
Baseline commit: 8c92fa0ae671

kvm / amd (Linux) (🌟 5.88x)

Top improvements

sandboxes/create_initialized_and_drop/medium — 🌟 5.88x faster

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`759.82 ns` (➖ 1.01x faster)
`vec_bytes`	`613.08 ns` (➖ 1.03x slower)
		`372.93 µs` (➖ 1.00x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`527.59 ns` (➖ 1.01x slower)
`65536`	`144.05 ns` (➖ 1.02x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`13.32 ms` (🌟 5.88x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`7.72 ns` (➖ 1.02x faster)	`7.72 ns` (➖ 1.09x faster)	`7.72 ns` (➖ 1.01x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`143.38 µs` (➖ 1.01x faster)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`7.35 µs` (➖ 1.04x slower)	`7.10 µs` (➖ 1.01x faster)
`65536`	`2.09 µs` (➖ 1.09x slower)	`1.93 µs` (➖ 1.08x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`6.23 µs` (➖ 1.02x faster)	`6.40 µs` (➖ 1.02x slower)
`8192`	`1.08 µs` (➖ 1.04x slower)	`1.06 µs` (➖ 1.00x slower)
`262144`		`26.84 µs` (➖ 1.03x faster)

kvm / intel (Linux) (🌟 5.62x)

Top improvements

sandboxes/create_initialized_and_drop/medium — 🌟 5.62x faster

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`814.26 ns` (➖ 1.01x faster)
`vec_bytes`	`610.42 ns` (➖ 1.02x faster)
		`773.26 µs` (➖ 1.02x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`601.58 ns` (➖ 1.00x faster)
`65536`	`162.57 ns` (➖ 1.01x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`14.97 ms` (🌟 5.62x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`8.42 ns` (➖ 1.02x slower)	`8.29 ns` (➖ 1.00x slower)	`8.34 ns` (➖ 1.01x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`80.81 µs` (➖ 1.04x faster)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`9.05 µs` (➖ 1.01x slower)	`9.19 µs` (➖ 1.02x slower)
`65536`	`2.55 µs` (➖ 1.01x slower)	`2.53 µs` (➖ 1.00x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.39 µs` (➖ 1.01x faster)	`8.44 µs` (➖ 1.00x faster)
`8192`	`881.68 ns` (➖ 1.01x faster)	`878.23 ns` (➖ 1.03x slower)
`262144`		`34.78 µs` (➖ 1.04x faster)

mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`939.87 ns` (➖ 1.03x faster)
`vec_bytes`	`710.45 ns` (➖ 1.00x faster)
		`271.31 µs` (➖ 1.12x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`727.24 ns` (➖ 1.00x slower)
`65536`	`192.99 ns` (➖ 1.00x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`62.76 ms` (➖ 1.05x slower)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`9.73 ns` (➖ 1.00x slower)	`9.72 ns` (➖ 1.00x slower)	`9.69 ns` (➖ 1.00x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`135.99 µs` (➖ 1.02x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`10.03 µs` (➖ 1.04x faster)	`9.89 µs` (➖ 1.01x faster)
`65536`	`2.43 µs` (➖ 1.01x faster)	`2.24 µs` (➖ 1.00x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.42 µs` (➖ 1.15x faster)	`8.39 µs` (➖ 1.22x faster)
`8192`	`1.28 µs` (➖ 1.03x faster)	`1.26 µs` (➖ 1.01x slower)
`262144`		`35.61 µs` (➖ 1.05x faster)

mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`946.92 ns` (➖ 1.05x faster)
`vec_bytes`	`696.09 ns` (➖ 1.04x slower)
		`725.60 µs` (➖ 1.18x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`625.58 ns` (➖ 1.02x faster)
`65536`	`166.14 ns` (➖ 1.01x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`84.72 ms` (➖ 1.23x slower)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`9.63 ns` (➖ 1.01x faster)	`8.78 ns` (➖ 1.00x faster)	`8.82 ns` (➖ 1.00x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`66.61 µs` (➖ 1.05x faster)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`8.00 µs` (➖ 1.00x faster)	`8.04 µs` (➖ 1.01x slower)
`65536`	`2.32 µs` (➖ 1.02x slower)	`2.30 µs` (➖ 1.00x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`7.61 µs` (➖ 1.00x slower)	`7.49 µs` (➖ 1.02x faster)
`8192`	`917.67 ns` (➖ 1.03x faster)	`922.88 ns` (➖ 1.01x slower)
`262144`		`39.06 µs` (➖ 1.04x faster)

hyperv-ws2025 / amd (Windows) (🌟 6.22x)

Top improvements

sandboxes/create_initialized_and_drop/medium — 🌟 6.22x faster

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.17 µs` (➖ 1.03x faster)
`vec_bytes`	`790.42 ns` (➖ 1.00x slower)
		`1.89 ms` (➖ 1.05x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`829.46 ns` (➖ 1.06x slower)
`65536`	`230.30 ns` (➖ 1.06x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`11.27 ms` (🌟 6.22x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`10.00 ns` (➖ 1.19x faster)	`10.11 ns` (➖ 1.04x faster)	`10.11 ns` (➖ 1.02x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`861.35 µs` (➖ 2.68x faster)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`9.12 µs` (➖ 1.02x faster)	`9.28 µs` (➖ 1.00x slower)
`65536`	`2.39 µs` (➖ 1.01x faster)	`2.39 µs` (➖ 1.02x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.86 µs` (➖ 1.01x faster)	`8.92 µs` (➖ 1.00x slower)
`8192`	`1.32 µs` (➖ 1.03x faster)	`1.32 µs` (➖ 1.00x faster)
`262144`		`38.61 µs` (➖ 1.00x slower)

hyperv-ws2025 / intel (Windows) (🌟 6.12x)

Top improvements

sandboxes/create_initialized_and_drop/medium — 🌟 6.12x faster

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.25 µs` (➖ 1.02x slower)
`vec_bytes`	`800.00 ns` (➖ 1.01x faster)
		`3.07 ms` (➖ 1.10x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`792.26 ns` (➖ 1.06x slower)
`65536`	`224.88 ns` (➖ 1.06x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`15.54 ms` (🌟 6.12x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`10.34 ns` (➖ 1.00x faster)	`10.58 ns` (➖ 1.05x slower)	`10.26 ns` (➖ 1.03x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`787.64 µs` (➖ 1.04x faster)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`7.85 µs` (➖ 1.02x slower)	`7.79 µs` (➖ 1.00x slower)
`65536`	`2.41 µs` (➖ 1.04x slower)	`2.33 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.41 µs` (➖ 1.07x faster)	`8.33 µs` (➖ 1.05x faster)
`8192`	`1.14 µs` (➖ 1.13x faster)	`1.11 µs` (➖ 1.01x faster)
`262144`		`50.95 µs` (➖ 1.12x slower)

_{Reported by cargo ci bench-report --candidate run:38301515311 --config-file bench_report.toml.}

ludfjig added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Oct 12, 2026

ludfjig requested review from andreiltd, danbugs, dblnz, devigned, jprendes, jsturtevant, simongdavies, squillace and syntactically as code owners October 12, 2026 20:01

ludfjig added this pull request to stack #1929 October 12, 2026 20:01

ludfjig mentioned this pull request Oct 12, 2026

Layered/incremental snapshots #1902

Closed

Conversation

ludfjig commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Full results

snapshots

guest_calls

guest_functions_with_large_parameters

sample_workloads

sandboxes

snapshot_files

Uh oh!

hyperlight-gh-bot Bot commented Oct 12, 2026

Benchmark Results

Top improvements

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Top improvements

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Top improvements

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Top improvements

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ludfjig commented Oct 12, 2026 •

edited

Loading