Check what LifeboatCache does for your model on your GPUs, with the vLLM or SGLang you already run. One line:
curl -fsSL https://raw.githubusercontent.com/IterateAI/LifeboatCacheTest/main/run.sh | bashIt finds your running vLLM / SGLang server, takes its model and flags, asks before stopping it, runs the tests, starts your server again with the same command, and prints a summary. The results are written as CSV files.
To skip the question (for example over a non-interactive SSH session), or to choose options:
curl -fsSL https://raw.githubusercontent.com/IterateAI/LifeboatCacheTest/main/run.sh | bash -s -- --yes
curl -fsSL https://raw.githubusercontent.com/IterateAI/LifeboatCacheTest/main/run.sh | bash -s -- --fullEach test runs on your engine three ways — without offload, with the engine's own offload, and with LifeboatCache:
| Engine | Configurations |
|---|---|
| vLLM on NVIDIA | no offload · vLLM native offload (CPU + disk) · LifeboatCache · LMCache · LMCache + LifeboatCache |
| SGLang on NVIDIA | no offload (radix cache) · SGLang HiCache with its file backend · LifeboatCache |
| vLLM on a Mac (CPU backend) | no offload · LifeboatCache (vLLM's own offload and LMCache need a GPU) |
| SGLang on a Mac (MLX backend) | no offload · LifeboatCache (the MLX runtime has no HiCache) |
| Test | What it shows |
|---|---|
| Capacity | How much KV fits on your GPU, and how many users of the test length fit at once |
| Concurrency | Total output tokens/s, tokens/s per user and time to first token at 1 to 64 users (to 256 with --full) |
| Returning users | Users come back with a new question after their KV left the GPU: time to first token and the share of the prompt served from cache |
| Overflow | About twice as many users as the GPU holds, so the GPU really evicts them |
| Restart | A returning user after the server restarts |
The prompt length (up to 64K tokens), the number of users and the cache sizes are picked from your
model's context length, the GPU's KV space, RAM, /dev/shm and free disk. A quick run takes about
30 minutes; --full about 90.
- Python packages: it installs
lifeboatcacheinto the Python environment that runs your engine. On vLLM it also tries LMCache 0.5.x, and skips it if installing would change anything. Your engine, torch and transformers are pinned, so their versions don't change. - Your server: it's stopped only with your consent (or
--yes) and started again at the end with the same command, environment and working directory, also if the test fails. Its output then goes to~/.lifeboatcache-verify/your-server-restarted.log. - Files: everything else stays in
~/.lifeboatcache-verify(results, logs, the temporary cache tiers, which are deleted after each test). Put it on local NVMe with--workdirif your home directory isn't.
Nothing is sent anywhere: no telemetry, and the results stay on your machine.
~/.lifeboatcache-verify/results-<time>/ and the same as a .tar.gz:
| File | Contents |
|---|---|
summary.csv |
One row per configuration: returning-user time to first token, speed-up over recompute and over the engine's own offload, overflow, restart, peak throughput |
returning.csv |
Every returning / overflow / restart phase: mean, p50, p90 and max time to first token, cached share, prefill rate, errors |
throughput.csv |
Every concurrency level per configuration |
system.csv |
GPU, driver, CPU, RAM, disk, engine and package versions, the engine flags used |
results.json |
Everything, including server log tails for configurations that didn't start |
Besides the one-line verifier, agentic/ holds the tools for an agentic workload
of real agent sessions replayed turn by turn across
configurations, host-DRAM sizes and numbers of agents — and reports/ the results
measured with them (per-request CSVs, per-cell summaries and the slides):
| Report | Hardware, model | 64 agents: vLLM alone → + LMCache → + LifeboatCache (mean TTFT / output tok/s) |
|---|---|---|
| 4 × RTX PRO 6000 | 4 × RTX PRO 6000, Qwen3-235B-A22B-FP8, 128 GB DRAM | 26.7 s / 94 → 17.1 s / 148 → 9.1 s / 219 |
| 1 × RTX PRO 6000 | 1 × RTX PRO 6000, Qwen3-30B-A3B, 32 GB DRAM | 25.4 s / 180 → 21.8 s / 211 → 13.0 s / 323 |
| gpt-oss-20b, 1 GPU | 1 × RTX PRO 6000, gpt-oss-20b, 24 GB DRAM (128 agents, 25 rounds; 256 agents in the report) | 0.49 s / 1,117 → 0.64 s / 1,144 → 0.44 s / 1,425 |
| AMD MI210 | 1 × AMD Instinct MI210, Qwen2.5-7B, 64 GB DRAM (128 agents, 10 rounds; vLLM's own offloading, SGLang and Lifeboat in the report) | 260.0 s / 45 → 158.7 s / 66 → 55.0 s / 147 |
At 128 agents on the four-GPU server LifeboatCache delivered 3.0x the output throughput of vLLM alone (208 vs 70 tok/s). With 25-round sessions it served 256 agents at 222.6 tok/s and 512 agents at 122.4 tok/s, about 90% of every prompt from cache. vLLM 0.32, LMCache 0.5.6, LifeboatCache 0.1.23. On one AMD Instinct MI210 it gave vLLM 3.2x (128 agents) and 3.6x (256) the output throughput of vLLM alone, and SGLang and Lifeboat 19–20% more at 128 agents.
The full set in one deck: reports/LifeboatCache_Performance_Customer.pptx
(NVIDIA GPUs in Intel Xeon servers, AMD Instinct MI210, Apple M4 Max).
Against the published Intel / HPE study. Metrum AI's public study Streamlined Agentic Inference with KV Cache Offload (September 2026; HPE ProLiant DL380a Gen12, Intel Xeon 6) ran the same model on 4 × RTX PRO 6000 with 25-round sessions and 512 output tokens, using vLLM 0.22 and LMCache 0.4.6. At 256 agents its best offload configuration (LMCache, DRAM + NVMe) gave 309.4 s mean TTFT and 124.0 output tok/s (GPU only: 839.5 s, 47.0); vLLM 0.32 + LifeboatCache 0.1.23 at the same settings gave 158.0 s and 222.6 tok/s, and at 512 agents 598.2 s / 122.4 tok/s against the study's 688.9 s / 111.2. Not a head-to-head run: the software versions and host differ (the study's host had 2 TB of RAM, ours 1 TB), and part of the gap can come from the newer vLLM.
Shared KV pool over RDMA (report, LifeboatCache 0.1.29): the memory pool that lets several servers share one KV cache, run over software RDMA (Soft-RoCE) on one host with UCX limited to RDMA verbs. Every block came back correct; the pool reached about 2/3 of that link's raw write rate and half its read rate (1.03 vs 1.53 GB/s, 0.66 vs 1.22 GB/s, 2 MB blocks). Soft-RoCE runs on the CPU: this verifies the RDMA path, not the speed of RDMA hardware, which has not been measured yet.
Apple M4 Max (Qwen3-0.6B, the verifier above, LifeboatCache 0.1.14 / 0.1.15): a returning user's time to first token went from 7.97 s to 1.48 s on SGLang's MLX backend and from 49.8 s to 1.11 s on vLLM's CPU backend. Peak throughput was unchanged on vLLM and 19% lower on SGLang MLX with the cache on. A functional check, not a capacity benchmark.
- Linux with NVIDIA GPUs and
nvidia-smi, or macOS on Apple silicon; Python 3 andcurl. - vLLM 0.32.x, or SGLang 0.5.19 or newer, installed, with the model you want to test. Other vLLM versions run, but LifeboatCache's native mode targets 0.32; a configuration that doesn't attach is reported as skipped rather than counted.
- Free disk for the cache tiers (the tool uses up to half the free space, at most 500 GB).
LifeboatCache (0.1.13+) runs with vLLM and SGLang on NVIDIA GPUs (measured), on AMD GPUs (measured on an MI210 with vLLM, SGLang and Lifeboat), on Intel GPUs (supported by the engines, not measured yet), with vLLM's CPU backend (Apple silicon and CPU servers) and with SGLang's MLX backend on Apple silicon (full-attention models, and hybrid models from 0.1.17).
The same line works on Apple silicon with vLLM's CPU build or SGLang's MLX runtime. Unified
memory holds the model, the engine's KV pool and its working buffers together, so the tests stay
small there: prompts up to 16K tokens, 2 to 3 returning users, no overflow test, and SGLang's test
servers start with --mem-fraction-static 0.3 unless your flags set it (larger settings ran a
64 GB M4 Max out of Metal memory with several long users). Measured with this line on an M4 Max,
Qwen3-0.6B (vLLM with lifeboatcache 0.1.14, SGLang with 0.1.15):
| Engine | Returning users | Without offload | With LifeboatCache | After a restart |
|---|---|---|---|---|
| vLLM 0.32 (CPU) | 3 x 8K | 49.8 s | 1.11 s (99% cached, 45x) | 0.55 s |
| SGLang 0.5.22 (MLX) | 2 x 16K | 7.9 s | 1.48 s (100% cached, 5.4x) | 0.88 s |
| Option | Meaning |
|---|---|
--yes |
Stop the running server without asking (it is started again afterwards) |
--full |
Longer run: bigger overflow, concurrency up to 256 |
--model PATH_OR_ID |
Model to test when no server is running (or a different one) |
--engine-python PATH |
The Python that runs vLLM / SGLang, if it isn't found automatically |
--pid PID |
Which running server to take the model and flags from, if there are several |
--workdir DIR |
Where results and temporary cache tiers go (default ~/.lifeboatcache-verify) |
--lifeboat-version X.Y.Z |
Test a specific lifeboatcache version (default: latest) |
--no-lmcache |
Don't try to install LMCache |
--only a,b |
Run only these configurations (vllm, vllm_offload, lifeboat, lmcache, lmcache_lifeboat, sglang, sglang_hicache) |
-- FLAGS |
Extra engine flags, e.g. -- --max-model-len 65536 --tensor-parallel-size 2 |
Without a running server: ... | bash -s -- --model Qwen/Qwen3-30B-A3B (the model must be
downloaded already, or downloadable by your engine).
- A server that renames its process (SGLang does) hides its environment: it is restarted with
the Python it runs in, its command line and this shell's environment (plus
SGLANG_USE_MLX=1on a Mac), and the tool says so before stopping it. - A server inside a container can't be stopped and restarted safely from outside. Run the line
inside that container, or stop the container and use
--model/--engine-pythonhere. - A server run by systemd: the tool prints the
systemctlcommands to stop it and start it again. - Hybrid (Mamba / linear-attention) models on SGLang's MLX backend: one request at a time; concurrent hybrid requests crash inside SGLang's MLX runtime, so the multi-user tests there will fail. On NVIDIA GPUs, hybrid restores after a restart work from 0.1.16.
Questions or results to share: open an issue in this repository.