moss-infer-vulkan runs MOSS-Image on Vulkan. It is the fallback for GPUs the CUDA engine does not support (AMD, Intel,
older NVIDIA cards): a standalone program in its own package (mossinfer-VERSION-linux-vulkan-x86_64.tar.gz,
mossinfer-VERSION-windows-vulkan-x86_64.zip) that needs a Vulkan driver and neither CUDA nor an NVIDIA driver. It is
slower than CUDA; on a supported NVIDIA GPU, use moss-infer.
It runs txt2img with the official Base and SFT bf16 files, the official euler sampler and schedule and classifier-free
guidance with the uncond skip layers. The tier is reference. Everything else is refused with a message that names
the option: other samplers and schedulers, LoRAs, img2img and inpainting, hires fix, control adapters, references,
the lossy accelerators, quantized or GGUF files, and the CUDA memory options other than --vram-budget. The tokenizer, the seed's initial noise
and the schedule are the engine's own, so a seed gives the same inputs as on CUDA.
moss-infer-vulkan --model MOSS-Image/base --prompt "a red apple on a wooden table" --size 1024x1024 -o apple.png
moss-infer-vulkan --list-devicesThe flags are those of moss-infer generate that the backend runs, with the same defaults (--negative, --size,
--width, --height, --steps, --cfg, --shift, --seed, --skip-layers, --max-seq, --latent-out,
--no-metadata, --quiet, --vram-budget; --tier accepts reference), checked with the engine's limits and
messages (at most 4.2 megapixels and 4096 pixels on a side, at most 10000 steps). --device N picks a device (default: the first usable one, a
discrete GPU first) and --kernels coop|coop-f16|plain the kernels (default: the first in that order the device has);
both are printed at the start of every run. Progress goes to stderr one line per event: text encoder, denoise <step>/<steps> <seconds>, vae, then a done: summary. The exit status is 0 on success, 2 for a usage error or an unsupported
request, 1 for any other failure. --help prints the usage to stdout, --version the version.
| Kernels | Needs | Used for |
|---|---|---|
coop |
Vulkan 1.3, VK_KHR_cooperative_matrix and VK_KHR_shader_bfloat16 with 16x16x16 bf16 fragments, a compute subgroup size fixable at 32, 40 KiB of shared memory |
GEMMs, attention and the VAE's convolutions on matrix cores |
coop-f16 |
the same with 16x16x16 f16 fragments instead of bf16 ones | devices with matrix cores but no bf16 fragments (Mesa's RADV exposes them only on RDNA4, for one): the GEMMs and attention on matrix cores, the VAE as plain |
plain |
Vulkan 1.2, buffer device addresses, 64-bit integers, 32 KiB of shared memory | every other device; no cooperative matrices, 16-bit arithmetic or particular subgroup size |
coop-f16 computes what coop computes. The weights and activations stay bf16; each GEMM operand row is scaled by a
power of two that brings it into f16's range as it is loaded, where a bf16 value is an f16 value, and the result is
scaled back, so the products are the same. Only attention's q, k and v convert unscaled: their rare values below 2^-14
lose bits, and a value above 65504 would become infinite (the model's stay far below). On the golden case's text encoder and on real DiT blocks it gives the same figures as coop, and the
same step 0 accuracy below; whole images then differ as any two backends' do (see below).
The text encoder streams its 32 layers from the model files, one layer on the device at a time. The DiT keeps as many
of its 42 blocks on the device as fit next to the run's activations, from the driver's memory budget
(VK_EXT_memory_budget). The other blocks stay in host memory and are copied on the transfer queue while the block
before them runs. So a GPU with less than the DiT's 9.5 GiB still runs it, at the cost of host memory and bus traffic.
The VAE decodes the whole image at once: about 1.6 GiB at 1024x1024, 3.1 GiB at 1456x1456 and 6 GiB at 2048x2048. A
decode the device cannot hold is refused before the text encoder runs. The text encoder and the DiT are on the device
one after the other, never together. --vram-budget GiB caps the device memory of the VAE and the DiT (their weights,
the DiT's activations and the decode), as moss-infer generate's does: the DiT's blocks past it stream, and a run it
cannot hold with every block streamed, or whose decode it cannot hold, is refused before the text encoder runs. With
-o the VAE's weights (about 0.1 GiB) count against it.
Long operations are split into submits of bounded work, so no submit runs long enough to trip the Windows display driver's watchdog on a slow GPU.
Against the official PyTorch pipeline's golden data at 1024x1024 (tests/golden/g_fridge_1024), on the coop path. All
figures on this page come from an RTX 4090 Laptop GPU (every path); CI runs the kernel checks on lavapipe (plain).
| Stage | Result |
|---|---|
| Text encoder, last hidden state | cosine 0.99988 (prompt), 0.99980 (negative prompt); torch itself with one kernel swapped: 0.99989, 0.99983 |
| DiT, step 0 prediction | cosine 0.999976 (cond), 0.999983 (uncond); the CUDA engine's test asks for 0.99988 |
| VAE decode of the golden latent | PSNR 69.4 dB against the golden PNG, 99.25 % identical bytes, max difference 1 |
Whole images keep the composition and the text but not the pixels: 50 sampling steps magnify any difference. The
golden 1024x1024 case through moss-infer ends with a final latent at cosine 0.974 to the official pipeline's (the CUDA
backend: 0.992, being nearly bit-identical to torch op by op) and an image at 30.8 dB PSNR. That is the sampler's
sensitivity, not an error of the backend: the same Vulkan steps on torch's own text embeddings instead of the Vulkan
encoder's (cosine 0.9998 apart, about torch's own variation when one of its kernels changes) end 0.971 apart from each
other. A 512x512, 20-step image with seed 7 is 32.9 dB PSNR from the CUDA engine's image of the same command: the same apple,
details redrawn.
vulkan/ is its own CMake project and needs no CUDA: Vulkan headers that define VK_KHR_shader_bfloat16 and a glslc
that compiles GL_EXT_bfloat16, as in the Vulkan SDK 1.4.357 the release builds use. The program opens the Vulkan
loader at run time, so it links nothing of Vulkan.
cmake -S vulkan -B build-vulkan -G Ninja -DMOSSINFER_VULKAN_TOOLS=OFF
cmake --build build-vulkan
cpack --config build-vulkan/CPackConfig.cmake -B build-vulkanWith the tools on (the default) it also builds the backend's checks and benchmarks; see vulkan/README.md.