Skip to content

Stop prefill at the last KV-owning layer for KV-shared models. - #1060

Open
copybara-service[bot] wants to merge 1 commit into
devfrom
test_997920331
Open

copybara-service[bot] wants to merge 1 commit into
devfrom
test_997920331

Conversation

@copybara-service

@copybara-service copybara-service Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Stop prefill at the last KV-owning layer for KV-shared models.

What:

  1. Truncate prefill after the last KV-owning layer (ModelConfig::NumPrefillLayers()),
    skipping trailing KV-shared layers unless RuntimeConfig::prefill_all_layers,
    embedding output (output_EOS), activation observers, or MTP decoding require
    running all layers.
  2. Add RuntimeConfig::prefill_last_token so prefill-only callers (max_generated_tokens = 0)
    can populate the KV cache of the final prompt token without running a full-depth
    decode step.
  3. Stop allocating tiles and kv_head_ptrs entries for KV-shared layers in
    tiled KVCache (layer_kv_head_offsets already aliases shared layers to
    their owner).

Why / Impact:
In KV-shared models such as Gemma 4 (2B shares layers 15..34 from 13/14; E4B
shares layers 24..41 from 22/23), trailing shared layers do not write K/V during
prefill and their prefill activations are never read again. Skipping them during
prefill and omitting their unused KV tiles yields on Gemma 4 2B (gemma4-2b-q4_0-it,
4 CPU threads):

  • 3.2x faster prefill:
    • 128 tokens: 3.4 s -> 1.1 s
    • 512 tokens: 14.3 s -> 4.4 s
    • 2048 tokens: 58.0 s -> 19.0 s
  • 57% less tiled KV cache memory on Gemma 4 2B (15 vs 35 layers allocated;
    43% less on E4B, 24 vs 42 layers).
  • Bit-identical own-KV cache and next-token logits (once the MatMul autotuner
    has converged).

Details & Tests:

  • RuntimeConfig::prefill_last_token asserts max_generated_tokens == 0,
    !use_continuous_batching, and no MTP layers, advances qbatch.MutablePos
    in StreamAndUpdateEOSAfterPrefill, and forces token-batched prefill
    (PrefillTBatch) when a batch would otherwise route to PrefillQBatch
    (which always stops at max_prompt_size - 1).
  • configs_test: tests NumPrefillLayers() on Gemma 4 2B (15/35), E4B (24/42),
    unshared models, and non-trailing shared layers.
  • kv_cache_test: verifies tiled KVCache allocates only KV-owning layers and
    routes shared layers to owner tiles via layer_kv_head_offsets.
  • prefill_truncation_test: loads the 270M checkpoint, injects KV sharing
    (layers 12..17 -> 6..11), warms MatMulEnv, and verifies bit-identical KV
    cache (default and tiled layouts), first-token logits, greedy token, and
    prefill_last_token cache export (single-query and QBatch for 1- and 2-token
    prompts).

@google-cla

google-cla Bot commented Oct 15, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

What:
1. Truncate prefill after the last KV-owning layer (`ModelConfig::NumPrefillLayers()`),
   skipping trailing KV-shared layers unless `RuntimeConfig::prefill_all_layers`,
   embedding output (`output_EOS`), activation observers, or MTP decoding require
   running all layers.
2. Add `RuntimeConfig::prefill_last_token` so prefill-only callers (`max_generated_tokens = 0`)
   can populate the KV cache of the final prompt token without running a full-depth
   decode step.
3. Stop allocating tiles and `kv_head_ptrs` entries for KV-shared layers in
   tiled `KVCache` (`layer_kv_head_offsets` already aliases shared layers to
   their owner).

Why / Impact:
In KV-shared models such as Gemma 4 (2B shares layers 15..34 from 13/14; E4B
shares layers 24..41 from 22/23), trailing shared layers do not write K/V during
prefill and their prefill activations are never read again. Skipping them during
prefill and omitting their unused KV tiles yields on Gemma 4 2B (`gemma4-2b-q4_0-it`,
4 CPU threads):
* 3.2x faster prefill:
  - 128 tokens:  3.4 s -> 1.1 s
  - 512 tokens:  14.3 s -> 4.4 s
  - 2048 tokens: 58.0 s -> 19.0 s
* 57% less tiled KV cache memory on Gemma 4 2B (15 vs 35 layers allocated;
  43% less on E4B, 24 vs 42 layers).
* Bit-identical own-KV cache and next-token logits (once the MatMul autotuner
  has converged).

Details & Tests:
* `RuntimeConfig::prefill_last_token` asserts `max_generated_tokens == 0`,
  `!use_continuous_batching`, and no MTP layers, advances `qbatch.MutablePos`
  in `StreamAndUpdateEOSAfterPrefill`, and forces token-batched prefill
  (`PrefillTBatch`) when a batch would otherwise route to `PrefillQBatch`
  (which always stops at `max_prompt_size - 1`).
* `configs_test`: tests `NumPrefillLayers()` on Gemma 4 2B (15/35), E4B (24/42),
  unshared models, and non-trailing shared layers.
* `kv_cache_test`: verifies tiled `KVCache` allocates only KV-owning layers and
  routes shared layers to owner tiles via `layer_kv_head_offsets`.
* `prefill_truncation_test`: loads the 270M checkpoint, injects KV sharing
  (layers 12..17 -> 6..11), warms `MatMulEnv`, and verifies bit-identical KV
  cache (default and tiled layouts), first-token logits, greedy token, and
  `prefill_last_token` cache export (single-query and QBatch for 1- and 2-token
  prompts).

PiperOrigin-RevId: 997920331
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants