Repository navigation
Stop prefill at the last KV-owning layer for KV-shared models. - #1060
Open
copybara-service[bot] wants to merge 1 commit into
Open
copybara-service[bot] wants to merge 1 commit into
copybara-service[bot] wants to merge 1 commit into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
What: 1. Truncate prefill after the last KV-owning layer (`ModelConfig::NumPrefillLayers()`), skipping trailing KV-shared layers unless `RuntimeConfig::prefill_all_layers`, embedding output (`output_EOS`), activation observers, or MTP decoding require running all layers. 2. Add `RuntimeConfig::prefill_last_token` so prefill-only callers (`max_generated_tokens = 0`) can populate the KV cache of the final prompt token without running a full-depth decode step. 3. Stop allocating tiles and `kv_head_ptrs` entries for KV-shared layers in tiled `KVCache` (`layer_kv_head_offsets` already aliases shared layers to their owner). Why / Impact: In KV-shared models such as Gemma 4 (2B shares layers 15..34 from 13/14; E4B shares layers 24..41 from 22/23), trailing shared layers do not write K/V during prefill and their prefill activations are never read again. Skipping them during prefill and omitting their unused KV tiles yields on Gemma 4 2B (`gemma4-2b-q4_0-it`, 4 CPU threads): * 3.2x faster prefill: - 128 tokens: 3.4 s -> 1.1 s - 512 tokens: 14.3 s -> 4.4 s - 2048 tokens: 58.0 s -> 19.0 s * 57% less tiled KV cache memory on Gemma 4 2B (15 vs 35 layers allocated; 43% less on E4B, 24 vs 42 layers). * Bit-identical own-KV cache and next-token logits (once the MatMul autotuner has converged). Details & Tests: * `RuntimeConfig::prefill_last_token` asserts `max_generated_tokens == 0`, `!use_continuous_batching`, and no MTP layers, advances `qbatch.MutablePos` in `StreamAndUpdateEOSAfterPrefill`, and forces token-batched prefill (`PrefillTBatch`) when a batch would otherwise route to `PrefillQBatch` (which always stops at `max_prompt_size - 1`). * `configs_test`: tests `NumPrefillLayers()` on Gemma 4 2B (15/35), E4B (24/42), unshared models, and non-trailing shared layers. * `kv_cache_test`: verifies tiled `KVCache` allocates only KV-owning layers and routes shared layers to owner tiles via `layer_kv_head_offsets`. * `prefill_truncation_test`: loads the 270M checkpoint, injects KV sharing (layers 12..17 -> 6..11), warms `MatMulEnv`, and verifies bit-identical KV cache (default and tiled layouts), first-token logits, greedy token, and `prefill_last_token` cache export (single-query and QBatch for 1- and 2-token prompts). PiperOrigin-RevId: 997920331
copybara-service
Bot
force-pushed
the
test_997920331
branch
from
October 15, 2026 12:51
719efe3 to
213ac39
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stop prefill at the last KV-owning layer for KV-shared models.
What:
ModelConfig::NumPrefillLayers()),skipping trailing KV-shared layers unless
RuntimeConfig::prefill_all_layers,embedding output (
output_EOS), activation observers, or MTP decoding requirerunning all layers.
RuntimeConfig::prefill_last_tokenso prefill-only callers (max_generated_tokens = 0)can populate the KV cache of the final prompt token without running a full-depth
decode step.
kv_head_ptrsentries for KV-shared layers intiled
KVCache(layer_kv_head_offsetsalready aliases shared layers totheir owner).
Why / Impact:
In KV-shared models such as Gemma 4 (2B shares layers 15..34 from 13/14; E4B
shares layers 24..41 from 22/23), trailing shared layers do not write K/V during
prefill and their prefill activations are never read again. Skipping them during
prefill and omitting their unused KV tiles yields on Gemma 4 2B (
gemma4-2b-q4_0-it,4 CPU threads):
43% less on E4B, 24 vs 42 layers).
has converged).
Details & Tests:
RuntimeConfig::prefill_last_tokenassertsmax_generated_tokens == 0,!use_continuous_batching, and no MTP layers, advancesqbatch.MutablePosin
StreamAndUpdateEOSAfterPrefill, and forces token-batched prefill(
PrefillTBatch) when a batch would otherwise route toPrefillQBatch(which always stops at
max_prompt_size - 1).configs_test: testsNumPrefillLayers()on Gemma 4 2B (15/35), E4B (24/42),unshared models, and non-trailing shared layers.
kv_cache_test: verifies tiledKVCacheallocates only KV-owning layers androutes shared layers to owner tiles via
layer_kv_head_offsets.prefill_truncation_test: loads the 270M checkpoint, injects KV sharing(layers 12..17 -> 6..11), warms
MatMulEnv, and verifies bit-identical KVcache (default and tiled layouts), first-token logits, greedy token, and
prefill_last_tokencache export (single-query and QBatch for 1- and 2-tokenprompts).