jitv2: KSEG0 direct-address fast path for the inline memory guard

CHANGELOG.md

@@ -64,6 +64,20 @@ hashes are given where a change is easiest to understand by reading the commit.
   page in six. IRIX on an R5000 O2 panicked with "init died" for the same
   reason; it now boots to the desktop.
 
+### CPU and JIT
+
+- **The jitv2 inline load/store guard takes a direct-address fast path for
+  KSEG0.** That segment is unmapped, so its physical address is arithmetic
+  (`virt_addr & 0x1FFF_FFFF`) and the nutlb probe was pure overhead on every
+  kernel load and store. One compare against a mirrored rule —
+  `(virt_addr >> 29) == core.jit_kseg0_rule`, recomputed from `Status` and
+  `Config` at the hooks those registers already have — now covers the address
+  shape, the kernel-only privilege check and KSEG0's cacheability; anything it
+  cannot prove takes the existing nutlb probe. Measured on `iris-bench`:
+  1.7x on the scalar-load chase, 1.5x on copy, 1.6x on fill, 1.3x on the
+  suite overall and 1.6x on DMIPS, with DRAM latency unchanged. See
+  [docs/jit-inline-memory.md](docs/jit-inline-memory.md) §6a.
+
 ### Physical address decoding
 
 - **The decode table is sparse and spans 40 bits.** `Physical` used to embed

FEATURES.md

@@ -485,6 +485,7 @@ and offline-tool controls are included so they are not mistaken for GUI settings
 | `IRIS_NO_FETCH_VERIFY` | 1 suppresses fetchverify checks in a fetchverify build. | [src/cpu/mips_exec.rs](src/cpu/mips_exec.rs) |
 | `IRIS_NO_IDLE` | Any presence disables idle parking. | [src/cpu/idle_park.rs](src/cpu/idle_park.rs) |
 | `IRIS_NO_INLINE_MEM` | Any presence disables JIT inline memory emission. | [src/cpu/jitv2/codegen.rs](src/cpu/jitv2/codegen.rs) |
+| `IRIS_NO_KSEG0_FAST` | Any presence disables the JIT's KSEG0 direct-address fast path (see [docs/jit-inline-memory.md](docs/jit-inline-memory.md)). Affects regions compiled afterwards; `j2 flush` applies it. | [src/cpu/jitv2/codegen.rs](src/cpu/jitv2/codegen.rs) |
 | `IRIS_NO_JIT` | Any presence disables the REX3 shader JIT and the IMPACT raster JIT; does not select the MIPS engine. | [src/dev/ng1/rex3.rs](src/dev/ng1/rex3.rs) |
 | `IRIS_OPT_SPEED` | Presence selects speed optimization in offline PCP/corpus measurements. | [src/bin/jitv2_pcp_dump.rs](src/bin/jitv2_pcp_dump.rs) |
 | `IRIS_PRINT_OFFSETS` | Presence runs the opt-in layout-offset print test. | [src/cpu/jitv2/mod.rs](src/cpu/jitv2/mod.rs) |

docs/jit-inline-memory.md

@@ -184,6 +184,87 @@ R4400 vs R5000), not hardcode R4400's.
 5. Re-measure on live IRIX; only then consider R5000 or inline fills.
 
 
+---
+
+## 6a. The KSEG0 direct-address fast path
+
+Status: **shipped, on by default.** `IRIS_NO_KSEG0_FAST=1` forces it off at
+emit time; a `j2 flush` is what makes a change take effect.
+
+KSEG0 is *unmapped*: `translate_32bit_impl` and `translate_64bit_impl` compute
+its physical address as `virt_addr & 0x1FFF_FFFF`, with no TLB entry, no ASID
+and no generation to consult. Probing the nutlb for it was therefore pure
+overhead — an index computation, two dependent loads and the validity-bitmask
+load — on every kernel load and store.
+
+The guard now tests one thing first:
+
+```
+if (virt_addr >> 29) == core.jit_kseg0_rule   ->  phys = virt_addr & 0x1FFF_FFFF
+else                                          ->  the nutlb probe, unchanged
+```
+
+`0xFFFF_FFFF_8800_0000 >> 29` is `0x7_FFFF_FFFC`: bits 63:32 all ones and bits
+31:29 = `0b100`. One compare covers both halves of the address test in 64-bit
+mode, which is exactly `translate_64bit_impl`'s xkseg compatibility arm — the
+only route to this translation there. In 32-bit mode the translator ignores
+bits 63:32, so the same compare accepts a *conservative subset* of what it
+translates (it demands the sign-extended form, which is what `lui`/`la`
+produce); anything else falls through to the nutlb probe.
+
+`jit_kseg0_rule` is derived state: compiled code reads it, and only Rust
+writes it. `MipsExecutor::refresh_kseg0_direct_rule` recomputes it from `Status` and
+`Config` at the two hooks those registers already have — `resync_privilege_state`
+(every Status change, by that function's own documented invariant) and
+`handle_cp0_side_effects` (every `MTC0 Config`) — and `kseg0_direct_rule` is the
+executable specification: kernel mode only (EXL/ERL force it, else KSU == 0),
+and `Config.K0` must select a cached algorithm in the *model's* table, so an
+uncached KSEG0 still goes to the bus. When either fails the rule is `1 << 63`,
+which `virt_addr >> 29` cannot produce (the shift result is at most 35 bits),
+so every access in that state takes the nutlb probe exactly as before.
+
+Everything downstream of `phys` — L1-D tag, dirty bit, LRU, tcache mapped test,
+swizzle, store generation bump — is the same code on both arms, so cache
+behaviour, VCE, aliases and the cache model are untouched. Only *how* `phys`
+was obtained differs.
+
+**Why a mirrored field and not a computed predicate.** The predicate evaluated
+per access (two CP0 loads, a dozen ALU ops) measured *worse than the nutlb probe
+it replaced*: an access in the tcache path is only a handful of host
+instructions, so a dozen more is a net loss. Hoisting the same computation to
+region entry was worse still — `iris-bench` enters regions often enough that a
+dozen instructions per entry costs more than it saves. One memory operand
+compare is the shape that wins.
+
+### Measured
+
+`iris-bench run --cpu r4400`, in-process, `lightning,jitv2`, median of three
+interleaved runs per binary; ns per work unit, lower is better.
+
+| kernel | before | after | ratio |
+|---|---:|---:|---:|
+| `mem/latency_l1` (scalar-load chase, 8 KB) | 8.81 | 5.28 | **1.67x** |
+| `mem/copy` (load+store stream) | 1.44 | 0.99 | **1.45x** |
+| `mem/fill` (store stream) | 1.58 | 0.98 | **1.61x** |
+| `mem/stream_copy` | 0.77 | 0.58 | **1.32x** |
+| `mem/random` (random-page scalar loads) | 100.7 | 51.3 | **1.96x** |
+| `mem/latency_dram` (chase) | 88.9 | 81.2 | **1.09x** |
+| `int/dhrystone` (string/record traffic) | 1088 | 662 | **1.64x** |
+| whole suite | 473 MIPS | 609 MIPS | **1.29x** |
+| whole suite, DMIPS | 526 | 859 | **1.63x** |
+
+With `--cache full` (where the L2 model, not the guard, is the dominant cost)
+the same build gives 1.02x on the suite, 1.70x on `mem/latency_l1`, 1.22x on
+`mem/random` and 1.06x on `mem/copy`.
+
+`sys/tlb_hit` (one word per 4 KB page) is 3-10% *slower*: nearly every access
+there is an L1-D tag miss, so the guard's compare is pure overhead before a
+callout that was going to happen anyway. `sys/uncached` pays a similar ~10%,
+for the same reason — a KSEG1 access declines past the compare. That is the
+trade the study predicted: the callout is the better path for miss-heavy work,
+and it is untouched. Individual non-memory kernels move by up to ~10% either
+way from host load and code layout.
+
 ---
 
 ## 7. Attempt 1: why the obvious emit is 9x SLOWER

src/cpu/jitv2/codegen.rs

@@ -3812,6 +3812,7 @@ pub static INLINE_MEM_DECLINED: std::sync::atomic::AtomicU64 = std::sync::atomic
 /// Offsets of the inline load/store fast-path pointers in `MipsCore`.
 fn core_offset_of_jit_dc_tags() -> i32 { std::mem::offset_of!(MipsCore, jit_dc_tags) as i32 }
 fn core_offset_of_jit_dc_lru() -> i32 { std::mem::offset_of!(MipsCore, jit_dc_lru) as i32 }
+fn core_offset_of_jit_kseg0_rule() -> i32 { std::mem::offset_of!(MipsCore, jit_kseg0_rule) as i32 }
 fn core_offset_of_jit_tc_base() -> i32 { std::mem::offset_of!(MipsCore, jit_tc_base) as i32 }
 fn core_offset_of_jit_tc_gen() -> i32 { std::mem::offset_of!(MipsCore, jit_tc_gen) as i32 }
 fn core_offset_of_jit_l2_tags() -> i32 { std::mem::offset_of!(MipsCore, jit_l2_tags) as i32 }
@@ -3844,10 +3845,26 @@ struct InlineMemPath {
     join_block: ir::Block,
 }
 
+/// KSEG0's physical-address mask: `translate_*bit_impl` computes a KSEG0
+/// access's physical address as `(va as u32) & 0x1FFF_FFFF`.
+const KSEG0_PHYS_MASK: i64 = 0x1FFF_FFFF;
+
+/// Whether the KSEG0 direct-address fast path is emitted. **On by default.**
+///
+/// Read at *emit* time only, like [`inline_mem_enabled`]: flipping it affects
+/// regions compiled afterwards, not ones already published, so `j2 flush` is
+/// what makes a change take effect. `IRIS_NO_KSEG0_FAST=1` forces it off at
+/// startup, for A/B runs on a binary that has no monitor.
+fn kseg0_fast_enabled() -> bool {
+    static ON: std::sync::OnceLock<bool> = std::sync::OnceLock::new();
+    *ON.get_or_init(|| std::env::var_os("IRIS_NO_KSEG0_FAST").is_none())
+}
+
 /// Emit the guard chain shared by inline loads and stores.
 ///
 /// Sequence (docs/jit-inline-memory.md §2):
-///   1. nutlb probe: VA -> phys, else slow
+///   0. KSEG0 direct-address fast path: `phys` is arithmetic, no entry to read
+///   1. otherwise nutlb probe: VA -> phys, else slow
 ///   2. L1D tag compare on phys, else slow
 ///   3. stores: line already dirty, else slow (the clean->dirty tag RMW stays
 ///      in Rust)
@@ -3877,6 +3894,10 @@ fn emit_inline_mem_guard<const STORE: bool>(
     let slow_block = ctx.builder.create_block();
     let join_block = ctx.builder.create_block();
     let fast_block = ctx.builder.create_block();
+    // Both translation arms deliver `phys` here; the cache-model half of the
+    // guard is written once and cannot tell which arm produced the address.
+    let phys_block = ctx.builder.create_block();
+    let phys_param = ctx.builder.append_block_param(phys_block, i64t);
 
     // ---- alignment ---------------------------------------------------
     // Misaligned accesses raise AdEL/AdES; that whole path stays in Rust.
@@ -3891,6 +3912,47 @@ fn emit_inline_mem_guard<const STORE: bool>(
         ctx.builder.seal_block(aligned_block);
     }
 
+    // ---- 0. KSEG0 direct-address fast path ---------------------------
+    //
+    // KSEG0 is *unmapped*: `translate_32bit_impl` and `translate_64bit_impl`
+    // compute its physical address as `(va & 0x1FFF_FFFF)` with no TLB entry,
+    // no ASID and no generation to consult. Probing the nutlb for it is
+    // therefore pure overhead — an index computation, two dependent loads and
+    // the validity-bitmask load — on every kernel load and store. Take the
+    // arithmetic translation directly when the conditions the translator would
+    // have checked all hold; anything else falls into the nutlb probe below,
+    // which is unchanged.
+    //
+    // `phys_block` is joined by both arms, so the cache-model half of the guard
+    // (tag check, dirty bit, LRU, tcache mapped test, swizzle, store
+    // generation bump) is identical on either path. Only how `phys` was
+    // obtained differs.
+    let direct_block = ctx.builder.create_block();
+    let xlat_block = ctx.builder.create_block();
+    if kseg0_fast_enabled() {
+        // `(va >> 29) == core.jit_kseg0_rule`, which is `KSEG0_RULE_HIT` only
+        // when this access is a cacheable KSEG0 translation in the current
+        // mode. See `MipsCore::jit_kseg0_rule` and `kseg0_direct_rule` for the
+        // rule itself, why it is mirrored rather than computed here, and why
+        // one compare covers both halves of the address test.
+        let rule = ctx.builder.ins().load(i64t, mem, ctx.core_ptr,
+            ir::immediates::Offset32::new(core_offset_of_jit_kseg0_rule()));
+        let shifted = ctx.builder.ins().ushr_imm_s(vaddr, 29);
+        let hit = ctx.builder.ins().icmp(IntCC::Equal, shifted, rule);
+        ctx.builder.ins().brif(hit, direct_block, &[], xlat_block, &[]);
+    } else {
+        ctx.builder.ins().jump(xlat_block, &[]);
+    }
+
+    ctx.builder.switch_to_block(direct_block);
+    ctx.builder.seal_block(direct_block);
+
+    let direct_phys = ctx.builder.ins().band_imm_s(vaddr, KSEG0_PHYS_MASK);
+    ctx.builder.ins().jump(phys_block, &[direct_phys.into()]);
+
+    ctx.builder.switch_to_block(xlat_block);
+    ctx.builder.seal_block(xlat_block);
+
     // ---- 1. nutlb probe ----------------------------------------------
     // Mirrors MipsExecutor::nutlb_translate exactly (mips_exec.rs). Any
     // divergence here is a correctness bug, not a performance one.
@@ -3957,7 +4019,13 @@ fn emit_inline_mem_guard<const STORE: bool>(
     // phys = (e.phys & !0xFFF) | (va & 0xFFF)
     let phys_page = ctx.builder.ins().band_imm_s(e_phys, !0xFFFi64);
     let va_off = ctx.builder.ins().band_imm_s(vaddr, 0xFFF);
-    let phys = ctx.builder.ins().bor(phys_page, va_off);
+    let xlat_phys = ctx.builder.ins().bor(phys_page, va_off);
+    ctx.builder.ins().jump(phys_block, &[xlat_phys.into()]);
+
+    // ---- both arms join here with `phys` -----------------------------
+    ctx.builder.switch_to_block(phys_block);
+    ctx.builder.seal_block(phys_block);
+    let phys = phys_param;
 
     if geom.tagless {
         return Some(emit_tagless_tail(ctx, phys, size, fast_block, slow_block, join_block));

src/cpu/mips_core.rs

@@ -1001,6 +1001,77 @@ pub struct MipsCore {
     /// is ever in flight.
     #[cfg(feature = "jitv2")]
     pub jit_read_scratch: u64,
+    /// KSEG0 direct-address rule for the JIT's inline load/store guard:
+    /// `(virt_addr >> 29) == jit_kseg0_rule` is true exactly for an access the
+    /// translator would serve from `va & 0x1FFF_FFFF` **and** cache.
+    ///
+    /// **Derived state, never written by compiled code.** It is recomputed by
+    /// `MipsExecutor::refresh_kseg0_direct_rule` from `cp0_status` and
+    /// `cp0_config` at the two points those can change — see
+    /// `mips_core::kseg0_direct_rule` for the predicate and
+    /// `MipsExecutor::resync_privilege_state` (every Status change, by that
+    /// function's own documented invariant) and
+    /// `MipsExecutor::handle_cp0_side_effects` (every Config write).
+    ///
+    /// Read by compiled code at a fixed `offset_of!` offset, so it stays a
+    /// plain field rather than moving into `hot`. It is deliberately adjacent
+    /// to the other jitv2 scratch fields: every inline load and store reads it,
+    /// and a cache line that is not already hot would cost more than the probe
+    /// it replaces.
+    ///
+    /// `KSEG0_RULE_NEVER` (not zero) is the "decline" value: `va >> 29` can be
+    /// zero, so a zeroed field would accept a KUSEG address. `MipsCore::new`
+    /// initialises it, and it is refreshed before any compiled code can run.
+    pub jit_kseg0_rule: u64,
+}
+
+/// The value of `virt_addr >> 29` for a sign-extended KSEG0 address —
+/// `0xFFFF_FFFF_8800_0000 >> 29`. One compare against it tests both halves of
+/// the address test in 64-bit mode; see [`kseg0_direct_rule`].
+pub const KSEG0_RULE_HIT: u64 = 0x7_FFFF_FFFC;
+
+/// The rule's "decline" value: `virt_addr >> 29` is at most 35 bits wide, so a
+/// value with bit 63 set can never match. Used instead of `0`, which a KUSEG
+/// address can produce.
+pub const KSEG0_RULE_NEVER: u64 = 1 << 63;
+
+/// The KSEG0 direct-address rule implied by a CP0 state, or
+/// [`KSEG0_RULE_NEVER`] when no KSEG0 access may take the inline path.
+///
+/// This is the executable specification of the JIT's KSEG0 fast path; the
+/// emitted test is a single `(virt_addr >> 29) == rule` compare against the
+/// value returned here. It answers the same three questions
+/// `translate_32bit_impl`'s `4 =>` arm and `translate_64bit_impl`'s xkseg
+/// compatibility arm answer, plus the cacheability check the inline guard owes
+/// on top:
+///
+/// 1. **Privilege** — `get_privilege_mode() == Kernel`. KSEG0 is kernel-only;
+///    user and supervisor accesses raise AdEL/AdES instead of translating.
+///    EXL or ERL force kernel; otherwise KSU must be 0.
+/// 2. **Address shape** — bits 63:32 all ones (the sign-extended form) and
+///    bits 31:29 = 0b100. In 64-bit mode that is exactly the xkseg
+///    compatibility arm; a zero-extended `0x0000_0000_8xxx_xxxx` is *xuseg*
+///    there and TLB-mapped, so it must not match. In 32-bit mode the
+///    translator ignores bits 63:32, so this is a conservative subset: an
+///    address that fails it falls through to the nutlb probe, which handles
+///    every 32-bit form.
+/// 3. **Cacheability** — `Config.K0` selects a cached algorithm, per
+///    `decode_cache_attr`'s per-model table. `k0_cached_mask` bit `c` is set
+///    when `Config.K0 == c` is cached on this model; see
+///    [`crate::cpu::mips_exec::k0_cached_mask_for`].
+pub fn kseg0_direct_rule(status: u32, config: u32, k0_cached_mask: u8) -> u64 {
+    // `get_privilege_mode`: EXL/ERL force kernel, otherwise KSU == 0 is
+    // kernel. KSU == 3 is reserved — `get_privilege_mode` calls it kernel, this
+    // calls it not-kernel, and declining is the safe direction: the nutlb probe
+    // still serves it.
+    let kernel = (status & (STATUS_EXL | STATUS_ERL)) != 0
+        || (status & STATUS_KSU_MASK) == 0;
+    let k0_cached = (k0_cached_mask >> (config & 0x7)) & 1 != 0;
+    if kernel && k0_cached {
+        KSEG0_RULE_HIT
+    } else {
+        KSEG0_RULE_NEVER
+    }
 }
 
 /// A completed real memory access, captured for `jitv2_lockstep`'s
@@ -1414,6 +1485,9 @@ impl MipsCore {
             gpr_scratch: 0,
             #[cfg(feature = "jitv2")]
             jit_read_scratch: 0,
+            // Declines the KSEG0 fast path until `MipsExecutor::new` derives the
+            // real value. `KSEG0_RULE_NEVER`, not 0: `va >> 29` can be zero.
+            jit_kseg0_rule: KSEG0_RULE_NEVER,
         };
         core.reset_registers(false);
         core