diff --git a/CHANGELOG.md b/CHANGELOG.md index dab0d0d4..f06f8c7c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -64,6 +64,20 @@ hashes are given where a change is easiest to understand by reading the commit. page in six. IRIX on an R5000 O2 panicked with "init died" for the same reason; it now boots to the desktop. +### CPU and JIT + +- **The jitv2 inline load/store guard takes a direct-address fast path for + KSEG0.** That segment is unmapped, so its physical address is arithmetic + (`virt_addr & 0x1FFF_FFFF`) and the nutlb probe was pure overhead on every + kernel load and store. One compare against a mirrored rule — + `(virt_addr >> 29) == core.jit_kseg0_rule`, recomputed from `Status` and + `Config` at the hooks those registers already have — now covers the address + shape, the kernel-only privilege check and KSEG0's cacheability; anything it + cannot prove takes the existing nutlb probe. Measured on `iris-bench`: + 1.7x on the scalar-load chase, 1.5x on copy, 1.6x on fill, 1.3x on the + suite overall and 1.6x on DMIPS, with DRAM latency unchanged. See + [docs/jit-inline-memory.md](docs/jit-inline-memory.md) §6a. + ### Physical address decoding - **The decode table is sparse and spans 40 bits.** `Physical` used to embed diff --git a/FEATURES.md b/FEATURES.md index d4518a65..fbfef252 100644 --- a/FEATURES.md +++ b/FEATURES.md @@ -485,6 +485,7 @@ and offline-tool controls are included so they are not mistaken for GUI settings | `IRIS_NO_FETCH_VERIFY` | 1 suppresses fetchverify checks in a fetchverify build. | [src/cpu/mips_exec.rs](src/cpu/mips_exec.rs) | | `IRIS_NO_IDLE` | Any presence disables idle parking. | [src/cpu/idle_park.rs](src/cpu/idle_park.rs) | | `IRIS_NO_INLINE_MEM` | Any presence disables JIT inline memory emission. | [src/cpu/jitv2/codegen.rs](src/cpu/jitv2/codegen.rs) | +| `IRIS_NO_KSEG0_FAST` | Any presence disables the JIT's KSEG0 direct-address fast path (see [docs/jit-inline-memory.md](docs/jit-inline-memory.md)). Affects regions compiled afterwards; `j2 flush` applies it. | [src/cpu/jitv2/codegen.rs](src/cpu/jitv2/codegen.rs) | | `IRIS_NO_JIT` | Any presence disables the REX3 shader JIT and the IMPACT raster JIT; does not select the MIPS engine. | [src/dev/ng1/rex3.rs](src/dev/ng1/rex3.rs) | | `IRIS_OPT_SPEED` | Presence selects speed optimization in offline PCP/corpus measurements. | [src/bin/jitv2_pcp_dump.rs](src/bin/jitv2_pcp_dump.rs) | | `IRIS_PRINT_OFFSETS` | Presence runs the opt-in layout-offset print test. | [src/cpu/jitv2/mod.rs](src/cpu/jitv2/mod.rs) | diff --git a/docs/jit-inline-memory.md b/docs/jit-inline-memory.md index aadbc742..dc7ef3a4 100644 --- a/docs/jit-inline-memory.md +++ b/docs/jit-inline-memory.md @@ -184,6 +184,87 @@ R4400 vs R5000), not hardcode R4400's. 5. Re-measure on live IRIX; only then consider R5000 or inline fills. +--- + +## 6a. The KSEG0 direct-address fast path + +Status: **shipped, on by default.** `IRIS_NO_KSEG0_FAST=1` forces it off at +emit time; a `j2 flush` is what makes a change take effect. + +KSEG0 is *unmapped*: `translate_32bit_impl` and `translate_64bit_impl` compute +its physical address as `virt_addr & 0x1FFF_FFFF`, with no TLB entry, no ASID +and no generation to consult. Probing the nutlb for it was therefore pure +overhead — an index computation, two dependent loads and the validity-bitmask +load — on every kernel load and store. + +The guard now tests one thing first: + +``` +if (virt_addr >> 29) == core.jit_kseg0_rule -> phys = virt_addr & 0x1FFF_FFFF +else -> the nutlb probe, unchanged +``` + +`0xFFFF_FFFF_8800_0000 >> 29` is `0x7_FFFF_FFFC`: bits 63:32 all ones and bits +31:29 = `0b100`. One compare covers both halves of the address test in 64-bit +mode, which is exactly `translate_64bit_impl`'s xkseg compatibility arm — the +only route to this translation there. In 32-bit mode the translator ignores +bits 63:32, so the same compare accepts a *conservative subset* of what it +translates (it demands the sign-extended form, which is what `lui`/`la` +produce); anything else falls through to the nutlb probe. + +`jit_kseg0_rule` is derived state: compiled code reads it, and only Rust +writes it. `MipsExecutor::refresh_kseg0_direct_rule` recomputes it from `Status` and +`Config` at the two hooks those registers already have — `resync_privilege_state` +(every Status change, by that function's own documented invariant) and +`handle_cp0_side_effects` (every `MTC0 Config`) — and `kseg0_direct_rule` is the +executable specification: kernel mode only (EXL/ERL force it, else KSU == 0), +and `Config.K0` must select a cached algorithm in the *model's* table, so an +uncached KSEG0 still goes to the bus. When either fails the rule is `1 << 63`, +which `virt_addr >> 29` cannot produce (the shift result is at most 35 bits), +so every access in that state takes the nutlb probe exactly as before. + +Everything downstream of `phys` — L1-D tag, dirty bit, LRU, tcache mapped test, +swizzle, store generation bump — is the same code on both arms, so cache +behaviour, VCE, aliases and the cache model are untouched. Only *how* `phys` +was obtained differs. + +**Why a mirrored field and not a computed predicate.** The predicate evaluated +per access (two CP0 loads, a dozen ALU ops) measured *worse than the nutlb probe +it replaced*: an access in the tcache path is only a handful of host +instructions, so a dozen more is a net loss. Hoisting the same computation to +region entry was worse still — `iris-bench` enters regions often enough that a +dozen instructions per entry costs more than it saves. One memory operand +compare is the shape that wins. + +### Measured + +`iris-bench run --cpu r4400`, in-process, `lightning,jitv2`, median of three +interleaved runs per binary; ns per work unit, lower is better. + +| kernel | before | after | ratio | +|---|---:|---:|---:| +| `mem/latency_l1` (scalar-load chase, 8 KB) | 8.81 | 5.28 | **1.67x** | +| `mem/copy` (load+store stream) | 1.44 | 0.99 | **1.45x** | +| `mem/fill` (store stream) | 1.58 | 0.98 | **1.61x** | +| `mem/stream_copy` | 0.77 | 0.58 | **1.32x** | +| `mem/random` (random-page scalar loads) | 100.7 | 51.3 | **1.96x** | +| `mem/latency_dram` (chase) | 88.9 | 81.2 | **1.09x** | +| `int/dhrystone` (string/record traffic) | 1088 | 662 | **1.64x** | +| whole suite | 473 MIPS | 609 MIPS | **1.29x** | +| whole suite, DMIPS | 526 | 859 | **1.63x** | + +With `--cache full` (where the L2 model, not the guard, is the dominant cost) +the same build gives 1.02x on the suite, 1.70x on `mem/latency_l1`, 1.22x on +`mem/random` and 1.06x on `mem/copy`. + +`sys/tlb_hit` (one word per 4 KB page) is 3-10% *slower*: nearly every access +there is an L1-D tag miss, so the guard's compare is pure overhead before a +callout that was going to happen anyway. `sys/uncached` pays a similar ~10%, +for the same reason — a KSEG1 access declines past the compare. That is the +trade the study predicted: the callout is the better path for miss-heavy work, +and it is untouched. Individual non-memory kernels move by up to ~10% either +way from host load and code layout. + --- ## 7. Attempt 1: why the obvious emit is 9x SLOWER diff --git a/src/cpu/jitv2/codegen.rs b/src/cpu/jitv2/codegen.rs index 89f8ba05..f32b2491 100644 --- a/src/cpu/jitv2/codegen.rs +++ b/src/cpu/jitv2/codegen.rs @@ -3812,6 +3812,7 @@ pub static INLINE_MEM_DECLINED: std::sync::atomic::AtomicU64 = std::sync::atomic /// Offsets of the inline load/store fast-path pointers in `MipsCore`. fn core_offset_of_jit_dc_tags() -> i32 { std::mem::offset_of!(MipsCore, jit_dc_tags) as i32 } fn core_offset_of_jit_dc_lru() -> i32 { std::mem::offset_of!(MipsCore, jit_dc_lru) as i32 } +fn core_offset_of_jit_kseg0_rule() -> i32 { std::mem::offset_of!(MipsCore, jit_kseg0_rule) as i32 } fn core_offset_of_jit_tc_base() -> i32 { std::mem::offset_of!(MipsCore, jit_tc_base) as i32 } fn core_offset_of_jit_tc_gen() -> i32 { std::mem::offset_of!(MipsCore, jit_tc_gen) as i32 } fn core_offset_of_jit_l2_tags() -> i32 { std::mem::offset_of!(MipsCore, jit_l2_tags) as i32 } @@ -3844,10 +3845,26 @@ struct InlineMemPath { join_block: ir::Block, } +/// KSEG0's physical-address mask: `translate_*bit_impl` computes a KSEG0 +/// access's physical address as `(va as u32) & 0x1FFF_FFFF`. +const KSEG0_PHYS_MASK: i64 = 0x1FFF_FFFF; + +/// Whether the KSEG0 direct-address fast path is emitted. **On by default.** +/// +/// Read at *emit* time only, like [`inline_mem_enabled`]: flipping it affects +/// regions compiled afterwards, not ones already published, so `j2 flush` is +/// what makes a change take effect. `IRIS_NO_KSEG0_FAST=1` forces it off at +/// startup, for A/B runs on a binary that has no monitor. +fn kseg0_fast_enabled() -> bool { + static ON: std::sync::OnceLock = std::sync::OnceLock::new(); + *ON.get_or_init(|| std::env::var_os("IRIS_NO_KSEG0_FAST").is_none()) +} + /// Emit the guard chain shared by inline loads and stores. /// /// Sequence (docs/jit-inline-memory.md §2): -/// 1. nutlb probe: VA -> phys, else slow +/// 0. KSEG0 direct-address fast path: `phys` is arithmetic, no entry to read +/// 1. otherwise nutlb probe: VA -> phys, else slow /// 2. L1D tag compare on phys, else slow /// 3. stores: line already dirty, else slow (the clean->dirty tag RMW stays /// in Rust) @@ -3877,6 +3894,10 @@ fn emit_inline_mem_guard( let slow_block = ctx.builder.create_block(); let join_block = ctx.builder.create_block(); let fast_block = ctx.builder.create_block(); + // Both translation arms deliver `phys` here; the cache-model half of the + // guard is written once and cannot tell which arm produced the address. + let phys_block = ctx.builder.create_block(); + let phys_param = ctx.builder.append_block_param(phys_block, i64t); // ---- alignment --------------------------------------------------- // Misaligned accesses raise AdEL/AdES; that whole path stays in Rust. @@ -3891,6 +3912,47 @@ fn emit_inline_mem_guard( ctx.builder.seal_block(aligned_block); } + // ---- 0. KSEG0 direct-address fast path --------------------------- + // + // KSEG0 is *unmapped*: `translate_32bit_impl` and `translate_64bit_impl` + // compute its physical address as `(va & 0x1FFF_FFFF)` with no TLB entry, + // no ASID and no generation to consult. Probing the nutlb for it is + // therefore pure overhead — an index computation, two dependent loads and + // the validity-bitmask load — on every kernel load and store. Take the + // arithmetic translation directly when the conditions the translator would + // have checked all hold; anything else falls into the nutlb probe below, + // which is unchanged. + // + // `phys_block` is joined by both arms, so the cache-model half of the guard + // (tag check, dirty bit, LRU, tcache mapped test, swizzle, store + // generation bump) is identical on either path. Only how `phys` was + // obtained differs. + let direct_block = ctx.builder.create_block(); + let xlat_block = ctx.builder.create_block(); + if kseg0_fast_enabled() { + // `(va >> 29) == core.jit_kseg0_rule`, which is `KSEG0_RULE_HIT` only + // when this access is a cacheable KSEG0 translation in the current + // mode. See `MipsCore::jit_kseg0_rule` and `kseg0_direct_rule` for the + // rule itself, why it is mirrored rather than computed here, and why + // one compare covers both halves of the address test. + let rule = ctx.builder.ins().load(i64t, mem, ctx.core_ptr, + ir::immediates::Offset32::new(core_offset_of_jit_kseg0_rule())); + let shifted = ctx.builder.ins().ushr_imm_s(vaddr, 29); + let hit = ctx.builder.ins().icmp(IntCC::Equal, shifted, rule); + ctx.builder.ins().brif(hit, direct_block, &[], xlat_block, &[]); + } else { + ctx.builder.ins().jump(xlat_block, &[]); + } + + ctx.builder.switch_to_block(direct_block); + ctx.builder.seal_block(direct_block); + + let direct_phys = ctx.builder.ins().band_imm_s(vaddr, KSEG0_PHYS_MASK); + ctx.builder.ins().jump(phys_block, &[direct_phys.into()]); + + ctx.builder.switch_to_block(xlat_block); + ctx.builder.seal_block(xlat_block); + // ---- 1. nutlb probe ---------------------------------------------- // Mirrors MipsExecutor::nutlb_translate exactly (mips_exec.rs). Any // divergence here is a correctness bug, not a performance one. @@ -3957,7 +4019,13 @@ fn emit_inline_mem_guard( // phys = (e.phys & !0xFFF) | (va & 0xFFF) let phys_page = ctx.builder.ins().band_imm_s(e_phys, !0xFFFi64); let va_off = ctx.builder.ins().band_imm_s(vaddr, 0xFFF); - let phys = ctx.builder.ins().bor(phys_page, va_off); + let xlat_phys = ctx.builder.ins().bor(phys_page, va_off); + ctx.builder.ins().jump(phys_block, &[xlat_phys.into()]); + + // ---- both arms join here with `phys` ----------------------------- + ctx.builder.switch_to_block(phys_block); + ctx.builder.seal_block(phys_block); + let phys = phys_param; if geom.tagless { return Some(emit_tagless_tail(ctx, phys, size, fast_block, slow_block, join_block)); diff --git a/src/cpu/mips_core.rs b/src/cpu/mips_core.rs index afa7f611..bf9e20d0 100644 --- a/src/cpu/mips_core.rs +++ b/src/cpu/mips_core.rs @@ -1001,6 +1001,77 @@ pub struct MipsCore { /// is ever in flight. #[cfg(feature = "jitv2")] pub jit_read_scratch: u64, + /// KSEG0 direct-address rule for the JIT's inline load/store guard: + /// `(virt_addr >> 29) == jit_kseg0_rule` is true exactly for an access the + /// translator would serve from `va & 0x1FFF_FFFF` **and** cache. + /// + /// **Derived state, never written by compiled code.** It is recomputed by + /// `MipsExecutor::refresh_kseg0_direct_rule` from `cp0_status` and + /// `cp0_config` at the two points those can change — see + /// `mips_core::kseg0_direct_rule` for the predicate and + /// `MipsExecutor::resync_privilege_state` (every Status change, by that + /// function's own documented invariant) and + /// `MipsExecutor::handle_cp0_side_effects` (every Config write). + /// + /// Read by compiled code at a fixed `offset_of!` offset, so it stays a + /// plain field rather than moving into `hot`. It is deliberately adjacent + /// to the other jitv2 scratch fields: every inline load and store reads it, + /// and a cache line that is not already hot would cost more than the probe + /// it replaces. + /// + /// `KSEG0_RULE_NEVER` (not zero) is the "decline" value: `va >> 29` can be + /// zero, so a zeroed field would accept a KUSEG address. `MipsCore::new` + /// initialises it, and it is refreshed before any compiled code can run. + pub jit_kseg0_rule: u64, +} + +/// The value of `virt_addr >> 29` for a sign-extended KSEG0 address — +/// `0xFFFF_FFFF_8800_0000 >> 29`. One compare against it tests both halves of +/// the address test in 64-bit mode; see [`kseg0_direct_rule`]. +pub const KSEG0_RULE_HIT: u64 = 0x7_FFFF_FFFC; + +/// The rule's "decline" value: `virt_addr >> 29` is at most 35 bits wide, so a +/// value with bit 63 set can never match. Used instead of `0`, which a KUSEG +/// address can produce. +pub const KSEG0_RULE_NEVER: u64 = 1 << 63; + +/// The KSEG0 direct-address rule implied by a CP0 state, or +/// [`KSEG0_RULE_NEVER`] when no KSEG0 access may take the inline path. +/// +/// This is the executable specification of the JIT's KSEG0 fast path; the +/// emitted test is a single `(virt_addr >> 29) == rule` compare against the +/// value returned here. It answers the same three questions +/// `translate_32bit_impl`'s `4 =>` arm and `translate_64bit_impl`'s xkseg +/// compatibility arm answer, plus the cacheability check the inline guard owes +/// on top: +/// +/// 1. **Privilege** — `get_privilege_mode() == Kernel`. KSEG0 is kernel-only; +/// user and supervisor accesses raise AdEL/AdES instead of translating. +/// EXL or ERL force kernel; otherwise KSU must be 0. +/// 2. **Address shape** — bits 63:32 all ones (the sign-extended form) and +/// bits 31:29 = 0b100. In 64-bit mode that is exactly the xkseg +/// compatibility arm; a zero-extended `0x0000_0000_8xxx_xxxx` is *xuseg* +/// there and TLB-mapped, so it must not match. In 32-bit mode the +/// translator ignores bits 63:32, so this is a conservative subset: an +/// address that fails it falls through to the nutlb probe, which handles +/// every 32-bit form. +/// 3. **Cacheability** — `Config.K0` selects a cached algorithm, per +/// `decode_cache_attr`'s per-model table. `k0_cached_mask` bit `c` is set +/// when `Config.K0 == c` is cached on this model; see +/// [`crate::cpu::mips_exec::k0_cached_mask_for`]. +pub fn kseg0_direct_rule(status: u32, config: u32, k0_cached_mask: u8) -> u64 { + // `get_privilege_mode`: EXL/ERL force kernel, otherwise KSU == 0 is + // kernel. KSU == 3 is reserved — `get_privilege_mode` calls it kernel, this + // calls it not-kernel, and declining is the safe direction: the nutlb probe + // still serves it. + let kernel = (status & (STATUS_EXL | STATUS_ERL)) != 0 + || (status & STATUS_KSU_MASK) == 0; + let k0_cached = (k0_cached_mask >> (config & 0x7)) & 1 != 0; + if kernel && k0_cached { + KSEG0_RULE_HIT + } else { + KSEG0_RULE_NEVER + } } /// A completed real memory access, captured for `jitv2_lockstep`'s @@ -1414,6 +1485,9 @@ impl MipsCore { gpr_scratch: 0, #[cfg(feature = "jitv2")] jit_read_scratch: 0, + // Declines the KSEG0 fast path until `MipsExecutor::new` derives the + // real value. `KSEG0_RULE_NEVER`, not 0: `va >> 29` can be zero. + jit_kseg0_rule: KSEG0_RULE_NEVER, }; core.reset_registers(false); core diff --git a/src/cpu/mips_exec.rs b/src/cpu/mips_exec.rs index a8a593a9..71d565d6 100644 --- a/src/cpu/mips_exec.rs +++ b/src/cpu/mips_exec.rs @@ -1061,6 +1061,59 @@ pub const TR_UNCACHED: u32 = 2; // C=2: Uncached pub const TR_CACHEABLE: u32 = 3; // C=3: Cacheable (write-back) pub const TR_CACHEABLE_COH: u32 = 5; // C=5: Cacheable Coherent Exclusive +/// Decode a 3-bit cache-coherency attribute with the model's tables passed as +/// data rather than as a const generic. [`MipsExecutor::decode_cache_attr`] is +/// the same function with `C` in scope; the split exists so +/// [`k0_cached_mask_for`] can build its membership mask from *this* table +/// instead of a second copy of it. +/// +/// `mips4` is `C::MIPS4`, `r10k` is `C::R10K_CACHE_OPS`; the table itself is +/// documented on `decode_cache_attr`. +pub(crate) const fn decode_cache_attr_for(c: u32, mips4: bool, r10k: bool) -> u32 { + if r10k { + // R10000 UM Table 4-1: MIPS IV like the R5000, but its own table. + // 4 and 5 are the coherent algorithms, not reserved: the O2's IRIX + // runs KSEG0 with K0=5, and reading that as uncached sent every + // kernel fetch, load and store to the bus. 7 (uncached accelerated) + // only gathers stores; a load is a plain uncached one. + match c & 0x7 { + 3 => TR_CACHEABLE, + 4 | 5 => TR_CACHEABLE_COH, + _ => TR_UNCACHED, // 2 and 7; 0, 1, 6 reserved + } + } else if mips4 { + // VR5000 Table 6-6. + match c & 0x7 { + 0 | 1 => TR_CACHEABLE, // write-through -> write-back (see above) + 3 => TR_CACHEABLE, + _ => TR_UNCACHED, // 2 = Uncached; 4,5,6,7 reserved + } + } else { + // R4400 Table 4-6. + match c & 0x7 { + 3 => TR_CACHEABLE, + 4 | 5 | 6 => TR_CACHEABLE_COH, // coherent variants + _ => TR_UNCACHED, // 2 = Uncached; 0,1,7 reserved + } + } +} + +/// `Config.K0` values that select a cached algorithm for a model with these +/// capability flags, one bit per 3-bit code. The JIT's KSEG0 direct-address +/// fast path consults it to decide whether a KSEG0 access may take the inline +/// path at all, exactly as the nutlb probe's cacheable test does. +pub(crate) const fn k0_cached_mask_for(mips4: bool, r10k: bool) -> u8 { + let mut mask = 0u8; + let mut c = 0u32; + while c < 8 { + if decode_cache_attr_for(c, mips4, r10k) != TR_UNCACHED { + mask |= 1u8 << c; + } + c += 1; + } + mask +} + /// **Contract:** these two flags are set only on the `translate_fn` slow /// path, and are read only by `nutlb_fill`. A nutlb *hit* returns them clear /// regardless of the underlying page, because a hit never fills and nothing @@ -2958,6 +3011,10 @@ impl MipsExecutor { { executor.core.fetch_verify_on = fetch_verify_env_default(); } executor.update_translate_fn(); executor.update_fpr_mode(); + // K0 is uncached at reset (see `config` above), so this starts at + // `KSEG0_RULE_NEVER` — but derive it rather than assume, so the rule can + // never disagree with the state the rest of this constructor just set. + executor.refresh_kseg0_direct_rule(); executor } @@ -3680,6 +3737,32 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", } } + /// Re-derive `core.jit_kseg0_rule` from the live `Status` and `Config`. + /// + /// The rule is the JIT inline load/store guard's KSEG0 fast path — see + /// [`crate::cpu::mips_core::kseg0_direct_rule`]. It is derived state, so + /// the only thing that matters is that every writer of the two registers it + /// reads runs this afterwards: + /// + /// * **Status** — every change goes through [`Self::resync_privilege_state`] + /// (that function's own doc comment states the invariant), which is why + /// the call sits there. That covers `MTC0 Status`, all exception-delivery + /// paths, `ERET`, soft reset, snapshot/digest restore and gdb. + /// * **Config** — `MTC0 Config` is the only writer (`MipsCore::write_cp0` + /// reg 16), and every one of them is followed by + /// [`Self::handle_cp0_side_effects`], which calls this. + /// + /// A missed refresh would be a correctness bug, not a slowdown, so both + /// call sites are on paths that already exist for the same reason (the + /// nutlb's privilege flush and its K0 flush respectively) rather than being + /// new hooks. `MipsCore::new` starts at `KSEG0_RULE_NEVER`, so anything + /// that fails to refresh declines the fast path instead of misusing it. + #[inline] + pub fn refresh_kseg0_direct_rule(&mut self) { + self.core.jit_kseg0_rule = crate::cpu::mips_core::kseg0_direct_rule( + self.core.cp0_status, self.core.cp0_config, Self::k0_cached_mask()); + } + /// Re-derive every piece of executor state cached off CP0 Status: `translate_fn` /// (privilege x 32/64-bit), the six `fpr_*` accessors (STATUS_FR), and the untagged /// translation caches that carry no permission bits of their own (nanotlb + nutlb + @@ -3704,6 +3787,7 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", pub fn resync_privilege_state(&mut self) { self.update_translate_fn(); self.update_fpr_mode(); + self.refresh_kseg0_direct_rule(); self.nanotlb_invalidate(); #[cfg(feature = "cache-ecc")] self.sync_cache_error_control(); @@ -4719,6 +4803,14 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", self.core.is_64bit_mode() && (virt_addr >> 62) != 2 } + /// `Config.K0` values that select a cached algorithm on this model, one bit + /// per 3-bit code — `decode_cache_attr`'s table as membership, for + /// `kseg0_direct_rule`. + #[inline] + pub(crate) const fn k0_cached_mask() -> u8 { + k0_cached_mask_for(C::MIPS4, C::R10K_CACHE_OPS) + } + /// Decode a 3-bit cache-coherency attribute (EntryLo's C field, or /// `Config.K0`, which uses the same encoding) into a `TR_*` value. /// @@ -4766,32 +4858,7 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", /// (`R10000 UM` Table 4-1), selected by `C::R10K_CACHE_OPS`. #[inline] pub(crate) fn decode_cache_attr(c: u32) -> u32 { - if C::R10K_CACHE_OPS { - // R10000 UM Table 4-1: MIPS IV like the R5000, but its own table. - // 4 and 5 are the coherent algorithms, not reserved: the O2's - // IRIX runs KSEG0 with K0=5, and reading that as uncached sent - // every kernel fetch, load and store to the bus. 7 (uncached - // accelerated) only gathers stores; a load is a plain uncached one. - match c & 0x7 { - 3 => TR_CACHEABLE, - 4 | 5 => TR_CACHEABLE_COH, - _ => TR_UNCACHED, // 2 and 7; 0, 1, 6 reserved - } - } else if C::MIPS4 { - // VR5000 Table 6-6. - match c & 0x7 { - 0 | 1 => TR_CACHEABLE, // write-through -> write-back (see above) - 3 => TR_CACHEABLE, - _ => TR_UNCACHED, // 2 = Uncached; 4,5,6,7 reserved - } - } else { - // R4400 Table 4-6. - match c & 0x7 { - 3 => TR_CACHEABLE, - 4 | 5 | 6 => TR_CACHEABLE_COH, // coherent variants - _ => TR_UNCACHED, // 2 = Uncached; 0,1,7 reserved - } - } + decode_cache_attr_for(c, C::MIPS4, C::R10K_CACHE_OPS) } /// KSEG0's cache attribute, from `Config.K0` (bits 2:0) rather than a @@ -7424,6 +7491,12 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", self.sysad.write32(crate::dev::testdev::TEST_DEV_BASE + crate::dev::testdev::REG_DUMP, self.core.cp0_ecc); } + // Every `MTC0/DMTC0` runs this hook, and Config (16) is the only writer + // of the K0 field the JIT's KSEG0 fast path reads — the same argument + // the K0 nutlb flush above rests on. Unconditional for the same reason: + // the events are a handful per boot, and a change gate would be another + // field to keep in sync across every path that writes `cp0_config`. + self.refresh_kseg0_direct_rule(); #[cfg(feature = "r5ksc_triton")] if reg == 16 { // CONFIG_SE (bit 12): wire L2 enable/disable to cache. diff --git a/src/cpu/mips_exec_test.rs b/src/cpu/mips_exec_test.rs index a40df28e..9291fab0 100644 --- a/src/cpu/mips_exec_test.rs +++ b/src/cpu/mips_exec_test.rs @@ -2,7 +2,10 @@ mod tests { use std::sync::Mutex; use std::collections::HashMap; - use crate::cpu::mips_core::{STATUS_KX, STATUS_CU1, STATUS_FR}; + use crate::cpu::mips_core::{ + STATUS_KX, STATUS_CU1, STATUS_FR, STATUS_EXL, STATUS_ERL, STATUS_KSU_SHIFT, + KSU_USER, KSU_SUPERVISOR, + }; use crate::cpu::mips_exec::{MipsExecutor, MipsCpuConfig, DecodedInstr, EXEC_COMPLETE, EXEC_BREAKPOINT, EXEC_IS_EXCEPTION, EXEC_IS_TLB_REFILL, exec_exception, EXC_SYS, EXC_BP, EXC_TR, EXC_OV, EXC_RI, EXC_ADEL}; use crate::cpu::mips_isa::*; use crate::cpu::mips_tlb::PassthroughTlb; @@ -6206,4 +6209,176 @@ mod tests { parked through an interrupt masked only by the spl"), } } + // ── KSEG0 direct-address rule (jitv2 inline load/store fast path) ────── + + /// The rule's hit constant must be exactly `KSEG0_VA >> 29` for any + /// sign-extended KSEG0 address. Pinned as a test because the constant is + /// easy to write one `F` short, which makes the fast path *never* fire — + /// a silent, total performance loss with no functional symptom. + #[test] + fn kseg0_rule_hit_constant_is_the_shifted_kseg0_base() { + use crate::cpu::mips_core::{KSEG0_RULE_HIT, KSEG0_RULE_NEVER}; + assert_eq!(KSEG0_RULE_HIT, 0xFFFF_FFFF_8800_0000u64 >> 29); + // Every KSEG0 address in the segment shifts to the same value... + for off in [0u64, 4, 0x1000, 0x1FFF_FFFF] { + assert_eq!(0xFFFF_FFFF_8000_0000u64.wrapping_add(off) >> 29, KSEG0_RULE_HIT); + } + for off in [0u64, 4, 0x1000, 0x17FF_FFFF] { + assert_eq!(0xFFFF_FFFF_8800_0000u64.wrapping_add(off) >> 29, KSEG0_RULE_HIT); + } + // ...and KSEG1, one segment up, must not. + assert_ne!(0xFFFF_FFFF_A000_0000u64 >> 29, KSEG0_RULE_HIT); + // ...and the "decline" value is unreachable, which is what lets a + // disallowed region keep the same emitted compare. + assert!(KSEG0_RULE_NEVER > 0x7_FFFF_FFFF, "never-value must not be a reachable shift result"); + } + + /// The rule must follow every writer of the two registers it reads: + /// Status through `resync_privilege_state` (the documented funnel for every + /// Status change), Config through `handle_cp0_side_effects`. A missed + /// refresh would let a user-mode access take the KSEG0 direct path, so this + /// asserts the mirror rather than only the pure function. + #[test] + fn kseg0_direct_rule_mirror_tracks_status_and_config() { + use crate::cpu::mips_core::{KSEG0_RULE_HIT, KSEG0_RULE_NEVER}; + let (mut exec, mem) = create_executor_with_r4000cache(); + + // Kernel, KX, K0 = 3 (the R4400 constructor's cacheable default). + exec.set_cp0_status(STATUS_KX); + assert_eq!(exec.core.jit_kseg0_rule, KSEG0_RULE_HIT, "kernel + K0=3"); + + // Supervisor and user mode must both decline: KSEG0 is kernel-only. + exec.set_cp0_status(STATUS_KX | (KSU_SUPERVISOR << STATUS_KSU_SHIFT)); + assert_eq!(exec.core.jit_kseg0_rule, KSEG0_RULE_NEVER, "supervisor"); + exec.set_cp0_status(STATUS_KX | (KSU_USER << STATUS_KSU_SHIFT)); + assert_eq!(exec.core.jit_kseg0_rule, KSEG0_RULE_NEVER, "user"); + + // EXL and ERL force kernel mode even with KSU still naming the + // interrupted mode — the syscall and TLB-refill case. + exec.set_cp0_status(STATUS_KX | (KSU_USER << STATUS_KSU_SHIFT) | STATUS_EXL); + assert_eq!(exec.core.jit_kseg0_rule, KSEG0_RULE_HIT, "user + EXL"); + exec.set_cp0_status(STATUS_KX | (KSU_USER << STATUS_KSU_SHIFT) | STATUS_ERL); + assert_eq!(exec.core.jit_kseg0_rule, KSEG0_RULE_HIT, "user + ERL"); + + // KSU = 3 is reserved; `get_privilege_mode` says kernel, the rule + // declines. Conservative, so assert the decline and the callout that + // still serves it. + exec.set_cp0_status(STATUS_KX | (3 << STATUS_KSU_SHIFT)); + assert_eq!(exec.core.jit_kseg0_rule, KSEG0_RULE_NEVER, "reserved KSU 3"); + + // Config.K0 goes through the *real* instruction path — `MTC0 Config` + // must leave the mirror consistent, since `handle_cp0_side_effects` is + // the only hook it runs through. + exec.set_cp0_status(STATUS_KX); + + // K0 = 2 (uncached) must decline: an uncached KSEG0 access belongs on + // the bus, never in an L1-D line. + exec.core.write_gpr(8, 0xc08482); + mem.set_word(0x00000000, 0x4088_8000); // MTC0 t0($8), Config($16) + exec.core.pc = 0xFFFFFFFF_80000000u64; + assert_eq!(exec.step_int(), EXEC_COMPLETE); + assert_eq!(exec.core.cp0_config & 0x7, 2, "K0=2"); + assert_eq!(exec.core.jit_kseg0_rule, KSEG0_RULE_NEVER, "K0 uncached"); + + // K0 = 5 is a coherent cached algorithm on an R4400... + exec.core.write_gpr(8, 0xc08485); + exec.core.pc = 0xFFFFFFFF_80000000u64; + assert_eq!(exec.step_int(), EXEC_COMPLETE); + assert_eq!(exec.core.jit_kseg0_rule, KSEG0_RULE_HIT, "R4400 K0=5 coherent"); + } + + /// On an R5000 the same `Config.K0` codes decode differently (VR5000 Table + /// 6-6: 0/1/3 cached, 4/5/6 reserved). The rule must use the model's table, + /// not R4400's — hence `k0_cached_mask_for` taking the model's flags. + #[test] + fn kseg0_direct_rule_uses_the_models_k0_table() { + use crate::cpu::mips_core::{KSEG0_RULE_HIT, KSEG0_RULE_NEVER}; + use crate::cpu::mips_cache_v2::R5000Cache; + let mem = Arc::new(MockMemory::new()); + let bus: Arc = mem.clone(); + let mut exec: MipsExecutor = + MipsExecutor::new(bus, PassthroughTlb::default(), &MipsCpuConfig::indy()); + exec.set_cp0_status(STATUS_KX); + exec.core.write_gpr(8, 0xc08485); // K0 = 5: coherent on R4400, reserved here + mem.set_word(0x00000000, 0x4088_8000); // MTC0 t0($8), Config($16) + exec.core.pc = 0xFFFFFFFF_80000000u64; + assert_eq!(exec.step_int(), EXEC_COMPLETE); + assert_eq!(exec.core.jit_kseg0_rule, KSEG0_RULE_NEVER, "R5000 K0=5 is uncached"); + exec.core.write_gpr(8, 0xc08483); // K0 = 3: cached on every model + exec.core.pc = 0xFFFFFFFF_80000000u64; + assert_eq!(exec.step_int(), EXEC_COMPLETE); + assert_eq!(exec.core.jit_kseg0_rule, KSEG0_RULE_HIT, "R5000 K0=3"); + } + + /// The address half of the rule, checked against the translator itself: + /// in 64-bit kernel mode the emitted compare must accept exactly the + /// addresses `translate_impl` serves from KSEG0 out of cache, and nothing + /// else. `TR_MAPPED` distinguishes the TLB-mapped arms (which set it) from + /// the unmapped ones, and `is_cached()` excludes KSEG1 and the ERL identity + /// window. + #[test] + fn kseg0_rule_address_test_agrees_with_the_translator() { + use crate::cpu::mips_core::KSEG0_RULE_HIT; + let (mut exec, _mem) = create_executor_with_r4000cache(); + exec.set_cp0_status(STATUS_KX); // 64-bit kernel, K0=3 + + let mut cases: Vec = Vec::new(); + for a in [ + 0x0000_0000_0000_0000u64, 0x0000_0000_1000_0000, 0x0000_0000_7FFF_FFFF, + 0x0000_0000_8000_0000, // KSEG0, zero-extended -> xuseg in 64-bit mode + 0x0000_0000_8800_0000, 0x0000_0000_9FFF_FFFF, + 0x0000_0000_A000_0000, // KSEG1, zero-extended + 0xFFFF_FFFF_8000_0000, 0xFFFF_FFFF_8800_0000, 0xFFFF_FFFF_9FFF_FFFC, + 0xFFFF_FFFF_A000_0000, // KSEG1 sign-extended + 0xFFFF_FFFF_BFFF_FFFF, + 0xFFFF_FFFF_C000_0000, // KSSEG + 0xFFFF_FFFF_E000_0000, // KSEG3 + 0x8000_0000_0000_0000, // xkphys + 0x0000_00FF_FFFF_FFFF, // top of xuseg + ] { + cases.push(a); + cases.push(a + 4); + cases.push(a & !0xFFF); + cases.push(a | 0xFFF); + } + + for va in cases { + let got = (va >> 29) == KSEG0_RULE_HIT; + let want = match exec.debug_translate(va) { + r if r.is_exception() => false, + r => { + let unmapped = (r.status & crate::cpu::mips_exec::TR_MAPPED) == 0; + unmapped && r.is_cached() + && r.phys == (va as u32 & 0x1FFF_FFFF) + } + }; + assert_eq!(got, want, "va={va:#018x}: rule says {got}, translator says {want}"); + } + } + + /// In 32-bit mode `translate_32bit_impl` ignores bits 63:32, so the rule is + /// allowed to be *narrower* than the translator (it demands the + /// sign-extended form) but must never be wider: a rule hit has to be an + /// access the translator would have served identically. + #[test] + fn kseg0_rule_is_a_subset_in_32bit_mode() { + use crate::cpu::mips_core::KSEG0_RULE_HIT; + let (mut exec, _mem) = create_executor_with_r4000cache(); + exec.set_cp0_status(0); // 32-bit kernel, K0=3 + + for va in [ + 0x0000_0000_8000_0000u64, 0x0000_0000_8800_1234, + 0xFFFF_FFFF_8000_0000, 0xFFFF_FFFF_8800_1234, + 0xFFFF_FFFF_9FFF_FFFF, 0xFFFF_FFFF_A000_0000, + 0x0000_0000_0000_1000, 0x0000_0000_7FFF_FFFF, + ] { + if (va >> 29) == KSEG0_RULE_HIT { + let r = exec.debug_translate(va); + assert!(!r.is_exception(), "va={va:#018x}: rule hit on a faulting address"); + assert_eq!(r.phys, va as u32 & 0x1FFF_FFFF, "va={va:#018x}"); + assert!(r.is_cached(), "va={va:#018x}: rule hit on an uncached translation"); + } + } + } + }