[belikh]

What this is

KSEG0 is unmapped: translate_32bit_impl and translate_64bit_impl compute its physical address as virt_addr & 0x1FFF_FFFF, with no TLB entry, ASID, generation or alias to consult. The jitv2 inline load/store guard still probed the nutlb for it on every kernel load and store — an index computation, two dependent loads and the validity-bitmask load — which a memory hot-path study measured as the bulk of a 15.8 ns L1 load.

The guard now tests one thing first:

(virt_addr >> 29) == core.jit_kseg0_rule   ->  phys = va & 0x1FFF_FFFF
else                                       ->  the existing nutlb probe, unchanged

0xFFFF_FFFF_8800_0000 >> 29 is 0x7_FFFF_FFFC — bits 63:32 all ones and bits 31:29 = 0b100 — so one compare covers both halves of the address test. Emitted on x86-64 that is movq 0x2520(%rbx),%rcx; shrq $0x1d,%rax; cmpq %rcx,%rax; jz.

Why it is exact

jit_kseg0_rule is derived state. Compiled code reads it through offset_of!; only Rust writes it, from the two registers it depends on, at the hooks those registers already own:

• Status — every change goes through resync_privilege_state, whose own doc comment states that invariant (MTC0 Status via status_changed_cb, all exception-delivery paths, ERET, soft reset, snapshot/digest restore, gdb). That function now refreshes the rule.
• Config — MTC0 Config is the only writer, and handle_cp0_side_effects is the single hook it runs through (the same hook the K0 nutlb flush already relies on). It now refreshes the rule.

kseg0_direct_rule is the executable specification, and it is a literal transcription of what the translator does plus the cacheability check the guard owes:

condition	rule	why
privilege	get_privilege_mode() == Kernel — EXL/ERL force it, else KSU == 0	KSEG0 is kernel-only; user/supervisor raise AdEL/AdES instead of translating
address shape	bits 63:32 all ones and bits 31:29 = 0b100	in 64-bit mode that is exactly translate_64bit_impl's xkseg compatibility arm; a zero-extended 0x0000_0000_8xxx_xxxx is xuseg there and TLB-mapped, so accepting it would be a wrong answer, not a slow one
cacheability	Config.K0 selects a cached algorithm in this model's table	an uncached KSEG0 access belongs on the bus and must never be served from (or into) an L1-D line

When either fails, the rule is 1 << 63, which virt_addr >> 29 cannot produce (the shift result is at most 35 bits), so every access in that state takes the nutlb probe exactly as before. In 32-bit mode the translator ignores bits 63:32, so the same compare accepts a conservative subset of what it translates — it demands the sign-extended form, which lui/la produce — and anything else falls back. Declining is always safe; the nutlb path is untouched.

Everything downstream of phys — L1-D tag, dirty bit, LRU, tcache mapped test, swizzle, store generation bump — is the same emitted code on both arms, so the cache model, VCE, aliasing and the page-generation checks are unchanged. Only how phys was obtained differs. TLB invalidation is irrelevant to KSEG0 by construction, and the TLB-mapped path is not modified in any way.

IRIS_NO_KSEG0_FAST=1 disables the path at emit time (like IRIS_NO_INLINE_MEM), so it can be A/B'd on a live system with j2 flush.

What did not work

Worth recording, because it is the reason the change looks the way it does: a predicate evaluated per access (two CP0 loads, a dozen ALU ops) measured worse than the nutlb probe it replaced — an access in the tcache path is only a handful of host instructions. Hoisting the same computation to region entry (entry_block, once per region) measured worse still, because iris-bench enters regions often enough that per-entry work costs more than it saves. One memory-operand compare is the shape that wins.

Measurement

Host: Intel i5-8500T, 6 cores, Linux x86_64, load average 3–5 during the runs. cargo build --release -p iris --features lightning,jitv2, nightly toolchain from the repo's flake. Two binaries (base = ed17f88, fast = this PR), three runs each, interleaved (base, fast, base, fast, base, fast), in-process:

iris-bench run --cpu r4400 --cache shadow   # the default for a jitv2 build

ns per work unit, median of three; lower is better. Spread is min–max of the three.

kernel	before	after	ratio	before spread	after spread
mem/latency_l1 (scalar-load chase, 8 KB)	8.81	5.28	1.67x	8.66–9.18	5.19–5.31
mem/copy (load+store stream)	1.44	0.99	1.45x	1.435–1.452	0.804–0.994
mem/fill (store stream)	1.58	0.98	1.61x	1.52–1.59	0.971–1.039
mem/stream_copy	0.77	0.58	1.32x	0.768–0.771	0.579–0.593
mem/random (random-page scalar loads)	100.7	51.3	1.96x	100.4–100.8	46.7–53.9
mem/latency_dram (chase)	88.9	81.2	1.09x	88.4–89.0	80.0–84.3
mem/latency_l2	15.85	11.64	1.36x	14.97–15.85	10.86–11.79
mem/unaligned	14.11	8.95	1.58x	13.91–14.25	8.77–9.26
int/dhrystone (string/record traffic)	1088	662	1.64x	1087–1089	653–674
codec/crc32 (scalar byte loads)	16.91	13.07	1.29x	16.72–16.98	12.85–13.97
whole suite	473 MIPS	609 MIPS	1.29x	468–479	609–613
whole suite, DMIPS	526	859	1.63x	—	—

--cache full (where the L2 model, not the guard, dominates): suite 1.02x, mem/latency_l1 1.70x, mem/random 1.22x, mem/copy 1.06x, mem/latency_dram 1.02x.

What it costs

• sys/tlb_hit is 3–10% slower (9.27 → 10.15 ns/op) and sys/uncached ~10% slower (3.84 → 4.27). Nearly every access in both is a callout, so the compare is pure overhead in front of work that was going to happen anyway. This is the trade the study predicted — the callout is the better path for miss-heavy work, and it is untouched.
• Individual non-memory kernels move by up to ~10% either way between runs from host load and code layout; with three runs and non-overlapping spreads only sys/tlb_hit, sys/uncached and fpu/scalar_s moved consistently, and the FP one has no mechanism (no memory access in the kernel).

Caveats

• These are bare-metal kernels, so every access is KSEG0 — the best case for this change. User processes on IRIX run in KUSEG and take the nutlb probe, paying only the extra compare (about two instructions per access). The kernel's own code and data are KSEG0, so real IRIX should still gain, but the headline 1.29x is a ceiling, not a prediction.
• The study's mem/load_l1 / mem/store_l1 micro-kernels are not in the checked-in suite, and this rig has no MIPS cross toolchain to add them; mem/latency_l1 (scalar loads), mem/fill (scalar stores), mem/copy/mem/stream_copy (load+store streams) and mem/random stand in for them. The 1.3x scalar-memory bar is met on all of those.
• Live IRIX was not boot-tested here: the only local disk image is one this session must not write to, and a copy-on-write config produced GIO timeouts before reaching the PROM. The evidence is the suite's 46 kernels at 100% accuracy (including sys/tlb_hit, sys/tlb_miss, sys/exception, sys/uncached, sys/llsc), the test battery below, and the --cache full/shadow runs agreeing.

Tests

• cargo test --workspace — 1351 passed, 0 failed.
• cargo test -p iris --lib --features lightning,jitv2 — 1752 passed, 0 failed (includes the jitv2 equivalence suite).
• cargo test -p iris --lib --features jitv2,jitv2_lockstep — 1785 passed, 4 failed; all four (every_instr_origin_class_is_tagged_in_traceback, fallback_successor_back_edge_is_still_recorded, lockstep_fpu_trunc_w_s_non_integer_value_sets_inexact, test_draw_debug_counts_every_word_of_a_batch) fail identically on upstream/main with the same feature set — verified by stashing this change and re-running them.
• New tests: kseg0_direct_rule checked against translate_impl itself over a grid of VAs spanning xuseg/KSEG0/KSEG1/KSSEG/KSEG3/xkphys, zero- and sign-extended, in both 32- and 64-bit modes (TR_MAPPED distinguishes the TLB-mapped arms); the R4400 vs R5000 K0 tables; the mirror tracking Status (kernel/supervisor/user, EXL, ERL, reserved KSU) and Config through a real MTC0; and a pinned check that KSEG0_RULE_HIT == 0xFFFF_FFFF_8800_0000 >> 29 — the constant is easy to write one F short, which makes the fast path silently never fire.
• cpu-tests was not run: no MIPS toolchain on this host (the flake does not provide one and the cross build needs a from-source binutils/gcc).

Docs: docs/jit-inline-memory.md §6a (design, predicate, measurement), FEATURES.md (the env var), CHANGELOG.md.

[techomancer]

Yeah, not sure about this. We need user space benchmarks. This adds a check in hot path for memory access which will slow down every TLB mapped transaction which are majority on a working system.
The benchmarks we have are running in kernel mode so they are not representative in this case.
It could be a minor win if we detect kernel pages and enable it only for these.