physical: a sparse, host-mapped decode table over a 40-bit address space

CHANGELOG.md

@@ -64,6 +64,19 @@ hashes are given where a change is easiest to understand by reading the commit.
   page in six. IRIX on an R5000 O2 panicked with "init died" for the same
   reason; it now boots to the desktop.
 
+### Physical address decoding
+
+- **The decode table is sparse and spans 40 bits.** `Physical` used to embed
+  a 1 MB array of device pointers covering a 32-bit physical space. It is now
+  a host-mapped table over a 40-bit space (`SparseDeviceMap`,
+  `src/ppmem/devmap.rs`, [design](docs/devmap-design.md)): pages of one
+  device share one 64 KB page, mixed pages come from a pool, and a page the
+  guest touches that nobody mapped is faulted in as the bus-error device with
+  a message naming the address. An Indy's table is about ten pages. Changes
+  swap whole pages in, so the DMA worker dispatching through the table during
+  a MEMCFG remap can no longer read a torn pointer. `phys map` in the monitor
+  shows the table. Groundwork for machines with a 40-bit address space.
+
 ### Memory configuration
 
 - **`banks` has eight entries.** Every machine today uses the first four and

HACKING.md

@@ -143,9 +143,16 @@ screenshots), `CpuDevice` (what `Machine` holds for the CPU).
 ### The physical address map
 
 `Physical` (`src/physical.rs`) decodes physical addresses through a table of
-64 KB pages (`device_map`, one entry per `addr >> 16`), so finding the device
-for an access is one indexed load. There is a lot of `unsafe` in there; it is
-deliberate. The hierarchy it models:
+64 KB slots (one `*const dyn BusDevice` per `addr >> 16`), so finding the
+device for an access is one indexed load. The table itself is sparse and
+host-mapped (`SparseDeviceMap`, `src/ppmem/devmap.rs`): it spans a 40-bit
+physical space as a 256 MB reservation, pages full of one pointer are views
+of one shared page, a change swaps a whole page in rather than storing into
+it (so threads dispatching through it never see a torn pointer), and a page
+nobody filled is faulted in as the bus-error device the first time the guest
+touches it. [docs/devmap-design.md](docs/devmap-design.md) has the details.
+There is a lot of `unsafe` in there; it is deliberate. The hierarchy it
+models:
 
 - **SysAD**: the CPU's 64-bit bus, with byte enables.
 - **MC** (Memory Controller, `src/dev/mc.rs`): the crossbar. DRAM, the GIO

README.md

@@ -152,7 +152,7 @@ Hardware, testing, and development references:
 | CPU test infrastructure | [Toolchain](cpu-tests/docs/toolchain.md), [memory map](cpu-tests/docs/memory-map.md), [oracle overview](cpu-tests/docs/oracle.md), [hardware oracle](cpu-tests/oracle/README.md), [R4600 notes](cpu-tests/docs/r4600.md), and [original test plan](cpu-tests/PLAN.md). |
 | Benchmarks and memory tests | [Benchmark suite](bench/README.md), [prebuilt assets](bench/prebuilt/README.md), [reference results](data/bench_reference.README.md), [GUI benchmark design](clanker/plans/gui-benchmark-plan.md), and [memory stress utility](test/memstress/README.md). |
 | Architecture and debugging | [HACKING.md](HACKING.md), [runtime debugging](debug.md), [agent instructions](clanker/CLANKER.md), and the [clanker/](clanker/INDEX.md) notes. |
-| Memory and JIT design | [Persistent JIT cache](docs/jitv2-persistent-cache.md), [inline memory](docs/jit-inline-memory.md), [JIT performance analysis](clanker/plans/jitv2_performance_analysis.md), [physical RAM](docs/ppmem-design.md), [transparent cache](docs/tcache-design.md), and [historical TLB design](docs/nutlb-design.md). |
+| Memory and JIT design | [Persistent JIT cache](docs/jitv2-persistent-cache.md), [inline memory](docs/jit-inline-memory.md), [JIT performance analysis](clanker/plans/jitv2_performance_analysis.md), [physical RAM](docs/ppmem-design.md), [address decode table](docs/devmap-design.md), [transparent cache](docs/tcache-design.md), and [historical TLB design](docs/nutlb-design.md). |
 | Plans and task briefs | [clanker/plans/](clanker/plans/): CHD sync, the NFS server, the networking GUI, the DaynaPort target, PCAP distribution, the NAT stack redesign, jitv2 performance analysis. Written as working documents; read their status notes first. |
 | HostGL tooling | [Khronos ABI registry tooling](iris-hostgl/tools/khronos/README.md). |
 

clanker/INDEX.md

@@ -97,6 +97,7 @@ Non-graphics devices: MC/MEMCFG and RAM banks, HPC3, VDMA, EEPROMs and NVRAM, PS
 | Note | Kind | Title |
 |---|---|---|
 | [cpal-alsa-busy-poll.md](devices/cpal-alsa-busy-poll.md) | rule | cpal's ALSA backend can spin a core: `[audio] host = "pulseaudio"` |
+| [decode-table-is-host-mapped.md](devices/decode-table-is-host-mapped.md) | rule | The physical decode table is a sparse host mapping over 40 bits: change it only through `fill`, set the default first, compare entries with `entry_eq` |
 | [extended-ram-memcfg.md](devices/extended-ram-memcfg.md) | rule | Extended RAM — MEMCFG synthesis for himem banks |
 | [hpc3-rx-channel-refused-while-active.md](devices/hpc3-rx-channel-refused-while-active.md) | rule | A receive channel that refuses while reporting itself active |
 | [indigo2-eeprom-byte-lane-gap.md](devices/indigo2-eeprom-byte-lane-gap.md) | rule | Indigo2 motherboard EEPROM: PROM writes via byte access, HPC3 only handled 32-bit |

clanker/devices/decode-table-is-host-mapped.md

@@ -0,0 +1,50 @@
+# The decode table is host-mapped and never written in place
+
+## Status: landed 2026-10-24 (`src/ppmem/devmap.rs`)
+
+`Physical::device_map`, the 64 KB-slot array of `*const dyn BusDevice`, is
+gone. The table is a `SparseDeviceMap`: a 256 MB `AddrSpace` reservation
+covering 40 bits of physical address, mapped page by page from shared
+constant pages (one per device pointer) and a pool of mixed pages. Design
+record: `docs/devmap-design.md`.
+
+## Rules
+
+- **Change entries only through `fill` / `fill_bytes`.** Every table view is
+  read-only; a store through `base()` segfaults. `fill` builds the new page
+  off-line and swaps it in, which is what keeps concurrent DMA dispatch
+  safe. Do not add a `*mut` path.
+- **Set the default before anything can dispatch.** `set_default` creates
+  the constant page the fault handler maps. Without it a fault on an
+  untouched page prints "no default device" and chains to the previous
+  handler, which kills the process. `Physical::init` does this first.
+- **A partial write to an untouched page pads with the default.** Writing a
+  device into a region nobody filled gives you that device there and the
+  default in the rest of the 256 MB page. That is what a 40-bit machine
+  wants; it means `fill` on an untouched page fails if no default is set.
+- **Pages are 64 KB on every host** (`granularity().max(64 KB)`), so one
+  table page covers 256 MB of guest space and every mapping is one page.
+  That is the Windows constraint (placeholders split on 64 KB, views cannot
+  be partially unmapped) applied everywhere, so Linux exercises the same
+  shape Windows needs.
+- **Compare entries with `entry_eq`, not `addr_eq`.** `UnmappedRam` is
+  zero-sized and shares its address with the next field of `Physical`; only
+  the vtable half of the fat pointer tells them apart. `addr_eq` would call
+  it equal to `AliasBus`.
+- **The fault handler must stay allocation-free.** It takes the table's
+  spin lock (bounded spin, gives up with a message), maps one page, writes
+  one line with `write(2)`. Keep it that way; a `println!` or `Vec` in there
+  is a deadlock waiting for a fault on the thread that holds the lock.
+- **Debuggers stop on SIGSEGV.** `handle SIGSEGV nostop noprint pass` (and
+  `SIGBUS`) in gdb. Only faults above 4 GB reach it on an Indy, so this
+  rarely matters until the Octane.
+
+## Not done
+
+- The `BusDevice` trait still takes `u32`. The table is 40-bit so the bus
+  can be widened without redoing the decode; nothing above 4 GB is reachable
+  from the CPU yet.
+- Windows backend written against the API, not run: same status as
+  `map_windows.rs`. The devmap unit tests are the acceptance test.
+- Shared pages are never freed (bounded by the number of distinct devices);
+  pool pages are. Fine for now.

debug.md

@@ -439,6 +439,7 @@ a forced termination; a normal stop saves them automatically.
 |---|---|
 | `phys mem <addr> [n]` / `mm` | Dump physical memory |
 | `phys dis <addr> [n]` / `md` | Disassemble physical memory |
+| `phys map` | The decode table: pages by kind (untouched, shared, private), memory in use, slots faulted in, and the table as runs of devices |
 | `phys trace <on\|off>` / `trace <on\|off>` | Log physical bus accesses |
 | `phys error <on\|off>` | Stop on bus errors |
 | `phys hole <on\|off>` | Stop on accesses to unmapped addresses |

docs/devmap-design.md

@@ -0,0 +1,125 @@
+# The physical address decode table
+
+`Physical` (`src/physical.rs`) finds the device behind a physical address
+with one indexed load: a table with one `*const dyn BusDevice` per 64 KB
+slot. This document describes how that table is stored since it grew from a
+32-bit to a 40-bit address space, and what happens when the guest touches an
+address nobody mapped. The code is `src/ppmem/devmap.rs`; it builds on the
+host-memory primitives ppmem already had ([ppmem-design.md](ppmem-design.md)
+§7).
+
+## 1. Why
+
+An Indy or Indigo2 decodes 32 bits of physical address. The table for that
+was a plain array: 65536 fat pointers, 1 MB, embedded in `Physical`. An
+Octane decodes 40 bits (XIO, the Bridge, memory beyond 4 GB). The same array
+for that space is 16 M entries and 256 MB, and almost all of it would hold the
+same few pointers: a bus-error sink, a memory bank, an empty slot.
+
+The table is therefore sparse in the same sense ppmem's RAM is: it is a
+reservation of host address space into which pages are mapped as needed,
+and the hot path does not know the difference.
+
+## 2. Shape
+
+- **One reservation**, 256 MB of virtual address space, `PROT_NONE` on Unix
+  and a placeholder on Windows (`AddrSpace::reserve`). Entry `i` is at
+  `base + 16 * i`. Nothing is committed by reserving.
+- **Table pages of 64 KB**, 4096 entries each, so one page decodes 256 MB of
+  guest address space. 64 KB is the Windows allocation granularity, and every
+  mapping is exactly one page, which keeps the Windows placeholder backend
+  (`map_windows.rs`) within what it can split and replace.
+- **Three kinds of page.** A page is *untouched* (never written, still
+  reserved), *shared* (a view of the constant page for one device pointer;
+  every page whose entries are all that pointer is a view of the same 64 KB),
+  or *private* (a page of mixed entries, from a pool of shared-memory
+  objects).
+- **Memory.** One 64 KB page per distinct device that fills a whole page, one
+  per mixed page, plus a 64 KB staging page. An Indy comes to about ten
+  pages, well under the 1 MB array it replaces, for 256 times the address
+  space.
+
+## 3. Writes never touch the table
+
+The MC's DMA worker, the SCSI and ethernet threads and the graphics boards
+dispatch through this table from their own threads while the CPU thread
+remaps banks on a MEMCFG write. With the old array that meant non-atomic
+16-byte stores under concurrent loads, and `plan_bank_slots` exists to write
+as few of them as possible.
+
+Now a change to any entry builds the new page contents off-line and swaps
+the page in:
+
+1. Read the page's current contents (from the table if private, from the
+   page's device pointer if shared, from the default device if untouched).
+2. Apply the change in a buffer. If the result is uniform, map the constant
+   page for that pointer. Otherwise take a free pool page, write the buffer
+   into it through the private *staging* view, and map the pool page.
+3. `MAP_FIXED` (Unix) or `MapViewOfFile3(MEM_REPLACE_PLACEHOLDER)` (Windows)
+   replaces the view atomically; a concurrent reader sees the old page or
+   the new one. The old pool page, if any, goes back on the free list.
+
+Every table view is read-only. A stray in-place store is a fault, not a
+silent corruption of a page shared with other regions.
+
+`plan_bank_slots` is kept. It still saves work, and a slot that ends up where
+it started is still never seen in a transient state.
+
+## 4. Untouched pages fault in
+
+A load from an untouched page is a host access violation. `devmap` installs
+a handler when the first table is created:
+
+- Unix: `sigaction` on `SIGSEGV` and `SIGBUS` (macOS raises `SIGBUS` for
+  `PROT_NONE` pages). Whatever handler was there before, which in a Rust
+  program is the standard library's stack-overflow detector, is saved and
+  chained to for every fault that is not in a table.
+- Windows: a vectored exception handler at the head of the chain. Faults
+  that are not in a table continue down the chain, so `crash_diag`'s handler
+  and the usual crash path are unchanged.
+
+For a fault inside a table the handler maps that page as a view of the
+constant page for the table's *default* device, prints one line naming the
+guest address, and returns; the load re-executes and finds the default. The
+default is set by the owner before anything dispatches (`Physical` uses its
+CPU bus-error sink, which reports to the MC). The constant page for it is
+created when the default is set, so the handler allocates nothing and takes
+only the table's spin lock, which is never held across anything that can
+block or fault.
+
+For the Indy and Indigo2 the whole 32-bit space is filled at `init`, exactly
+as before, so this path is reached only above 4 GB: nothing the CPU can
+address today. It is what lets a 40-bit machine leave its unused address
+space unmapped and still survive a probe.
+
+The monitor shows the table with `phys map`: page counts by kind, memory in
+use, how many faults filled a page in and where, and the table as runs of
+named devices ([debug.md](../debug.md)).
+
+## 5. Debugging with it
+
+A debugger stops on every `SIGSEGV` by default. Under gdb:
+
+```
+handle SIGSEGV nostop noprint pass
+handle SIGBUS  nostop noprint pass
+```
+
+and under lldb `process handle SIGSEGV -s false -n false -p true`. A fault in
+a table is reported on stderr either way; a fault anywhere else still
+reaches the previous handler and the debugger as before.
+
+Linux limits a process to about 65 000 mappings (`vm.max_map_count`). Each
+mapped table page is one; filling a full 40-bit space page by page would
+need 4096, which is fine, and the guests we run map far less.
+
+## 6. What this is not
+
+It is not device mapping through page faults (ppmem-design.md §11): the
+fault handler here maps a *table page*, not a guest device, and it runs once
+per untouched 256 MB region, not per access. The hot path is still the
+indexed load and the virtual call.
+
+The bus trait is still addressed with `u32`. The table reaches 40 bits so
+that the Octane work can widen the bus without also redesigning the decode;
+the Indy and Indigo2 use the low 4 GB of it unchanged.

docs/ppmem-design.md

@@ -570,6 +570,15 @@ demonstrated. Windows is **designed but unverified** — no Windows host here.
 Explicitly **not** part of this work, and not phase 2 either. Recorded here so
 the idea and its blockers aren't relearned.
 
+*Update, October 2026:* a fault handler now exists, but for the **decode
+table**, not for devices. `src/ppmem/devmap.rs` keeps `Physical`'s
+address-to-device table as a sparse reservation over a 40-bit space and
+faults untouched table pages in as the bus-error device; see
+[devmap-design.md](devmap-design.md). It settles the portability question
+for this narrow use (one handler per OS, no instruction decoding, since the
+retried access is the table load itself) and leaves everything below as
+open as before.
+
 ### 11.1 The interesting use: map devices, delete the bus abstraction
 
 The obvious uses (dirty-page snapshot tracking, lazy/COW bank materialisation

src/main.rs

@@ -50,14 +50,17 @@ fn main() {
     cfg.debug.apply_env();
     cfg.jitv2.apply_env();
 
-    // Machine::new() allocates >1MB on the stack (Physical device_map), which overflows
-    // the default stack on Windows (1MB). We spawn a thread with a larger stack to create it.
+    // Machine::new() has several large temporaries on the stack, which
+    // overflows the default stack on Windows (1MB). We spawn a thread with a
+    // larger stack to create it.
     let mut machine = std::thread::Builder::new()
         // Rex3Context embeds the HOSTRW data-port array (HOSTRW_BUF_QWORDS u64s,
-            // 1 MiB), and construction has several context-sized temporaries in
-            // flight before the machine is boxed. This is virtual address space,
-            // lazily committed, so the large reservation has no real cost.
-            .stack_size(64 * 1024 * 1024)
+        // 1 MiB), and construction has several context-sized temporaries in
+        // flight before the machine is boxed. (The physical decode table used
+        // to be another 1 MB here; it is host-mapped now.) This is virtual
+        // address space, lazily committed, so the large reservation has no
+        // real cost.
+        .stack_size(64 * 1024 * 1024)
         .spawn(move || iris::workstation::build(cfg))
         .unwrap()
         .join()