From ff0fa802695fde63f09f48dee4a2c7061a4365ad Mon Sep 17 00:00:00 2001 From: technomancer Date: Sat, 24 Oct 2026 21:13:02 -0700 Subject: [PATCH] physical: a sparse, host-mapped decode table over a 40-bit address space Physical found the device behind an address with one indexed load from a 1 MB array of fat pointers covering 32 bits. An Octane decodes 40 bits, which would make that array 256 MB, almost all of it the same few pointers. The table is now SparseDeviceMap (src/ppmem/devmap.rs), built on ppmem's AddrSpace and SharedMem: a 256 MB reservation, 64 KB table pages (4096 entries, 256 MB of guest space each, one mapping per page so the Windows placeholder backend works unchanged), pages of one device sharing one constant page, mixed pages from a pool. The hot path is still one load. Changes build the new page through a staging view and swap it in with MAP_FIXED; every table view is read-only. The DMA worker dispatching through the table during a MEMCFG remap sees whole pages, never a torn 16-byte pointer. Entries are compared on both halves, since the zero-sized UnmappedRam shares its address with the next field. A page nobody filled stays reserved. A load from one is a host fault; a SIGSEGV/SIGBUS handler (Unix, chaining to the previous handler for foreign faults) or a vectored exception handler (Windows) maps the page to the table's default device, prints the guest address, and resumes. The Indy and Indigo2 fill their 32-bit space at init as before, so this path only matters above 4 GB. An Indy's table is four committed pages. Monitor: `phys map` shows the table as named runs, page counts and faults. The BusDevice trait still takes u32; widening it is the next step. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 13 + HACKING.md | 13 +- README.md | 2 +- clanker/INDEX.md | 1 + .../devices/decode-table-is-host-mapped.md | 50 + debug.md | 1 + docs/devmap-design.md | 125 +++ docs/ppmem-design.md | 9 + src/main.rs | 15 +- src/physical.rs | 284 +++--- src/ppmem/devmap.rs | 915 ++++++++++++++++++ src/ppmem/mod.rs | 5 +- 12 files changed, 1311 insertions(+), 122 deletions(-) create mode 100644 clanker/devices/decode-table-is-host-mapped.md create mode 100644 docs/devmap-design.md create mode 100644 src/ppmem/devmap.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d3eb91b..dab0d0d4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -64,6 +64,19 @@ hashes are given where a change is easiest to understand by reading the commit. page in six. IRIX on an R5000 O2 panicked with "init died" for the same reason; it now boots to the desktop. +### Physical address decoding + +- **The decode table is sparse and spans 40 bits.** `Physical` used to embed + a 1 MB array of device pointers covering a 32-bit physical space. It is now + a host-mapped table over a 40-bit space (`SparseDeviceMap`, + `src/ppmem/devmap.rs`, [design](docs/devmap-design.md)): pages of one + device share one 64 KB page, mixed pages come from a pool, and a page the + guest touches that nobody mapped is faulted in as the bus-error device with + a message naming the address. An Indy's table is about ten pages. Changes + swap whole pages in, so the DMA worker dispatching through the table during + a MEMCFG remap can no longer read a torn pointer. `phys map` in the monitor + shows the table. Groundwork for machines with a 40-bit address space. + ### Memory configuration - **`banks` has eight entries.** Every machine today uses the first four and diff --git a/HACKING.md b/HACKING.md index 45306e28..6298e4ac 100644 --- a/HACKING.md +++ b/HACKING.md @@ -143,9 +143,16 @@ screenshots), `CpuDevice` (what `Machine` holds for the CPU). ### The physical address map `Physical` (`src/physical.rs`) decodes physical addresses through a table of -64 KB pages (`device_map`, one entry per `addr >> 16`), so finding the device -for an access is one indexed load. There is a lot of `unsafe` in there; it is -deliberate. The hierarchy it models: +64 KB slots (one `*const dyn BusDevice` per `addr >> 16`), so finding the +device for an access is one indexed load. The table itself is sparse and +host-mapped (`SparseDeviceMap`, `src/ppmem/devmap.rs`): it spans a 40-bit +physical space as a 256 MB reservation, pages full of one pointer are views +of one shared page, a change swaps a whole page in rather than storing into +it (so threads dispatching through it never see a torn pointer), and a page +nobody filled is faulted in as the bus-error device the first time the guest +touches it. [docs/devmap-design.md](docs/devmap-design.md) has the details. +There is a lot of `unsafe` in there; it is deliberate. The hierarchy it +models: - **SysAD**: the CPU's 64-bit bus, with byte enables. - **MC** (Memory Controller, `src/dev/mc.rs`): the crossbar. DRAM, the GIO diff --git a/README.md b/README.md index bc39d047..14bd6d98 100644 --- a/README.md +++ b/README.md @@ -152,7 +152,7 @@ Hardware, testing, and development references: | CPU test infrastructure | [Toolchain](cpu-tests/docs/toolchain.md), [memory map](cpu-tests/docs/memory-map.md), [oracle overview](cpu-tests/docs/oracle.md), [hardware oracle](cpu-tests/oracle/README.md), [R4600 notes](cpu-tests/docs/r4600.md), and [original test plan](cpu-tests/PLAN.md). | | Benchmarks and memory tests | [Benchmark suite](bench/README.md), [prebuilt assets](bench/prebuilt/README.md), [reference results](data/bench_reference.README.md), [GUI benchmark design](clanker/plans/gui-benchmark-plan.md), and [memory stress utility](test/memstress/README.md). | | Architecture and debugging | [HACKING.md](HACKING.md), [runtime debugging](debug.md), [agent instructions](clanker/CLANKER.md), and the [clanker/](clanker/INDEX.md) notes. | -| Memory and JIT design | [Persistent JIT cache](docs/jitv2-persistent-cache.md), [inline memory](docs/jit-inline-memory.md), [JIT performance analysis](clanker/plans/jitv2_performance_analysis.md), [physical RAM](docs/ppmem-design.md), [transparent cache](docs/tcache-design.md), and [historical TLB design](docs/nutlb-design.md). | +| Memory and JIT design | [Persistent JIT cache](docs/jitv2-persistent-cache.md), [inline memory](docs/jit-inline-memory.md), [JIT performance analysis](clanker/plans/jitv2_performance_analysis.md), [physical RAM](docs/ppmem-design.md), [address decode table](docs/devmap-design.md), [transparent cache](docs/tcache-design.md), and [historical TLB design](docs/nutlb-design.md). | | Plans and task briefs | [clanker/plans/](clanker/plans/): CHD sync, the NFS server, the networking GUI, the DaynaPort target, PCAP distribution, the NAT stack redesign, jitv2 performance analysis. Written as working documents; read their status notes first. | | HostGL tooling | [Khronos ABI registry tooling](iris-hostgl/tools/khronos/README.md). | diff --git a/clanker/INDEX.md b/clanker/INDEX.md index 411e3c1e..10cce2ac 100644 --- a/clanker/INDEX.md +++ b/clanker/INDEX.md @@ -97,6 +97,7 @@ Non-graphics devices: MC/MEMCFG and RAM banks, HPC3, VDMA, EEPROMs and NVRAM, PS | Note | Kind | Title | |---|---|---| | [cpal-alsa-busy-poll.md](devices/cpal-alsa-busy-poll.md) | rule | cpal's ALSA backend can spin a core: `[audio] host = "pulseaudio"` | +| [decode-table-is-host-mapped.md](devices/decode-table-is-host-mapped.md) | rule | The physical decode table is a sparse host mapping over 40 bits: change it only through `fill`, set the default first, compare entries with `entry_eq` | | [extended-ram-memcfg.md](devices/extended-ram-memcfg.md) | rule | Extended RAM — MEMCFG synthesis for himem banks | | [hpc3-rx-channel-refused-while-active.md](devices/hpc3-rx-channel-refused-while-active.md) | rule | A receive channel that refuses while reporting itself active | | [indigo2-eeprom-byte-lane-gap.md](devices/indigo2-eeprom-byte-lane-gap.md) | rule | Indigo2 motherboard EEPROM: PROM writes via byte access, HPC3 only handled 32-bit | diff --git a/clanker/devices/decode-table-is-host-mapped.md b/clanker/devices/decode-table-is-host-mapped.md new file mode 100644 index 00000000..d358cea7 --- /dev/null +++ b/clanker/devices/decode-table-is-host-mapped.md @@ -0,0 +1,50 @@ +# The decode table is host-mapped and never written in place + +## Status: landed 2026-10-24 (`src/ppmem/devmap.rs`) + +`Physical::device_map`, the 64 KB-slot array of `*const dyn BusDevice`, is +gone. The table is a `SparseDeviceMap`: a 256 MB `AddrSpace` reservation +covering 40 bits of physical address, mapped page by page from shared +constant pages (one per device pointer) and a pool of mixed pages. Design +record: `docs/devmap-design.md`. + +## Rules + +- **Change entries only through `fill` / `fill_bytes`.** Every table view is + read-only; a store through `base()` segfaults. `fill` builds the new page + off-line and swaps it in, which is what keeps concurrent DMA dispatch + safe. Do not add a `*mut` path. +- **Set the default before anything can dispatch.** `set_default` creates + the constant page the fault handler maps. Without it a fault on an + untouched page prints "no default device" and chains to the previous + handler, which kills the process. `Physical::init` does this first. +- **A partial write to an untouched page pads with the default.** Writing a + device into a region nobody filled gives you that device there and the + default in the rest of the 256 MB page. That is what a 40-bit machine + wants; it means `fill` on an untouched page fails if no default is set. +- **Pages are 64 KB on every host** (`granularity().max(64 KB)`), so one + table page covers 256 MB of guest space and every mapping is one page. + That is the Windows constraint (placeholders split on 64 KB, views cannot + be partially unmapped) applied everywhere, so Linux exercises the same + shape Windows needs. +- **Compare entries with `entry_eq`, not `addr_eq`.** `UnmappedRam` is + zero-sized and shares its address with the next field of `Physical`; only + the vtable half of the fat pointer tells them apart. `addr_eq` would call + it equal to `AliasBus`. +- **The fault handler must stay allocation-free.** It takes the table's + spin lock (bounded spin, gives up with a message), maps one page, writes + one line with `write(2)`. Keep it that way; a `println!` or `Vec` in there + is a deadlock waiting for a fault on the thread that holds the lock. +- **Debuggers stop on SIGSEGV.** `handle SIGSEGV nostop noprint pass` (and + `SIGBUS`) in gdb. Only faults above 4 GB reach it on an Indy, so this + rarely matters until the Octane. + +## Not done + +- The `BusDevice` trait still takes `u32`. The table is 40-bit so the bus + can be widened without redoing the decode; nothing above 4 GB is reachable + from the CPU yet. +- Windows backend written against the API, not run: same status as + `map_windows.rs`. The devmap unit tests are the acceptance test. +- Shared pages are never freed (bounded by the number of distinct devices); + pool pages are. Fine for now. diff --git a/debug.md b/debug.md index 328f6fbc..59081f96 100644 --- a/debug.md +++ b/debug.md @@ -439,6 +439,7 @@ a forced termination; a normal stop saves them automatically. |---|---| | `phys mem [n]` / `mm` | Dump physical memory | | `phys dis [n]` / `md` | Disassemble physical memory | +| `phys map` | The decode table: pages by kind (untouched, shared, private), memory in use, slots faulted in, and the table as runs of devices | | `phys trace ` / `trace ` | Log physical bus accesses | | `phys error ` | Stop on bus errors | | `phys hole ` | Stop on accesses to unmapped addresses | diff --git a/docs/devmap-design.md b/docs/devmap-design.md new file mode 100644 index 00000000..63c97d51 --- /dev/null +++ b/docs/devmap-design.md @@ -0,0 +1,125 @@ +# The physical address decode table + +`Physical` (`src/physical.rs`) finds the device behind a physical address +with one indexed load: a table with one `*const dyn BusDevice` per 64 KB +slot. This document describes how that table is stored since it grew from a +32-bit to a 40-bit address space, and what happens when the guest touches an +address nobody mapped. The code is `src/ppmem/devmap.rs`; it builds on the +host-memory primitives ppmem already had ([ppmem-design.md](ppmem-design.md) +§7). + +## 1. Why + +An Indy or Indigo2 decodes 32 bits of physical address. The table for that +was a plain array: 65536 fat pointers, 1 MB, embedded in `Physical`. An +Octane decodes 40 bits (XIO, the Bridge, memory beyond 4 GB). The same array +for that space is 16 M entries and 256 MB, and almost all of it would hold the +same few pointers: a bus-error sink, a memory bank, an empty slot. + +The table is therefore sparse in the same sense ppmem's RAM is: it is a +reservation of host address space into which pages are mapped as needed, +and the hot path does not know the difference. + +## 2. Shape + +- **One reservation**, 256 MB of virtual address space, `PROT_NONE` on Unix + and a placeholder on Windows (`AddrSpace::reserve`). Entry `i` is at + `base + 16 * i`. Nothing is committed by reserving. +- **Table pages of 64 KB**, 4096 entries each, so one page decodes 256 MB of + guest address space. 64 KB is the Windows allocation granularity, and every + mapping is exactly one page, which keeps the Windows placeholder backend + (`map_windows.rs`) within what it can split and replace. +- **Three kinds of page.** A page is *untouched* (never written, still + reserved), *shared* (a view of the constant page for one device pointer; + every page whose entries are all that pointer is a view of the same 64 KB), + or *private* (a page of mixed entries, from a pool of shared-memory + objects). +- **Memory.** One 64 KB page per distinct device that fills a whole page, one + per mixed page, plus a 64 KB staging page. An Indy comes to about ten + pages, well under the 1 MB array it replaces, for 256 times the address + space. + +## 3. Writes never touch the table + +The MC's DMA worker, the SCSI and ethernet threads and the graphics boards +dispatch through this table from their own threads while the CPU thread +remaps banks on a MEMCFG write. With the old array that meant non-atomic +16-byte stores under concurrent loads, and `plan_bank_slots` exists to write +as few of them as possible. + +Now a change to any entry builds the new page contents off-line and swaps +the page in: + +1. Read the page's current contents (from the table if private, from the + page's device pointer if shared, from the default device if untouched). +2. Apply the change in a buffer. If the result is uniform, map the constant + page for that pointer. Otherwise take a free pool page, write the buffer + into it through the private *staging* view, and map the pool page. +3. `MAP_FIXED` (Unix) or `MapViewOfFile3(MEM_REPLACE_PLACEHOLDER)` (Windows) + replaces the view atomically; a concurrent reader sees the old page or + the new one. The old pool page, if any, goes back on the free list. + +Every table view is read-only. A stray in-place store is a fault, not a +silent corruption of a page shared with other regions. + +`plan_bank_slots` is kept. It still saves work, and a slot that ends up where +it started is still never seen in a transient state. + +## 4. Untouched pages fault in + +A load from an untouched page is a host access violation. `devmap` installs +a handler when the first table is created: + +- Unix: `sigaction` on `SIGSEGV` and `SIGBUS` (macOS raises `SIGBUS` for + `PROT_NONE` pages). Whatever handler was there before, which in a Rust + program is the standard library's stack-overflow detector, is saved and + chained to for every fault that is not in a table. +- Windows: a vectored exception handler at the head of the chain. Faults + that are not in a table continue down the chain, so `crash_diag`'s handler + and the usual crash path are unchanged. + +For a fault inside a table the handler maps that page as a view of the +constant page for the table's *default* device, prints one line naming the +guest address, and returns; the load re-executes and finds the default. The +default is set by the owner before anything dispatches (`Physical` uses its +CPU bus-error sink, which reports to the MC). The constant page for it is +created when the default is set, so the handler allocates nothing and takes +only the table's spin lock, which is never held across anything that can +block or fault. + +For the Indy and Indigo2 the whole 32-bit space is filled at `init`, exactly +as before, so this path is reached only above 4 GB: nothing the CPU can +address today. It is what lets a 40-bit machine leave its unused address +space unmapped and still survive a probe. + +The monitor shows the table with `phys map`: page counts by kind, memory in +use, how many faults filled a page in and where, and the table as runs of +named devices ([debug.md](../debug.md)). + +## 5. Debugging with it + +A debugger stops on every `SIGSEGV` by default. Under gdb: + +``` +handle SIGSEGV nostop noprint pass +handle SIGBUS nostop noprint pass +``` + +and under lldb `process handle SIGSEGV -s false -n false -p true`. A fault in +a table is reported on stderr either way; a fault anywhere else still +reaches the previous handler and the debugger as before. + +Linux limits a process to about 65 000 mappings (`vm.max_map_count`). Each +mapped table page is one; filling a full 40-bit space page by page would +need 4096, which is fine, and the guests we run map far less. + +## 6. What this is not + +It is not device mapping through page faults (ppmem-design.md §11): the +fault handler here maps a *table page*, not a guest device, and it runs once +per untouched 256 MB region, not per access. The hot path is still the +indexed load and the virtual call. + +The bus trait is still addressed with `u32`. The table reaches 40 bits so +that the Octane work can widen the bus without also redesigning the decode; +the Indy and Indigo2 use the low 4 GB of it unchanged. diff --git a/docs/ppmem-design.md b/docs/ppmem-design.md index 5de57696..3893464c 100644 --- a/docs/ppmem-design.md +++ b/docs/ppmem-design.md @@ -570,6 +570,15 @@ demonstrated. Windows is **designed but unverified** — no Windows host here. Explicitly **not** part of this work, and not phase 2 either. Recorded here so the idea and its blockers aren't relearned. +*Update, October 2026:* a fault handler now exists, but for the **decode +table**, not for devices. `src/ppmem/devmap.rs` keeps `Physical`'s +address-to-device table as a sparse reservation over a 40-bit space and +faults untouched table pages in as the bus-error device; see +[devmap-design.md](devmap-design.md). It settles the portability question +for this narrow use (one handler per OS, no instruction decoding, since the +retried access is the table load itself) and leaves everything below as +open as before. + ### 11.1 The interesting use: map devices, delete the bus abstraction The obvious uses (dirty-page snapshot tracking, lazy/COW bank materialisation diff --git a/src/main.rs b/src/main.rs index 100e244d..108fecb8 100644 --- a/src/main.rs +++ b/src/main.rs @@ -50,14 +50,17 @@ fn main() { cfg.debug.apply_env(); cfg.jitv2.apply_env(); - // Machine::new() allocates >1MB on the stack (Physical device_map), which overflows - // the default stack on Windows (1MB). We spawn a thread with a larger stack to create it. + // Machine::new() has several large temporaries on the stack, which + // overflows the default stack on Windows (1MB). We spawn a thread with a + // larger stack to create it. let mut machine = std::thread::Builder::new() // Rex3Context embeds the HOSTRW data-port array (HOSTRW_BUF_QWORDS u64s, - // 1 MiB), and construction has several context-sized temporaries in - // flight before the machine is boxed. This is virtual address space, - // lazily committed, so the large reservation has no real cost. - .stack_size(64 * 1024 * 1024) + // 1 MiB), and construction has several context-sized temporaries in + // flight before the machine is boxed. (The physical decode table used + // to be another 1 MB here; it is host-mapped now.) This is virtual + // address space, lazily committed, so the large reservation has no + // real cost. + .stack_size(64 * 1024 * 1024) .spawn(move || iris::workstation::build(cfg)) .unwrap() .join() diff --git a/src/physical.rs b/src/physical.rs index 7995df17..f338f469 100644 --- a/src/physical.rs +++ b/src/physical.rs @@ -8,6 +8,7 @@ use crate::exp::eval_const_expr; use crate::cpu::mips_dis; use crate::dev::mem::{BlackHoleRegion, UnmappedRam}; use crate::ppmem::{MappedMemory, PpMemSpace, PpMemory}; +use crate::ppmem::devmap::{entry_eq, Entry, SparseDeviceMap, SLOT_SHIFT}; use crate::dev::prom::PromPort; use crate::dev::mc::MemoryController; @@ -196,7 +197,7 @@ const PROM_END: u32 = 0x1FD00000; // per MC spec: "The bottom 512KB of memory is just an alias for the memory located // at address 0x08000000 to 0x0807ffff." // Implemented as an AliasBus that adds LOMEM_BASE to the incoming address, so -// accesses go through the normal lomem device_map entries — no direct bank pointer needed. +// accesses go through the normal lomem decode-table entries — no direct bank pointer needed. const ALIAS_BASE: u32 = 0x00000000; const ALIAS_END: u32 = 0x00080000; @@ -218,14 +219,14 @@ fn alias_offset_for(ip28: bool) -> u32 { } -/// What one 64 KB `device_map` slot should point at after a MEMCFG write. +/// What one 64 KB decode-table slot should point at after a MEMCFG write. #[derive(Clone, Copy, Debug, PartialEq, Eq)] enum BankSlot { Unmapped, Bank(usize), } -/// Decide every `device_map` slot a MEMCFG write can affect: the lomem and +/// Decide every decode-table slot a MEMCFG write can affect: the lomem and /// himem windows, each bank's placement, and wherever a bank sat outside /// those windows last time. Returns one entry per slot, in slot order, plus /// the slots this placement puts outside the windows, for the next call. @@ -233,11 +234,13 @@ enum BankSlot { /// A plan rather than a sequence of stores, for two reasons: /// /// - **Only the differences get written.** Wiping both 256 MB windows and -/// mapping the banks back over them is about 8200 stores per MEMCFG write, -/// each a non-atomic 16-byte fat pointer, while the MC's DMA worker may be -/// dispatching through the table from its own thread. Collapsing to one -/// entry per slot means a slot that ends up where it started is never -/// touched, not wiped and rewritten. +/// mapping the banks back over them is about 8200 slots per MEMCFG write, +/// while the MC's DMA worker may be dispatching through the table from its +/// own thread. Collapsing to one entry per slot means a slot that ends up +/// where it started is never touched, not wiped and rewritten, and the +/// table is never seen half-way through a change. (The table now swaps +/// whole pages atomically, so a transient would no longer be a torn +/// pointer; it would still be a slot briefly answering as unmapped.) /// - **A bank placed outside the windows is unmapped again when it moves.** /// MEMCFG's base field reaches far beyond lomem and himem, and a wipe of /// only those two left such a slot pointing at a bank the MC had since @@ -285,17 +288,19 @@ const MYSTERY_HOLE_END: u32 = 0x02090000; /// Acts as the central hub for connecting devices. /// Routes all bus accesses to the appropriate device based on address. /// Devices are stored directly in the struct for zero-overhead access. -/// Uses a 64KB granularity lookup table for O(1) address decoding. +/// Decodes with one indexed load from a table of 64 KB slots; the table +/// itself is sparse and host-MMU-backed (`SparseDeviceMap`), so it spans a +/// 40-bit space while costing a few pages. /// /// Memory is split into 4 × 128MB banks (0..3). MEMCFG0/1 in the MC /// configure at which physical address each bank appears. On MEMCFG write, -/// the MC invokes remap_banks() which updates device_map and each bank's +/// the MC invokes remap_banks() which updates the decode table and each bank's /// base_addr. Banks 0/1 occupy lomem slots; banks 2/3 occupy himem slots. pub struct Physical { // 4 × 128MB RAM banks (0,1 in lomem; 2,3 in himem). base_addr set by remap_banks(). banks: [RamBank; 4], /// ppmem's 4GB window. Banks are mapped into it by `remap_banks` in - /// addition to being installed in `device_map`, so the mapping-based fast + /// addition to being installed in the decode table, so the mapping-based fast /// path and the bus path stay in agreement. `None` if the window could not /// be reserved, in which case only the bus path is used. ppmem_space: Option, @@ -338,9 +343,14 @@ pub struct Physical { vino_gio_alias: AliasBus, // GIO aperture at 0x1F080000 → VINO at 0x00080000 black_hole: BlackHoleRegion, - // Lookup table: 64KB granularity (65536 entries = 512KB on 64-bit) - // Maps (address >> 16) to device pointer (non-null, always valid) - device_map: [*const dyn BusDevice; 65536], + /// The decode table: one device per 64 KB slot of a 40-bit physical + /// space, sparse and host-MMU-backed (`src/ppmem/devmap.rs`). Slots the + /// Indy and Indigo2 decode are filled in `init`; a slot nobody filled is + /// faulted in as the CPU bus-error device the first time it is touched. + devmap: SparseDeviceMap, + /// `devmap.base()`, cached so the hot path is a load off `self` and one + /// indexed load. + map_base: *const Entry, /// 64 KB slots the last `remap_banks` placed a RAM bank in outside the /// lomem and himem windows. MEMCFG's base field reaches well beyond those @@ -357,7 +367,7 @@ pub struct Physical { host_freq: u64, } -// Safety: Physical owns all devices and the pointers in device_map point to owned data +// Safety: Physical owns all devices and the pointers in the decode table point to owned data // All the devices themselves are Send+Sync, and we never mutate through the pointers unsafe impl Send for Physical {} unsafe impl Sync for Physical {} @@ -430,17 +440,14 @@ impl Physical { let vino_gio_alias = AliasBus::new(std::ptr::null::(), 0xE1000000u32); let black_hole = BlackHoleRegion::new(); - // The lookup table is filled in init(). Until then it points at a - // bus-error device rather than at null, although every slot is - // written before the guest runs: the MC's DMA worker dispatches - // through this table from its own thread while `remap_banks` rewrites - // its 16-byte fat pointers non-atomically from the CPU's, and a torn - // read that pairs a null data pointer with a live vtable is a - // segfault, not a bus error. A stateless static costs nothing and - // takes null out of the table for good. - static BOOT_ERR: ErrorBus = ErrorBus { debug: AtomicBool::new(false) }; - const BOOT_PTR: *const dyn BusDevice = &BOOT_ERR; - let device_map: [*const dyn BusDevice; 65536] = [BOOT_PTR; 65536]; + // The decode table. Only reserved here: `init` fills it once `self` + // is in its final place, since the entries point at fields of `self`. + // Nothing dispatches before that; a dispatch that did would fault + // with a message naming the address rather than read a null. + let devmap = SparseDeviceMap::new().unwrap_or_else(|e| { + panic!("iris: cannot reserve the physical address decode table: {e}") + }); + let map_base = devmap.base(); // ppmem: reserve the 4GB window over these banks. A failure here is // not fatal — the bus path works regardless — so log and carry on @@ -488,7 +495,8 @@ impl Physical { alias_bus, vino_gio_alias, black_hole, - device_map, + devmap, + map_base, banks_outside_windows: Vec::new(), alias_offset: alias_offset_for(ip28), is_ip28: ip28, @@ -519,18 +527,29 @@ impl Physical { let prom_ptr: *const dyn BusDevice = &self.prom; let black_hole_ptr: *const dyn BusDevice = &self.black_hole; - // Layer 1: fill entire table with CPU bus error device - for i in 0..65536usize { - self.device_map[i] = cpu_err_ptr; - } + let map = &self.devmap; + let set = |base: u32, end: u32, dev: Entry| { + map.fill_bytes(base as u64, end as u64, dev) + .unwrap_or_else(|e| panic!("physical: mapping {base:#010x}..{end:#010x}: {e}")); + }; + + // Whatever is never mapped below answers as a CPU bus error. Within + // the 32-bit space that is filled in now (layer 1); the rest of the + // 40-bit table stays untouched until the guest reaches it, and the + // first access to a slot up there faults it in as the same device, + // with a message naming the address. + map.set_default(cpu_err_ptr) + .unwrap_or_else(|e| panic!("physical: setting the bus-error default: {e}")); + + // Layer 1: the whole 32-bit space, CPU bus error device + map.fill(0, 1 << (32 - SLOT_SHIFT), cpu_err_ptr) + .unwrap_or_else(|e| panic!("physical: filling the 32-bit space: {e}")); // Layer 2: overlay GIO space (0x18000000..0x1FA00000) with GIO timeout device // This covers: reserved future GIO (0x18000000..0x1F000000) + Newport slot // (0x1F000000..0x1F400000) + GIO expansion slots 0/1 (0x1F400000..0x1FA00000) // Real devices will overlay their own ranges on top in layer 3. - for i in (0x1800_0000u32 >> 16)..(0x1FA0_0000u32 >> 16) { - self.device_map[i as usize] = gio_err_ptr; - } + set(0x1800_0000, 0x1FA0_0000, gio_err_ptr); // Memory banks are NOT mapped here — MEMCFG0/1 control their placement. // remap_banks() is called by the MC when MEMCFG is written. @@ -539,17 +558,13 @@ impl Physical { // Layer 3: real devices overlaid on top // Map VINO (physical 0x00080000, one 64KB slot) - self.device_map[(crate::dev::vino::VINO_BASE >> 16) as usize] = vino_ptr; + set(crate::dev::vino::VINO_BASE, crate::dev::vino::VINO_BASE + (1 << SLOT_SHIFT), vino_ptr); // Map Mystery Hole - for i in (MYSTERY_HOLE_BASE >> 16)..((MYSTERY_HOLE_END - 1) >> 16) + 1 { - self.device_map[i as usize] = black_hole_ptr; - } + set(MYSTERY_HOLE_BASE, MYSTERY_HOLE_END, black_hole_ptr); // 2nd hpc - for i in (0x1F980000 >> 16)..((0x1F990000 - 1) >> 16) + 1 { - self.device_map[i as usize] = black_hole_ptr; - } + set(0x1F980000, 0x1F990000, black_hole_ptr); // HPC1 region (0x1FB00000–0x1FB80000) — older HPC chip iris doesn't // emulate. IRIX still probes here during normal operation (visible as @@ -560,47 +575,33 @@ impl Physical { // black hole (reads as zero, writes silently accepted) prevents the // bus error from firing and avoids the panic — without implementing // real HPC1 semantics, which would be a much larger emulation gap. - for i in (0x1FB00000u32 >> 16)..((HPC3_BASE - 1) >> 16) + 1 { - self.device_map[i as usize] = black_hole_ptr; - } + set(0x1FB00000, HPC3_BASE, black_hole_ptr); // Map Newport/REX3 (4MB GIO slot at 0x1F000000) — only if graphics enabled if let Some(rex3_ptr) = rex3_ptr { - for i in (NEWPORT_BASE >> 16)..((NEWPORT_END - 1) >> 16) + 1 { - self.device_map[i as usize] = rex3_ptr; - } + set(NEWPORT_BASE, NEWPORT_END, rex3_ptr); } else if let Some(gr2_ptr) = gr2_ptr { // GR2 (XZ / Extreme): the whole 4 MB gfx slot; see src/dev/gr2. - for i in (NEWPORT_BASE >> 16)..((NEWPORT_END - 1) >> 16) + 1 { - self.device_map[i as usize] = gr2_ptr; - } + set(NEWPORT_BASE, NEWPORT_END, gr2_ptr); } else if let Some(mgras_ptr) = mgras_ptr { // IMPACT in the graphics slot. The expansion slots stay unmapped so // their probes bus-error, as empty slots do. - for i in (NEWPORT_BASE >> 16)..((NEWPORT_END - 1) >> 16) + 1 { - self.device_map[i as usize] = mgras_ptr; - } + set(NEWPORT_BASE, NEWPORT_END, mgras_ptr); } // else: GIO timeout from layer 2 already covers the Newport slot // Second Newport head at GIO expansion slot 1 (dual-head). if let Some(h1_ptr) = rex3_head1_ptr { - for i in (GIO_SLOT1_BASE >> 16)..((GIO_SLOT1_END - 1) >> 16) + 1 { - self.device_map[i as usize] = h1_ptr; - } + set(GIO_SLOT1_BASE, GIO_SLOT1_END, h1_ptr); } // GIO expansion slot 0 (0x1F400000–0x1F5FFFFF): N64 dev board if enabled if let Some(u64_ptr) = ultra64_ptr { use crate::dev::ultra64::{GIO_SLOT0_BASE, RAMROM_BASE, RAMROM_SIZE}; // Control registers: 0x1F400000–0x1F4FFFFF (16 × 64KB slots) - for i in (GIO_SLOT0_BASE >> 16)..((RAMROM_BASE - 1) >> 16) + 1 { - self.device_map[i as usize] = u64_ptr; - } + set(GIO_SLOT0_BASE, RAMROM_BASE, u64_ptr); // RAMROM window: 0x1F500000–0x1F5FFFFF (16 × 64KB slots) - for i in (RAMROM_BASE >> 16)..((RAMROM_BASE + RAMROM_SIZE - 1) >> 16) + 1 { - self.device_map[i as usize] = u64_ptr; - } + set(RAMROM_BASE, RAMROM_BASE + RAMROM_SIZE, u64_ptr); } // GIO expansion slot 1 — second Newport when rex3_head1 absent: GIO timeout remains @@ -609,51 +610,40 @@ impl Physical { if let Some(td) = self.testdev.as_deref() { let td_ptr: *const dyn BusDevice = td; use crate::dev::testdev::{TEST_DEV_BASE, TEST_DEV_SIZE}; - for i in (TEST_DEV_BASE >> 16)..((TEST_DEV_BASE + TEST_DEV_SIZE - 1) >> 16) + 1 { - self.device_map[i as usize] = td_ptr; - } + set(TEST_DEV_BASE, TEST_DEV_BASE + TEST_DEV_SIZE, td_ptr); } // Map MC registers (128KB at 0x1FA00000) - for i in (MC_BASE >> 16)..((MC_END - 1) >> 16) + 1 { - self.device_map[i as usize] = mc_ptr; - } + set(MC_BASE, MC_END, mc_ptr); // Map HPC3 (512KB at 0x1FB80000) - for i in (HPC3_BASE >> 16)..((HPC3_END - 1) >> 16) + 1 { - self.device_map[i as usize] = hpc3_ptr; - } + set(HPC3_BASE, HPC3_END, hpc3_ptr); // Map PROM (1MB at 0x1FC00000) - for i in (PROM_BASE >> 16)..((PROM_END - 1) >> 16) + 1 { - self.device_map[i as usize] = prom_ptr; - } + set(PROM_BASE, PROM_END, prom_ptr); // Alias: points back into Physical itself with `alias_offset()` added. // So alias accesses go: AliasBus → Physical::read/write(addr + LOMEM_BASE) - // → device_map lookup → whichever bank is mapped at LOMEM_BASE. + // → table lookup → whichever bank is mapped at LOMEM_BASE. // This way alias automatically tracks whatever MEMCFG maps at LOMEM_BASE. self.alias_bus.target = self as *const Physical as *const dyn BusDevice; let alias_ptr: *const dyn BusDevice = &self.alias_bus; - for i in (ALIAS_BASE >> 16)..(ALIAS_END >> 16) { - self.device_map[i as usize] = alias_ptr; - } + set(ALIAS_BASE, ALIAS_END, alias_ptr); // VINO GIO alias: 0x1F080000 → 0x00080000 - // Routes through Physical again with 0xFF000000 added (wrapping subtraction of 0x1F000000) + // Routes through Physical again with 0xE1000000 added (wrapping subtraction of 0x1F000000) // so the re-dispatched address falls into VINO's primary slot at 0x0008xxxx. self.vino_gio_alias.target = self as *const Physical as *const dyn BusDevice; let vino_gio_alias_ptr: *const dyn BusDevice = &self.vino_gio_alias; - self.device_map[(0x1F080000u32 >> 16) as usize] = vino_gio_alias_ptr; - + set(0x1F080000, 0x1F090000, vino_gio_alias_ptr); } - /// Remap memory banks in device_map. + /// Remap memory banks in the decode table. /// /// `bank_addrs[i]` is `Some((base, addr_mask, limit))` if bank i is valid, or `None`. /// - `base` — physical base address /// - `addr_mask` — applied inside Memory for aliasing (SIMM wrapping); equals size_mb*1MB-1 - /// - `limit` — number of bytes to map in the device_map; slots beyond limit stay as + /// - `limit` — number of bytes to map in the decode table; slots beyond limit stay as /// UnmappedRam and return 0, matching the hardware behaviour of reads past /// the real SIMM boundary /// @@ -740,27 +730,38 @@ impl Physical { } } - // Now point the table at the banks, whose masks are set, storing only - // the slots whose contents actually change. The DMA worker can be - // dispatching through this table right now, and each store is a - // non-atomic 16-byte fat pointer; see `plan_bank_slots`. + // Now point the table at the banks, whose masks are set. The plan + // is one entry per slot; apply it as runs of one target, and leave + // a run alone that already answers as planned. The DMA worker can be + // dispatching through the table right now: a run that does change + // is swapped in whole pages by `SparseDeviceMap`, so it sees the old + // page or the new one, never a torn pointer. See `plan_bank_slots`. let (plan, outside) = plan_bank_slots(&bank_addrs, &self.banks_outside_windows); self.banks_outside_windows = outside; - for (idx, target) in plan { + let mut i = 0; + while i < plan.len() { + let (start, target) = plan[i]; + let mut j = i + 1; + while j < plan.len() && plan[j].1 == target && plan[j].0 == plan[j - 1].0 + 1 { + j += 1; + } let ptr = match target { BankSlot::Unmapped => unmapped_ptr, BankSlot::Bank(b) => bank_ptrs[b], }; - let slot = &mut self.device_map[idx as usize]; - if !std::ptr::addr_eq(*slot, ptr) { - *slot = ptr; + let count = (j - i) as u64; + if !self.devmap.all_are(start as u64, count, ptr) { + self.devmap.fill(start as u64, count, ptr).unwrap_or_else(|e| { + panic!("physical: remapping slots {start:#x}+{count:#x}: {e}") + }); } + i = j; } // ppmem: the low-512KB alias of bank 0 (MC spec — the bottom 512KB // mirrors 0x08000000..0x0807ffff). Mapped as real pages rather than // routed through AliasBus, so it is the same physical memory with no - // re-dispatch. AliasBus stays installed in device_map as the bus-path + // re-dispatch. AliasBus stays installed in the decode table as the bus-path // equivalent; both see identical memory. if let (true, Some(sp)) = (ppmem_placement_changed, &self.ppmem_space) { let bank0_mapped = bank_addrs[0].is_some_and(|(base, _, _)| base == self.alias_offset); @@ -795,6 +796,46 @@ impl Physical { } } + /// The device behind `addr`: one indexed load from the decode table. + #[inline(always)] + fn dev_at(&self, addr: u32) -> Entry { + // SAFETY: `map_base` spans the whole table for the life of `devmap`; + // a page nobody filled is faulted in by devmap's handler, so every + // load lands on a readable page holding a live device pointer. + unsafe { *self.map_base.add((addr >> SLOT_SHIFT) as usize) } + } + + /// Name a decode-table entry for the monitor. + fn device_name(&self, dev: Entry) -> String { + fn e(t: &T) -> Entry { t } + let mut known: Vec<(Entry, String)> = vec![ + (e(&self.cpu_bus_error), "cpu bus error".into()), + (e(&self.gio_bus_error), "gio timeout".into()), + (e(&self.unmapped_ram), "unmapped ram (reads 0)".into()), + (e(&self.alias_bus), "low-memory alias".into()), + (e(&self.vino_gio_alias), "vino gio alias".into()), + (e(&self.black_hole), "black hole".into()), + (e(&self.vino), "vino".into()), + (e(&self.mc), "mc".into()), + (e(&self.hpc3), "hpc3".into()), + (e(&self.prom), "prom".into()), + ]; + for (i, b) in self.banks.iter().enumerate() { + known.push((e(b), format!("bank {i}"))); + } + if let Some(d) = self.rex3.as_deref() { known.push((e(d), "newport (rex3)".into())); } + if let Some(d) = self.rex3_head1.as_deref() { known.push((e(d), "newport head 1".into())); } + if let Some(d) = self.gr2.as_deref() { known.push((e(d), "gr2".into())); } + if let Some(d) = self.mgras.as_deref() { known.push((e(d), "impact (mgras)".into())); } + if let Some(d) = self.ultra64.as_deref() { known.push((e(d), "ultra64".into())); } + if let Some(d) = self.testdev.as_deref() { known.push((e(d), "test device".into())); } + known + .into_iter() + .find(|(p, _)| entry_eq(*p, dev)) + .map(|(_, n)| n) + .unwrap_or_else(|| format!("{:p}", dev as *const ())) + } + /// Is `addr` backed by a whole directly-mapped 64MB region? /// /// This is the quick test the design calls for: one load of the bitmap, @@ -874,7 +915,7 @@ impl Device for Physical { fn register_commands(&self) -> Vec<(String, String)> { let cmds = vec![ - ("phys".to_string(), "Physical Bus commands: mem, dis, trace, error , hole , bench".to_string()), + ("phys".to_string(), "Physical Bus commands: mem, dis, map, trace, error , hole , bench".to_string()), ("mm".to_string(), "Alias for phys mem".to_string()), ("md".to_string(), "Alias for phys dis".to_string()), ("trace".to_string(), "Enable/disable tracing: trace ".to_string()), @@ -923,6 +964,27 @@ impl Device for Physical { if r.is_ok() { writeln!(writer, "{}", mips_dis::disassemble(r.data, addr, None)).unwrap(); } else { writeln!(writer, "Could not fetch instruction at {:016x}", addr).unwrap(); } } + "map" => { + let s = self.devmap.stats(); + writeln!(writer, + "decode table: {} pages of {} KB ({} MB of guest space each): {} untouched, {} shared, {} private", + s.pages, s.page_bytes >> 10, ((s.page_bytes as u64 / 16) << SLOT_SHIFT) >> 20, + s.unmapped, s.shared, s.private).unwrap(); + writeln!(writer, + " {} constant page(s), {} pool page(s) ({} free), {} KB in use, {} fault(s)", + s.const_pages, s.pool_pages, s.pool_free, s.in_use_bytes() >> 10, s.faults).unwrap(); + for a in &s.first_faults { + writeln!(writer, " faulted in: {:#012x}", a).unwrap(); + } + for r in self.devmap.runs() { + let name = match r.dev { + Some(d) => self.device_name(d), + None => "(untouched)".to_string(), + }; + writeln!(writer, " {:#012x}-{:#012x} {}", + r.start << SLOT_SHIFT, (r.end << SLOT_SHIFT) - 1, name).unwrap(); + } + } "error" => { if actual_args.is_empty() { return Err("Usage: error ".to_string()); @@ -998,7 +1060,7 @@ impl Device for Physical { impl BusDevice for Physical { #[inline(always)] fn read8(&self, addr: u32) -> BusRead8 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); let r = unsafe { (*device_ptr).read8(addr) }; #[cfg(not(feature = "lightning"))] if self.trace.load(Ordering::Relaxed) { @@ -1010,7 +1072,7 @@ impl BusDevice for Physical { #[inline(always)] fn write8(&self, addr: u32, val: u8) -> u32 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); let ws = unsafe { (*device_ptr).write8(addr, val) }; #[cfg(not(feature = "lightning"))] if self.trace.load(Ordering::Relaxed) { println!("PHYS8 Write {:08x} val={:02x} -> {:08x}", addr, val, ws); } @@ -1019,7 +1081,7 @@ impl BusDevice for Physical { #[inline(always)] fn read16(&self, addr: u32) -> BusRead16 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); let r = unsafe { (*device_ptr).read16(addr) }; #[cfg(not(feature = "lightning"))] if self.trace.load(Ordering::Relaxed) { @@ -1031,7 +1093,7 @@ impl BusDevice for Physical { #[inline(always)] fn write16(&self, addr: u32, val: u16) -> u32 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); let ws = unsafe { (*device_ptr).write16(addr, val) }; #[cfg(not(feature = "lightning"))] if self.trace.load(Ordering::Relaxed) { println!("PHYS16 Write {:08x} val={:04x} -> {:08x}", addr, val, ws); } @@ -1040,7 +1102,7 @@ impl BusDevice for Physical { #[inline(always)] fn read32(&self, addr: u32) -> BusRead32 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); let r = unsafe { (*device_ptr).read32(addr) }; #[cfg(not(feature = "lightning"))] if self.trace.load(Ordering::Relaxed) { @@ -1052,7 +1114,7 @@ impl BusDevice for Physical { #[inline(always)] fn write32(&self, addr: u32, val: u32) -> u32 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); let ws = unsafe { (*device_ptr).write32(addr, val) }; #[cfg(not(feature = "lightning"))] if self.trace.load(Ordering::Relaxed) { println!("PHYS32 Write {:08x} val={:08x} -> {:08x}", addr, val, ws); } @@ -1061,7 +1123,7 @@ impl BusDevice for Physical { #[inline(always)] fn read64(&self, addr: u32) -> BusRead64 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); let r = unsafe { (*device_ptr).read64(addr) }; #[cfg(not(feature = "lightning"))] if self.trace.load(Ordering::Relaxed) { @@ -1073,7 +1135,7 @@ impl BusDevice for Physical { #[inline(always)] fn write64(&self, addr: u32, val: u64) -> u32 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); let ws = unsafe { (*device_ptr).write64(addr, val) }; #[cfg(not(feature = "lightning"))] if self.trace.load(Ordering::Relaxed) { println!("PHYS64 Write {:08x} val={:016x} -> {:08x}", addr, val, ws); } @@ -1082,7 +1144,7 @@ impl BusDevice for Physical { #[inline(always)] fn write64_masked(&self, addr: u32, val: u64, mask: u64) -> u32 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); unsafe { (*device_ptr).write64_masked(addr, val, mask) } } @@ -1093,13 +1155,13 @@ impl BusDevice for Physical { // any DMA-specific override (e.g. Rex3::dma_read64) entirely. #[inline(always)] fn dma_read64(&self, addr: u32) -> BusRead64 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); unsafe { (*device_ptr).dma_read64(addr) } } #[inline(always)] fn dma_write64(&self, addr: u32, val: u64) -> u32 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); unsafe { (*device_ptr).dma_write64(addr, val) } } @@ -1110,13 +1172,13 @@ impl BusDevice for Physical { // (Rex3's single-token push) actually gets a chance to run. #[inline(always)] fn dma_write64_bulk(&self, addr: u32, vals: &[u64]) -> u32 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); unsafe { (*device_ptr).dma_write64_bulk(addr, vals) } } #[inline(always)] fn dma_read64_bulk(&self, addr: u32, out: &mut [u64]) -> u32 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); unsafe { (*device_ptr).dma_read64_bulk(addr, out) } } @@ -1130,7 +1192,7 @@ impl BusDevice for Physical { if let Some(p) = self.ppmem_gen_ptr(addr) { return p; } - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); unsafe { (*device_ptr).gen_ptr(addr) } } @@ -1145,7 +1207,7 @@ impl BusDevice for Physical { if let Some(p) = self.ppmem_ptr(addr) { return Some(p as *const u64); } - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); unsafe { (*device_ptr).mem_ptr(addr) } } @@ -1161,7 +1223,7 @@ impl BusDevice for Physical { } return BUS_OK; } - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); unsafe { (*device_ptr).read_block(addr, buf) } } @@ -1179,7 +1241,7 @@ impl BusDevice for Physical { self.ppmem_bump_gen_range(addr, buf.len()); return BUS_OK; } - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); unsafe { (*device_ptr).write_block(addr, buf) } } } @@ -1405,7 +1467,7 @@ mod bank_plan_tests { Some((base, size - 1, size)) } - /// `device_map` in miniature: apply a plan the way `remap_banks` does and + /// The decode table in miniature: apply a plan the way `remap_banks` does and /// count the stores that actually happen. fn apply(map: &mut [BankSlot], plan: &[(u32, BankSlot)]) -> usize { let mut stores = 0; diff --git a/src/ppmem/devmap.rs b/src/ppmem/devmap.rs new file mode 100644 index 00000000..2870aaff --- /dev/null +++ b/src/ppmem/devmap.rs @@ -0,0 +1,915 @@ +//! A sparse, host-MMU-backed physical address decode table. +//! +//! `Physical` finds the device behind a physical address with one indexed +//! load: `table[addr >> 16]`, a `*const dyn BusDevice` per 64 KB. As a plain +//! array that is 1 MB for a 32-bit space and 256 MB for the 40-bit space an +//! Octane has, almost all of it the same few pointers repeated. This keeps +//! the one-load lookup and makes the array virtual: +//! +//! - **Reserved, not allocated.** The whole table is one `AddrSpace` +//! reservation (`PROT_NONE` / a placeholder). A page nobody has written is +//! not backed by anything. +//! - **Shared where it repeats.** A table page whose entries are all the same +//! pointer is a view of one shared "constant page" for that pointer. Four +//! banks and three error sinks cost seven pages however much address space +//! they cover. +//! - **Built off-line, swapped in whole.** A mixed page lives in a pool of +//! shared-memory pages. To change any entry the new contents are written +//! into a fresh pool page through a private staging view, then mapped over +//! the table page with `MAP_FIXED`. Readers on other threads (the MC's DMA +//! worker dispatches through this table) see the old page or the new one, +//! never a torn 16-byte fat pointer: the table is never written in place, +//! and every table view is read-only so that cannot happen by accident. +//! - **Faults fill in the rest.** A load from a page that was never written +//! is a host access violation. The handler installed here recognises the +//! reservation, maps that page to the table's default device (the +//! bus-error sink), prints the guest address once, and resumes the load. +//! So an unmapped region costs nothing until the guest probes it, and a +//! probe costs one fault. +//! +//! One table page is 64 KB (the Windows allocation granularity, and a +//! multiple of every Unix page size), 4096 entries, 256 MB of guest address +//! space. Every mapping is exactly one page, which is what the Windows +//! placeholder backend in `map_windows.rs` can split and replace. +//! +//! Who may call what: `fill` and `set_default` are for the owner's setup and +//! MEMCFG remaps, on the CPU thread; the fault path may run on any thread +//! that dispatches through the table. Both take the table's spin lock, which +//! is held only across mapping calls, never across anything that can block +//! or fault. + +use super::map::{granularity, AddrSpace, Prot, SharedMem}; +use crate::traits::BusDevice; +use std::cell::UnsafeCell; +use std::io; +use std::sync::atomic::{AtomicBool, AtomicPtr, AtomicU64, Ordering}; + +/// One table entry: the device that answers a 64 KB slot. +pub type Entry = *const dyn BusDevice; + +/// Width of the physical address space the table covers. +pub const PHYS_BITS: u32 = 40; +/// Bytes of guest address space per entry: 64 KB. +pub const SLOT_SHIFT: u32 = 16; +/// Entries in the table. +pub const SLOTS: u64 = 1 << (PHYS_BITS - SLOT_SHIFT); + +const ENTRY_BYTES: usize = std::mem::size_of::(); +const TABLE_BYTES: usize = SLOTS as usize * ENTRY_BYTES; +/// A table page is at least this, whatever the host page size. +const MIN_PAGE_BYTES: usize = 64 * 1024; +/// Pool pages come in objects of this many pages. +const POOL_PAGES: usize = 16; +/// How many faulting addresses are remembered for the monitor. +const FIRST_FAULTS: usize = 8; + +const _: () = assert!(ENTRY_BYTES == 2 * std::mem::size_of::(), "fat pointer layout"); + +/// Compare both halves of a fat pointer. Comparing only the data address is +/// wrong here: a zero-sized device (`UnmappedRam`) shares its address with +/// whatever field follows it, and only the vtable tells them apart. +#[inline] +fn same(a: Entry, b: Entry) -> bool { + // SAFETY: `Entry` is two words, asserted above; reading them as integers + // is the comparison `std::ptr::eq` would do on a stable `ptr_metadata`. + unsafe { + std::mem::transmute::(a) == std::mem::transmute::(b) + } +} + +/// Do two entries name the same device through the same vtable? +pub fn entry_eq(a: Entry, b: Entry) -> bool { + same(a, b) +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum PageState { + /// Never written: reserved, faults on access. + Unmapped, + /// A view of constant page `consts[k]`. + Const(u32), + /// A view of pool page `slot` (object `slot / POOL_PAGES`, page within it + /// `slot % POOL_PAGES`). + Private(u32), +} + +struct ConstObj { + dev: Entry, + mem: SharedMem, +} + +struct State { + pages: Vec, + consts: Vec, + pool: Vec, + free: Vec, + /// One page of private address space the mutator writes new page contents + /// through. Mapped read-write over whichever pool or constant page is + /// being built; the table's own views stay read-only. + staging: AddrSpace, + /// The device an untouched slot resolves to: what the fault handler maps, + /// and what fills the rest of a page that is written only in part. The + /// constant page for it is created by `set_default`, so the fault path + /// allocates nothing. + default: Option, +} + +struct Inner { + table: AddrSpace, + page_bytes: usize, + per_page: usize, + lock: AtomicBool, + st: UnsafeCell, + faults: AtomicU64, + /// Guest addresses of the first faults, `u64::MAX` where empty. + first_faults: [AtomicU64; FIRST_FAULTS], + /// Index in the fault registry, so `Drop` can clear it. + reg: usize, +} + +/// The table. See the module documentation. +pub struct SparseDeviceMap { + // Boxed so the fault registry can hold a stable pointer while the owner + // moves the map into place. + inner: Box, +} + +// SAFETY: the raw pointers inside are device addresses the owner keeps alive +// for as long as the table; `State` is guarded by `lock`. +unsafe impl Send for SparseDeviceMap {} +unsafe impl Sync for SparseDeviceMap {} + +/// One run of the table with one answer, for the monitor. +#[derive(Clone, Copy, Debug)] +pub struct Run { + /// First slot (guest address `>> SLOT_SHIFT`). + pub start: u64, + /// One past the last slot. + pub end: u64, + /// The device, or `None` for a run of pages nobody has written. + pub dev: Option, +} + +/// Counts for the monitor. +#[derive(Clone, Debug, Default)] +pub struct MapStats { + pub page_bytes: usize, + pub pages: usize, + pub unmapped: usize, + pub shared: usize, + pub private: usize, + pub const_pages: usize, + pub pool_pages: usize, + pub pool_free: usize, + pub faults: u64, + pub first_faults: Vec, +} + +impl MapStats { + /// Host memory the table has in use: every constant page, every pool + /// page a table page views, and the staging page. Free pool pages are + /// address space in a shared object, not memory, until written. + pub fn in_use_bytes(&self) -> usize { + (self.const_pages + self.pool_pages - self.pool_free + 1) * self.page_bytes + } +} + +struct Guard<'a>(&'a AtomicBool); + +impl Drop for Guard<'_> { + fn drop(&mut self) { + self.0.store(false, Ordering::Release); + } +} + +impl Inner { + fn lock(&self) -> Guard<'_> { + while self + .lock + .compare_exchange_weak(false, true, Ordering::Acquire, Ordering::Relaxed) + .is_err() + { + std::hint::spin_loop(); + } + Guard(&self.lock) + } + + /// As `lock`, but gives up: the fault handler must not hang a thread + /// forever if it ever runs with the lock already held. + fn try_lock_spinning(&self, spins: u32) -> Option> { + for _ in 0..spins { + if self + .lock + .compare_exchange_weak(false, true, Ordering::Acquire, Ordering::Relaxed) + .is_ok() + { + return Some(Guard(&self.lock)); + } + std::hint::spin_loop(); + } + None + } + + #[inline] + fn entries(&self) -> *const Entry { + self.table.base() as *const Entry + } + + fn npages(&self) -> usize { + TABLE_BYTES / self.page_bytes + } + + /// Map table page `page` as a read-only view of `mem[off..off+page]`. + unsafe fn view(&self, page: usize, mem: &SharedMem, off: usize) -> io::Result<()> { + unsafe { self.table.map(page * self.page_bytes, self.page_bytes, mem, off as u64, Prot::ReadOnly) } + } + + /// Write `buf` into `mem[off..]` through the staging view. + unsafe fn write_through_staging(&self, st: &State, mem: &SharedMem, off: usize, buf: &[Entry]) -> io::Result<()> { + debug_assert_eq!(buf.len(), self.per_page); + unsafe { + st.staging.map(0, self.page_bytes, mem, off as u64, Prot::ReadWrite)?; + std::ptr::copy_nonoverlapping(buf.as_ptr(), st.staging.base() as *mut Entry, self.per_page); + } + Ok(()) + } + + /// Index of the constant page for `dev`, created on first use. + fn const_for(&self, st: &mut State, dev: Entry) -> io::Result { + if let Some(k) = st.consts.iter().position(|c| same(c.dev, dev)) { + return Ok(k as u32); + } + let mem = SharedMem::new(self.page_bytes)?; + let buf = vec![dev; self.per_page]; + unsafe { self.write_through_staging(st, &mem, 0, &buf)? }; + st.consts.push(ConstObj { dev, mem }); + Ok((st.consts.len() - 1) as u32) + } + + fn alloc_slot(&self, st: &mut State) -> io::Result { + if let Some(s) = st.free.pop() { + return Ok(s); + } + let obj = st.pool.len() as u32; + st.pool.push(SharedMem::new(POOL_PAGES * self.page_bytes)?); + // Hand out low pages first; the order only matters for the tests. + for p in (0..POOL_PAGES as u32).rev() { + st.free.push(obj * POOL_PAGES as u32 + p); + } + Ok(st.free.pop().unwrap()) + } + + fn release(&self, st: &mut State, old: PageState) { + if let PageState::Private(s) = old { + st.free.push(s); + } + } + + /// Point a whole page at `dev`. + fn map_const(&self, st: &mut State, page: usize, dev: Entry) -> io::Result<()> { + let k = self.const_for(st, dev)?; + let old = st.pages[page]; + if old == PageState::Const(k) { + return Ok(()); + } + unsafe { self.view(page, &st.consts[k as usize].mem, 0)? }; + st.pages[page] = PageState::Const(k); + self.release(st, old); + Ok(()) + } + + /// Current contents of `page` as the mutator sees them: what is mapped, + /// or the default for a page nobody has written. + fn contents(&self, st: &State, page: usize) -> io::Result> { + Ok(match st.pages[page] { + PageState::Unmapped => { + let Some(k) = st.default else { + return Err(io::Error::other( + "devmap: a partial write to an untouched page needs a default device", + )); + }; + vec![st.consts[k as usize].dev; self.per_page] + } + PageState::Const(k) => vec![st.consts[k as usize].dev; self.per_page], + PageState::Private(_) => { + let mut v = Vec::with_capacity(self.per_page); + // SAFETY: a private page is mapped and readable; only this + // thread (lock held) replaces views. + unsafe { + let p = self.entries().add(page * self.per_page); + v.extend_from_slice(std::slice::from_raw_parts(p, self.per_page)); + } + v + } + }) + } + + /// Set entries `lo..hi` of `page` to `dev`. + fn modify(&self, st: &mut State, page: usize, lo: usize, hi: usize, dev: Entry) -> io::Result<()> { + let mut buf = self.contents(st, page)?; + if buf[lo..hi].iter().all(|&e| same(e, dev)) && st.pages[page] != PageState::Unmapped { + return Ok(()); + } + buf[lo..hi].fill(dev); + let first = buf[0]; + if buf.iter().all(|&e| same(e, first)) { + return self.map_const(st, page, first); + } + let slot = self.alloc_slot(st)?; + let (obj, idx) = (slot as usize / POOL_PAGES, slot as usize % POOL_PAGES); + let off = idx * self.page_bytes; + let r = unsafe { + self.write_through_staging(st, &st.pool[obj], off, &buf) + .and_then(|()| self.view(page, &st.pool[obj], off)) + }; + if let Err(e) = r { + st.free.push(slot); + return Err(e); + } + let old = st.pages[page]; + st.pages[page] = PageState::Private(slot); + self.release(st, old); + Ok(()) + } + + /// The fault path. `addr` is the host address that faulted. Returns true + /// if it was inside this table and the page is now mapped, so the access + /// can be retried. + fn fault(&self, addr: usize) -> bool { + let base = self.table.base() as usize; + if addr < base || addr >= base + TABLE_BYTES { + return false; + } + let page = (addr - base) / self.page_bytes; + let slot = ((addr - base) / ENTRY_BYTES) as u64; + // The lock is only ever held across mapping calls, which do not + // touch unmapped table pages, so it is never held by the faulting + // thread; the bound is a backstop against hanging in a handler. + let Some(_g) = self.try_lock_spinning(50_000_000) else { + fault_message(b"devmap: fault while the table lock is held; giving up\n", slot << SLOT_SHIFT); + return false; + }; + // SAFETY: lock held. + let st = unsafe { &mut *self.st.get() }; + if st.pages[page] != PageState::Unmapped { + // Another thread mapped it between the fault and the lock. + return true; + } + let Some(k) = st.default else { + fault_message(b"devmap: unmapped physical region and no default device at ", slot << SLOT_SHIFT); + return false; + }; + if unsafe { self.view(page, &st.consts[k as usize].mem, 0) }.is_err() { + fault_message(b"devmap: could not map the default device at ", slot << SLOT_SHIFT); + return false; + } + st.pages[page] = PageState::Const(k); + drop(_g); + let n = self.faults.fetch_add(1, Ordering::Relaxed) as usize; + if n < FIRST_FAULTS { + self.first_faults[n].store(slot << SLOT_SHIFT, Ordering::Relaxed); + } + fault_message(b"iris: physical access to an unmapped region at ", slot << SLOT_SHIFT); + true + } +} + +impl SparseDeviceMap { + /// Reserve the table. Nothing is mapped yet; set a default before the + /// first dispatch. + pub fn new() -> io::Result { + let page_bytes = granularity().max(MIN_PAGE_BYTES); + let table = AddrSpace::reserve(TABLE_BYTES)?; + let staging = AddrSpace::reserve(page_bytes)?; + let npages = TABLE_BYTES / page_bytes; + let inner = Box::new(Inner { + table, + page_bytes, + per_page: page_bytes / ENTRY_BYTES, + lock: AtomicBool::new(false), + st: UnsafeCell::new(State { + pages: vec![PageState::Unmapped; npages], + consts: Vec::new(), + pool: Vec::new(), + free: Vec::new(), + staging, + default: None, + }), + faults: AtomicU64::new(0), + first_faults: [const { AtomicU64::new(u64::MAX) }; FIRST_FAULTS], + reg: usize::MAX, + }); + let mut map = Self { inner }; + fault::install(); + map.inner.reg = fault::register(&*map.inner)?; + Ok(map) + } + + /// Base of the table: entry `i` is at `base().add(i)`. Valid for the + /// life of the map; the owner caches it for the hot path. + #[inline] + pub fn base(&self) -> *const Entry { + self.inner.entries() + } + + /// Entries per table page. + pub fn page_slots(&self) -> u64 { + self.inner.per_page as u64 + } + + /// The device an untouched slot answers with. Required before a partial + /// write to a page nobody has written, and before any dispatch may fault. + pub fn set_default(&self, dev: Entry) -> io::Result<()> { + let _g = self.inner.lock(); + let st = unsafe { &mut *self.inner.st.get() }; + let k = self.inner.const_for(st, dev)?; + st.default = Some(k); + Ok(()) + } + + /// Point slots `first..first + count` at `dev`. + pub fn fill(&self, first: u64, count: u64, dev: Entry) -> io::Result<()> { + let end = first.checked_add(count).filter(|&e| e <= SLOTS).unwrap_or_else(|| { + panic!("devmap: slots [{first:#x}, +{count:#x}) escape the {PHYS_BITS}-bit space") + }); + let per = self.inner.per_page as u64; + let _g = self.inner.lock(); + let st = unsafe { &mut *self.inner.st.get() }; + let mut slot = first; + while slot < end { + let page = (slot / per) as usize; + let page_start = page as u64 * per; + let hi = end.min(page_start + per); + if slot == page_start && hi == page_start + per { + self.inner.map_const(st, page, dev)?; + } else { + self.inner.modify(st, page, (slot - page_start) as usize, (hi - page_start) as usize, dev)?; + } + slot = hi; + } + Ok(()) + } + + /// Point the slots covering guest bytes `start..end` at `dev`. `end` is + /// exclusive and rounded up to a slot boundary. + pub fn fill_bytes(&self, start: u64, end: u64, dev: Entry) -> io::Result<()> { + assert!(end > start, "devmap: empty range {start:#x}..{end:#x}"); + let first = start >> SLOT_SHIFT; + let last = (end - 1) >> SLOT_SHIFT; + self.fill(first, last - first + 1, dev) + } + + /// Whether the page holding `slot` has been written (or faulted in). + pub fn is_mapped(&self, slot: u64) -> bool { + let _g = self.inner.lock(); + let st = unsafe { &*self.inner.st.get() }; + st.pages[(slot / self.inner.per_page as u64) as usize] != PageState::Unmapped + } + + /// The entry at `slot`, without faulting an untouched page in. + pub fn get(&self, slot: u64) -> Option { + if !self.is_mapped(slot) { + return None; + } + // SAFETY: mapped pages stay mapped (views are replaced, never + // removed), and the load is one the hot path does without a lock. + Some(unsafe { *self.inner.entries().add(slot as usize) }) + } + + /// Whether every slot in `first..first + count` already answers with + /// `dev`. Untouched pages count as not matching. + pub fn all_are(&self, first: u64, count: u64, dev: Entry) -> bool { + let per = self.inner.per_page as u64; + let end = first + count; + let _g = self.inner.lock(); + let st = unsafe { &*self.inner.st.get() }; + let mut slot = first; + while slot < end { + let page = (slot / per) as usize; + let hi = end.min((page as u64 + 1) * per); + match st.pages[page] { + PageState::Unmapped => return false, + PageState::Const(k) => { + if !same(st.consts[k as usize].dev, dev) { + return false; + } + } + PageState::Private(_) => unsafe { + let p = self.inner.entries(); + for s in slot..hi { + if !same(*p.add(s as usize), dev) { + return false; + } + } + }, + } + slot = hi; + } + true + } + + /// The table as runs of equal entries, untouched pages included as + /// `dev: None`. + pub fn runs(&self) -> Vec { + let per = self.inner.per_page as u64; + let _g = self.inner.lock(); + let st = unsafe { &*self.inner.st.get() }; + let mut out: Vec = Vec::new(); + let mut push = |start: u64, end: u64, dev: Option| { + if let Some(last) = out.last_mut() { + let joins = match (last.dev, dev) { + (None, None) => true, + (Some(a), Some(b)) => same(a, b), + _ => false, + }; + if joins && last.end == start { + last.end = end; + return; + } + } + out.push(Run { start, end, dev }); + }; + for (page, &state) in st.pages.iter().enumerate() { + let start = page as u64 * per; + match state { + PageState::Unmapped => push(start, start + per, None), + PageState::Const(k) => push(start, start + per, Some(st.consts[k as usize].dev)), + PageState::Private(_) => unsafe { + let p = self.inner.entries().add(start as usize); + let mut run_start = start; + let mut cur = *p; + for i in 1..per { + let e = *p.add(i as usize); + if !same(e, cur) { + push(run_start, start + i, Some(cur)); + run_start = start + i; + cur = e; + } + } + push(run_start, start + per, Some(cur)); + }, + } + } + out + } + + pub fn stats(&self) -> MapStats { + let _g = self.inner.lock(); + let st = unsafe { &*self.inner.st.get() }; + let mut s = MapStats { + page_bytes: self.inner.page_bytes, + pages: st.pages.len(), + const_pages: st.consts.len(), + pool_pages: st.pool.len() * POOL_PAGES, + pool_free: st.free.len(), + faults: self.inner.faults.load(Ordering::Relaxed), + ..Default::default() + }; + for p in &st.pages { + match p { + PageState::Unmapped => s.unmapped += 1, + PageState::Const(_) => s.shared += 1, + PageState::Private(_) => s.private += 1, + } + } + s.first_faults = self + .inner + .first_faults + .iter() + .map(|a| a.load(Ordering::Relaxed)) + .take_while(|&a| a != u64::MAX) + .collect(); + s + } +} + +impl Drop for SparseDeviceMap { + fn drop(&mut self) { + fault::unregister(self.inner.reg); + } +} + +/// Print `what` followed by the guest address, from a fault handler: a fixed +/// buffer and one raw write, no allocation and no locks. +fn fault_message(what: &[u8], phys: u64) { + let mut buf = [0u8; 160]; + let mut n = 0; + let mut push = |b: &[u8]| { + let take = b.len().min(buf.len() - n); + buf[n..n + take].copy_from_slice(&b[..take]); + n += take; + }; + push(what); + push(b"0x"); + let digits = (PHYS_BITS as usize).div_ceil(4); + let mut hex = [0u8; 16]; + for (i, d) in hex.iter_mut().take(digits).enumerate() { + let nib = (phys >> (4 * (digits - 1 - i))) & 0xF; + *d = b"0123456789abcdef"[nib as usize]; + } + push(&hex[..digits]); + push(b" (64 KB slot now answers as the bus-error device)\n"); + fault::emit(&buf[..n]); +} + +/// Process-wide: the host fault handler and the registry of live tables it +/// consults. +mod fault { + use super::Inner; + use std::io; + use std::sync::atomic::{AtomicPtr, Ordering}; + use std::sync::Once; + + // Generous: a test binary builds many machines at once, each with a table. + const MAX_MAPS: usize = 64; + static REGISTRY: [AtomicPtr; MAX_MAPS] = [const { AtomicPtr::new(std::ptr::null_mut()) }; MAX_MAPS]; + static INSTALL: Once = Once::new(); + + pub(super) fn register(inner: *const Inner) -> io::Result { + for (i, slot) in REGISTRY.iter().enumerate() { + if slot + .compare_exchange(std::ptr::null_mut(), inner as *mut Inner, Ordering::AcqRel, Ordering::Relaxed) + .is_ok() + { + return Ok(i); + } + } + Err(io::Error::other(format!("devmap: more than {MAX_MAPS} tables alive"))) + } + + pub(super) fn unregister(i: usize) { + if i < MAX_MAPS { + REGISTRY[i].store(std::ptr::null_mut(), Ordering::Release); + } + } + + /// Offer a faulting host address to every live table. + pub(super) fn dispatch(addr: usize) -> bool { + REGISTRY.iter().any(|slot| { + let p = slot.load(Ordering::Acquire); + // SAFETY: a registered table stays alive until it unregisters in + // `Drop`, and nothing dispatches through a table being dropped. + !p.is_null() && unsafe { (*p).fault(addr) } + }) + } + + pub(super) fn install() { + INSTALL.call_once(imp::install); + } + + pub(super) fn emit(bytes: &[u8]) { + imp::emit(bytes); + } + + #[cfg(unix)] + mod imp { + //! `SIGSEGV` (Linux, and macOS for unmapped pages) and `SIGBUS` + //! (macOS for `PROT_NONE` pages). Whatever handler was installed + //! before — Rust's own stack-overflow detector — is chained to for + //! every fault that is not ours, so nothing else changes. + use std::cell::UnsafeCell; + + struct Prev(UnsafeCell<[libc::sigaction; 2]>); + // SAFETY: written once under `INSTALL`, read only from the handler. + unsafe impl Sync for Prev {} + static PREV: Prev = Prev(UnsafeCell::new([unsafe { std::mem::zeroed() }; 2])); + + const SIGNALS: [libc::c_int; 2] = [libc::SIGSEGV, libc::SIGBUS]; + + unsafe extern "C" fn handler(sig: libc::c_int, info: *mut libc::siginfo_t, ctx: *mut libc::c_void) { + if !info.is_null() { + let addr = unsafe { (*info).si_addr() } as usize; + if super::dispatch(addr) { + return; // the load re-executes against the new page + } + } + let idx = SIGNALS.iter().position(|&s| s == sig).unwrap_or(0); + let prev = unsafe { (*PREV.0.get())[idx] }; + if prev.sa_flags & libc::SA_SIGINFO != 0 { + let f: unsafe extern "C" fn(libc::c_int, *mut libc::siginfo_t, *mut libc::c_void) = + unsafe { std::mem::transmute(prev.sa_sigaction) }; + unsafe { f(sig, info, ctx) }; + } else if prev.sa_sigaction == libc::SIG_DFL || prev.sa_sigaction == libc::SIG_IGN { + // Restore the default and return: the access repeats and the + // default action (a core dump) takes it from there. + unsafe { libc::sigaction(sig, &prev, std::ptr::null_mut()) }; + } else { + let f: unsafe extern "C" fn(libc::c_int) = unsafe { std::mem::transmute(prev.sa_sigaction) }; + unsafe { f(sig) }; + } + } + + pub(super) fn install() { + unsafe { + let mut sa: libc::sigaction = std::mem::zeroed(); + // Through `*const ()`: see `hwwatch.rs` on why not a direct cast. + sa.sa_sigaction = handler as *const () as usize; + // ONSTACK so a thread with an alternate stack keeps using it, + // as Rust's handler expects; NODEFER so a fault inside the + // chained handler is not silently blocked. + sa.sa_flags = libc::SA_SIGINFO | libc::SA_ONSTACK | libc::SA_NODEFER; + libc::sigemptyset(&mut sa.sa_mask); + for (i, &sig) in SIGNALS.iter().enumerate() { + libc::sigaction(sig, &sa, &mut (*PREV.0.get())[i]); + } + } + } + + pub(super) fn emit(bytes: &[u8]) { + unsafe { libc::write(2, bytes.as_ptr() as *const libc::c_void, bytes.len()) }; + } + } + + #[cfg(windows)] + mod imp { + //! A vectored exception handler, added at the head of the chain. + //! `crash_diag` adds its own at the head earlier in `main`, so this + //! one runs first and a handled fault never reaches its log; every + //! other exception continues down the chain untouched. + use windows_sys::Win32::System::Diagnostics::Debug::{AddVectoredExceptionHandler, EXCEPTION_POINTERS}; + + const EXCEPTION_ACCESS_VIOLATION: i32 = 0xC000_0005u32 as i32; + const EXCEPTION_CONTINUE_EXECUTION: i32 = -1; + const EXCEPTION_CONTINUE_SEARCH: i32 = 0; + + unsafe extern "system" fn handler(info: *mut EXCEPTION_POINTERS) -> i32 { + if info.is_null() { + return EXCEPTION_CONTINUE_SEARCH; + } + let rec = unsafe { (*info).ExceptionRecord }; + if rec.is_null() { + return EXCEPTION_CONTINUE_SEARCH; + } + let rec = unsafe { &*rec }; + if rec.ExceptionCode == EXCEPTION_ACCESS_VIOLATION && rec.NumberParameters >= 2 { + // ExceptionInformation[0] is the access kind, [1] the address. + let addr = rec.ExceptionInformation[1]; + if super::dispatch(addr) { + return EXCEPTION_CONTINUE_EXECUTION; + } + } + EXCEPTION_CONTINUE_SEARCH + } + + pub(super) fn install() { + unsafe { + let _ = AddVectoredExceptionHandler(1, Some(handler)); + } + } + + pub(super) fn emit(bytes: &[u8]) { + use std::io::Write; + let _ = std::io::stderr().write_all(bytes); + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A device with a body, so two of them never share an address. + struct Dummy(#[allow(dead_code)] u64); + impl BusDevice for Dummy {} + + fn dev(tag: u64) -> Entry { + Box::leak(Box::new(Dummy(tag))) as *const Dummy as Entry + } + + fn entry_at(m: &SparseDeviceMap, slot: u64) -> Entry { + unsafe { std::ptr::read_volatile(m.base().add(slot as usize)) } + } + + #[test] + fn a_uniform_fill_shares_one_constant_page() { + let m = SparseDeviceMap::new().unwrap(); + let (d0, d1) = (dev(0), dev(1)); + m.set_default(d0).unwrap(); + let per = m.page_slots(); + m.fill(0, 4 * per, d1).unwrap(); + let s = m.stats(); + assert_eq!((s.shared, s.private, s.const_pages, s.pool_pages), (4, 0, 2, 0)); + for slot in [0, per - 1, per, 3 * per + 7, 4 * per - 1] { + assert!(same(entry_at(&m, slot), d1), "slot {slot:#x}"); + } + assert!(!m.is_mapped(4 * per)); + // A second identical fill maps nothing new. + m.fill(0, 4 * per, d1).unwrap(); + assert_eq!(m.stats().const_pages, 2); + } + + #[test] + fn a_partial_fill_makes_a_private_page_and_uniformity_frees_it() { + let m = SparseDeviceMap::new().unwrap(); + let (d0, d1, d2) = (dev(0), dev(1), dev(2)); + m.set_default(d0).unwrap(); + let per = m.page_slots(); + m.fill(0, per, d1).unwrap(); + m.fill(5, 4, d2).unwrap(); + let s = m.stats(); + assert_eq!((s.shared, s.private, s.pool_pages, s.pool_free), (0, 1, POOL_PAGES, POOL_PAGES - 1)); + assert!(same(entry_at(&m, 4), d1)); + assert!(same(entry_at(&m, 5), d2)); + assert!(same(entry_at(&m, 8), d2)); + assert!(same(entry_at(&m, 9), d1)); + assert!(m.all_are(5, 4, d2)); + assert!(!m.all_are(4, 4, d2)); + // Changing one entry of a private page swaps in a fresh pool page. + m.fill(7, 1, d1).unwrap(); + assert!(same(entry_at(&m, 7), d1)); + assert_eq!(m.stats().pool_free, POOL_PAGES - 1, "the old pool page was released"); + // Back to uniform: the page becomes a constant view again. + m.fill(5, 4, d1).unwrap(); + let s = m.stats(); + assert_eq!((s.shared, s.private, s.pool_free), (1, 0, POOL_PAGES)); + assert!(same(entry_at(&m, 6), d1)); + } + + #[test] + fn a_partial_fill_of_an_untouched_page_is_padded_with_the_default() { + let m = SparseDeviceMap::new().unwrap(); + let (d0, d1) = (dev(0), dev(1)); + assert!(m.fill(10, 2, d1).is_err(), "no default yet"); + m.set_default(d0).unwrap(); + m.fill(10, 2, d1).unwrap(); + let per = m.page_slots(); + assert!(same(entry_at(&m, 0), d0)); + assert!(same(entry_at(&m, 9), d0)); + assert!(same(entry_at(&m, 10), d1)); + assert!(same(entry_at(&m, 11), d1)); + assert!(same(entry_at(&m, per - 1), d0)); + assert!(!m.is_mapped(per)); + assert_eq!(m.get(per), None); + assert!(same(m.get(10).unwrap(), d1)); + } + + #[test] + fn a_load_from_an_untouched_page_faults_in_the_default() { + let m = SparseDeviceMap::new().unwrap(); + let (d0, d1) = (dev(0), dev(1)); + m.set_default(d0).unwrap(); + let per = m.page_slots(); + m.fill(0, per, d1).unwrap(); + // Well above anything written, as an Octane probe above 4 GB would be. + let slot = (1u64 << (PHYS_BITS - SLOT_SHIFT - 1)) + 3 * per + 17; + assert!(!m.is_mapped(slot)); + let e = entry_at(&m, slot); + assert!(same(e, d0), "the faulted-in page answers with the default"); + assert!(m.is_mapped(slot)); + let s = m.stats(); + assert_eq!(s.faults, 1); + assert_eq!(s.first_faults, vec![slot << SLOT_SHIFT]); + // The page is a shared view of the default's constant page. + assert_eq!((s.shared, s.private), (2, 0)); + // Loading it again is an ordinary load. + assert!(same(entry_at(&m, slot + 1), d0)); + assert_eq!(m.stats().faults, 1); + } + + #[test] + fn runs_coalesce_across_pages_and_report_gaps() { + let m = SparseDeviceMap::new().unwrap(); + let (d0, d1, d2) = (dev(0), dev(1), dev(2)); + m.set_default(d0).unwrap(); + let per = m.page_slots(); + m.fill(0, 2 * per, d1).unwrap(); + m.fill(2 * per - 3, 5, d2).unwrap(); // straddles pages 1 and 2 + m.fill_bytes(0x1FC0_0000, 0x1FD0_0000, d0).unwrap(); // 16 slots inside page 0 + let runs = m.runs(); + let mapped: Vec<_> = runs.iter().filter(|r| r.dev.is_some()).collect(); + let starts: Vec<(u64, u64)> = mapped.iter().map(|r| (r.start, r.end)).collect(); + assert_eq!( + starts, + vec![ + (0, 0x1FC0), + (0x1FC0, 0x1FD0), + (0x1FD0, 2 * per - 3), + (2 * per - 3, 2 * per + 2), + (2 * per + 2, 3 * per), + ] + ); + assert!(same(mapped[1].dev.unwrap(), d0)); + assert!(same(mapped[3].dev.unwrap(), d2)); + assert!(same(mapped[4].dev.unwrap(), d0), "rest of page 2 is the default"); + // Everything after page 2 is one untouched run to the end. + let last = runs.last().unwrap(); + assert!(last.dev.is_none()); + assert_eq!((last.start, last.end), (3 * per, SLOTS)); + } + + #[test] + fn same_tells_zero_sized_devices_apart() { + struct A; + struct B; + impl BusDevice for A {} + impl BusDevice for B {} + #[repr(C)] + struct Both(A, B); + let both = Both(A, B); + let a: Entry = &both.0; + let b: Entry = &both.1; + assert!(std::ptr::addr_eq(a, b), "the test needs them to share an address"); + assert!(!same(a, b)); + assert!(same(a, a)); + } +} diff --git a/src/ppmem/mod.rs b/src/ppmem/mod.rs index 4bef4f04..828c7470 100644 --- a/src/ppmem/mod.rs +++ b/src/ppmem/mod.rs @@ -4,14 +4,17 @@ //! (`BusDevice`, `Resettable`, and `Memory`'s inherent methods) but backs the //! guest's physical address space with real host virtual memory, so that //! SIMM mirroring and the low-512KB alias become *mappings* rather than -//! address arithmetic on every access. +//! address arithmetic on every access. `devmap` applies the same idea to +//! the physical address decode table itself. //! //! Full design and rationale: `docs/ppmem-design.md`. //! Verified platform findings: `clanker/build/mmap-rs-fixed-address-aliasing.md`. +pub mod devmap; pub mod map; #[allow(clippy::module_inception)] pub mod ppmem; pub use map::{align_up, granularity, is_aligned, AddrSpace, Prot, SharedMem}; +pub use devmap::SparseDeviceMap; pub use ppmem::{MappedMemory, PpMemSpace, PpMemory, BITMAP_SHIFT, WINDOW_SIZE};