diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d3eb91b..dab0d0d4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -64,6 +64,19 @@ hashes are given where a change is easiest to understand by reading the commit. page in six. IRIX on an R5000 O2 panicked with "init died" for the same reason; it now boots to the desktop. +### Physical address decoding + +- **The decode table is sparse and spans 40 bits.** `Physical` used to embed + a 1 MB array of device pointers covering a 32-bit physical space. It is now + a host-mapped table over a 40-bit space (`SparseDeviceMap`, + `src/ppmem/devmap.rs`, [design](docs/devmap-design.md)): pages of one + device share one 64 KB page, mixed pages come from a pool, and a page the + guest touches that nobody mapped is faulted in as the bus-error device with + a message naming the address. An Indy's table is about ten pages. Changes + swap whole pages in, so the DMA worker dispatching through the table during + a MEMCFG remap can no longer read a torn pointer. `phys map` in the monitor + shows the table. Groundwork for machines with a 40-bit address space. + ### Memory configuration - **`banks` has eight entries.** Every machine today uses the first four and diff --git a/HACKING.md b/HACKING.md index 45306e28..6298e4ac 100644 --- a/HACKING.md +++ b/HACKING.md @@ -143,9 +143,16 @@ screenshots), `CpuDevice` (what `Machine` holds for the CPU). ### The physical address map `Physical` (`src/physical.rs`) decodes physical addresses through a table of -64 KB pages (`device_map`, one entry per `addr >> 16`), so finding the device -for an access is one indexed load. There is a lot of `unsafe` in there; it is -deliberate. The hierarchy it models: +64 KB slots (one `*const dyn BusDevice` per `addr >> 16`), so finding the +device for an access is one indexed load. The table itself is sparse and +host-mapped (`SparseDeviceMap`, `src/ppmem/devmap.rs`): it spans a 40-bit +physical space as a 256 MB reservation, pages full of one pointer are views +of one shared page, a change swaps a whole page in rather than storing into +it (so threads dispatching through it never see a torn pointer), and a page +nobody filled is faulted in as the bus-error device the first time the guest +touches it. [docs/devmap-design.md](docs/devmap-design.md) has the details. +There is a lot of `unsafe` in there; it is deliberate. The hierarchy it +models: - **SysAD**: the CPU's 64-bit bus, with byte enables. - **MC** (Memory Controller, `src/dev/mc.rs`): the crossbar. DRAM, the GIO diff --git a/README.md b/README.md index bc39d047..14bd6d98 100644 --- a/README.md +++ b/README.md @@ -152,7 +152,7 @@ Hardware, testing, and development references: | CPU test infrastructure | [Toolchain](cpu-tests/docs/toolchain.md), [memory map](cpu-tests/docs/memory-map.md), [oracle overview](cpu-tests/docs/oracle.md), [hardware oracle](cpu-tests/oracle/README.md), [R4600 notes](cpu-tests/docs/r4600.md), and [original test plan](cpu-tests/PLAN.md). | | Benchmarks and memory tests | [Benchmark suite](bench/README.md), [prebuilt assets](bench/prebuilt/README.md), [reference results](data/bench_reference.README.md), [GUI benchmark design](clanker/plans/gui-benchmark-plan.md), and [memory stress utility](test/memstress/README.md). | | Architecture and debugging | [HACKING.md](HACKING.md), [runtime debugging](debug.md), [agent instructions](clanker/CLANKER.md), and the [clanker/](clanker/INDEX.md) notes. | -| Memory and JIT design | [Persistent JIT cache](docs/jitv2-persistent-cache.md), [inline memory](docs/jit-inline-memory.md), [JIT performance analysis](clanker/plans/jitv2_performance_analysis.md), [physical RAM](docs/ppmem-design.md), [transparent cache](docs/tcache-design.md), and [historical TLB design](docs/nutlb-design.md). | +| Memory and JIT design | [Persistent JIT cache](docs/jitv2-persistent-cache.md), [inline memory](docs/jit-inline-memory.md), [JIT performance analysis](clanker/plans/jitv2_performance_analysis.md), [physical RAM](docs/ppmem-design.md), [address decode table](docs/devmap-design.md), [transparent cache](docs/tcache-design.md), and [historical TLB design](docs/nutlb-design.md). | | Plans and task briefs | [clanker/plans/](clanker/plans/): CHD sync, the NFS server, the networking GUI, the DaynaPort target, PCAP distribution, the NAT stack redesign, jitv2 performance analysis. Written as working documents; read their status notes first. | | HostGL tooling | [Khronos ABI registry tooling](iris-hostgl/tools/khronos/README.md). | diff --git a/clanker/INDEX.md b/clanker/INDEX.md index 411e3c1e..10cce2ac 100644 --- a/clanker/INDEX.md +++ b/clanker/INDEX.md @@ -97,6 +97,7 @@ Non-graphics devices: MC/MEMCFG and RAM banks, HPC3, VDMA, EEPROMs and NVRAM, PS | Note | Kind | Title | |---|---|---| | [cpal-alsa-busy-poll.md](devices/cpal-alsa-busy-poll.md) | rule | cpal's ALSA backend can spin a core: `[audio] host = "pulseaudio"` | +| [decode-table-is-host-mapped.md](devices/decode-table-is-host-mapped.md) | rule | The physical decode table is a sparse host mapping over 40 bits: change it only through `fill`, set the default first, compare entries with `entry_eq` | | [extended-ram-memcfg.md](devices/extended-ram-memcfg.md) | rule | Extended RAM — MEMCFG synthesis for himem banks | | [hpc3-rx-channel-refused-while-active.md](devices/hpc3-rx-channel-refused-while-active.md) | rule | A receive channel that refuses while reporting itself active | | [indigo2-eeprom-byte-lane-gap.md](devices/indigo2-eeprom-byte-lane-gap.md) | rule | Indigo2 motherboard EEPROM: PROM writes via byte access, HPC3 only handled 32-bit | diff --git a/clanker/devices/decode-table-is-host-mapped.md b/clanker/devices/decode-table-is-host-mapped.md new file mode 100644 index 00000000..d358cea7 --- /dev/null +++ b/clanker/devices/decode-table-is-host-mapped.md @@ -0,0 +1,50 @@ +# The decode table is host-mapped and never written in place + +## Status: landed 2026-10-24 (`src/ppmem/devmap.rs`) + +`Physical::device_map`, the 64 KB-slot array of `*const dyn BusDevice`, is +gone. The table is a `SparseDeviceMap`: a 256 MB `AddrSpace` reservation +covering 40 bits of physical address, mapped page by page from shared +constant pages (one per device pointer) and a pool of mixed pages. Design +record: `docs/devmap-design.md`. + +## Rules + +- **Change entries only through `fill` / `fill_bytes`.** Every table view is + read-only; a store through `base()` segfaults. `fill` builds the new page + off-line and swaps it in, which is what keeps concurrent DMA dispatch + safe. Do not add a `*mut` path. +- **Set the default before anything can dispatch.** `set_default` creates + the constant page the fault handler maps. Without it a fault on an + untouched page prints "no default device" and chains to the previous + handler, which kills the process. `Physical::init` does this first. +- **A partial write to an untouched page pads with the default.** Writing a + device into a region nobody filled gives you that device there and the + default in the rest of the 256 MB page. That is what a 40-bit machine + wants; it means `fill` on an untouched page fails if no default is set. +- **Pages are 64 KB on every host** (`granularity().max(64 KB)`), so one + table page covers 256 MB of guest space and every mapping is one page. + That is the Windows constraint (placeholders split on 64 KB, views cannot + be partially unmapped) applied everywhere, so Linux exercises the same + shape Windows needs. +- **Compare entries with `entry_eq`, not `addr_eq`.** `UnmappedRam` is + zero-sized and shares its address with the next field of `Physical`; only + the vtable half of the fat pointer tells them apart. `addr_eq` would call + it equal to `AliasBus`. +- **The fault handler must stay allocation-free.** It takes the table's + spin lock (bounded spin, gives up with a message), maps one page, writes + one line with `write(2)`. Keep it that way; a `println!` or `Vec` in there + is a deadlock waiting for a fault on the thread that holds the lock. +- **Debuggers stop on SIGSEGV.** `handle SIGSEGV nostop noprint pass` (and + `SIGBUS`) in gdb. Only faults above 4 GB reach it on an Indy, so this + rarely matters until the Octane. + +## Not done + +- The `BusDevice` trait still takes `u32`. The table is 40-bit so the bus + can be widened without redoing the decode; nothing above 4 GB is reachable + from the CPU yet. +- Windows backend written against the API, not run: same status as + `map_windows.rs`. The devmap unit tests are the acceptance test. +- Shared pages are never freed (bounded by the number of distinct devices); + pool pages are. Fine for now. diff --git a/debug.md b/debug.md index 328f6fbc..59081f96 100644 --- a/debug.md +++ b/debug.md @@ -439,6 +439,7 @@ a forced termination; a normal stop saves them automatically. |---|---| | `phys mem [n]` / `mm` | Dump physical memory | | `phys dis [n]` / `md` | Disassemble physical memory | +| `phys map` | The decode table: pages by kind (untouched, shared, private), memory in use, slots faulted in, and the table as runs of devices | | `phys trace ` / `trace ` | Log physical bus accesses | | `phys error ` | Stop on bus errors | | `phys hole ` | Stop on accesses to unmapped addresses | diff --git a/docs/devmap-design.md b/docs/devmap-design.md new file mode 100644 index 00000000..63c97d51 --- /dev/null +++ b/docs/devmap-design.md @@ -0,0 +1,125 @@ +# The physical address decode table + +`Physical` (`src/physical.rs`) finds the device behind a physical address +with one indexed load: a table with one `*const dyn BusDevice` per 64 KB +slot. This document describes how that table is stored since it grew from a +32-bit to a 40-bit address space, and what happens when the guest touches an +address nobody mapped. The code is `src/ppmem/devmap.rs`; it builds on the +host-memory primitives ppmem already had ([ppmem-design.md](ppmem-design.md) +§7). + +## 1. Why + +An Indy or Indigo2 decodes 32 bits of physical address. The table for that +was a plain array: 65536 fat pointers, 1 MB, embedded in `Physical`. An +Octane decodes 40 bits (XIO, the Bridge, memory beyond 4 GB). The same array +for that space is 16 M entries and 256 MB, and almost all of it would hold the +same few pointers: a bus-error sink, a memory bank, an empty slot. + +The table is therefore sparse in the same sense ppmem's RAM is: it is a +reservation of host address space into which pages are mapped as needed, +and the hot path does not know the difference. + +## 2. Shape + +- **One reservation**, 256 MB of virtual address space, `PROT_NONE` on Unix + and a placeholder on Windows (`AddrSpace::reserve`). Entry `i` is at + `base + 16 * i`. Nothing is committed by reserving. +- **Table pages of 64 KB**, 4096 entries each, so one page decodes 256 MB of + guest address space. 64 KB is the Windows allocation granularity, and every + mapping is exactly one page, which keeps the Windows placeholder backend + (`map_windows.rs`) within what it can split and replace. +- **Three kinds of page.** A page is *untouched* (never written, still + reserved), *shared* (a view of the constant page for one device pointer; + every page whose entries are all that pointer is a view of the same 64 KB), + or *private* (a page of mixed entries, from a pool of shared-memory + objects). +- **Memory.** One 64 KB page per distinct device that fills a whole page, one + per mixed page, plus a 64 KB staging page. An Indy comes to about ten + pages, well under the 1 MB array it replaces, for 256 times the address + space. + +## 3. Writes never touch the table + +The MC's DMA worker, the SCSI and ethernet threads and the graphics boards +dispatch through this table from their own threads while the CPU thread +remaps banks on a MEMCFG write. With the old array that meant non-atomic +16-byte stores under concurrent loads, and `plan_bank_slots` exists to write +as few of them as possible. + +Now a change to any entry builds the new page contents off-line and swaps +the page in: + +1. Read the page's current contents (from the table if private, from the + page's device pointer if shared, from the default device if untouched). +2. Apply the change in a buffer. If the result is uniform, map the constant + page for that pointer. Otherwise take a free pool page, write the buffer + into it through the private *staging* view, and map the pool page. +3. `MAP_FIXED` (Unix) or `MapViewOfFile3(MEM_REPLACE_PLACEHOLDER)` (Windows) + replaces the view atomically; a concurrent reader sees the old page or + the new one. The old pool page, if any, goes back on the free list. + +Every table view is read-only. A stray in-place store is a fault, not a +silent corruption of a page shared with other regions. + +`plan_bank_slots` is kept. It still saves work, and a slot that ends up where +it started is still never seen in a transient state. + +## 4. Untouched pages fault in + +A load from an untouched page is a host access violation. `devmap` installs +a handler when the first table is created: + +- Unix: `sigaction` on `SIGSEGV` and `SIGBUS` (macOS raises `SIGBUS` for + `PROT_NONE` pages). Whatever handler was there before, which in a Rust + program is the standard library's stack-overflow detector, is saved and + chained to for every fault that is not in a table. +- Windows: a vectored exception handler at the head of the chain. Faults + that are not in a table continue down the chain, so `crash_diag`'s handler + and the usual crash path are unchanged. + +For a fault inside a table the handler maps that page as a view of the +constant page for the table's *default* device, prints one line naming the +guest address, and returns; the load re-executes and finds the default. The +default is set by the owner before anything dispatches (`Physical` uses its +CPU bus-error sink, which reports to the MC). The constant page for it is +created when the default is set, so the handler allocates nothing and takes +only the table's spin lock, which is never held across anything that can +block or fault. + +For the Indy and Indigo2 the whole 32-bit space is filled at `init`, exactly +as before, so this path is reached only above 4 GB: nothing the CPU can +address today. It is what lets a 40-bit machine leave its unused address +space unmapped and still survive a probe. + +The monitor shows the table with `phys map`: page counts by kind, memory in +use, how many faults filled a page in and where, and the table as runs of +named devices ([debug.md](../debug.md)). + +## 5. Debugging with it + +A debugger stops on every `SIGSEGV` by default. Under gdb: + +``` +handle SIGSEGV nostop noprint pass +handle SIGBUS nostop noprint pass +``` + +and under lldb `process handle SIGSEGV -s false -n false -p true`. A fault in +a table is reported on stderr either way; a fault anywhere else still +reaches the previous handler and the debugger as before. + +Linux limits a process to about 65 000 mappings (`vm.max_map_count`). Each +mapped table page is one; filling a full 40-bit space page by page would +need 4096, which is fine, and the guests we run map far less. + +## 6. What this is not + +It is not device mapping through page faults (ppmem-design.md §11): the +fault handler here maps a *table page*, not a guest device, and it runs once +per untouched 256 MB region, not per access. The hot path is still the +indexed load and the virtual call. + +The bus trait is still addressed with `u32`. The table reaches 40 bits so +that the Octane work can widen the bus without also redesigning the decode; +the Indy and Indigo2 use the low 4 GB of it unchanged. diff --git a/docs/ppmem-design.md b/docs/ppmem-design.md index 5de57696..3893464c 100644 --- a/docs/ppmem-design.md +++ b/docs/ppmem-design.md @@ -570,6 +570,15 @@ demonstrated. Windows is **designed but unverified** — no Windows host here. Explicitly **not** part of this work, and not phase 2 either. Recorded here so the idea and its blockers aren't relearned. +*Update, October 2026:* a fault handler now exists, but for the **decode +table**, not for devices. `src/ppmem/devmap.rs` keeps `Physical`'s +address-to-device table as a sparse reservation over a 40-bit space and +faults untouched table pages in as the bus-error device; see +[devmap-design.md](devmap-design.md). It settles the portability question +for this narrow use (one handler per OS, no instruction decoding, since the +retried access is the table load itself) and leaves everything below as +open as before. + ### 11.1 The interesting use: map devices, delete the bus abstraction The obvious uses (dirty-page snapshot tracking, lazy/COW bank materialisation diff --git a/src/main.rs b/src/main.rs index 100e244d..108fecb8 100644 --- a/src/main.rs +++ b/src/main.rs @@ -50,14 +50,17 @@ fn main() { cfg.debug.apply_env(); cfg.jitv2.apply_env(); - // Machine::new() allocates >1MB on the stack (Physical device_map), which overflows - // the default stack on Windows (1MB). We spawn a thread with a larger stack to create it. + // Machine::new() has several large temporaries on the stack, which + // overflows the default stack on Windows (1MB). We spawn a thread with a + // larger stack to create it. let mut machine = std::thread::Builder::new() // Rex3Context embeds the HOSTRW data-port array (HOSTRW_BUF_QWORDS u64s, - // 1 MiB), and construction has several context-sized temporaries in - // flight before the machine is boxed. This is virtual address space, - // lazily committed, so the large reservation has no real cost. - .stack_size(64 * 1024 * 1024) + // 1 MiB), and construction has several context-sized temporaries in + // flight before the machine is boxed. (The physical decode table used + // to be another 1 MB here; it is host-mapped now.) This is virtual + // address space, lazily committed, so the large reservation has no + // real cost. + .stack_size(64 * 1024 * 1024) .spawn(move || iris::workstation::build(cfg)) .unwrap() .join() diff --git a/src/physical.rs b/src/physical.rs index 7995df17..f338f469 100644 --- a/src/physical.rs +++ b/src/physical.rs @@ -8,6 +8,7 @@ use crate::exp::eval_const_expr; use crate::cpu::mips_dis; use crate::dev::mem::{BlackHoleRegion, UnmappedRam}; use crate::ppmem::{MappedMemory, PpMemSpace, PpMemory}; +use crate::ppmem::devmap::{entry_eq, Entry, SparseDeviceMap, SLOT_SHIFT}; use crate::dev::prom::PromPort; use crate::dev::mc::MemoryController; @@ -196,7 +197,7 @@ const PROM_END: u32 = 0x1FD00000; // per MC spec: "The bottom 512KB of memory is just an alias for the memory located // at address 0x08000000 to 0x0807ffff." // Implemented as an AliasBus that adds LOMEM_BASE to the incoming address, so -// accesses go through the normal lomem device_map entries — no direct bank pointer needed. +// accesses go through the normal lomem decode-table entries — no direct bank pointer needed. const ALIAS_BASE: u32 = 0x00000000; const ALIAS_END: u32 = 0x00080000; @@ -218,14 +219,14 @@ fn alias_offset_for(ip28: bool) -> u32 { } -/// What one 64 KB `device_map` slot should point at after a MEMCFG write. +/// What one 64 KB decode-table slot should point at after a MEMCFG write. #[derive(Clone, Copy, Debug, PartialEq, Eq)] enum BankSlot { Unmapped, Bank(usize), } -/// Decide every `device_map` slot a MEMCFG write can affect: the lomem and +/// Decide every decode-table slot a MEMCFG write can affect: the lomem and /// himem windows, each bank's placement, and wherever a bank sat outside /// those windows last time. Returns one entry per slot, in slot order, plus /// the slots this placement puts outside the windows, for the next call. @@ -233,11 +234,13 @@ enum BankSlot { /// A plan rather than a sequence of stores, for two reasons: /// /// - **Only the differences get written.** Wiping both 256 MB windows and -/// mapping the banks back over them is about 8200 stores per MEMCFG write, -/// each a non-atomic 16-byte fat pointer, while the MC's DMA worker may be -/// dispatching through the table from its own thread. Collapsing to one -/// entry per slot means a slot that ends up where it started is never -/// touched, not wiped and rewritten. +/// mapping the banks back over them is about 8200 slots per MEMCFG write, +/// while the MC's DMA worker may be dispatching through the table from its +/// own thread. Collapsing to one entry per slot means a slot that ends up +/// where it started is never touched, not wiped and rewritten, and the +/// table is never seen half-way through a change. (The table now swaps +/// whole pages atomically, so a transient would no longer be a torn +/// pointer; it would still be a slot briefly answering as unmapped.) /// - **A bank placed outside the windows is unmapped again when it moves.** /// MEMCFG's base field reaches far beyond lomem and himem, and a wipe of /// only those two left such a slot pointing at a bank the MC had since @@ -285,17 +288,19 @@ const MYSTERY_HOLE_END: u32 = 0x02090000; /// Acts as the central hub for connecting devices. /// Routes all bus accesses to the appropriate device based on address. /// Devices are stored directly in the struct for zero-overhead access. -/// Uses a 64KB granularity lookup table for O(1) address decoding. +/// Decodes with one indexed load from a table of 64 KB slots; the table +/// itself is sparse and host-MMU-backed (`SparseDeviceMap`), so it spans a +/// 40-bit space while costing a few pages. /// /// Memory is split into 4 × 128MB banks (0..3). MEMCFG0/1 in the MC /// configure at which physical address each bank appears. On MEMCFG write, -/// the MC invokes remap_banks() which updates device_map and each bank's +/// the MC invokes remap_banks() which updates the decode table and each bank's /// base_addr. Banks 0/1 occupy lomem slots; banks 2/3 occupy himem slots. pub struct Physical { // 4 × 128MB RAM banks (0,1 in lomem; 2,3 in himem). base_addr set by remap_banks(). banks: [RamBank; 4], /// ppmem's 4GB window. Banks are mapped into it by `remap_banks` in - /// addition to being installed in `device_map`, so the mapping-based fast + /// addition to being installed in the decode table, so the mapping-based fast /// path and the bus path stay in agreement. `None` if the window could not /// be reserved, in which case only the bus path is used. ppmem_space: Option, @@ -338,9 +343,14 @@ pub struct Physical { vino_gio_alias: AliasBus, // GIO aperture at 0x1F080000 → VINO at 0x00080000 black_hole: BlackHoleRegion, - // Lookup table: 64KB granularity (65536 entries = 512KB on 64-bit) - // Maps (address >> 16) to device pointer (non-null, always valid) - device_map: [*const dyn BusDevice; 65536], + /// The decode table: one device per 64 KB slot of a 40-bit physical + /// space, sparse and host-MMU-backed (`src/ppmem/devmap.rs`). Slots the + /// Indy and Indigo2 decode are filled in `init`; a slot nobody filled is + /// faulted in as the CPU bus-error device the first time it is touched. + devmap: SparseDeviceMap, + /// `devmap.base()`, cached so the hot path is a load off `self` and one + /// indexed load. + map_base: *const Entry, /// 64 KB slots the last `remap_banks` placed a RAM bank in outside the /// lomem and himem windows. MEMCFG's base field reaches well beyond those @@ -357,7 +367,7 @@ pub struct Physical { host_freq: u64, } -// Safety: Physical owns all devices and the pointers in device_map point to owned data +// Safety: Physical owns all devices and the pointers in the decode table point to owned data // All the devices themselves are Send+Sync, and we never mutate through the pointers unsafe impl Send for Physical {} unsafe impl Sync for Physical {} @@ -430,17 +440,14 @@ impl Physical { let vino_gio_alias = AliasBus::new(std::ptr::null::(), 0xE1000000u32); let black_hole = BlackHoleRegion::new(); - // The lookup table is filled in init(). Until then it points at a - // bus-error device rather than at null, although every slot is - // written before the guest runs: the MC's DMA worker dispatches - // through this table from its own thread while `remap_banks` rewrites - // its 16-byte fat pointers non-atomically from the CPU's, and a torn - // read that pairs a null data pointer with a live vtable is a - // segfault, not a bus error. A stateless static costs nothing and - // takes null out of the table for good. - static BOOT_ERR: ErrorBus = ErrorBus { debug: AtomicBool::new(false) }; - const BOOT_PTR: *const dyn BusDevice = &BOOT_ERR; - let device_map: [*const dyn BusDevice; 65536] = [BOOT_PTR; 65536]; + // The decode table. Only reserved here: `init` fills it once `self` + // is in its final place, since the entries point at fields of `self`. + // Nothing dispatches before that; a dispatch that did would fault + // with a message naming the address rather than read a null. + let devmap = SparseDeviceMap::new().unwrap_or_else(|e| { + panic!("iris: cannot reserve the physical address decode table: {e}") + }); + let map_base = devmap.base(); // ppmem: reserve the 4GB window over these banks. A failure here is // not fatal — the bus path works regardless — so log and carry on @@ -488,7 +495,8 @@ impl Physical { alias_bus, vino_gio_alias, black_hole, - device_map, + devmap, + map_base, banks_outside_windows: Vec::new(), alias_offset: alias_offset_for(ip28), is_ip28: ip28, @@ -519,18 +527,29 @@ impl Physical { let prom_ptr: *const dyn BusDevice = &self.prom; let black_hole_ptr: *const dyn BusDevice = &self.black_hole; - // Layer 1: fill entire table with CPU bus error device - for i in 0..65536usize { - self.device_map[i] = cpu_err_ptr; - } + let map = &self.devmap; + let set = |base: u32, end: u32, dev: Entry| { + map.fill_bytes(base as u64, end as u64, dev) + .unwrap_or_else(|e| panic!("physical: mapping {base:#010x}..{end:#010x}: {e}")); + }; + + // Whatever is never mapped below answers as a CPU bus error. Within + // the 32-bit space that is filled in now (layer 1); the rest of the + // 40-bit table stays untouched until the guest reaches it, and the + // first access to a slot up there faults it in as the same device, + // with a message naming the address. + map.set_default(cpu_err_ptr) + .unwrap_or_else(|e| panic!("physical: setting the bus-error default: {e}")); + + // Layer 1: the whole 32-bit space, CPU bus error device + map.fill(0, 1 << (32 - SLOT_SHIFT), cpu_err_ptr) + .unwrap_or_else(|e| panic!("physical: filling the 32-bit space: {e}")); // Layer 2: overlay GIO space (0x18000000..0x1FA00000) with GIO timeout device // This covers: reserved future GIO (0x18000000..0x1F000000) + Newport slot // (0x1F000000..0x1F400000) + GIO expansion slots 0/1 (0x1F400000..0x1FA00000) // Real devices will overlay their own ranges on top in layer 3. - for i in (0x1800_0000u32 >> 16)..(0x1FA0_0000u32 >> 16) { - self.device_map[i as usize] = gio_err_ptr; - } + set(0x1800_0000, 0x1FA0_0000, gio_err_ptr); // Memory banks are NOT mapped here — MEMCFG0/1 control their placement. // remap_banks() is called by the MC when MEMCFG is written. @@ -539,17 +558,13 @@ impl Physical { // Layer 3: real devices overlaid on top // Map VINO (physical 0x00080000, one 64KB slot) - self.device_map[(crate::dev::vino::VINO_BASE >> 16) as usize] = vino_ptr; + set(crate::dev::vino::VINO_BASE, crate::dev::vino::VINO_BASE + (1 << SLOT_SHIFT), vino_ptr); // Map Mystery Hole - for i in (MYSTERY_HOLE_BASE >> 16)..((MYSTERY_HOLE_END - 1) >> 16) + 1 { - self.device_map[i as usize] = black_hole_ptr; - } + set(MYSTERY_HOLE_BASE, MYSTERY_HOLE_END, black_hole_ptr); // 2nd hpc - for i in (0x1F980000 >> 16)..((0x1F990000 - 1) >> 16) + 1 { - self.device_map[i as usize] = black_hole_ptr; - } + set(0x1F980000, 0x1F990000, black_hole_ptr); // HPC1 region (0x1FB00000–0x1FB80000) — older HPC chip iris doesn't // emulate. IRIX still probes here during normal operation (visible as @@ -560,47 +575,33 @@ impl Physical { // black hole (reads as zero, writes silently accepted) prevents the // bus error from firing and avoids the panic — without implementing // real HPC1 semantics, which would be a much larger emulation gap. - for i in (0x1FB00000u32 >> 16)..((HPC3_BASE - 1) >> 16) + 1 { - self.device_map[i as usize] = black_hole_ptr; - } + set(0x1FB00000, HPC3_BASE, black_hole_ptr); // Map Newport/REX3 (4MB GIO slot at 0x1F000000) — only if graphics enabled if let Some(rex3_ptr) = rex3_ptr { - for i in (NEWPORT_BASE >> 16)..((NEWPORT_END - 1) >> 16) + 1 { - self.device_map[i as usize] = rex3_ptr; - } + set(NEWPORT_BASE, NEWPORT_END, rex3_ptr); } else if let Some(gr2_ptr) = gr2_ptr { // GR2 (XZ / Extreme): the whole 4 MB gfx slot; see src/dev/gr2. - for i in (NEWPORT_BASE >> 16)..((NEWPORT_END - 1) >> 16) + 1 { - self.device_map[i as usize] = gr2_ptr; - } + set(NEWPORT_BASE, NEWPORT_END, gr2_ptr); } else if let Some(mgras_ptr) = mgras_ptr { // IMPACT in the graphics slot. The expansion slots stay unmapped so // their probes bus-error, as empty slots do. - for i in (NEWPORT_BASE >> 16)..((NEWPORT_END - 1) >> 16) + 1 { - self.device_map[i as usize] = mgras_ptr; - } + set(NEWPORT_BASE, NEWPORT_END, mgras_ptr); } // else: GIO timeout from layer 2 already covers the Newport slot // Second Newport head at GIO expansion slot 1 (dual-head). if let Some(h1_ptr) = rex3_head1_ptr { - for i in (GIO_SLOT1_BASE >> 16)..((GIO_SLOT1_END - 1) >> 16) + 1 { - self.device_map[i as usize] = h1_ptr; - } + set(GIO_SLOT1_BASE, GIO_SLOT1_END, h1_ptr); } // GIO expansion slot 0 (0x1F400000–0x1F5FFFFF): N64 dev board if enabled if let Some(u64_ptr) = ultra64_ptr { use crate::dev::ultra64::{GIO_SLOT0_BASE, RAMROM_BASE, RAMROM_SIZE}; // Control registers: 0x1F400000–0x1F4FFFFF (16 × 64KB slots) - for i in (GIO_SLOT0_BASE >> 16)..((RAMROM_BASE - 1) >> 16) + 1 { - self.device_map[i as usize] = u64_ptr; - } + set(GIO_SLOT0_BASE, RAMROM_BASE, u64_ptr); // RAMROM window: 0x1F500000–0x1F5FFFFF (16 × 64KB slots) - for i in (RAMROM_BASE >> 16)..((RAMROM_BASE + RAMROM_SIZE - 1) >> 16) + 1 { - self.device_map[i as usize] = u64_ptr; - } + set(RAMROM_BASE, RAMROM_BASE + RAMROM_SIZE, u64_ptr); } // GIO expansion slot 1 — second Newport when rex3_head1 absent: GIO timeout remains @@ -609,51 +610,40 @@ impl Physical { if let Some(td) = self.testdev.as_deref() { let td_ptr: *const dyn BusDevice = td; use crate::dev::testdev::{TEST_DEV_BASE, TEST_DEV_SIZE}; - for i in (TEST_DEV_BASE >> 16)..((TEST_DEV_BASE + TEST_DEV_SIZE - 1) >> 16) + 1 { - self.device_map[i as usize] = td_ptr; - } + set(TEST_DEV_BASE, TEST_DEV_BASE + TEST_DEV_SIZE, td_ptr); } // Map MC registers (128KB at 0x1FA00000) - for i in (MC_BASE >> 16)..((MC_END - 1) >> 16) + 1 { - self.device_map[i as usize] = mc_ptr; - } + set(MC_BASE, MC_END, mc_ptr); // Map HPC3 (512KB at 0x1FB80000) - for i in (HPC3_BASE >> 16)..((HPC3_END - 1) >> 16) + 1 { - self.device_map[i as usize] = hpc3_ptr; - } + set(HPC3_BASE, HPC3_END, hpc3_ptr); // Map PROM (1MB at 0x1FC00000) - for i in (PROM_BASE >> 16)..((PROM_END - 1) >> 16) + 1 { - self.device_map[i as usize] = prom_ptr; - } + set(PROM_BASE, PROM_END, prom_ptr); // Alias: points back into Physical itself with `alias_offset()` added. // So alias accesses go: AliasBus → Physical::read/write(addr + LOMEM_BASE) - // → device_map lookup → whichever bank is mapped at LOMEM_BASE. + // → table lookup → whichever bank is mapped at LOMEM_BASE. // This way alias automatically tracks whatever MEMCFG maps at LOMEM_BASE. self.alias_bus.target = self as *const Physical as *const dyn BusDevice; let alias_ptr: *const dyn BusDevice = &self.alias_bus; - for i in (ALIAS_BASE >> 16)..(ALIAS_END >> 16) { - self.device_map[i as usize] = alias_ptr; - } + set(ALIAS_BASE, ALIAS_END, alias_ptr); // VINO GIO alias: 0x1F080000 → 0x00080000 - // Routes through Physical again with 0xFF000000 added (wrapping subtraction of 0x1F000000) + // Routes through Physical again with 0xE1000000 added (wrapping subtraction of 0x1F000000) // so the re-dispatched address falls into VINO's primary slot at 0x0008xxxx. self.vino_gio_alias.target = self as *const Physical as *const dyn BusDevice; let vino_gio_alias_ptr: *const dyn BusDevice = &self.vino_gio_alias; - self.device_map[(0x1F080000u32 >> 16) as usize] = vino_gio_alias_ptr; - + set(0x1F080000, 0x1F090000, vino_gio_alias_ptr); } - /// Remap memory banks in device_map. + /// Remap memory banks in the decode table. /// /// `bank_addrs[i]` is `Some((base, addr_mask, limit))` if bank i is valid, or `None`. /// - `base` — physical base address /// - `addr_mask` — applied inside Memory for aliasing (SIMM wrapping); equals size_mb*1MB-1 - /// - `limit` — number of bytes to map in the device_map; slots beyond limit stay as + /// - `limit` — number of bytes to map in the decode table; slots beyond limit stay as /// UnmappedRam and return 0, matching the hardware behaviour of reads past /// the real SIMM boundary /// @@ -740,27 +730,38 @@ impl Physical { } } - // Now point the table at the banks, whose masks are set, storing only - // the slots whose contents actually change. The DMA worker can be - // dispatching through this table right now, and each store is a - // non-atomic 16-byte fat pointer; see `plan_bank_slots`. + // Now point the table at the banks, whose masks are set. The plan + // is one entry per slot; apply it as runs of one target, and leave + // a run alone that already answers as planned. The DMA worker can be + // dispatching through the table right now: a run that does change + // is swapped in whole pages by `SparseDeviceMap`, so it sees the old + // page or the new one, never a torn pointer. See `plan_bank_slots`. let (plan, outside) = plan_bank_slots(&bank_addrs, &self.banks_outside_windows); self.banks_outside_windows = outside; - for (idx, target) in plan { + let mut i = 0; + while i < plan.len() { + let (start, target) = plan[i]; + let mut j = i + 1; + while j < plan.len() && plan[j].1 == target && plan[j].0 == plan[j - 1].0 + 1 { + j += 1; + } let ptr = match target { BankSlot::Unmapped => unmapped_ptr, BankSlot::Bank(b) => bank_ptrs[b], }; - let slot = &mut self.device_map[idx as usize]; - if !std::ptr::addr_eq(*slot, ptr) { - *slot = ptr; + let count = (j - i) as u64; + if !self.devmap.all_are(start as u64, count, ptr) { + self.devmap.fill(start as u64, count, ptr).unwrap_or_else(|e| { + panic!("physical: remapping slots {start:#x}+{count:#x}: {e}") + }); } + i = j; } // ppmem: the low-512KB alias of bank 0 (MC spec — the bottom 512KB // mirrors 0x08000000..0x0807ffff). Mapped as real pages rather than // routed through AliasBus, so it is the same physical memory with no - // re-dispatch. AliasBus stays installed in device_map as the bus-path + // re-dispatch. AliasBus stays installed in the decode table as the bus-path // equivalent; both see identical memory. if let (true, Some(sp)) = (ppmem_placement_changed, &self.ppmem_space) { let bank0_mapped = bank_addrs[0].is_some_and(|(base, _, _)| base == self.alias_offset); @@ -795,6 +796,46 @@ impl Physical { } } + /// The device behind `addr`: one indexed load from the decode table. + #[inline(always)] + fn dev_at(&self, addr: u32) -> Entry { + // SAFETY: `map_base` spans the whole table for the life of `devmap`; + // a page nobody filled is faulted in by devmap's handler, so every + // load lands on a readable page holding a live device pointer. + unsafe { *self.map_base.add((addr >> SLOT_SHIFT) as usize) } + } + + /// Name a decode-table entry for the monitor. + fn device_name(&self, dev: Entry) -> String { + fn e(t: &T) -> Entry { t } + let mut known: Vec<(Entry, String)> = vec![ + (e(&self.cpu_bus_error), "cpu bus error".into()), + (e(&self.gio_bus_error), "gio timeout".into()), + (e(&self.unmapped_ram), "unmapped ram (reads 0)".into()), + (e(&self.alias_bus), "low-memory alias".into()), + (e(&self.vino_gio_alias), "vino gio alias".into()), + (e(&self.black_hole), "black hole".into()), + (e(&self.vino), "vino".into()), + (e(&self.mc), "mc".into()), + (e(&self.hpc3), "hpc3".into()), + (e(&self.prom), "prom".into()), + ]; + for (i, b) in self.banks.iter().enumerate() { + known.push((e(b), format!("bank {i}"))); + } + if let Some(d) = self.rex3.as_deref() { known.push((e(d), "newport (rex3)".into())); } + if let Some(d) = self.rex3_head1.as_deref() { known.push((e(d), "newport head 1".into())); } + if let Some(d) = self.gr2.as_deref() { known.push((e(d), "gr2".into())); } + if let Some(d) = self.mgras.as_deref() { known.push((e(d), "impact (mgras)".into())); } + if let Some(d) = self.ultra64.as_deref() { known.push((e(d), "ultra64".into())); } + if let Some(d) = self.testdev.as_deref() { known.push((e(d), "test device".into())); } + known + .into_iter() + .find(|(p, _)| entry_eq(*p, dev)) + .map(|(_, n)| n) + .unwrap_or_else(|| format!("{:p}", dev as *const ())) + } + /// Is `addr` backed by a whole directly-mapped 64MB region? /// /// This is the quick test the design calls for: one load of the bitmap, @@ -874,7 +915,7 @@ impl Device for Physical { fn register_commands(&self) -> Vec<(String, String)> { let cmds = vec![ - ("phys".to_string(), "Physical Bus commands: mem, dis, trace, error , hole , bench".to_string()), + ("phys".to_string(), "Physical Bus commands: mem, dis, map, trace, error , hole , bench".to_string()), ("mm".to_string(), "Alias for phys mem".to_string()), ("md".to_string(), "Alias for phys dis".to_string()), ("trace".to_string(), "Enable/disable tracing: trace ".to_string()), @@ -923,6 +964,27 @@ impl Device for Physical { if r.is_ok() { writeln!(writer, "{}", mips_dis::disassemble(r.data, addr, None)).unwrap(); } else { writeln!(writer, "Could not fetch instruction at {:016x}", addr).unwrap(); } } + "map" => { + let s = self.devmap.stats(); + writeln!(writer, + "decode table: {} pages of {} KB ({} MB of guest space each): {} untouched, {} shared, {} private", + s.pages, s.page_bytes >> 10, ((s.page_bytes as u64 / 16) << SLOT_SHIFT) >> 20, + s.unmapped, s.shared, s.private).unwrap(); + writeln!(writer, + " {} constant page(s), {} pool page(s) ({} free), {} KB in use, {} fault(s)", + s.const_pages, s.pool_pages, s.pool_free, s.in_use_bytes() >> 10, s.faults).unwrap(); + for a in &s.first_faults { + writeln!(writer, " faulted in: {:#012x}", a).unwrap(); + } + for r in self.devmap.runs() { + let name = match r.dev { + Some(d) => self.device_name(d), + None => "(untouched)".to_string(), + }; + writeln!(writer, " {:#012x}-{:#012x} {}", + r.start << SLOT_SHIFT, (r.end << SLOT_SHIFT) - 1, name).unwrap(); + } + } "error" => { if actual_args.is_empty() { return Err("Usage: error ".to_string()); @@ -998,7 +1060,7 @@ impl Device for Physical { impl BusDevice for Physical { #[inline(always)] fn read8(&self, addr: u32) -> BusRead8 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); let r = unsafe { (*device_ptr).read8(addr) }; #[cfg(not(feature = "lightning"))] if self.trace.load(Ordering::Relaxed) { @@ -1010,7 +1072,7 @@ impl BusDevice for Physical { #[inline(always)] fn write8(&self, addr: u32, val: u8) -> u32 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); let ws = unsafe { (*device_ptr).write8(addr, val) }; #[cfg(not(feature = "lightning"))] if self.trace.load(Ordering::Relaxed) { println!("PHYS8 Write {:08x} val={:02x} -> {:08x}", addr, val, ws); } @@ -1019,7 +1081,7 @@ impl BusDevice for Physical { #[inline(always)] fn read16(&self, addr: u32) -> BusRead16 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); let r = unsafe { (*device_ptr).read16(addr) }; #[cfg(not(feature = "lightning"))] if self.trace.load(Ordering::Relaxed) { @@ -1031,7 +1093,7 @@ impl BusDevice for Physical { #[inline(always)] fn write16(&self, addr: u32, val: u16) -> u32 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); let ws = unsafe { (*device_ptr).write16(addr, val) }; #[cfg(not(feature = "lightning"))] if self.trace.load(Ordering::Relaxed) { println!("PHYS16 Write {:08x} val={:04x} -> {:08x}", addr, val, ws); } @@ -1040,7 +1102,7 @@ impl BusDevice for Physical { #[inline(always)] fn read32(&self, addr: u32) -> BusRead32 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); let r = unsafe { (*device_ptr).read32(addr) }; #[cfg(not(feature = "lightning"))] if self.trace.load(Ordering::Relaxed) { @@ -1052,7 +1114,7 @@ impl BusDevice for Physical { #[inline(always)] fn write32(&self, addr: u32, val: u32) -> u32 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); let ws = unsafe { (*device_ptr).write32(addr, val) }; #[cfg(not(feature = "lightning"))] if self.trace.load(Ordering::Relaxed) { println!("PHYS32 Write {:08x} val={:08x} -> {:08x}", addr, val, ws); } @@ -1061,7 +1123,7 @@ impl BusDevice for Physical { #[inline(always)] fn read64(&self, addr: u32) -> BusRead64 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); let r = unsafe { (*device_ptr).read64(addr) }; #[cfg(not(feature = "lightning"))] if self.trace.load(Ordering::Relaxed) { @@ -1073,7 +1135,7 @@ impl BusDevice for Physical { #[inline(always)] fn write64(&self, addr: u32, val: u64) -> u32 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); let ws = unsafe { (*device_ptr).write64(addr, val) }; #[cfg(not(feature = "lightning"))] if self.trace.load(Ordering::Relaxed) { println!("PHYS64 Write {:08x} val={:016x} -> {:08x}", addr, val, ws); } @@ -1082,7 +1144,7 @@ impl BusDevice for Physical { #[inline(always)] fn write64_masked(&self, addr: u32, val: u64, mask: u64) -> u32 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); unsafe { (*device_ptr).write64_masked(addr, val, mask) } } @@ -1093,13 +1155,13 @@ impl BusDevice for Physical { // any DMA-specific override (e.g. Rex3::dma_read64) entirely. #[inline(always)] fn dma_read64(&self, addr: u32) -> BusRead64 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); unsafe { (*device_ptr).dma_read64(addr) } } #[inline(always)] fn dma_write64(&self, addr: u32, val: u64) -> u32 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); unsafe { (*device_ptr).dma_write64(addr, val) } } @@ -1110,13 +1172,13 @@ impl BusDevice for Physical { // (Rex3's single-token push) actually gets a chance to run. #[inline(always)] fn dma_write64_bulk(&self, addr: u32, vals: &[u64]) -> u32 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); unsafe { (*device_ptr).dma_write64_bulk(addr, vals) } } #[inline(always)] fn dma_read64_bulk(&self, addr: u32, out: &mut [u64]) -> u32 { - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); unsafe { (*device_ptr).dma_read64_bulk(addr, out) } } @@ -1130,7 +1192,7 @@ impl BusDevice for Physical { if let Some(p) = self.ppmem_gen_ptr(addr) { return p; } - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); unsafe { (*device_ptr).gen_ptr(addr) } } @@ -1145,7 +1207,7 @@ impl BusDevice for Physical { if let Some(p) = self.ppmem_ptr(addr) { return Some(p as *const u64); } - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); unsafe { (*device_ptr).mem_ptr(addr) } } @@ -1161,7 +1223,7 @@ impl BusDevice for Physical { } return BUS_OK; } - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); unsafe { (*device_ptr).read_block(addr, buf) } } @@ -1179,7 +1241,7 @@ impl BusDevice for Physical { self.ppmem_bump_gen_range(addr, buf.len()); return BUS_OK; } - let device_ptr = self.device_map[(addr >> 16) as usize]; + let device_ptr = self.dev_at(addr); unsafe { (*device_ptr).write_block(addr, buf) } } } @@ -1405,7 +1467,7 @@ mod bank_plan_tests { Some((base, size - 1, size)) } - /// `device_map` in miniature: apply a plan the way `remap_banks` does and + /// The decode table in miniature: apply a plan the way `remap_banks` does and /// count the stores that actually happen. fn apply(map: &mut [BankSlot], plan: &[(u32, BankSlot)]) -> usize { let mut stores = 0; diff --git a/src/ppmem/devmap.rs b/src/ppmem/devmap.rs new file mode 100644 index 00000000..2870aaff --- /dev/null +++ b/src/ppmem/devmap.rs @@ -0,0 +1,915 @@ +//! A sparse, host-MMU-backed physical address decode table. +//! +//! `Physical` finds the device behind a physical address with one indexed +//! load: `table[addr >> 16]`, a `*const dyn BusDevice` per 64 KB. As a plain +//! array that is 1 MB for a 32-bit space and 256 MB for the 40-bit space an +//! Octane has, almost all of it the same few pointers repeated. This keeps +//! the one-load lookup and makes the array virtual: +//! +//! - **Reserved, not allocated.** The whole table is one `AddrSpace` +//! reservation (`PROT_NONE` / a placeholder). A page nobody has written is +//! not backed by anything. +//! - **Shared where it repeats.** A table page whose entries are all the same +//! pointer is a view of one shared "constant page" for that pointer. Four +//! banks and three error sinks cost seven pages however much address space +//! they cover. +//! - **Built off-line, swapped in whole.** A mixed page lives in a pool of +//! shared-memory pages. To change any entry the new contents are written +//! into a fresh pool page through a private staging view, then mapped over +//! the table page with `MAP_FIXED`. Readers on other threads (the MC's DMA +//! worker dispatches through this table) see the old page or the new one, +//! never a torn 16-byte fat pointer: the table is never written in place, +//! and every table view is read-only so that cannot happen by accident. +//! - **Faults fill in the rest.** A load from a page that was never written +//! is a host access violation. The handler installed here recognises the +//! reservation, maps that page to the table's default device (the +//! bus-error sink), prints the guest address once, and resumes the load. +//! So an unmapped region costs nothing until the guest probes it, and a +//! probe costs one fault. +//! +//! One table page is 64 KB (the Windows allocation granularity, and a +//! multiple of every Unix page size), 4096 entries, 256 MB of guest address +//! space. Every mapping is exactly one page, which is what the Windows +//! placeholder backend in `map_windows.rs` can split and replace. +//! +//! Who may call what: `fill` and `set_default` are for the owner's setup and +//! MEMCFG remaps, on the CPU thread; the fault path may run on any thread +//! that dispatches through the table. Both take the table's spin lock, which +//! is held only across mapping calls, never across anything that can block +//! or fault. + +use super::map::{granularity, AddrSpace, Prot, SharedMem}; +use crate::traits::BusDevice; +use std::cell::UnsafeCell; +use std::io; +use std::sync::atomic::{AtomicBool, AtomicPtr, AtomicU64, Ordering}; + +/// One table entry: the device that answers a 64 KB slot. +pub type Entry = *const dyn BusDevice; + +/// Width of the physical address space the table covers. +pub const PHYS_BITS: u32 = 40; +/// Bytes of guest address space per entry: 64 KB. +pub const SLOT_SHIFT: u32 = 16; +/// Entries in the table. +pub const SLOTS: u64 = 1 << (PHYS_BITS - SLOT_SHIFT); + +const ENTRY_BYTES: usize = std::mem::size_of::(); +const TABLE_BYTES: usize = SLOTS as usize * ENTRY_BYTES; +/// A table page is at least this, whatever the host page size. +const MIN_PAGE_BYTES: usize = 64 * 1024; +/// Pool pages come in objects of this many pages. +const POOL_PAGES: usize = 16; +/// How many faulting addresses are remembered for the monitor. +const FIRST_FAULTS: usize = 8; + +const _: () = assert!(ENTRY_BYTES == 2 * std::mem::size_of::(), "fat pointer layout"); + +/// Compare both halves of a fat pointer. Comparing only the data address is +/// wrong here: a zero-sized device (`UnmappedRam`) shares its address with +/// whatever field follows it, and only the vtable tells them apart. +#[inline] +fn same(a: Entry, b: Entry) -> bool { + // SAFETY: `Entry` is two words, asserted above; reading them as integers + // is the comparison `std::ptr::eq` would do on a stable `ptr_metadata`. + unsafe { + std::mem::transmute::(a) == std::mem::transmute::(b) + } +} + +/// Do two entries name the same device through the same vtable? +pub fn entry_eq(a: Entry, b: Entry) -> bool { + same(a, b) +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum PageState { + /// Never written: reserved, faults on access. + Unmapped, + /// A view of constant page `consts[k]`. + Const(u32), + /// A view of pool page `slot` (object `slot / POOL_PAGES`, page within it + /// `slot % POOL_PAGES`). + Private(u32), +} + +struct ConstObj { + dev: Entry, + mem: SharedMem, +} + +struct State { + pages: Vec, + consts: Vec, + pool: Vec, + free: Vec, + /// One page of private address space the mutator writes new page contents + /// through. Mapped read-write over whichever pool or constant page is + /// being built; the table's own views stay read-only. + staging: AddrSpace, + /// The device an untouched slot resolves to: what the fault handler maps, + /// and what fills the rest of a page that is written only in part. The + /// constant page for it is created by `set_default`, so the fault path + /// allocates nothing. + default: Option, +} + +struct Inner { + table: AddrSpace, + page_bytes: usize, + per_page: usize, + lock: AtomicBool, + st: UnsafeCell, + faults: AtomicU64, + /// Guest addresses of the first faults, `u64::MAX` where empty. + first_faults: [AtomicU64; FIRST_FAULTS], + /// Index in the fault registry, so `Drop` can clear it. + reg: usize, +} + +/// The table. See the module documentation. +pub struct SparseDeviceMap { + // Boxed so the fault registry can hold a stable pointer while the owner + // moves the map into place. + inner: Box, +} + +// SAFETY: the raw pointers inside are device addresses the owner keeps alive +// for as long as the table; `State` is guarded by `lock`. +unsafe impl Send for SparseDeviceMap {} +unsafe impl Sync for SparseDeviceMap {} + +/// One run of the table with one answer, for the monitor. +#[derive(Clone, Copy, Debug)] +pub struct Run { + /// First slot (guest address `>> SLOT_SHIFT`). + pub start: u64, + /// One past the last slot. + pub end: u64, + /// The device, or `None` for a run of pages nobody has written. + pub dev: Option, +} + +/// Counts for the monitor. +#[derive(Clone, Debug, Default)] +pub struct MapStats { + pub page_bytes: usize, + pub pages: usize, + pub unmapped: usize, + pub shared: usize, + pub private: usize, + pub const_pages: usize, + pub pool_pages: usize, + pub pool_free: usize, + pub faults: u64, + pub first_faults: Vec, +} + +impl MapStats { + /// Host memory the table has in use: every constant page, every pool + /// page a table page views, and the staging page. Free pool pages are + /// address space in a shared object, not memory, until written. + pub fn in_use_bytes(&self) -> usize { + (self.const_pages + self.pool_pages - self.pool_free + 1) * self.page_bytes + } +} + +struct Guard<'a>(&'a AtomicBool); + +impl Drop for Guard<'_> { + fn drop(&mut self) { + self.0.store(false, Ordering::Release); + } +} + +impl Inner { + fn lock(&self) -> Guard<'_> { + while self + .lock + .compare_exchange_weak(false, true, Ordering::Acquire, Ordering::Relaxed) + .is_err() + { + std::hint::spin_loop(); + } + Guard(&self.lock) + } + + /// As `lock`, but gives up: the fault handler must not hang a thread + /// forever if it ever runs with the lock already held. + fn try_lock_spinning(&self, spins: u32) -> Option> { + for _ in 0..spins { + if self + .lock + .compare_exchange_weak(false, true, Ordering::Acquire, Ordering::Relaxed) + .is_ok() + { + return Some(Guard(&self.lock)); + } + std::hint::spin_loop(); + } + None + } + + #[inline] + fn entries(&self) -> *const Entry { + self.table.base() as *const Entry + } + + fn npages(&self) -> usize { + TABLE_BYTES / self.page_bytes + } + + /// Map table page `page` as a read-only view of `mem[off..off+page]`. + unsafe fn view(&self, page: usize, mem: &SharedMem, off: usize) -> io::Result<()> { + unsafe { self.table.map(page * self.page_bytes, self.page_bytes, mem, off as u64, Prot::ReadOnly) } + } + + /// Write `buf` into `mem[off..]` through the staging view. + unsafe fn write_through_staging(&self, st: &State, mem: &SharedMem, off: usize, buf: &[Entry]) -> io::Result<()> { + debug_assert_eq!(buf.len(), self.per_page); + unsafe { + st.staging.map(0, self.page_bytes, mem, off as u64, Prot::ReadWrite)?; + std::ptr::copy_nonoverlapping(buf.as_ptr(), st.staging.base() as *mut Entry, self.per_page); + } + Ok(()) + } + + /// Index of the constant page for `dev`, created on first use. + fn const_for(&self, st: &mut State, dev: Entry) -> io::Result { + if let Some(k) = st.consts.iter().position(|c| same(c.dev, dev)) { + return Ok(k as u32); + } + let mem = SharedMem::new(self.page_bytes)?; + let buf = vec![dev; self.per_page]; + unsafe { self.write_through_staging(st, &mem, 0, &buf)? }; + st.consts.push(ConstObj { dev, mem }); + Ok((st.consts.len() - 1) as u32) + } + + fn alloc_slot(&self, st: &mut State) -> io::Result { + if let Some(s) = st.free.pop() { + return Ok(s); + } + let obj = st.pool.len() as u32; + st.pool.push(SharedMem::new(POOL_PAGES * self.page_bytes)?); + // Hand out low pages first; the order only matters for the tests. + for p in (0..POOL_PAGES as u32).rev() { + st.free.push(obj * POOL_PAGES as u32 + p); + } + Ok(st.free.pop().unwrap()) + } + + fn release(&self, st: &mut State, old: PageState) { + if let PageState::Private(s) = old { + st.free.push(s); + } + } + + /// Point a whole page at `dev`. + fn map_const(&self, st: &mut State, page: usize, dev: Entry) -> io::Result<()> { + let k = self.const_for(st, dev)?; + let old = st.pages[page]; + if old == PageState::Const(k) { + return Ok(()); + } + unsafe { self.view(page, &st.consts[k as usize].mem, 0)? }; + st.pages[page] = PageState::Const(k); + self.release(st, old); + Ok(()) + } + + /// Current contents of `page` as the mutator sees them: what is mapped, + /// or the default for a page nobody has written. + fn contents(&self, st: &State, page: usize) -> io::Result> { + Ok(match st.pages[page] { + PageState::Unmapped => { + let Some(k) = st.default else { + return Err(io::Error::other( + "devmap: a partial write to an untouched page needs a default device", + )); + }; + vec![st.consts[k as usize].dev; self.per_page] + } + PageState::Const(k) => vec![st.consts[k as usize].dev; self.per_page], + PageState::Private(_) => { + let mut v = Vec::with_capacity(self.per_page); + // SAFETY: a private page is mapped and readable; only this + // thread (lock held) replaces views. + unsafe { + let p = self.entries().add(page * self.per_page); + v.extend_from_slice(std::slice::from_raw_parts(p, self.per_page)); + } + v + } + }) + } + + /// Set entries `lo..hi` of `page` to `dev`. + fn modify(&self, st: &mut State, page: usize, lo: usize, hi: usize, dev: Entry) -> io::Result<()> { + let mut buf = self.contents(st, page)?; + if buf[lo..hi].iter().all(|&e| same(e, dev)) && st.pages[page] != PageState::Unmapped { + return Ok(()); + } + buf[lo..hi].fill(dev); + let first = buf[0]; + if buf.iter().all(|&e| same(e, first)) { + return self.map_const(st, page, first); + } + let slot = self.alloc_slot(st)?; + let (obj, idx) = (slot as usize / POOL_PAGES, slot as usize % POOL_PAGES); + let off = idx * self.page_bytes; + let r = unsafe { + self.write_through_staging(st, &st.pool[obj], off, &buf) + .and_then(|()| self.view(page, &st.pool[obj], off)) + }; + if let Err(e) = r { + st.free.push(slot); + return Err(e); + } + let old = st.pages[page]; + st.pages[page] = PageState::Private(slot); + self.release(st, old); + Ok(()) + } + + /// The fault path. `addr` is the host address that faulted. Returns true + /// if it was inside this table and the page is now mapped, so the access + /// can be retried. + fn fault(&self, addr: usize) -> bool { + let base = self.table.base() as usize; + if addr < base || addr >= base + TABLE_BYTES { + return false; + } + let page = (addr - base) / self.page_bytes; + let slot = ((addr - base) / ENTRY_BYTES) as u64; + // The lock is only ever held across mapping calls, which do not + // touch unmapped table pages, so it is never held by the faulting + // thread; the bound is a backstop against hanging in a handler. + let Some(_g) = self.try_lock_spinning(50_000_000) else { + fault_message(b"devmap: fault while the table lock is held; giving up\n", slot << SLOT_SHIFT); + return false; + }; + // SAFETY: lock held. + let st = unsafe { &mut *self.st.get() }; + if st.pages[page] != PageState::Unmapped { + // Another thread mapped it between the fault and the lock. + return true; + } + let Some(k) = st.default else { + fault_message(b"devmap: unmapped physical region and no default device at ", slot << SLOT_SHIFT); + return false; + }; + if unsafe { self.view(page, &st.consts[k as usize].mem, 0) }.is_err() { + fault_message(b"devmap: could not map the default device at ", slot << SLOT_SHIFT); + return false; + } + st.pages[page] = PageState::Const(k); + drop(_g); + let n = self.faults.fetch_add(1, Ordering::Relaxed) as usize; + if n < FIRST_FAULTS { + self.first_faults[n].store(slot << SLOT_SHIFT, Ordering::Relaxed); + } + fault_message(b"iris: physical access to an unmapped region at ", slot << SLOT_SHIFT); + true + } +} + +impl SparseDeviceMap { + /// Reserve the table. Nothing is mapped yet; set a default before the + /// first dispatch. + pub fn new() -> io::Result { + let page_bytes = granularity().max(MIN_PAGE_BYTES); + let table = AddrSpace::reserve(TABLE_BYTES)?; + let staging = AddrSpace::reserve(page_bytes)?; + let npages = TABLE_BYTES / page_bytes; + let inner = Box::new(Inner { + table, + page_bytes, + per_page: page_bytes / ENTRY_BYTES, + lock: AtomicBool::new(false), + st: UnsafeCell::new(State { + pages: vec![PageState::Unmapped; npages], + consts: Vec::new(), + pool: Vec::new(), + free: Vec::new(), + staging, + default: None, + }), + faults: AtomicU64::new(0), + first_faults: [const { AtomicU64::new(u64::MAX) }; FIRST_FAULTS], + reg: usize::MAX, + }); + let mut map = Self { inner }; + fault::install(); + map.inner.reg = fault::register(&*map.inner)?; + Ok(map) + } + + /// Base of the table: entry `i` is at `base().add(i)`. Valid for the + /// life of the map; the owner caches it for the hot path. + #[inline] + pub fn base(&self) -> *const Entry { + self.inner.entries() + } + + /// Entries per table page. + pub fn page_slots(&self) -> u64 { + self.inner.per_page as u64 + } + + /// The device an untouched slot answers with. Required before a partial + /// write to a page nobody has written, and before any dispatch may fault. + pub fn set_default(&self, dev: Entry) -> io::Result<()> { + let _g = self.inner.lock(); + let st = unsafe { &mut *self.inner.st.get() }; + let k = self.inner.const_for(st, dev)?; + st.default = Some(k); + Ok(()) + } + + /// Point slots `first..first + count` at `dev`. + pub fn fill(&self, first: u64, count: u64, dev: Entry) -> io::Result<()> { + let end = first.checked_add(count).filter(|&e| e <= SLOTS).unwrap_or_else(|| { + panic!("devmap: slots [{first:#x}, +{count:#x}) escape the {PHYS_BITS}-bit space") + }); + let per = self.inner.per_page as u64; + let _g = self.inner.lock(); + let st = unsafe { &mut *self.inner.st.get() }; + let mut slot = first; + while slot < end { + let page = (slot / per) as usize; + let page_start = page as u64 * per; + let hi = end.min(page_start + per); + if slot == page_start && hi == page_start + per { + self.inner.map_const(st, page, dev)?; + } else { + self.inner.modify(st, page, (slot - page_start) as usize, (hi - page_start) as usize, dev)?; + } + slot = hi; + } + Ok(()) + } + + /// Point the slots covering guest bytes `start..end` at `dev`. `end` is + /// exclusive and rounded up to a slot boundary. + pub fn fill_bytes(&self, start: u64, end: u64, dev: Entry) -> io::Result<()> { + assert!(end > start, "devmap: empty range {start:#x}..{end:#x}"); + let first = start >> SLOT_SHIFT; + let last = (end - 1) >> SLOT_SHIFT; + self.fill(first, last - first + 1, dev) + } + + /// Whether the page holding `slot` has been written (or faulted in). + pub fn is_mapped(&self, slot: u64) -> bool { + let _g = self.inner.lock(); + let st = unsafe { &*self.inner.st.get() }; + st.pages[(slot / self.inner.per_page as u64) as usize] != PageState::Unmapped + } + + /// The entry at `slot`, without faulting an untouched page in. + pub fn get(&self, slot: u64) -> Option { + if !self.is_mapped(slot) { + return None; + } + // SAFETY: mapped pages stay mapped (views are replaced, never + // removed), and the load is one the hot path does without a lock. + Some(unsafe { *self.inner.entries().add(slot as usize) }) + } + + /// Whether every slot in `first..first + count` already answers with + /// `dev`. Untouched pages count as not matching. + pub fn all_are(&self, first: u64, count: u64, dev: Entry) -> bool { + let per = self.inner.per_page as u64; + let end = first + count; + let _g = self.inner.lock(); + let st = unsafe { &*self.inner.st.get() }; + let mut slot = first; + while slot < end { + let page = (slot / per) as usize; + let hi = end.min((page as u64 + 1) * per); + match st.pages[page] { + PageState::Unmapped => return false, + PageState::Const(k) => { + if !same(st.consts[k as usize].dev, dev) { + return false; + } + } + PageState::Private(_) => unsafe { + let p = self.inner.entries(); + for s in slot..hi { + if !same(*p.add(s as usize), dev) { + return false; + } + } + }, + } + slot = hi; + } + true + } + + /// The table as runs of equal entries, untouched pages included as + /// `dev: None`. + pub fn runs(&self) -> Vec { + let per = self.inner.per_page as u64; + let _g = self.inner.lock(); + let st = unsafe { &*self.inner.st.get() }; + let mut out: Vec = Vec::new(); + let mut push = |start: u64, end: u64, dev: Option| { + if let Some(last) = out.last_mut() { + let joins = match (last.dev, dev) { + (None, None) => true, + (Some(a), Some(b)) => same(a, b), + _ => false, + }; + if joins && last.end == start { + last.end = end; + return; + } + } + out.push(Run { start, end, dev }); + }; + for (page, &state) in st.pages.iter().enumerate() { + let start = page as u64 * per; + match state { + PageState::Unmapped => push(start, start + per, None), + PageState::Const(k) => push(start, start + per, Some(st.consts[k as usize].dev)), + PageState::Private(_) => unsafe { + let p = self.inner.entries().add(start as usize); + let mut run_start = start; + let mut cur = *p; + for i in 1..per { + let e = *p.add(i as usize); + if !same(e, cur) { + push(run_start, start + i, Some(cur)); + run_start = start + i; + cur = e; + } + } + push(run_start, start + per, Some(cur)); + }, + } + } + out + } + + pub fn stats(&self) -> MapStats { + let _g = self.inner.lock(); + let st = unsafe { &*self.inner.st.get() }; + let mut s = MapStats { + page_bytes: self.inner.page_bytes, + pages: st.pages.len(), + const_pages: st.consts.len(), + pool_pages: st.pool.len() * POOL_PAGES, + pool_free: st.free.len(), + faults: self.inner.faults.load(Ordering::Relaxed), + ..Default::default() + }; + for p in &st.pages { + match p { + PageState::Unmapped => s.unmapped += 1, + PageState::Const(_) => s.shared += 1, + PageState::Private(_) => s.private += 1, + } + } + s.first_faults = self + .inner + .first_faults + .iter() + .map(|a| a.load(Ordering::Relaxed)) + .take_while(|&a| a != u64::MAX) + .collect(); + s + } +} + +impl Drop for SparseDeviceMap { + fn drop(&mut self) { + fault::unregister(self.inner.reg); + } +} + +/// Print `what` followed by the guest address, from a fault handler: a fixed +/// buffer and one raw write, no allocation and no locks. +fn fault_message(what: &[u8], phys: u64) { + let mut buf = [0u8; 160]; + let mut n = 0; + let mut push = |b: &[u8]| { + let take = b.len().min(buf.len() - n); + buf[n..n + take].copy_from_slice(&b[..take]); + n += take; + }; + push(what); + push(b"0x"); + let digits = (PHYS_BITS as usize).div_ceil(4); + let mut hex = [0u8; 16]; + for (i, d) in hex.iter_mut().take(digits).enumerate() { + let nib = (phys >> (4 * (digits - 1 - i))) & 0xF; + *d = b"0123456789abcdef"[nib as usize]; + } + push(&hex[..digits]); + push(b" (64 KB slot now answers as the bus-error device)\n"); + fault::emit(&buf[..n]); +} + +/// Process-wide: the host fault handler and the registry of live tables it +/// consults. +mod fault { + use super::Inner; + use std::io; + use std::sync::atomic::{AtomicPtr, Ordering}; + use std::sync::Once; + + // Generous: a test binary builds many machines at once, each with a table. + const MAX_MAPS: usize = 64; + static REGISTRY: [AtomicPtr; MAX_MAPS] = [const { AtomicPtr::new(std::ptr::null_mut()) }; MAX_MAPS]; + static INSTALL: Once = Once::new(); + + pub(super) fn register(inner: *const Inner) -> io::Result { + for (i, slot) in REGISTRY.iter().enumerate() { + if slot + .compare_exchange(std::ptr::null_mut(), inner as *mut Inner, Ordering::AcqRel, Ordering::Relaxed) + .is_ok() + { + return Ok(i); + } + } + Err(io::Error::other(format!("devmap: more than {MAX_MAPS} tables alive"))) + } + + pub(super) fn unregister(i: usize) { + if i < MAX_MAPS { + REGISTRY[i].store(std::ptr::null_mut(), Ordering::Release); + } + } + + /// Offer a faulting host address to every live table. + pub(super) fn dispatch(addr: usize) -> bool { + REGISTRY.iter().any(|slot| { + let p = slot.load(Ordering::Acquire); + // SAFETY: a registered table stays alive until it unregisters in + // `Drop`, and nothing dispatches through a table being dropped. + !p.is_null() && unsafe { (*p).fault(addr) } + }) + } + + pub(super) fn install() { + INSTALL.call_once(imp::install); + } + + pub(super) fn emit(bytes: &[u8]) { + imp::emit(bytes); + } + + #[cfg(unix)] + mod imp { + //! `SIGSEGV` (Linux, and macOS for unmapped pages) and `SIGBUS` + //! (macOS for `PROT_NONE` pages). Whatever handler was installed + //! before — Rust's own stack-overflow detector — is chained to for + //! every fault that is not ours, so nothing else changes. + use std::cell::UnsafeCell; + + struct Prev(UnsafeCell<[libc::sigaction; 2]>); + // SAFETY: written once under `INSTALL`, read only from the handler. + unsafe impl Sync for Prev {} + static PREV: Prev = Prev(UnsafeCell::new([unsafe { std::mem::zeroed() }; 2])); + + const SIGNALS: [libc::c_int; 2] = [libc::SIGSEGV, libc::SIGBUS]; + + unsafe extern "C" fn handler(sig: libc::c_int, info: *mut libc::siginfo_t, ctx: *mut libc::c_void) { + if !info.is_null() { + let addr = unsafe { (*info).si_addr() } as usize; + if super::dispatch(addr) { + return; // the load re-executes against the new page + } + } + let idx = SIGNALS.iter().position(|&s| s == sig).unwrap_or(0); + let prev = unsafe { (*PREV.0.get())[idx] }; + if prev.sa_flags & libc::SA_SIGINFO != 0 { + let f: unsafe extern "C" fn(libc::c_int, *mut libc::siginfo_t, *mut libc::c_void) = + unsafe { std::mem::transmute(prev.sa_sigaction) }; + unsafe { f(sig, info, ctx) }; + } else if prev.sa_sigaction == libc::SIG_DFL || prev.sa_sigaction == libc::SIG_IGN { + // Restore the default and return: the access repeats and the + // default action (a core dump) takes it from there. + unsafe { libc::sigaction(sig, &prev, std::ptr::null_mut()) }; + } else { + let f: unsafe extern "C" fn(libc::c_int) = unsafe { std::mem::transmute(prev.sa_sigaction) }; + unsafe { f(sig) }; + } + } + + pub(super) fn install() { + unsafe { + let mut sa: libc::sigaction = std::mem::zeroed(); + // Through `*const ()`: see `hwwatch.rs` on why not a direct cast. + sa.sa_sigaction = handler as *const () as usize; + // ONSTACK so a thread with an alternate stack keeps using it, + // as Rust's handler expects; NODEFER so a fault inside the + // chained handler is not silently blocked. + sa.sa_flags = libc::SA_SIGINFO | libc::SA_ONSTACK | libc::SA_NODEFER; + libc::sigemptyset(&mut sa.sa_mask); + for (i, &sig) in SIGNALS.iter().enumerate() { + libc::sigaction(sig, &sa, &mut (*PREV.0.get())[i]); + } + } + } + + pub(super) fn emit(bytes: &[u8]) { + unsafe { libc::write(2, bytes.as_ptr() as *const libc::c_void, bytes.len()) }; + } + } + + #[cfg(windows)] + mod imp { + //! A vectored exception handler, added at the head of the chain. + //! `crash_diag` adds its own at the head earlier in `main`, so this + //! one runs first and a handled fault never reaches its log; every + //! other exception continues down the chain untouched. + use windows_sys::Win32::System::Diagnostics::Debug::{AddVectoredExceptionHandler, EXCEPTION_POINTERS}; + + const EXCEPTION_ACCESS_VIOLATION: i32 = 0xC000_0005u32 as i32; + const EXCEPTION_CONTINUE_EXECUTION: i32 = -1; + const EXCEPTION_CONTINUE_SEARCH: i32 = 0; + + unsafe extern "system" fn handler(info: *mut EXCEPTION_POINTERS) -> i32 { + if info.is_null() { + return EXCEPTION_CONTINUE_SEARCH; + } + let rec = unsafe { (*info).ExceptionRecord }; + if rec.is_null() { + return EXCEPTION_CONTINUE_SEARCH; + } + let rec = unsafe { &*rec }; + if rec.ExceptionCode == EXCEPTION_ACCESS_VIOLATION && rec.NumberParameters >= 2 { + // ExceptionInformation[0] is the access kind, [1] the address. + let addr = rec.ExceptionInformation[1]; + if super::dispatch(addr) { + return EXCEPTION_CONTINUE_EXECUTION; + } + } + EXCEPTION_CONTINUE_SEARCH + } + + pub(super) fn install() { + unsafe { + let _ = AddVectoredExceptionHandler(1, Some(handler)); + } + } + + pub(super) fn emit(bytes: &[u8]) { + use std::io::Write; + let _ = std::io::stderr().write_all(bytes); + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A device with a body, so two of them never share an address. + struct Dummy(#[allow(dead_code)] u64); + impl BusDevice for Dummy {} + + fn dev(tag: u64) -> Entry { + Box::leak(Box::new(Dummy(tag))) as *const Dummy as Entry + } + + fn entry_at(m: &SparseDeviceMap, slot: u64) -> Entry { + unsafe { std::ptr::read_volatile(m.base().add(slot as usize)) } + } + + #[test] + fn a_uniform_fill_shares_one_constant_page() { + let m = SparseDeviceMap::new().unwrap(); + let (d0, d1) = (dev(0), dev(1)); + m.set_default(d0).unwrap(); + let per = m.page_slots(); + m.fill(0, 4 * per, d1).unwrap(); + let s = m.stats(); + assert_eq!((s.shared, s.private, s.const_pages, s.pool_pages), (4, 0, 2, 0)); + for slot in [0, per - 1, per, 3 * per + 7, 4 * per - 1] { + assert!(same(entry_at(&m, slot), d1), "slot {slot:#x}"); + } + assert!(!m.is_mapped(4 * per)); + // A second identical fill maps nothing new. + m.fill(0, 4 * per, d1).unwrap(); + assert_eq!(m.stats().const_pages, 2); + } + + #[test] + fn a_partial_fill_makes_a_private_page_and_uniformity_frees_it() { + let m = SparseDeviceMap::new().unwrap(); + let (d0, d1, d2) = (dev(0), dev(1), dev(2)); + m.set_default(d0).unwrap(); + let per = m.page_slots(); + m.fill(0, per, d1).unwrap(); + m.fill(5, 4, d2).unwrap(); + let s = m.stats(); + assert_eq!((s.shared, s.private, s.pool_pages, s.pool_free), (0, 1, POOL_PAGES, POOL_PAGES - 1)); + assert!(same(entry_at(&m, 4), d1)); + assert!(same(entry_at(&m, 5), d2)); + assert!(same(entry_at(&m, 8), d2)); + assert!(same(entry_at(&m, 9), d1)); + assert!(m.all_are(5, 4, d2)); + assert!(!m.all_are(4, 4, d2)); + // Changing one entry of a private page swaps in a fresh pool page. + m.fill(7, 1, d1).unwrap(); + assert!(same(entry_at(&m, 7), d1)); + assert_eq!(m.stats().pool_free, POOL_PAGES - 1, "the old pool page was released"); + // Back to uniform: the page becomes a constant view again. + m.fill(5, 4, d1).unwrap(); + let s = m.stats(); + assert_eq!((s.shared, s.private, s.pool_free), (1, 0, POOL_PAGES)); + assert!(same(entry_at(&m, 6), d1)); + } + + #[test] + fn a_partial_fill_of_an_untouched_page_is_padded_with_the_default() { + let m = SparseDeviceMap::new().unwrap(); + let (d0, d1) = (dev(0), dev(1)); + assert!(m.fill(10, 2, d1).is_err(), "no default yet"); + m.set_default(d0).unwrap(); + m.fill(10, 2, d1).unwrap(); + let per = m.page_slots(); + assert!(same(entry_at(&m, 0), d0)); + assert!(same(entry_at(&m, 9), d0)); + assert!(same(entry_at(&m, 10), d1)); + assert!(same(entry_at(&m, 11), d1)); + assert!(same(entry_at(&m, per - 1), d0)); + assert!(!m.is_mapped(per)); + assert_eq!(m.get(per), None); + assert!(same(m.get(10).unwrap(), d1)); + } + + #[test] + fn a_load_from_an_untouched_page_faults_in_the_default() { + let m = SparseDeviceMap::new().unwrap(); + let (d0, d1) = (dev(0), dev(1)); + m.set_default(d0).unwrap(); + let per = m.page_slots(); + m.fill(0, per, d1).unwrap(); + // Well above anything written, as an Octane probe above 4 GB would be. + let slot = (1u64 << (PHYS_BITS - SLOT_SHIFT - 1)) + 3 * per + 17; + assert!(!m.is_mapped(slot)); + let e = entry_at(&m, slot); + assert!(same(e, d0), "the faulted-in page answers with the default"); + assert!(m.is_mapped(slot)); + let s = m.stats(); + assert_eq!(s.faults, 1); + assert_eq!(s.first_faults, vec![slot << SLOT_SHIFT]); + // The page is a shared view of the default's constant page. + assert_eq!((s.shared, s.private), (2, 0)); + // Loading it again is an ordinary load. + assert!(same(entry_at(&m, slot + 1), d0)); + assert_eq!(m.stats().faults, 1); + } + + #[test] + fn runs_coalesce_across_pages_and_report_gaps() { + let m = SparseDeviceMap::new().unwrap(); + let (d0, d1, d2) = (dev(0), dev(1), dev(2)); + m.set_default(d0).unwrap(); + let per = m.page_slots(); + m.fill(0, 2 * per, d1).unwrap(); + m.fill(2 * per - 3, 5, d2).unwrap(); // straddles pages 1 and 2 + m.fill_bytes(0x1FC0_0000, 0x1FD0_0000, d0).unwrap(); // 16 slots inside page 0 + let runs = m.runs(); + let mapped: Vec<_> = runs.iter().filter(|r| r.dev.is_some()).collect(); + let starts: Vec<(u64, u64)> = mapped.iter().map(|r| (r.start, r.end)).collect(); + assert_eq!( + starts, + vec![ + (0, 0x1FC0), + (0x1FC0, 0x1FD0), + (0x1FD0, 2 * per - 3), + (2 * per - 3, 2 * per + 2), + (2 * per + 2, 3 * per), + ] + ); + assert!(same(mapped[1].dev.unwrap(), d0)); + assert!(same(mapped[3].dev.unwrap(), d2)); + assert!(same(mapped[4].dev.unwrap(), d0), "rest of page 2 is the default"); + // Everything after page 2 is one untouched run to the end. + let last = runs.last().unwrap(); + assert!(last.dev.is_none()); + assert_eq!((last.start, last.end), (3 * per, SLOTS)); + } + + #[test] + fn same_tells_zero_sized_devices_apart() { + struct A; + struct B; + impl BusDevice for A {} + impl BusDevice for B {} + #[repr(C)] + struct Both(A, B); + let both = Both(A, B); + let a: Entry = &both.0; + let b: Entry = &both.1; + assert!(std::ptr::addr_eq(a, b), "the test needs them to share an address"); + assert!(!same(a, b)); + assert!(same(a, a)); + } +} diff --git a/src/ppmem/mod.rs b/src/ppmem/mod.rs index 4bef4f04..828c7470 100644 --- a/src/ppmem/mod.rs +++ b/src/ppmem/mod.rs @@ -4,14 +4,17 @@ //! (`BusDevice`, `Resettable`, and `Memory`'s inherent methods) but backs the //! guest's physical address space with real host virtual memory, so that //! SIMM mirroring and the low-512KB alias become *mappings* rather than -//! address arithmetic on every access. +//! address arithmetic on every access. `devmap` applies the same idea to +//! the physical address decode table itself. //! //! Full design and rationale: `docs/ppmem-design.md`. //! Verified platform findings: `clanker/build/mmap-rs-fixed-address-aliasing.md`. +pub mod devmap; pub mod map; #[allow(clippy::module_inception)] pub mod ppmem; pub use map::{align_up, granularity, is_aligned, AddrSpace, Prot, SharedMem}; +pub use devmap::SparseDeviceMap; pub use ppmem::{MappedMemory, PpMemSpace, PpMemory, BITMAP_SHIFT, WINDOW_SIZE};