Repository navigation
Conversation
|
|
||
| r[items.extern.static.size] | ||
| The actual memory that the extern static resolves to must have *at least* the size and alignment of the type that it was declared with in the extern block. | ||
| If the actual memory is bigger, then it is permitted to access that extra memory by creating a raw pointer to the extern static and then doing appropriate pointer arithmetic. |
There was a problem hiding this comment.
Perhaps be more explicit about the need to use &raw (and avoid an intermediate reference)?
There was a problem hiding this comment.
I have removed the &raw requirement, anticipating #2338. I don't think we want subobject provenance to get in the way here, so this should be fine (if the static is big enough to read element N):
extern { static mut X: (); }
let ptr = &X as *const () as *const i32;
ptr.add(N).read();This is a bad idea because of aliasing rules (the pointer gets invalidated on writes), but we already lint against that.
3a54061 to
45fbcfe
Compare
|
This would formally resolve/answer rust-lang/unsafe-code-guidelines#259 and maybe some of rust-lang/unsafe-code-guidelines#546 right? |
|
The latter has already been resolved by #1657. And this does resolve the former, yes. Thanks for digging that up. |
| * Incorrect use of inline assembly. For more details, refer to the [rules] to follow when writing code that uses inline assembly. | ||
|
|
||
| r[undefined.extern-static] | ||
| * Declaring an `extern static` with some size/alignment/mutability, when the actual symbol this resolves to is smaller / less aligned / less mutable. |
There was a problem hiding this comment.
This new kind of UB is kind of remarkable in that it does not require any code to trigger. But I don't see an alternative...
There was a problem hiding this comment.
seems we sure did declare extern "abi" to be unsafe for a good reason.
ad86228 to
ec8f4ef
Compare
|
|
||
| r[items.extern.static.size] | ||
| The actual memory that the extern static resolves to [must have][extern-static-ub] *at least* the size and alignment of the type that it was declared with in the extern block. | ||
| If the actual memory is bigger, then it is permitted to access that extra memory. |
There was a problem hiding this comment.
For dynamic libraries that may be used by a PIE executable, the size given by the symbol must be exact given that the PIE executable will emit a copy relocation that copies a block with exactly the size the symbol had at link time to memory the executable image has reserved for this and redirect all accesses to the static to this copy. This way the executable can avoid GOT indirection, which is a slight perf win. And yes, this means adding elements to a static array in a dylib (or otherwise changing the size) is an ABI breaking change on Linux.
There was a problem hiding this comment.
Uh... I understand like maybe half of those words. (Can I have some 🥧 please? :D )
"Copying" sounds wrong, statics are places and if you copy them, well, you have two copies so that can't be right?
But it sounds like you are saying linkme and inventory are unsound? IIRC they rely on extern statics that are bigger than declared, filled in by the linker.
There was a problem hiding this comment.
Basically when linking the executable, space is reserved for each static referenced from a dylib inside the executable based on symbol size. Then at runtime the dynamic linker will copy the contents of the static in the dylib to the reserved space in the executable and redirect all references to this static to the version in the executable. So there are two copies in memory, but only the one inside the executable is observable by the user. The copy inside the dylib is unused and would be safe to unmap from memory after the dynamic linker has copied it.
linkme and inventory don't work across dylib boundaries anyway as they need the section they emit to be contiguous in memory. They would only see items defined inside the same dylib/executable as the one where the access code happened to be codegened.
There was a problem hiding this comment.
Thanks!
And what does that mean for crates like linkme/inventory?
There was a problem hiding this comment.
linkme and inventory only see items defined in the same dylib/executable as the access code is codegened. So if you depend on them to have a global view, you will get misbehavior.
There was a problem hiding this comment.
Only ELF has copy relocations for PIE executables. The other object file formats don't implement this "optimization".
There was a problem hiding this comment.
Ah, I see.
Not sure if we should make our UB-rules format-dependent, though.
There was a problem hiding this comment.
There are two cases here, IIUC:
- Rust is a dylib using the static. Access to the static goes through GOT lookup, independent of whether Rust is the definition site of the static or not. When linking to the dylib, the linker fixups must supply a memory region at least the declared size.
- Rust is the binary using a static from a dylib. Rust supplies the memory region for the static matching the declared size. At dynamic linking time (before main), the dynamic linker writes the initialization value to the provided memory region for the static.
The memory region of the static always matches the declaration exactly if Rustc is the one allocating it; it's only when Rust is linking to an external memory region that it may potentially be larger. The case where Rust allocates the static and dynamic linking initializes it could be declared morally equivalent to life-before-main w.r.t. the abstract machine, and thus the initialization going out of bounds of the static is an out-of-bounds access unsoundness. Notably, if the Rust code declares a larger static that doesn't require the entire static value to be initialized, it's perfectly fine for the linker to copy initialize a smaller value in that memory region.
In conclusion, I think ELF copy relocation doesn't preclude allowing Rust extern static from linking to a memory region that ends up being larger than declared. But it is another weird case of where the wrong signature in an unsafe extern block can lead to UB even if the symbols go unused by Rust. Which isn't pretty, but it is the reality of today's compiler/linker plumbing.
There was a problem hiding this comment.
So... do you think the wording in the PR right now is accurate?
Note that this is about extern blocks. If you define a static in a Rust crate, compile that as a dylib, build a binary against that, and then load a different dylib into the binary, then there are all sorts of limitations that we have not even begun to document. That's not what I am trying to do here.
There was a problem hiding this comment.
Maybe change r[items.extern.attributes.link.kind-raw-dylib.size] to clarify that the size needs to match exactly, but that is effectively already implied when combined with r[items.extern.static.size], so fine either way.
9e483d7 to
199eef1
Compare
| * Incorrect use of inline assembly. For more details, refer to the [rules] to follow when writing code that uses inline assembly. | ||
|
|
||
| r[undefined.extern-static] | ||
| * Declaring an `extern static` with some size/alignment/mutability, when the actual symbol this resolves to is smaller / less aligned / less mutable. |
There was a problem hiding this comment.
What is the ground truth with respect to which "less mutable" is evaluated? I understand what "declaring an extern static with some... mutability" means, but what does the mutability of an "actual symbol" refer to?
There was a problem hiding this comment.
It refers to whether that symbol is in actually mutable memory. My understanding is that in ELF there are flags indicating which sections are mutable and which are not, and I presume other binary formats have something similar.
There was a problem hiding this comment.
Possible alternative wording instead of symbol: the linking-resolved memory region.
There was a problem hiding this comment.
I went for "The actual memory that the extern static resolves to ", which is also what I used in the other file.
|
I guess this does make new commitments, we are officially allowing symbols-bigger-than-the-static-they-are-imported-with. So this should probably be FCP'd. I guess t-opsem is enough? |
|
We probably want to rope the linker-format experts in to the FCP as well at least, given potential limiting factors like ELF copy relocation could exist in other object formats. (Although I don't think ELF copy relocation precludes allowing this.) |
37cef41 to
68f8440
Compare
|
I think this is ready to go now... what's the best way to make progress @traviscross ? Some sort of FCP for opsem or opsem+lang? |
Sounds right. I'll check this over editorially, then propose dual FCP merge. |
The new rules talked about "the actual memory" that an extern static resolves to. It'd be better to talk about the *allocation*, as the Reference already does elsewhere. Let's do that, and let's talk about allocations being *larger* rather than *bigger*.
Every word in the Reference is important, so we generally don't emphasize words such as *exactly*, *at least*, and *not*. This helps us reserve emphasis for when we're defining a term. Let's remove the stray emphasis added in the new rules.
The bytes used for an extern static will often fall within a linker *section* that starts earlier. Let's clarify that the *allocation* starts at the address of the extern static.
|
|
||
| r[items.extern.static.size] | ||
| The allocation that an extern static resolves to must have at least the size and alignment of the type it is declared with (see [undefined.extern-static]). If the allocation is larger, accessing the extra bytes is allowed. | ||
| An extern static resolves to an allocation that starts at the address of the static; the allocation must have at least the size and alignment of the type it is declared with (see [undefined.extern-static]). If the allocation is larger, accessing the extra bytes is allowed. |
There was a problem hiding this comment.
I am not sure whether it really has to start there. Having extra bytes to the left seems fine? (As long as the bytes are still only in one allocation, of course.)
Cc @nikic does LLVM have any trouble with statics having dereferenceable bytes before their start address?
There was a problem hiding this comment.
Cc @nikic does LLVM have any trouble with statics having dereferenceable bytes before their start address?
Global variables must point to the start of the allocation.
| r[items.extern.static.allocations-disjoint] | ||
| The allocation that an extern static resolves to does not include the bytes of any other static, even when the linker places the statics in the same section. | ||
|
|
||
| > [!NOTE] | ||
| > Pointer arithmetic from one static cannot reach another. In particular, an extern static that marks the start of a linker section cannot be used to reach other statics placed in that section. |
There was a problem hiding this comment.
If multiple extern statics have the same link_name, then they are all valid aliases of each other AFAIK.
I don't know if there is a way to "remap" symbol names and legally alias two statics with different names?
There was a problem hiding this comment.
You can use a linker script to assign the address of one symbol as address of another symbol. Or just assign them both the same fixed address.
There was a problem hiding this comment.
I assume that will be UB as LLVM assumes statics do not alias, but let's check with @nikic to confirm.
There was a problem hiding this comment.
If multiple extern statics have the same link_name, then they are all valid aliases of each other AFAIK.
Thanks. I've clarified this in the rules.
You can use a linker script to assign the address of one symbol as address of another symbol. Or just assign them both the same fixed address.
It's not clear that the behavior of that is defined.1
use core::ffi::c_char;
use core::hint::{assert_unchecked, black_box};
unsafe extern "C" {
// Glibc makes one of these a weak alias for the other.
static environ: *const *const c_char;
static __environ: *const *const c_char;
}
fn main() {
let (a, b) = (&raw const environ, &raw const __environ);
assert!(a != b);
assert!(black_box(a) == black_box(b));
if black_box(a) == black_box(b) {
unsafe { assert_unchecked(a == b) }; //~ UB.
}
}Footnotes
| The allocation that an extern static resolves to does not include the bytes of any other static, even when the linker places the statics in the same section. | ||
|
|
||
| > [!NOTE] | ||
| > Pointer arithmetic from one static cannot reach another. In particular, an extern static that marks the start of a linker section cannot be used to reach other statics placed in that section. |
There was a problem hiding this comment.
In particular, an extern static that marks the start of a linker section cannot be used to reach other statics placed in that section.
Does this include walking the entire contents of the section, in the case of sections that have well-defined contents? For instance, one common stunt with linker sections is to have a section that accumulates pointers to things, and walk the section (from a pointer to the beginning to a pointer to the end), looking at each pointer-sized value within. Logically, this isn't accessing "the bytes of another static", it's accessing the contents of the whole section. How can we do that in Rust? Use raw pointers to the start/end of the section?
There was a problem hiding this comment.
Where is this particular restriction coming from? People do this in C all the time, and there's presumably a way to convince LLVM/clang to accept that.
How can we craft this language such that there is a correct way to do this in Rust, interoperating with the same mechanisms used in C, using #[link_section] and not having to use inline assembly?
Note in particular that the exact ordering of the section is not a guarantee. The usual reason for not allowing walks off the end of one object into another is that we don't actually guarantee that they're placed adjacent to each other. But if unsafe code ensures that everything in the section is the same type, or is otherwise walkable in sequence (e.g. every record has a length at the beginning), that should be sufficient for code to walk it from beginning to end. (A combination of linker scripts and section naming can also affect ordering, such as by merging numbered sections in numeric order.)
There was a problem hiding this comment.
I mean, I guess you can always use the wrapping_* functions to compare the pointers. You just can't use add() or sub() which require both pointers to be in the same allocation.
There was a problem hiding this comment.
Does this include walking the entire contents of the section, in the case of sections that have well-defined contents? For instance, one common stunt with linker sections is to have a section that accumulates pointers to things, and walk the section (from a pointer to the beginning to a pointer to the end), looking at each pointer-sized value within. Logically, this isn't accessing "the bytes of another static", it's accessing the contents of the whole section. How can we do that in Rust? Use raw pointers to the start/end of the section?
That's completely fine as long as those bytes are not part of any other static.
What's not okay is have two statics alias each other, where you have
extern "C" {
static mut A: [u8; 16];
static mut B: [u8; 16];
}and then you access the same byte (i.e., the same absolute address) via both of them.
What is completely okay is to have
extern "C" {
static mut A: ();
static mut B: ();
}and then say that at A an allocation starts that reaches all the way to B, and at B a zero-sized allocation starts. That's my model for this "beginning and end marker" pattern. There just can't be any other static in between these two then, and pointers derived from B must not be used to access memory (or to do inbounds pointer arithmetic).
Where is this particular restriction coming from?
IIUC, LLVM alias analysis assumes that different statics do not alias.
There was a problem hiding this comment.
People do this in C all the time
I do wonder if it's actually defined in C either, or "happens to work", since provenance is a thing there too, even if the compiler might not in practice -- especially on weak vendor compilers -- take advantage of it.
I do think there's probably a way to do it even if the allocations are different so long as you pick up the correct provenance for the pointers. I could imagine you iterating by integer address, then using from_exposed_provenance for each item you're trying to read to ensure you get the correct provenance on the pointer -- assuming either that linker script addresses are deemed exposed or that you otherwise have exposed those pointers in whatever created those allocations.
There was a problem hiding this comment.
@RalfJung To be explicit, the case I'm thinking of would have two pointers (not Rust allocations, more like raw pointers) provided as symbols by the linker and imported via Rust as extern, and then walked as raw pointers via unsafe code. And then separately you have constant (non-mut) statics that define a non-mutable frozen data structure (e.g. a pointer to an init function) that's being read, not written by any part of the program (it is defined with a fixed value and never changed).
What is the mechanism to define and implement that pattern without using any inline assembly?
There was a problem hiding this comment.
We're discussing this on Zulip: #t-opsem > linker section slice iteration
That some extern statics may be used to access more bytes than would be implied by the size of their types does not imply that they may be used to access the bytes of other statics (with different symbols), even if those bytes are in the same linker section. Let's make this explicit. And since statics with the same symbol do resolve to the same allocation, let's add a rule making that explicit too.
6818de5 to
d3a2589
Compare
|
@rfcbot reviewed |
|
@rfcbot concern document-pattern-for-section-iteration (Filing this about the ongoing conversation iterating on how to correctly do section iteration, without requiring any form of asm. I expect to resolve it and check a box the moment we have a documented incantation for that.) |
|
Ongoing conversation in https://rust-lang.zulipchat.com/#narrow/channel/136281-t-opsem/topic/linker.20section.20slice.20iteration , and the general trend seems to be "yeah, this should work", with some thought going in to making sure that's the case. |
| > | ||
| > In particular, an extern static that marks the start of a linker section cannot be used to access statics with different symbols placed in that section. | ||
|
|
||
| ```rust,standalone_crate |
There was a problem hiding this comment.
This example is pretty subtle. It'd be good to, before this example, precisely document the conditions under which a static can be used to access bytes. Currently the only prose that exists says when it can't be:
In particular, an extern static that marks the start of a linker section cannot be used to access statics with different symbols placed in that section.
| ```rust,standalone_crate | ||
| // A static placed in a section is an allocation of its own, which the | ||
| // section's start symbol cannot be used to read. | ||
|
|
||
| // A static placed in a section named `entries`. | ||
| # #[cfg(target_os = "linux")] | ||
| #[used] | ||
| #[unsafe(link_section = "entries")] | ||
| static ENTRY: [u8; 3] = [1, 2, 3]; | ||
|
|
||
| # #[cfg(target_os = "linux")] | ||
| unsafe extern "C" { | ||
| // On ELF targets, the linker defines these symbols at the start | ||
| // and the end of the `entries` section. | ||
| static __start_entries: [u8; 0]; | ||
| static __stop_entries: [u8; 0]; | ||
| } | ||
|
|
||
| fn main() { | ||
| # #[cfg(target_os = "linux")] { | ||
| let start = &raw const __start_entries as *const u8; | ||
| let stop = &raw const __stop_entries as *const u8; | ||
| // The section holds exactly the bytes of `ENTRY`. | ||
| assert_eq!(stop.addr() - start.addr(), 3); | ||
| assert_eq!(start.addr(), (&raw const ENTRY).addr()); | ||
| // But those bytes are the allocation of `ENTRY`, not of | ||
| // `__start_entries`, so reading them through `start` is undefined | ||
| // behavior. | ||
| //let x = unsafe { *start }; // UB. | ||
| # } | ||
| } | ||
| ``` |
There was a problem hiding this comment.
This is exactly the pattern @joshtriplett asked about, right?
There was a problem hiding this comment.
Yes, it is. I'm hoping the conversation on Zulip turns up a pattern we can document as successfully allowing this. Not least of which because it won't use mut, which should make Rust okay with it, so we just need to make sure LLVM is too.
|
We talked about this in today's lang meeting. I'm hoping that this is a relatively straightforward thing to confirm with LLVM. If it is indeed straightforward, I'm hoping we can document it rather than explicitly restricting it (rather than before, when we implicitly restricted it). If it turns out to not be straightforward, then given that people already do it anyway in C, and need to be able to interoperate with that pattern from Rust, then I can drop the concern and we can proceed with what we have. That said, it'd be helpful to have a non-normative note saying essentially "we know this pattern exists, this currently makes it UB but we're trying to document an interoperable way to interface with it that isn't UB", so that this doesn't get lost. (For calibration on importance, interoperating with this pattern will be essential for writing some types of Linux kernel code.) |
I hope so too. I'm waiting for confirmation from @nikic. |
I wonder about other backends. E.g., GCC closed a similar issue, involving Linux kernel code, as INVALID:1
The kernel side discussion was here. Linus wrote:
And:
Nor does it seem that GCC's position has shifted here.2 When the matter was raised on 2026-08-13, a GCC maintainer responded:
And:
GCC has an issue for documenting the correct way to iterate a "section array"; the proposal is to document the pattern of using inline assembly as an optimization barrier, as they suggested to the kernel developers. cc @rust-lang/wg-gcc-backend Footnotes |
/// Allows registering static items globally, possibly across crates, to iterate over at runtime.
(unstable, global_registration, "CURRENT_RUSTC_VERSION", Some(125119)), |
|
IMO it is the responsibility of GCC to properly model Rust's semantics, not the other way around. This is assuming that the semantics work fine in LLVM, which demonstrates that the semantics can reasonably be implemented. |
I don't disagree. |
|
Looking at the issue, it seems like GCC folks claim that comparing unrelated pointers, such as pointers to two different statics, is UB in C. In think they are wrong even in C, but they are definitely wrong in Rust. This isn't just about statics, or overlapping statics. Pointer comparison is always well-defined in Rust and the GCC backend will have to find a way to deal with this. (Subtracting unrelated pointers is indeed UB, and on the Rust side that would be |
IIUC, |
|
The issue is about == / !=, so the other operators are not relevant here.
|
Sometimes it's useful for multiple crates (or external compilation units) to be able to each add entries at link time to a single table that can be iterated over at runtime. Let's show how to soundly encode this pattern using the guarantees that we're adding.
|
The guarantees we're making here support writing the decentralized registration pattern. I've pushed a guide-like example section that demonstrates how to soundly encode this, including the optimized form, with place-relative offsets, that's used by the Linux kernel. This more-optimized form saves up to 28 bytes per table entry and allows the pages backing the table to be marked read-only, which lets them be shared across processes. Please have a look. My feeling is that even if we were to make the additional guarantee that we're discussing, people would be better off not using it, since there's no way to use |
A table entry is usually ptr-sized, i.e., 4/8 bytes. How can one save 28 bytes over that...? |
See the text in the commit. The relocation record is up to 24 bytes. |
|
Ah so this is about on-disk binary size, not runtime RAM usage? TBH I feel like that example has way too much detail for a section of the Rust reference.^^ Going into so much detail distracts from the main content. But that's a separate question. |
There are three aspects of this form that save RAM:
Yes, there's some tension there. I have ideas about how to better organize or present this long term. I do think this sort of material adds value to the Reference. (Also, even today, looking at the rendered form rather than the Markdown may affect one's sense of this: it's set off visually from the main body of the section in a kind of admonition designed for these asides.) |
I am fairly sure we discussed at some point whether it's okay to access memory beyond the declared size of an
extern static, and @nikic confirmed that this is fine. But I couldn't find any official place where we document this. So, let's add it to the Reference.I wondered if we also need the converse on the UB page: it is UB to even start the program if an extern static ends up being backed by memory that is smaller or less-aligned than the declared type. What do you think?
Cc @rust-lang/opsem
Fixes rust-lang/unsafe-code-guidelines#259
Fixes rust-lang/unsafe-code-guidelines#554