document that extern statics may be bigger than the declared size

[RalfJung]

I am fairly sure we discussed at some point whether it's okay to access memory beyond the declared size of an extern static, and @nikic confirmed that this is fine. But I couldn't find any official place where we document this. So, let's add it to the Reference.

I wondered if we also need the converse on the UB page: it is UB to even start the program if an extern static ends up being backed by memory that is smaller or less-aligned than the declared type. What do you think?

Cc @rust-lang/opsem
Fixes rust-lang/unsafe-code-guidelines#259
Fixes rust-lang/unsafe-code-guidelines#554

[Jules-Bertholet]

Perhaps be more explicit about the need to use &raw (and avoid an intermediate reference)?

View changes since the review

[RalfJung]

I have removed the &raw requirement, anticipating #2338. I don't think we want subobject provenance to get in the way here, so this should be fine (if the static is big enough to read element N):

extern { static mut X: (); }

let ptr = &X as *const () as *const i32;
ptr.add(N).read();

This is a bad idea because of aliasing rules (the pointer gets invalidated on writes), but we already lint against that.

[DianaNites]

This would formally resolve/answer rust-lang/unsafe-code-guidelines#259 and maybe some of rust-lang/unsafe-code-guidelines#546 right?

[RalfJung]

The latter has already been resolved by #1657.

And this does resolve the former, yes. Thanks for digging that up.

[RalfJung]

This new kind of UB is kind of remarkable in that it does not require any code to trigger. But I don't see an alternative...

View changes since the review

[bjorn3]

For dynamic libraries that may be used by a PIE executable, the size given by the symbol must be exact given that the PIE executable will emit a copy relocation that copies a block with exactly the size the symbol had at link time to memory the executable image has reserved for this and redirect all accesses to the static to this copy. This way the executable can avoid GOT indirection, which is a slight perf win. And yes, this means adding elements to a static array in a dylib (or otherwise changing the size) is an ABI breaking change on Linux.

View changes since the review

[RalfJung]

Uh... I understand like maybe half of those words. (Can I have some 🥧 please? :D )
"Copying" sounds wrong, statics are places and if you copy them, well, you have two copies so that can't be right?

But it sounds like you are saying linkme and inventory are unsound? IIRC they rely on extern statics that are bigger than declared, filled in by the linker.

[bjorn3]

Basically when linking the executable, space is reserved for each static referenced from a dylib inside the executable based on symbol size. Then at runtime the dynamic linker will copy the contents of the static in the dylib to the reserved space in the executable and redirect all references to this static to the version in the executable. So there are two copies in memory, but only the one inside the executable is observable by the user. The copy inside the dylib is unused and would be safe to unmap from memory after the dynamic linker has copied it.

linkme and inventory don't work across dylib boundaries anyway as they need the section they emit to be contiguous in memory. They would only see items defined inside the same dylib/executable as the one where the access code happened to be codegened.

[RalfJung]

Thanks!

And what does that mean for crates like linkme/inventory?

[bjorn3]

linkme and inventory only see items defined in the same dylib/executable as the access code is codegened. So if you depend on them to have a global view, you will get misbehavior.

[bjorn3]

Only ELF has copy relocations for PIE executables. The other object file formats don't implement this "optimization".

[RalfJung]

Ah, I see.

Not sure if we should make our UB-rules format-dependent, though.

[CAD97]

There are two cases here, IIUC:

• Rust is a dylib using the static. Access to the static goes through GOT lookup, independent of whether Rust is the definition site of the static or not. When linking to the dylib, the linker fixups must supply a memory region at least the declared size.
• Rust is the binary using a static from a dylib. Rust supplies the memory region for the static matching the declared size. At dynamic linking time (before main), the dynamic linker writes the initialization value to the provided memory region for the static.

The memory region of the static always matches the declaration exactly if Rustc is the one allocating it; it's only when Rust is linking to an external memory region that it may potentially be larger. The case where Rust allocates the static and dynamic linking initializes it could be declared morally equivalent to life-before-main w.r.t. the abstract machine, and thus the initialization going out of bounds of the static is an out-of-bounds access unsoundness. Notably, if the Rust code declares a larger static that doesn't require the entire static value to be initialized, it's perfectly fine for the linker to copy initialize a smaller value in that memory region.

In conclusion, I think ELF copy relocation doesn't preclude allowing Rust extern static from linking to a memory region that ends up being larger than declared. But it is another weird case of where the wrong signature in an unsafe extern block can lead to UB even if the symbols go unused by Rust. Which isn't pretty, but it is the reality of today's compiler/linker plumbing.

[RalfJung]

So... do you think the wording in the PR right now is accurate?
Note that this is about extern blocks. If you define a static in a Rust crate, compile that as a dylib, build a binary against that, and then load a different dylib into the binary, then there are all sorts of limitations that we have not even begun to document. That's not what I am trying to do here.

[bjorn3]

Maybe change r[items.extern.attributes.link.kind-raw-dylib.size] to clarify that the size needs to match exactly, but that is effectively already implied when combined with r[items.extern.static.size], so fine either way.

[joshlf]

What is the ground truth with respect to which "less mutable" is evaluated? I understand what "declaring an extern static with some... mutability" means, but what does the mutability of an "actual symbol" refer to?

View changes since the review

[RalfJung]

It refers to whether that symbol is in actually mutable memory. My understanding is that in ELF there are flags indicating which sections are mutable and which are not, and I presume other binary formats have something similar.

[CAD97]

Possible alternative wording instead of symbol: the linking-resolved memory region.

[RalfJung]

I went for "The actual memory that the extern static resolves to ", which is also what I used in the other file.

[RalfJung]

I guess this does make new commitments, we are officially allowing symbols-bigger-than-the-static-they-are-imported-with. So this should probably be FCP'd. I guess t-opsem is enough?

[CAD97]

We probably want to rope the linker-format experts in to the FCP as well at least, given potential limiting factors like ELF copy relocation could exist in other object formats. (Although I don't think ELF copy relocation precludes allowing this.)

[RalfJung]

I think this is ready to go now... what's the best way to make progress @traviscross ? Some sort of FCP for opsem or opsem+lang?

[traviscross]

I think this is ready to go now... what's the best way to make progress @traviscross ? Some sort of FCP for opsem or opsem+lang?

Sounds right. I'll check this over editorially, then propose dual FCP merge.

[traviscross]

@RalfJung: I've pushed some editorial revisions. Please let me know if anything doesn't look right to you.

[traviscross]

Let's propose to adopt these rules.

@rfcbot fcp merge lang,opsem

[rust-rfcbot]

@traviscross has proposed to merge this. The next step is review by the rest of the tagged team members:

• @CAD97
• @RalfJung
• @digama0
• @joshlf
• @joshtriplett
• @nikomatsakis
• @saethlin
• @scottmcm
• @tmandry
• @traviscross

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

cc @rust-lang/lang-advisors: FCP proposed for lang, please feel free to register concerns.
See this document for info about what commands tagged team members can give me.

[RalfJung]

@rfcbot reviewed

[RalfJung]

Say that extern statics resolve to allocations

Hm, this is an interesting one.
Purely on the Rust side, I agree.
OTOH, this tells people what they have to do on the linker level, and there "allocations" isn't really a thing. You have to set up some segments with the right name or size or so, I don't really know.

View changes since the review

[Darksonn]

If a static is associated with a linker section, it's pretty common for said static to point in the middle of the allocation - you often have a static for every entry in the linker section. It's also quite common to have a my_linker_section_end static pointing at the end of said linker section, which is used to iterate over the linker section (together with my_linker_section_begin).

[traviscross]

@RalfJung: Agreed it's not a linker-level term. But whoever provides the symbol has to translate from our terms, so the rule's job is to state the thing they need to satisfy on our side. That thing is an allocation, something that has a base address, size, and mutability and within which pointer arithmetic is possible (which is why the permission to access the extra bytes is meaningful).

See items.static.intro, where we say that a static "represents an allocation in the program". This goes back to your PR #1657 and was later adjusted to your terminology in rust-lang/rust#141224 (see core::ptr, which names statics as an example of an allocation). And the LangRef states the requirement as "the allocation size and alignment of the definition it resolves to must be greater than or equal to that of the declaration".

[traviscross]

@Darksonn: What I'd suggest is that the allocation starts at the address of the extern static rather than at the start of the linker section. I've pushed a commit to clarify this.

[RalfJung]

If a static is associated with a linker section, it's pretty common for said static to point in the middle of the allocation - you often have a static for every entry in the linker section. It's also quite common to have a my_linker_section_end static pointing at the end of said linker section, which is used to iterate over the linker section (together with my_linker_section_begin).

These will all become separate allocations in Rust. You cannot do pointer arithmetic between statics, even if they are in the same underlying linker section.

[traviscross]

I've pushed a rule that makes this disjointness explicit.

[RalfJung]

I see where you are coming from in terms of splitting this into two items. What I dislike about this is that it makes the two rules seem entirely unrelated in the structure of the page, they are just 2 items in a kind of long list.

View changes since the review

[traviscross]

Agreed there is that tension. It's not the only place it comes up in the Reference. Adding identifiers to each rule tends to push things in this direction. I resisted it for some time, for the kind of reasons that you mention. But there is value in the regularity. (And, here in particular, there's value in each rule being stated with the same orientation.)

As a pattern, it helps to close some of the distance between the Reference and the FLS.

[RalfJung]

I am not sure whether it really has to start there. Having extra bytes to the left seems fine? (As long as the bytes are still only in one allocation, of course.)

Cc @nikic does LLVM have any trouble with statics having dereferenceable bytes before their start address?

View changes since the review

[nikic]

Cc @nikic does LLVM have any trouble with statics having dereferenceable bytes before their start address?

Yes: https://github.com/llvm/llvm-project/blob/7159ce54c436f91a9437eced53800b8fcb4c34bc/llvm/lib/Analysis/AliasAnalysis.cpp#L927-L933

Global variables must point to the start of the allocation.

[RalfJung]

If multiple extern statics have the same link_name, then they are all valid aliases of each other AFAIK.

I don't know if there is a way to "remap" symbol names and legally alias two statics with different names?

View changes since the review

[bjorn3]

You can use a linker script to assign the address of one symbol as address of another symbol. Or just assign them both the same fixed address.

[RalfJung]

I assume that will be UB as LLVM assumes statics do not alias, but let's check with @nikic to confirm.

[traviscross]

If multiple extern statics have the same link_name, then they are all valid aliases of each other AFAIK.

Thanks. I've clarified this in the rules.

You can use a linker script to assign the address of one symbol as address of another symbol. Or just assign them both the same fixed address.

It's not clear that the behavior of that is defined.¹

use core::ffi::c_char;
use core::hint::{assert_unchecked, black_box};

unsafe extern "C" {
    // Glibc makes one of these a weak alias for the other.
    static environ: *const *const c_char;
    static __environ: *const *const c_char;
}

fn main() {
    let (a, b) = (&raw const environ, &raw const __environ);
    assert!(a != b);
    assert!(black_box(a) == black_box(b));
    if black_box(a) == black_box(b) {
        unsafe { assert_unchecked(a == b) }; //~ UB.
    }
}

Playground link

Footnotes

1. Edit: I hadn't refreshed and so hadn't seen @RalfJung's concurrent comments. ↩

[joshtriplett]

In particular, an extern static that marks the start of a linker section cannot be used to reach other statics placed in that section.

Does this include walking the entire contents of the section, in the case of sections that have well-defined contents? For instance, one common stunt with linker sections is to have a section that accumulates pointers to things, and walk the section (from a pointer to the beginning to a pointer to the end), looking at each pointer-sized value within. Logically, this isn't accessing "the bytes of another static", it's accessing the contents of the whole section. How can we do that in Rust? Use raw pointers to the start/end of the section?

View changes since the review

[joshtriplett]

Where is this particular restriction coming from? People do this in C all the time, and there's presumably a way to convince LLVM/clang to accept that.

How can we craft this language such that there is a correct way to do this in Rust, interoperating with the same mechanisms used in C, using #[link_section] and not having to use inline assembly?

Note in particular that the exact ordering of the section is not a guarantee. The usual reason for not allowing walks off the end of one object into another is that we don't actually guarantee that they're placed adjacent to each other. But if unsafe code ensures that everything in the section is the same type, or is otherwise walkable in sequence (e.g. every record has a length at the beginning), that should be sufficient for code to walk it from beginning to end. (A combination of linker scripts and section naming can also affect ordering, such as by merging numbered sections in numeric order.)

[Darksonn]

I mean, I guess you can always use the wrapping_* functions to compare the pointers. You just can't use add() or sub() which require both pointers to be in the same allocation.

[RalfJung]

Does this include walking the entire contents of the section, in the case of sections that have well-defined contents? For instance, one common stunt with linker sections is to have a section that accumulates pointers to things, and walk the section (from a pointer to the beginning to a pointer to the end), looking at each pointer-sized value within. Logically, this isn't accessing "the bytes of another static", it's accessing the contents of the whole section. How can we do that in Rust? Use raw pointers to the start/end of the section?

That's completely fine as long as those bytes are not part of any other static.

What's not okay is have two statics alias each other, where you have

extern "C" {
  static mut A: [u8; 16];
  static mut B: [u8; 16];
}

and then you access the same byte (i.e., the same absolute address) via both of them.

What is completely okay is to have

extern "C" {
  static mut A: ();
  static mut B: ();
}

and then say that at A an allocation starts that reaches all the way to B, and at B a zero-sized allocation starts. That's my model for this "beginning and end marker" pattern. There just can't be any other static in between these two then, and pointers derived from B must not be used to access memory (or to do inbounds pointer arithmetic).

Where is this particular restriction coming from?

IIUC, LLVM alias analysis assumes that different statics do not alias.