[NativeAOT-LLVM] Null reference exceptions and transient byrefs

[SingleAccretion]

We have this comment in codegen:

// The frontend's contract with the backend is that it will not insert null checks for accesses which
// are inside the "[0..compMaxUncheckedOffsetForNullObject]" range. Thus, we usually need to check not
// just for "null", but "null + small offset". However, for TYP_REF, we know it will either be a valid
// object on heap, or null, and can utilize the more direct form.

Using the "long form" nullcheck costs a lot of code size (measured to be > 1.7% on WasmDebugging), both due to the explicitly longer WASM code sequence and the opaqueness of the check to LLVM.

Thus, the tasks in this issue are as follows:

1. Rationalize the current "transient invalid byrefs allowed... but only briefly" null checking model.
2. Stop using the long form checks for WASM by setting compMaxUncheckedOffsetForNullObject to 0. This would also allow us to delete the wasm-global-base argument and the (implicit) ABI coupling it creates.
3. Fix all (upstream) RyuJit bugs due to this. Ensure it is tested upstream (there is already a stress mode, but the value used is a bit larger).
4. Fix all (upstream) RyuJit optimization deficiencies due to this. Currently it results in a 1.8% code size increase.

[SingleAccretion]

For !fgIsBigOffset(...) offsets, the transformation that happens in fgMorphField loses information about the original shape of the null check.

Consider:

*(p + 4) // ldloc p; ldc.i32 4; add; ldind.i

The semantics of this are:

1. Add p and 4.
2. Null-check the address.
3. Perform the dereference.

There is no requirement for the addition to not overflow. p could be nuint.MaxValue - 3. Without additional ABI guarantees, we cannot make this into a short-form null check even for byrefs.

Now consider:

p->Field; // offsetof(Field) == 4

This involves ldfld/ldflda where the underlying null check is always p == null (assuming p is not a transient byref). However, the transformation leaves no information of this behind for struct fields (for class fields there are field sequences), making it equivalent in the IR to the p + 4 case above.

Side note on class field sequences

Why do class field sequences always permit short-form null-checking? After all, TYP_REF values can be pinned and arbitrary arithmetic performed on them just like with byrefs.

A TYP_REF is either null or points to at least sizeof(<underlying object type>) bytes. With field sequences, we can validate the offset to not be outside those bytes. It is notable that we currently also allow "one past end" LEAs too, which would violate the "physical" null checking model given an object allocated exactly at the end of the address space.

It is useful to consider how this transformation changes semantics in corner cases. We go from p != null; hardware_load(p + offset) to p + offset < null_page_size; hardware_load(p + offset) - the difference is in the check. In both cases we assume that p is not a transient byref, i. e. p not in (0, null_page_size). So:

1. p != null

   • p == null
     • NRE
   • p != null
     • hardware_load(p + offset)

2. p + offset < null_page_size

   • p == null
     • offset < null_page_size: NRE
     • offset >= null_page_size: hardware_load(p + offset)
   • p + offset does not overflow
     • hardware_load(p + offset)
   • p + offset does overflow
     • p + offset < null_page_size: NRE
     • p + offset >= null_page_size: hardware_load(p + offset)

Since the p == null && offset >= null_page_size case is obviously not equivalent and so the transformation not performed for it, we can exclude it:
2) p + offset < null_page_size

• p == null
  • offset < null_page_size: NRE
• p + offset does not overflow
  • hardware_load(p + offset)
• p + offset does overflow
  • NRE (p + offset always less than null_page_size)

We can see that the only change is for the "p + offset does overflow (offset < null_page_size)" case:

1. hardware_load(p + offset) => loading from the zero page.
2. NRE.

In practice, this kind of overflow mostly cannot happen 'legally' on mainstream targets since the upper part of the VA range is inaccessible, therefore supplying such an invalid pointer to ldfld[a] is immediate UB. It is not fully self-consistent, however. CoreCLR's struct size limit is ~128MB, whereas Windows x86 allows pretty "high" pointers, making it possible to induce this overflow (however, only with offsets larger than null_page_size).

Concluding from all of the above, performing the reverse transformation, which is what we want to do with short-form null checks, implies the following change in behavior:

• If offset < null_page_size and p + offset overflows, the load will succeed instead of throwing an NRE.

[SingleAccretion]

For reference, here are some statistics on null checks (1 is short-form, 2 is long-form):

; WasmDebugging
       <=          1 ===>    3345 count ( 60% of total)
        2 ..       2 ===>    2146 count (100% of total)

; HelloWasm
       <=          1 ===>   14253 count ( 73% of total)
        2 ..       2 ===>    5239 count (100% of total)

A lot of long-form null checking occurs due to the structByRef->Field pattern. However, some also occur due to dynamic offset.