Direct Mapping Supercharged

[Lichtso]

Problem

Direct mapping is very complex and still buggy. So far, we have been trying to match the existing behavior exactly, including things which are undefined behavior from the programs (not the runtimes) perspective. E.g. the runtime allows reads (in the realloc padding) beyond the current length of an account. From a programs perspective this is uninitialized memory and no sane program should be reading from it before writing to it first. If we were to stop supporting this and instead throw InstructionError::AccountDataTooSmall we could significantly simplify the direct mapping feature.

Summary of Changes

Removes:

• Support for noncontiguous memops syscalls (memcpy, memmov, memset, memcmp)
• Support for noncontiguous memory mapping in SBPF
• Automatic update of MemoryRegion::host_addr and MemoryRegion::writable in SBPF
• Interior mutability of MemoryRegion (Cell<> around host_addr and writable) in SBPF, which prevented declaring proper lifetimes on the results of map / translate calls in this repo
• Hacks involving Vec::spare_capacity_mut
• Hacks which temporarily marked MemoryRegion as writable and then reverted that using scopeguard::defer
• Zeroing of 10 KiB realloc padding during serialization (serialize_parameters_aligned)
• Zeroing of truncated account length during CPI return (update_caller_account)
• Zeroing of the spare capacity during CPI return (update_caller_account) when the underlying Vec was reallocated
• Copying of the realloc padding back to the runtime in CPI call (update_callee_account)
• Copying of the realloc padding back to the caller in CPI return (update_caller_account)
• Copying of the realloc padding back to the runtime during deserialization (deserialize_parameters_aligned)
• Mapping the realloc padding as a MemoryRegion
• Unused Cargo dependencies, code and tests thereof

Adds:

• New key for the feature gate
• Requires a new SBPF release for: Refactor - AccessViolationHandler sbpf#42
• SBPF memory access violation callback parameters: Address space reserved for the region, access type, VM start address and length
• Two phase translation of mutable references (AccessType::Store) in syscalls which enables use of the borrow checker to ensure that the AccessViolationHandler can not invalidate translated references during the syscall
• Realloc and zero fill account immediately in TransactionContext::access_violation_handler() when resize padding is written to (AccessType::Store)
• EbpfError::AccessViolation error translation to InstructionError::AccountDataTooSmall and InstructionError::InvalidRealloc
• test_deny_access_beyond_current_length() for InstructionError::AccountDataTooSmall and InstructionError::InvalidRealloc
• test_access_violation_handler() for the interaction of TransactionContext::access_violation_handler() and create_memory_region_of_account()

Feature Gate Issue: https://github.com/anza-xyz/feature-gate-tracker/issues/16

[mergify]

The Firedancer team maintains a line-for-line reimplementation of the
native programs, and until native programs are moved to BPF, those
implementations must exactly match their Agave counterparts.
If this PR represents a change to a native program implementation (not
tests), please include a reviewer from the Firedancer team. And please
keep refactors to a minimum.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 84.64646% with 76 lines in your changes missing coverage. Please review.

Project coverage is 82.8%. Comparing base (2d83436) to head (7e2054b).
Report is 72 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##           master    #5871     +/-   ##
=========================================
  Coverage    82.8%    82.8%             
=========================================
  Files         847      847             
  Lines      379509   378425   -1084     
=========================================
- Hits       314372   313572    -800     
+ Misses      65137    64853    -284     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mergify]

Backports to the beta branch are to be avoided unless absolutely necessary for fixing bugs, security issues, and perf regressions. Changes intended for backport should be structured such that a minimum effective diff can be committed separately from any refactoring, plumbing, cleanup, etc that are not strictly necessary to achieve the goal. Any of the latter should go only into master and ride the normal stabilization schedule. Exceptions include CI/metrics changes, CLI improvements and documentation updates on a case by case basis.

[Lichtso]

We could remove BorrowedAccount::make_data_mut() as well, as it only makes sure to also reserve additional 10 KiB while unsharing / making an account unique. The unsharing still is guaranteed to happen by the other operation which make_data_mut precedes. Since all these special methods on BorrowedAccount are used by built-ins, which don't necessarily go though the serializing ABI at all (as they can be top level instructions) and because we don't need to map in zeros in the reserved address space anymore, this might simply be unnecessary. But, in the interest of keeping this PR smaller, it would be removed later.

[LucasSte]

It would be nice to explicitly say that when using translate_mut all previous translations will be invalidated.

[Lichtso]

The borrow checker now enforces this via lifetimes, so you can not get to the situation where a previous translation would be invalidated.

[LucasSte]

Nit: Not necessary for this PR, but renaming aligned would clear a source of confusion.

[Lichtso]

Yep aligned here does not refer to the MemoryMapping but the unaligned ABIv0 of loader-v1.
That name has already been in place before this PR, so I would change it separately later.