svm: re-land Firedancer testing harness without git submodules

[cmoyes-jump]

Problem

We need a fast, reliable way to catch regressions across SVM components as we push forward on direct mapping, ABI v2, and other major runtime changes.

This approach does not clone the test-vectors repo, or run test-vectors in CI, and vendors protobuf using a crate instead of a git submodule.

Summary of Changes

• Integrates Firedancer's instr fuzz harness directly into SVM.
• Removes submodule dependency by pulling protosol as a crate instead of a git submodule, eliminating the previous build.rs proto compilation step.
• Ensures the long-running test vectors script does not run in CI.

[mergify]

The Firedancer team maintains a line-for-line reimplementation of the
native programs, and until native programs are moved to BPF, those
implementations must exactly match their Agave counterparts.
If this PR represents a change to a native program implementation (not
tests), please include a reviewer from the Firedancer team. And please
keep refactors to a minimum.

[buffalojoec]

@cmoyes-jump @mjain-jump @topointon-jump Thanks for putting this back up guys.

I think we just need to iterate on the protobuf library, to get rid of those weird wrapper types.

What you can do is actually use cargo links and just keep the protos in a crate in Git almost exactly like you're doing. You just need a little sugar to pull it off.

Check this out: https://github.com/buffalojoec/protolib-example

[buffalojoec]

Ah, this stinks. I have an idea that might work for this.

[cmoyes-jump]

Updated to use Cargo links

[LucasSte]

Isn't this going to panic whenever we run cargo build without the environment variable set? I think that is a nuisance to Agave developers, and a potential problem for CI.

[buffalojoec]

Check out my example above (here). It's not a variable we set, but a build step by the dependency crate. It should never panic.

[cmoyes-jump]

This variable gets automatically set by Cargo during the build process.

[buffalojoec]

Ship it! 🚢

For anyone on-looking, a few use-cases for this library, which will come with us to the new SVM repo:

• Replace all usages of Bank in programs/sbf
• Mature tooling for regression testing two versions of a program (ie. a Pinocchio-ified version vs. the original)
• Regression tests between SVM versions

Similar intentions apply to the rest of SolFuzz-Agave.

[buffalojoec]

Ahh, looks like just some minor CI stuff. Gotta fix clippy/fmt and then there's a Windows-specific issue resulting from just the path getting clobbered.

error: failed to run custom build command for `protosol v1.0.1 (https://github.com/firedancer-io/protosol?tag=v1.0.1#c842af8d)`

Caused by:
  process didn't exit successfully: `D:\a\agave\agave\target\debug\build\protosol-e5a6d5a49ffd6ab7\build-script-build` (exit code: 1)
  --- stdout
  cargo:rerun-if-changed=\\?\C:\Users\runneradmin\.cargo\git\checkouts\protosol-696d8300f4248d03\c842af8\proto
  cargo:PROTO_DIR=\\?\C:\Users\runneradmin\.cargo\git\checkouts\protosol-696d8300f4248d03\c842af8\proto
  cargo:rerun-if-changed=\\?\C:\Users\runneradmin\.cargo\git\checkouts\protosol-696d8300f4248d03\c842af8\proto\block.proto
  cargo:rerun-if-changed=\\?\C:\Users\runneradmin\.cargo\git\checkouts\protosol-696d8300f4248d03\c842af8\proto\context.proto
  cargo:rerun-if-changed=\\?\C:\Users\runneradmin\.cargo\git\checkouts\protosol-696d8300f4248d03\c842af8\proto\elf.proto
  cargo:rerun-if-changed=\\?\C:\Users\runneradmin\.cargo\git\checkouts\protosol-696d8300f4248d03\c842af8\proto\invoke.proto
  cargo:rerun-if-changed=\\?\C:\Users\runneradmin\.cargo\git\checkouts\protosol-696d8300f4248d03\c842af8\proto\metadata.proto
  cargo:rerun-if-changed=\\?\C:\Users\runneradmin\.cargo\git\checkouts\protosol-696d8300f4248d03\c842af8\proto\pack.proto
  cargo:rerun-if-changed=\\?\C:\Users\runneradmin\.cargo\git\checkouts\protosol-696d8300f4248d03\c842af8\proto\serialize.proto
  cargo:rerun-if-changed=\\?\C:\Users\runneradmin\.cargo\git\checkouts\protosol-696d8300f4248d03\c842af8\proto\shred.proto
  cargo:rerun-if-changed=\\?\C:\Users\runneradmin\.cargo\git\checkouts\protosol-696d8300f4248d03\c842af8\proto\txn.proto
  cargo:rerun-if-changed=\\?\C:\Users\runneradmin\.cargo\git\checkouts\protosol-696d8300f4248d03\c842af8\proto\type.proto
  cargo:rerun-if-changed=\\?\C:\Users\runneradmin\.cargo\git\checkouts\protosol-696d8300f4248d03\c842af8\proto\vm.proto

  --- stderr
  Error: Custom { kind: Other, error: "protoc failed: block.proto: File not found.\r\n" }

I think you just need to adjust your path logic to support windows.

https://github.com/firedancer-io/protosol/blob/c842af8d921397243ec225d8bfdfa9f002ce0203/build.rs#L5-L9

You should probably either get rid of .canonicalize() or use #[cfg(windows)] to strip the leading \\?\ for Windows paths.