Skip to content

[Feature] End-to-end testing of native package installation #6841

@ItsHarper

Description

@ItsHarper
  • I'd be willing to implement this feature (contributing guide)
    • Potentially, but I'm probably not going to find the time in the near-term
  • This feature is important to have in this repository; a contrib plugin wouldn't do

Describe the user story

Users expect that yarn will install the correct version of native dependencies, but the current integration tests do not do a sufficient job of guaranteeing this (see #6840).

The integration test suite has a decent amount of coverage for making sure that the installed packages match what yarn thinks the current platform is, but insufficient coverage (if any at all?) for verifying that its information about the current platform is correct.

Describe the solution you'd like

  • real-native-X packages should be added to pkg-tests-fixtures, where X represents combinations of <os>-<cpu>-<libc>
    • I don't think GitHub actions supports any OSes other than Windows, macOS, and Linux, and all others are pretty rare anyway, so we can skip those.
    • <libc> only applies to Linux
    • Each of these packages should contain a pre-compiled binary that only runs on that platform combination. This should probably just be a "hello world" binary written in C (some languages like go make direct syscalls instead of using libc).
  • A real-native package should be added to pkg-tests-fixtures that specifies all of the real-native-X packages in optionalDependencies, and includes a simple JS script (specified via bin) that tries to run whichever binary got installed.
    *An integration test should be added that uses yarn dlx to execute real-native.
  • An integration test should be added that tries to run all of the real-native-x binaries directly, and expects all of them to fail except the one that matches yarn's view of the current system
  • Alpine Linux should be added to the integration tests matrix to enable testing against musl

Describe the drawbacks of your solution

I dislike the idea of trusting checked-in binaries, especially one that only runs during tests and never gets updated (the xz backdoor was snuck in a similar way).

Describe alternatives you've considered

Perhaps we could cross-compile all of the binaries before kicking off the big integration test matrix. To avoid having to add a dependency on a C compiler when running the test suite locally, we can just skip those tests altogether when not running in CI.

It's important that we don't just compile the binaries within each integration test matrix runner, as it would be easy to accidentally configure it to build the binary for the currently-running system, even if we're ostensibly building a package that doesn't match the currently-running system.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions