How can I embed tests into my binary with ginkgo/v2 to have a command that runs them in parallel

[tnozicka]

Hi @onsi,

thank you for building such a great framework! 🙇

For a long time, since I bumped into ginkgo in Kubernetes, this is my choice for e2e tests in Golang, often also targeting Kubernetes clusters. In the days of ginkgo/v1, OpenShift (a distribution of Kubernetes) and some other projects started shipping embedded tests suites in their binaries, so these tests can be run from a container image. With that, you can run a container in a Kubernetes cluster (or locally) and assess whether say the conformance tests pass or something is broken. Obviously, at that place you can't run ginkgo CLI with a path to the source code (which is not there) and you kind of also want to wrap the CLI into something that defines suites / presets some focus strings and such.

I haven't figured out how to reuse the ginkgo/v2 facilities to build something similar without rewriting or hacking ginkgo. Effectively, all the parallel abstractions are hidden in internal/parallel_support which can't be vendored, so it's very hard to rebuild running tests in parallel.

What I've seen done (or done myself) was:
a) instead of vendoring, copy and rename the internal package and duplicate the command execution logic 😞
https://github.com/scylladb/scylla-operator/blob/67c1ce99a003ec530da47f5d711f7888825b41aa/pkg/cmd/tests/tests_run.go#L233-L432
b) OpenShift has build a custom parallel queue
https://github.com/openshift/origin/blob/0060ccdd4d01b70980ee8b8108829e16b8714fa7/pkg/test/ginkgo/cmd_runsuite.go#L368-L369

Is there a chance some of these abstraction could be exposed so people don't have to suffer through rebuilding this?

Or am I missing how this could be done with the existing code base?

[onsi]

My sense is that, in general, stuff like this ends up getting shipped in a docker container. Of course shipping source and adding a compile step wouldn't make much sense. But you can ship the ginkgo cli and precompiled test suites (via ginkgo build) in the image. You could then simply invoke ginkgo -p path/to/binary. Ginkgo will spin that binary up and run the tests in parallel. And you could ship a third binary that orchestrates everything that defines the suites/presets you're describing. (To the extent that this is end-user facing it makes sense to me that you would wrap the whole thing in a binary or web ui that knows how to invoke/orchestrate ginkgo and the test binaries under the hood).

The parallel test support is indeed packaged up in an internal library and I don't have plans for exposing it. the main reason being that Ginkgo's isolation model requires you to run the different parallel test runners in separate processes. I could imagine adding all that logic into a binary that refers to itself and knows how to spin itself up in either server/cli mode (to aggregate the results) or in test runner mode (to run the suite) - and I could imagine making that a straightforward process with a supported ginkgo subpackage that does all that for you. That would be a chunk of work that I won't be able to get to in the near-term.

Given that - maybe you can share if there are issues with the ginkgo + test binaries approach I outlined above that make that approach a dealbreaker?

[fischerman]

The parallel test support is indeed packaged up in an internal library and I don't have plans for exposing it.

I believe, we have a different use case for this feature. We currently use Ginkgo for integration tests (Kubernetes) and use plain Go for unit tests. Because we do both in one package, we call go test to run both types. It would be really useful to run the Ginkgo tests in parallel. But as far as I understand we need the Ginkgo CLI for that. So we would run Ginkgo tests with Ginkgo und Go tests with Go and there is no nice way to "combine" them.