CMake: Port test suite from rspec to CMake

[FtZPetruska]

This ports the test suite that was located under spec/ and fully integrates it into CMake. This removes the dependency on Ruby, and fits better in the build system.

This currently does not run on CI, since the test suite requires commercial WADs to run (DOOM2, HERETIC, and HEXEN). Github provides options that could perhaps make this feasible. But it feels out of scope for this PR.

---

As to how it's done, here are the details:

• An ENABLE_TESTS option is added to control whether or not to include the tests.
• CMake's add_test only allows for a single command to be run, in order for this to work, the tests command are invocations of "TestRunner" scripts, which play back the demos and process the results.
• Each test is run from its own subdirectory, meaning tests can be run in parallel without all writing the analysis.txt and levelstat.txt files to the same directory.
• Tests are only added if the necessary WADs are present.
• When cross-compiling, running through an emulator is supported. Typically add_test would already inject CMAKE_CROSSCOMPILING_EMULATOR in the command, however, since the test commands are just script invocation, this must be done manually. This works pretty seamlessly too, I managed to make a Windows build from my mac and run the entire test suite by just setting the emulator to wine!

[rfomin]

Thank you for your effort, but why should we switch to CMake? Just to eliminate one dependency? I'm not an expert in either the CMake or Ruby languages, so modifying the CMake version would be just as difficult for me as modifying the Ruby version.

CMake's add_test only allows for a single command to be run, in order for this to work, the tests command are invocations of "TestRunner" scripts, which play back the demos and process the results.

That sounds complicated. Other languages support parallel processes just fine. For example, we use Python for small CI test: https://github.com/fabiangreffrath/woof/tree/master/demotest

[Pedro-Beirao]

Its pretty cool to see this implemented in CMake, but I have to agree with rfomin, better to use Python.
Not only because of how easy it is to set it up for the CI, but it can also be very easily expanded into a full testing system outside of this repo.

I want to automatically get demos from dsdarchive and have all the necessary info to perform tests. Ive added some decent api calls to the website, which will allow us to do this (just need to populate a new levelstat field for all demos, and a few other things).

So thank you for the effort, but I dont think this is the way.
Although maybe we could merge this for now?

[FtZPetruska]

This python script from Woof looks neat! And I really hope you can get that test suite based on the dsdarchive website running, that would be awesome! (Although I'm curious about how you would go about demos of commercial WADs).

Here, I went with CMake just so it would integrate neatly into the current build system without relying on another tool. Personally, I had never heard of ruby's rspec and never really looked into the spec directory, so I was quite surprised to find a whole test suite hidden in there! The idea of moving all this inside CMake was to make the workflow look something like this, where running the test suite is just another cmake command:

cmake -S prboom2 -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release
ctest --test-dir build -C Release -j
cmake --install build --config Release

With this PR, I went through the process of moving all the tests into CMake, so they were all listed individually, and in case of failure, could be re-run selectively. On the other hand, the existing rspec test suite could also be run from CMake as such:

add_test(
  NAME demo_test_suite
  COMMAND bundle exec rspec
  WORKING_DIRECTORY "${PROJECT_SOURCE_DIR}/.."
)

With the main caveats being that CMake would only list it as a single test, and that it would require to have a proper ruby environment setup beforehand.

CMake's add_test only allows for a single command to be run, in order for this to work, the tests command are invocations of "TestRunner" scripts, which play back the demos and process the results.

That sounds complicated. Other languages support parallel processes just fine.

Just to clarify that part, CTest is able to run all the tests in parallel. I was mainly referring to how tests are defined within CMake itself. The add_test function only takes a single command for a test, and that command could be anything, an executable, a shell script with several commands, or even invoking another test driver:

find_package(Python COMPONENTS Interpreter)
add_test(
  NAME running_tests_with_python
  COMMAND Python::Interpreter my_python_test_suite.py
)

I understand this is not the way you want testing to look like in the future, and again, I look forward to see what you come up with! So, in the meantime, I'll let y'all decide what you want to do about this PR, there is no harm in closing it if you think it's not worth merging. 👍

[rfomin]

I tried installing Ruby on Windows and running the rspec command. It worked quite nicely with many processes and everything I expected. I'm not against merging this PR, but it doesn't seem to make much of a difference.

And I really hope you can get that test suite based on the dsdarchive website running, that would be awesome! (Although I'm curious about how you would go about demos of commercial WADs).

Since we can't host commercial WADs, we will have to run tests like this locally.

[Pedro-Beirao]

And I really hope you can get that test suite based on the dsdarchive website running, that would be awesome! (Although I'm curious about how you would go about demos of commercial WADs).

Only a a few selected pwad demos would be run in the CI (like woof), the rest would be run locally and have different types of tests. You could choose to do only heretic, or only a certain pwad, or even test against the whole archive if you have the space and patience.

And the demos/wads could also be cached in your machine so that you dont need to download them again every time.

Anyway, that will still take a while, since populating the website db will need a lot of manual overseeing. So I am ok with merging this PR.