Automated Testing and CI/CD

[andrewjong]

Want automated system level and node level tests. Want to be able to run colcon test to automatically run tests. Want to generate reports in an HTML page. Want to be able to test across different environments. Want to be able to scale this to run dozens or hundreds of tests at once.

[andrewjong]

We should use the launch_testing to test nodes https://docs.ros.org/en/rolling/p/launch_testing/
https://github.com/ros2/launch/tree/master/launch_testing

examples from ROS using the launch_testing package to test ROS itself https://github.com/ros2/launch_ros/tree/master/launch_testing_ros

https://autowarefoundation.github.io/autoware-documentation/main/contributing/testing-guidelines/integration-testing/

[andrewjong]

Idea: Composable Tests

I wonder if we could specify a test that says “when the takeoff button is pressed, the robot should reach a height of N meters”
Which actually composes:

1. When the button is pressed, the BT should send a service request to the takeoff landing planner
2. When the TOL planner receives a service request, it should publish a trajectory segment going up
3. When the trajectory controller receives a trajectory, it should move the tracking point up
4. When the tracking point is moved, position_setpoint_pub.py should send mavros commands
5. When the mavros commands are sent, the drone’s odometry should reach the takeoff height

Although it seems like a lot up front, this actually creates tests for many functionalities that can then be easily composed for other types of tests later.
Another advantage of composing it this way is that if it fails, we know exactly at what point it fails

[andrewjong]

[andrewjong]

Brainstorming

Test Env: There's a "test-environment" component. Which could be sim, could be a bag, could be a GUI, could be real world sensors. Basically whatever is providing inputs to our thing under testing.
Test Module: Then there's the thing we want to test. Perhaps a ROS node, or several ROS nodes. The test module takes inputs and produces outputs.
Tester: Then there's the actual tester, which is monitoring the inputs and outputs, and checking produced that the outputs against expected outputs. It can also check to make sure that inputs come in the first place. There can be multiple of these checking for different things.
Test Orchestrator: The thing that launches all the above. Gets fed a specification file to decide what specific tests to launch.
Test Specification: a specification of which test environments, modules, and tests that the orchestrator should spin up.

We could use docker compose to spin up the test-environment, test module, and test monitor all together.
We could provide some parameter list that specifies which testers are spun up?
Perhaps the first one should be for the takeoff/landing planner.
The test should perhaps specify a list of test environments, or test parameters.

GUI Interface

For test reporting.
Can click on a test, and see the list of runs that this test passes, and which fail.

Can also click on a specific run and see which tests pass and which fail. Can play back the bag.

[andrewjong]

I found out this AI tool can solve failing CI/CD tests
https://github.com/All-Hands-AI/OpenHands

[kavin-cmu]

@andrewjong Any given tester would need the runtime configuration (node name, ns, remapped topics etc.) of the node that it would be testing. I am planning to get these by parsing the bingup launch.xml for each respective package. Right now we have a central autonomy bringup launch file which directly calls the nodes, but I would like to have individual package bringup launches which are then called by the overall bringup launch.

Planning to add the custom launch parser util to a new testing_helper package in autonomy/common.

Do you have any suggestions/advice?