Add option to launch each test in separate process #238

virtuald · 2025-01-29T06:27:24Z

Either specify directly on command line, or add this to pyproject.toml:

[tool.robotpy.pyfrc]
isolated = true

If it works out, expect to make this default in 2026. Replaces #236

Currently, this almost works, but I haven't tried it on Windows yet.

The examples/Timed project works
The examples/Physics project does not work yet (any project that has a embedded 'tests' directory). Need to get rid of directory changing as auscompgeek suggested.
This will work if the project has no test directory and the user runs robotpy tests --builtin

pyfrc/mains/cli_test.py

virtuald · 2025-01-31T05:16:12Z

This works in all the cases I tried on Linux.

virtuald · 2025-01-31T05:43:07Z

Potentially blocked on robotpy/mostrobotpy#146

auscompgeek

I tried this on my team's robot code today. Unit tests run fine, but the pyfrc tests mysteriously fail with no further information.

❱ .venv/bin/robotpy test --isolated -- --maxfail=0
12:22:55:462 INFO    : faulthandler        : registered SIGUSR2 for PID 153549
[phoenix] CANbus Connected: sim
[phoenix] CANbus Network Up: sim
============================= test session starts ==============================
platform linux -- Python 3.13.1, pytest-8.3.3, pluggy-1.5.0 -- /home/davo/dev/frc/thedropbears/pyreefscape/.venv/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(PosixPath('/home/davo/dev/frc/thedropbears/pyreefscape/.hypothesis/examples'))
rootdir: /home/davo/dev/frc/thedropbears/pyreefscape
configfile: pyproject.toml
testpaths: tests
plugins: integration-0.2.3, reraise-2.1.2, hypothesis-6.119.3, typeguard-4.3.0
collected 25 items                                                             

tests/pyfrc_test.py::test_disabled FAILED                                [  4%]
tests/pyfrc_test.py::test_operator_control FAILED                        [  8%]
tests/test_caching.py::test_cache_per_loop PASSED                        [ 12%]
tests/test_constrain_angle.py::test_happy PASSED                         [ 16%]
tests/test_constrain_angle.py::test_all PASSED                           [ 20%]
tests/test_constrain_angle.py::test_zero PASSED                          [ 24%]
tests/test_constrain_angle.py::test_edge_pos PASSED                      [ 28%]
tests/test_constrain_angle.py::test_edge_neg PASSED                      [ 32%]
tests/test_constrain_angle.py::test_revolution_pos PASSED                [ 36%]
tests/test_constrain_angle.py::test_revolution_neg PASSED                [ 40%]
tests/test_constrain_angle.py::test_one_wrap_positive_half PASSED        [ 44%]
tests/test_constrain_angle.py::test_one_wrap_negative_half PASSED        [ 48%]
tests/test_functions.py::test_rate_limit2d PASSED                        [ 52%]
tests/test_functions.py::test_rate_limit_2d_limit PASSED                 [ 56%]
tests/test_functions.py::test_clamp2d_noconstrain PASSED                 [ 60%]
tests/test_functions.py::test_clamp2d_constrain PASSED                   [ 64%]
tests/test_scalers.py::test_deadzone PASSED                              [ 68%]
tests/test_scalers.py::test_deadzone_zero_threshold PASSED               [ 72%]
tests/test_scalers.py::test_exponential PASSED                           [ 76%]
tests/test_scalers.py::test_scale_value PASSED                           [ 80%]
tests/fuzz_test.py::test_fuzz[Blue2] SKIPPED (Integration tests skipped) [ 84%]
tests/fuzz_test.py::test_fuzz[Red1] SKIPPED (Integration tests skipped)  [ 88%]
tests/fuzz_test.py::test_fuzz_test SKIPPED (Integration tests skipped)   [ 92%]
tests/autonomous_test.py::test_all_autonomous[Red] SKIPPED (Slow int...) [ 96%]
tests/autonomous_test.py::test_all_autonomous[Blue] SKIPPED (Slow in...) [100%]

=================================== FAILURES ===================================
________________________________ test_disabled _________________________________
Test failed in subprocess: tests/pyfrc_test.py::test_disabled
----------------------------- Captured stdout call -----------------------------
[phoenix] CANbus Connected: sim
[phoenix] CANbus Network Up: sim
============================= test session starts ==============================
collecting ... collected 1 item

tests/pyfrc_test.py::test_disabled FAILED                                [100%]

============================== 1 failed in 0.12s ===============================
----------------------------- Captured stderr call -----------------------------
12:22:56:843 INFO    : pyfrc.physics       : Physics support successfully enabled
[phoenix-diagnostics] Server shutdown cleanly. (dur:0)
	
[phoenix] Library shutdown cleanly
	
____________________________ test_operator_control _____________________________
Test failed in subprocess: tests/pyfrc_test.py::test_operator_control
----------------------------- Captured stdout call -----------------------------
[phoenix] CANbus Connected: sim
[phoenix] CANbus Network Up: sim
============================= test session starts ==============================
collecting ... collected 1 item

tests/pyfrc_test.py::test_operator_control FAILED                        [100%]

============================== 1 failed in 0.14s ===============================
----------------------------- Captured stderr call -----------------------------
12:22:58:254 INFO    : pyfrc.physics       : Physics support successfully enabled
[phoenix] Library initialization is complete.
	
[phoenix-diagnostics] Server shutdown cleanly. (dur:0)
	
[phoenix] Library shutdown cleanly
	
=========================== short test summary info ============================
FAILED tests/pyfrc_test.py::test_disabled - Failed: Test failed in subprocess: tests/pyfrc_test.py::test_disabled
FAILED tests/pyfrc_test.py::test_operator_control - Failed: Test failed in subprocess: tests/pyfrc_test.py::test_operator_control
=================== 2 failed, 18 passed, 5 skipped in 29.43s ===================
[phoenix-diagnostics] Server shutdown cleanly. (dur:0)
	
[phoenix] Library shutdown cleanly

virtuald · 2025-02-08T06:48:34Z

The --no-summary flag caused the error information to not be displayed. I removed it, now it shows an actionable error. @auscompgeek can you take a look and see if this is good for you now?

auscompgeek · 2025-02-08T13:08:53Z

Got the failure details now:

❱ .venv/bin/robotpy test --isolated -- -x
00:07:58:188 INFO    : faulthandler        : registered SIGUSR2 for PID 38788
[phoenix] CANbus Connected: sim
[phoenix] CANbus Network Up: sim
======================================= test session starts =======================================
platform darwin -- Python 3.13.1, pytest-8.3.3, pluggy-1.5.0 -- /Users/davidv/dev/frc/thedropbears/pyreefscape/.venv/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(PosixPath('/Users/davidv/dev/frc/thedropbears/pyreefscape/.hypothesis/examples'))
rootdir: /Users/davidv/dev/frc/thedropbears/pyreefscape
configfile: pyproject.toml
testpaths: tests
plugins: integration-0.2.3, hypothesis-6.119.3, reraise-2.1.2, typeguard-4.3.0
collected 26 items

tests/pyfrc_test.py::test_disabled FAILED                                                   [  3%]

============================================ FAILURES =============================================
__________________________________________ test_disabled __________________________________________
Test failed in subprocess: tests/pyfrc_test.py::test_disabled (exit code 1)
-------------------------------------- Captured stdout call ---------------------------------------
[phoenix] CANbus Connected: sim
[phoenix] CANbus Network Up: sim
============================= test session starts ==============================
collecting ... collected 1 item

tests/pyfrc_test.py::test_disabled FAILED                                [100%]

=================================== FAILURES ===================================
________________________________ test_disabled _________________________________

self = <pyfrc.test_support.controller.TestController object at 0x11aac30e0>
robot = <weakproxy at 0x11ab1d260; to 'robot.MyRobot' at 0x11aace270>

    def _robot_thread(self, robot):
        with self._cond:
            self._robot_started = True
            self._cond.notify_all()

        with self._reraise(catch=True):
            assert robot is not None  # shouldn't happen...

            robot._TestRobot__robotInitialized = self._on_robot_initialized

            try:
>               robot.startCompetition()

../../robotpy/pyfrc/pyfrc/test_support/controller.py:42:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.venv/lib/python3.13/site-packages/magicbot/magicrobot.py:365: in startCompetition
    self.robotInit()
../../robotpy/pyfrc/pyfrc/test_support/pytest_plugin.py:53: in robotInit
    super().robotInit()
.venv/lib/python3.13/site-packages/magicbot/magicrobot.py:111: in robotInit
    self._automodes = AutonomousModeSelector("autonomous")
.venv/lib/python3.13/site-packages/robotpy_ext/autonomous/selector.py:145: in __init__
    instance = obj(*args, **kwargs)
autonomous/coral_auto.py:8: in __init__
    super().__init__("Coral")
autonomous/coral_auto_base.py:28: in __init__
    self.trajectory = choreo.load_swerve_trajectory(trajectory_name)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

trajectory_name = 'Coral'

    def load_swerve_trajectory(trajectory_name: str) -> SwerveTrajectory:
        """Load a swerve trajectory from a file.

        Parameter ``trajectory_name``:
            The path name in Choreo, which matches the file name in the deploy
            directory. Do not include ".traj" here.
        """
>       with open(
            os.path.join(getDeployDirectory(), "choreo", trajectory_name + ".traj"),
            "r",
            encoding="utf-8",
        ) as trajectory_file:
E       FileNotFoundError: [Errno 2] No such file or directory: '/Users/davidv/dev/frc/thedropbears/pyreefscape/.venv/bin/deploy/choreo/Coral.traj'

.venv/lib/python3.13/site-packages/choreo/__init__.py:139: FileNotFoundError

During handling of the above exception, another exception occurred:

item = <Function test_disabled>

    @pytest.hookimpl(hookwrapper=True, tryfirst=True)
    def pytest_runtest_call(item: Item):
        result = yield
        if hasattr(item, "funcargs") and "reraise" in item.funcargs:
            reraise = item.funcargs["reraise"]

            # Override any non-re-raised exception in the main thread by calling `reraise()`
            if result.excinfo is None or not hasattr(
                result.excinfo[1], "_is_from_pytest_reraise"
            ):
>               reraise()

.venv/lib/python3.13/site-packages/pytest_reraise/reraise.py:136:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.venv/lib/python3.13/site-packages/pytest_reraise/reraise.py:72: in __call__
    raise e
../../robotpy/pyfrc/pyfrc/test_support/controller.py:46: in _robot_thread
    robot.endCompetition()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <robot.MyRobot object at 0x11aace270>

    def endCompetition(self) -> None:
        self.__done = True
>       self._automodes.endCompetition()
E       AttributeError: 'MyRobot' object has no attribute '_automodes'

.venv/lib/python3.13/site-packages/magicbot/magicrobot.py:384: AttributeError
---------------------------- Captured stdout setup -----------------------------
Not loading CameraServerShared
----------------------------- Captured stdout call -----------------------------
DataLog: Logging to '/Users/davidv/dev/frc/thedropbears/pyreefscape/logs/FRC_TBD_fd0c1ee96732ae68.wpilog' (12.9 GiB free space)
----------------------------- Captured stderr call -----------------------------
00:07:58:862 INFO    : autonomous          : Begin initializing autonomous mode switcher
------------------------------ Captured log call -------------------------------
INFO     autonomous:selector.py:86 Begin initializing autonomous mode switcher
=========================== short test summary info ============================
FAILED tests/pyfrc_test.py::test_disabled - AttributeError: 'MyRobot' object ...
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
============================== 1 failed in 0.11s ===============================
-------------------------------------- Captured stderr call ---------------------------------------
00:07:58:795 INFO    : pyfrc.physics       : Physics support successfully enabled
[phoenix-diagnostics] Server shutdown cleanly. (dur:0)

[phoenix] Library shutdown cleanly

===================================== short test summary info =====================================
FAILED tests/pyfrc_test.py::test_disabled - Failed: Test failed in subprocess: tests/pyfrc_test.py::test_disabled (exit code 1)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================== 1 failed in 0.77s ========================================
[phoenix-diagnostics] Server shutdown cleanly. (dur:0)

[phoenix] Library shutdown cleanly

Either specify directly on command line, or add tool.robotpy.pyfrc.multiprocess to pyproject.toml. If it works out, expect to make this default in 2026

virtuald requested review from auscompgeek and TheTripleV January 29, 2025 06:27

virtuald mentioned this pull request Jan 29, 2025

Use pytest-xdist to isolate tests #236

Closed

virtuald commented Jan 29, 2025

View reviewed changes

pyfrc/mains/cli_test.py Outdated Show resolved Hide resolved

virtuald force-pushed the separate-process branch 3 times, most recently from d255b31 to 1c67297 Compare January 31, 2025 05:16

virtuald force-pushed the separate-process branch from 1c67297 to cb80e36 Compare January 31, 2025 05:20

auscompgeek reviewed Feb 2, 2025

View reviewed changes

virtuald force-pushed the separate-process branch from 2212aeb to 34bcb1b Compare February 8, 2025 06:49

virtuald force-pushed the separate-process branch from 4581ed3 to 9c7267a Compare February 9, 2025 03:53

Add option to launch each test in separate process

e4046c9

Either specify directly on command line, or add tool.robotpy.pyfrc.multiprocess to pyproject.toml. If it works out, expect to make this default in 2026

virtuald force-pushed the separate-process branch from e313b17 to e4046c9 Compare February 12, 2025 05:43

virtuald merged commit 27fe979 into main Feb 12, 2025
18 checks passed

virtuald deleted the separate-process branch February 12, 2025 05:45

This was referenced Feb 16, 2025

[BUG] Cleanup doesn't occur correctly if tests fail #229

Closed

Run tests in subprocess instead of reloading robot code #230

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add option to launch each test in separate process #238

Add option to launch each test in separate process #238

Uh oh!

virtuald commented Jan 29, 2025 •

edited

Loading

Uh oh!

Uh oh!

virtuald commented Jan 31, 2025

Uh oh!

virtuald commented Jan 31, 2025

Uh oh!

auscompgeek left a comment •

edited

Loading

Uh oh!

virtuald commented Feb 8, 2025

Uh oh!

auscompgeek commented Feb 8, 2025

Uh oh!

Uh oh!

Uh oh!

Add option to launch each test in separate process #238

Add option to launch each test in separate process #238

Uh oh!

Conversation

virtuald commented Jan 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

virtuald commented Jan 31, 2025

Uh oh!

virtuald commented Jan 31, 2025

Uh oh!

auscompgeek left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

virtuald commented Feb 8, 2025

Uh oh!

auscompgeek commented Feb 8, 2025

Uh oh!

Uh oh!

Uh oh!

virtuald commented Jan 29, 2025 •

edited

Loading

auscompgeek left a comment •

edited

Loading