[tools/cli_tests] introduce Multipass CLI tests

[xmkg]

CLI tests aim to test the high-level, user-facing Multipass functionality
via the CLI commands. The tests are written in Python, and the suite uses
pytest as the test executor.

See README.md for more details.

Signed-off-by: Mustafa Kemal Gilor mustafa.gilor@canonical.com

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.97%. Comparing base (4980c78) to head (b215d53).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4220   +/-   ##
=======================================
  Coverage   88.97%   88.97%           
=======================================
  Files         238      238           
  Lines       15253    15253           
=======================================
  Hits        13572    13572           
  Misses       1681     1681           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[xmkg]

@Sploder12, this is ready for another round of review in Windows. I've addressed Windows privilege and path-related issues and made a couple of resiliency measures in code. Let me know how it runs on your end :)

[xmkg]

Hi @sharder996, this should be ready for review in Mac, too. I've tested with my MBP '19 (Intel) with non-privileged user, the tests work as expected. Also, the Mac CI job now runs smoothly too. Please let me know if you encounter any issues.

[Sploder12]

Looking good on Windows! I marked resolved comments resolved, only some nitpicks remain. I have attached the full output again.

log.txt

[Sploder12]

return of the super nitpick! There is an extra newline here

[Sploder12]

idem

[sharder996]

I'm still having trouble on macOS. I don't get queried a second for root authentication, but there still seems to be some trouble accessing the Multipass daemon.

 % pytest tools/cli_tests/cli_version_test.py
======================================= test session starts ========================================
platform darwin -- Python 3.13.7, pytest-8.4.2, pluggy-1.6.0
rootdir: /Users/scott/Documents/dev/multipass/tools/cli_tests
configfile: pyproject.toml
collected 2 items                                                                                  

tools/cli_tests/cli_version_test.py E.                                                       [100%]

============================================== ERRORS ==============================================
_______________________ ERROR at setup of TestVersion.test_version_no_daemon _______________________

store_config = None

    @pytest.fixture(scope="function")
    def multipassd(store_config):
>       with multipassd_impl() as daemon:
             ^^^^^^^^^^^^^^^^^

tools/cli_tests/conftest.py:548: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/contextlib.py:141: in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
tools/cli_tests/conftest.py:538: in multipassd_impl
    wait_for_future(loop.run(governor.start_async()))
tools/cli_tests/utilities/threadutils.py:263: in wait_for_future
    if fut.exception():
       ^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Future at 0x1070c20d0 state=cancelled>, timeout = None

    def exception(self, timeout=None):
        """Return the exception raised by the call that the future represents.
    
        Args:
            timeout: The number of seconds to wait for the exception if the
                future isn't done. If None, then there is no limit on the wait
                time.
    
        Returns:
            The exception raised by the call that the future represents or None
            if the call completed without raising.
    
        Raises:
            CancelledError: If the future was cancelled.
            TimeoutError: If the future didn't finish executing before the given
                timeout.
        """
    
        with self._condition:
            if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:
>               raise CancelledError()
E               concurrent.futures._base.CancelledError

/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/concurrent/futures/_base.py:483: CancelledError
-------------------------------------- Captured stderr setup ---------------------------------------
[WARNING] multipassd-governor :: monitor task -- unable to determine daemon exit code
---------------------------------------- Captured log setup ----------------------------------------
WARNING  root:multipassd_governor.py:108 multipassd-governor :: monitor task -- unable to determine daemon exit code
===================================== short test summary info ======================================
ERROR tools/cli_tests/cli_version_test.py::TestVersion::test_version_no_daemon - concurrent.futures._base.CancelledError
=================================== 1 passed, 1 error in 20.34s ====================================

[xmkg]

I'm still having trouble on macOS. I don't get queried a second for root authentication, but there still seems to be some trouble accessing the Multipass daemon.

 % pytest tools/cli_tests/cli_version_test.py
======================================= test session starts ========================================
platform darwin -- Python 3.13.7, pytest-8.4.2, pluggy-1.6.0
rootdir: /Users/scott/Documents/dev/multipass/tools/cli_tests
configfile: pyproject.toml
collected 2 items                                                                                  

tools/cli_tests/cli_version_test.py E.                                                       [100%]

============================================== ERRORS ==============================================
_______________________ ERROR at setup of TestVersion.test_version_no_daemon _______________________

store_config = None

    @pytest.fixture(scope="function")
    def multipassd(store_config):
>       with multipassd_impl() as daemon:
             ^^^^^^^^^^^^^^^^^

tools/cli_tests/conftest.py:548: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/contextlib.py:141: in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
tools/cli_tests/conftest.py:538: in multipassd_impl
    wait_for_future(loop.run(governor.start_async()))
tools/cli_tests/utilities/threadutils.py:263: in wait_for_future
    if fut.exception():
       ^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Future at 0x1070c20d0 state=cancelled>, timeout = None

    def exception(self, timeout=None):
        """Return the exception raised by the call that the future represents.
    
        Args:
            timeout: The number of seconds to wait for the exception if the
                future isn't done. If None, then there is no limit on the wait
                time.
    
        Returns:
            The exception raised by the call that the future represents or None
            if the call completed without raising.
    
        Raises:
            CancelledError: If the future was cancelled.
            TimeoutError: If the future didn't finish executing before the given
                timeout.
        """
    
        with self._condition:
            if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:
>               raise CancelledError()
E               concurrent.futures._base.CancelledError

/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/concurrent/futures/_base.py:483: CancelledError
-------------------------------------- Captured stderr setup ---------------------------------------
[WARNING] multipassd-governor :: monitor task -- unable to determine daemon exit code
---------------------------------------- Captured log setup ----------------------------------------
WARNING  root:multipassd_governor.py:108 multipassd-governor :: monitor task -- unable to determine daemon exit code
===================================== short test summary info ======================================
ERROR tools/cli_tests/cli_version_test.py::TestVersion::test_version_no_daemon - concurrent.futures._base.CancelledError
=================================== 1 passed, 1 error in 20.34s ====================================

It is strange -- the same command line works just fine here:

mustafakemalgilor@Mustafas-MacBook-Pro multipass % pytest tools/cli_tests/cli_version_test.py 
======================================= test session starts ========================================
platform darwin -- Python 3.12.11, pytest-8.4.1, pluggy-1.6.0
rootdir: /Users/mustafakemalgilor/Workspace/multipass/tools/cli_tests
configfile: pyproject.toml
collected 2 items                                                                                  

tools/cli_tests/cli_version_test.py Password:
..                                                       [100%]

======================================== 2 passed in 33.56s ========================================
mustafakemalgilor@Mustafas-MacBook-Pro multipass % 

Are you running on Tahoe? Also could you re-run the tests with the following parameters added and paste the output here:

pytest -s tools/cli_tests/cli_version_test.py --print-all-output -o log_cli_level=DEBUG -vvv

Thanks!