Multiple targets in one env.yaml

[siredmar]

Hi!

I'm currently having dozens of environment YAMLs lying around each with the default target called main.

I want to put them all in one big file and let the targets name be more useful.

I've seen this here https://labgrid.readthedocs.io/en/latest/configuration.html#environment-configuration

Now my question: How would i select the right target when running pytest?
Currently pytest seems to load the main target only if you pass a env file with --lg-env.

[Emantor]

If you want to test with multiple targets in one pytest you'll need to provide your own fixtures for the additional targets.
Note that if you just want to unify multiple targets into one YAML file, we very much want to have separate YAML files since you can specify target dependent configuration (features, image files,…) as well.

[ca-ja]

Hi @siredmar,

I have a similar experience as @Emantor describes. Generally, you need to extract the necessary drivers/resources from the config YAML for the relevant target and further process them in fixtures, which are then used in the tests. I can share a piece of code that I recently used in a more complex setup.

config_remote.yaml:

targets:
  wb-rpi-1:
    resources:
      RemotePlace:
        name: !template 'wb-rpi-1'
    drivers:
      - SerialDriver:
          name: 'dmm'
          bindings:
            port: 'DMM'
  beta:
    resources:
      RemotePlace:
        name: !template 'beta'
    drivers:
      - SerialDriver:
          name: 'PWR1'
          bindings:
            port: 'PWR1'

Where pytest was called by pytest --lg-env ./config_remote.yaml ...

conftest.py:

@pytest.fixture(scope="session")
def env(request):
    config_file = request.config.getoption("--lg-env")
    env = Environment(config_file=config_file)
    return env

@pytest.fixture(scope='session')
def rnd_pwr(env):
    rnd = RNDpower(env,"beta",["RND1"])
    yield rnd

@pytest.fixture(scope="session")
def dmm_read(env):
    dmm = DMMread(env,"wb-rpi-1","dmm")
    yield dmm

test_example.py:

class SomeDut:
    def test_checkout(self, dmm_read, rnd_pwr):
        # do something
        assert ...

RNDpower and DMMread are driver replacements with the corresponding backend, which is something I'm trying to fix at the moment...

[siredmar]

@ca-ja thanks for your answer!

I think we are kind of on the same page here. What i don't understand right is the thing with the driver replacements... Can you elaborate a little more? Thanks!

[ca-ja]

Yes. I mentioned RNDpower and DMMread for completeness, to make it clear that in this example there are some custom classes providing IO functionality with devices on the given targets.

This should ideally be provided by a proper driver. For example, if it was a Tektronix TDS2000 connected via USB, the remote config would have usbtmcdriver using backend TDS2000 so these custom classes would not be needed and the fixture under discussion would just provide this driver to the test for further use.

I am trying to discuss this in this post for instruments communicating over LAN. In fact, this option is quite missing in Labgrid, so i would like to improve it...

[siredmar]

Alright folks, i got it working. Maybe this help other people facing similar issues.

The use case

We are having multiple devices we want to run tests on. Each device is independent and does not run tests together with other devices. So we decided to have a 1:1 relationship between a remote place and a DUT and also a 1:1 between a remote place and a target.

The goal

The goal is to have one big file of targets. We use labgrid in multiple repos we want to run tests in with the CI. So keeping track of one file is way more simple instead of having dozens of files that needs to be kept in sync among several repos.

How to run devices on a group of devices?

Using tags we group together a set of places and therefore DUTs, e.g. we have a testcase called foo, we tag e.g. 4 devices with foo=true. So it's well defined which device are used if we run our CI in parallel for 4 devices. Using the tags and the the reserve, wait and lock feature from labgrid-client we can easily acquire one of these devices. Then we find out which remote place it is and this is the target we need to use in the test.

#!/usr/bin/env bash
set -euo pipefail

error_report() {
    echo "Error: command \"${BASH_COMMAND}\" failed with exit code $?"
    echo "In script: $0"
    echo "At line: ${BASH_LINENO[0]}"
    exit 1
}
trap error_report ERR

WORK_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

LG_COORDINATOR=${1:-labgrid-coordinator:20408}
TOKEN=$(labgrid-client -x ${LG_COORDINATOR} reserve foo=true | yq '.[].token')

cleanup() {
    labgrid-client -x ${LG_COORDINATOR} -p +${TOKEN} unlock
}
trap cleanup EXIT


labgrid-client -x ${LG_COORDINATOR} wait ${TOKEN}
labgrid-client -x ${LG_COORDINATOR} -p +${TOKEN} lock
REMOTE=$(labgrid-client -x ${LG_COORDINATOR} reservations | yq "to_entries[] | select(.value.token == \"${TOKEN}\") | .value.allocations.main")

REMOTE=${REMOTE} pytest -s  --log-level=DEBUG --lg-coordinator=$LG_COORDINATOR --lg-env ${WORK_DIR}/remotes/environment.yaml --html=${WORK_DIR}/labgrid.html --self-contained-html ${WORK_DIR}/tests

How does the config look like?

There is no main target. Instead there are several targets that look something like this:

targets:
  mydevice1:
    resources:
      RemotePlace:
        name: mydevice1
  mydevice2:
    resources:
      RemotePlace:
        name: mydevice2

However, running pytest using this environment configuration leads to errors as there currently there is no way to pass a specific target. Instead it looks for main.

One can fix this by passing an environment variable e.g. REMOTE to pytest and have a fixture overwriting the target.

import os
import pytest

from labgrid import Environment

@pytest.fixture(scope="session")
def target(request):
    remote = os.environ.get('REMOTE')
    config_file = request.config.getoption("--lg-env")
    env = Environment(config_file=config_file)
    print("environment: ", env)
    mytarget = env.get_target(remote)
    yield mytarget

This reads the environment and returns the target from the one big file.

The test itself isn't changed at all as the fixture returns the right target.

def test_ping(target):
    ssh = target.get_driver('SSHDriver', name="ssh")
    stdout, stderr, exitcode = ssh.run("ping -c 4 8.8.8.8", timeout=15)
    assert exitcode == 0, (
        f"ping failed (exit {exitcode}):\n"
        f"STDOUT:\n" + "\n".join(stdout) + "\n"
        f"STDERR:\n" + "\n".join(stderr)
    )