Run tier tests on Testing Farm

[ansasaki]

Add a reusable workflow that schedule tests on testing farm using the CLI. If the specified plan is not found, the test is ignored and passes.

The tier tests are executed sequentially using the reusable workflow. The tests from the tier1.yaml plan are executed on a single distro, currently Fedora 37.

If the 'tier' input is set as 'tier2', the tests from the tier2.yaml plan are executed in multiple distros, currently Fedora 37 and 38.

The tests are triggered when a pull request is opened or reopened.

[kkaarreell]

Also, while you are testing this, I think you can delete .packit.yaml since those tests are not needed for this PR.

[phracek]

hi @ansasaki Please have a look at our GitHub Action https://github.com/sclorg/testing-farm-as-github-action. It is already available on MarketPlace https://github.com/marketplace/actions/schedule-tests-on-testing-farm. If something is missing and we can improve it. Let's us now :)

[ansasaki]

hi @ansasaki Please have a look at our GitHub Action https://github.com/sclorg/testing-farm-as-github-action. It is already available on MarketPlace https://github.com/marketplace/actions/schedule-tests-on-testing-farm. If something is missing and we can improve it. Let's us now :)

@phracek Hello, I saw the action and shamelessly copied some parts to experiment. I'll try later to create what I want using only your action. I'd be happy to contribute too!

[kkaarreell]

Hi @ansasaki , regarding those test failures, I believe this is because some tests not relevant to Fedora were not excluded because adjust rules in tmt are case sensitive and therefore distro=Fedora-37 (provided) is different from distro=fedora-37 (expected). I am wondering how to address that, I think I will consult tmt developers here because it doesn't make really sense to always operate with both options explicitly.

[ansasaki]

e case sensitive a

OK, I'll make it to pass both compose and distro separately for now. But I agree it can be improved.

[ansasaki]

Note that the code in this PR requires each contributor to have their own Testing Farm token, which is not ideal. To avoid this, a 2-step trick is needed:

1. A workflow is triggered when a PR is opened. This workflow will not actually run the tests as the repository secrets are not accessible. It will only store the PR info as an artifact
2. A workflow is triggered when the first workflow is completed. This will actually run the tests as it will have access to the secrets, but it will use the workflow code from the repository (i.e. code already merged). This means that PRs that changes workflows cannot be tested this way (will only be effective after merging). Since this second workflow is not tied with a PR (as it was triggered not by the "pull_request" event, but by the finish of the first workflow), the results of the tests will not be shown on the PR that originated the tests. To make that connection, we need to implement the "check" API from github, to show the test results on the PR.

This 2-step trick is implemented on #340, but the check API is not implemented yet.

I'll close this PR, it already served the purpose of a PoC for the table showing the artifacts links. The work on the 2-step trick will continue on #340.