Continuous Integration with multiple git repos? (using west or similar)

[marc-hb]

This is not a Zephyr-specific question. This is multi-repo question that should matter to users of west and other similar git wrappers. There is no https://github.com/zephyrproject-rtos/west/discussions and won't be in the near future so it's been recommended to post it here.

The short question is: how to perform Continuous Integration for projects with multiple repos?

While the Continuous Integration of the open-source Zephyr project is one of the best I've seen, it does not really support multiple repos - because it does not really need to. Its west projects are organized with a "star" repo at the "top" (Zephyr), and all other repos are second class. Getting test coverage for pull requests in secondary repos requires an indirect, two-step process documented at https://docs.zephyrproject.org/latest/guides/modules.html#changes-to-existing-module
I think this assumes that faster-paced testing on secondary repos happens elsewhere: in different places where the secondary repos are the "star" there (or don't use west at all). Once that "local" validation has been performed modules can be upgraded in Zephyr using the slower and indirect process documented above.

Keeping with the official west terminology (not a huge fan of it sorry), this question is about "forest topologies" where all repos are treated equally. The simplest example is a product repo and a test repo combined in a west manifest. No tightly coupled unit tests in the test repo but higher level tests instead that can be used to compare product versions. In such an example CI should ideally be able to combine a pull request on any repo with the latest version of every other repo with little user interface overhead. Has anyone already achieved this and how?

[marc-hb]

Chromium does this:
https://dev.chromium.org/developers/testing/chromium-build-infrastructure/tour-of-the-chromium-buildbot
It is (was?) based on buildbot. It's a very powerful but very complex system and apparently product specific, not sure much can be re-used from it.

Any idea how Android people do this?

Speaking of buildbot I'm sure writing a lot of Jenkins scripts is a valid answer; cause you can really do anything with Jenkins! However I'm more interested in the lighter-weight .github/ / .gitlab/ model and the re-use of as much "free" infrastructure as possible.

[mbolivar-nordic]

I think, but am not sure, that Android relies on the fact that Gerrit owns all the git repositories and can atomically merge PRs across multiple repositories after doing some testing. Wasn't there some sort of android automerger bot that was involved in this process?

[marc-hb]

https://chromium.googlesource.com/chromiumos/docs/+/master/contributing.md#cq-depend

Merges across repos is not atomic due to Gerrit limitations. There is a small window where syncs may pull a partial set of changes.

(a bit off-topic)

[marc-hb]

Here's one (untested) idea. In the .github/workflows/ of every repo implement this logic:

• Save the pull/12345/merge commit that the Github Checkout Action provides by default.
• Initialize a west workspace around the existing git clone provided by the same.
• west update everything to the greatest and latest. This will reset the initial git repo; I don't think west update has any "update all but this repo" option simple to use.
• git reset the tested repo to the pull/12345/merge commit saved in first step
• Run tests, profit?

This should let users issue pull requests "as usual". Would this work? Any better way?

[mbolivar-nordic]

Separate comment just to address this side point:

* I don't think `west update` has any "update all but this repo" option simple to use.

It doesn't and it would be difficult (maybe impossible) to write one in the general case.

manifest:
  # ...
  projects:
  - name: foo
    revision: main
    import:
     name-allowlist: [bar]

What does it mean to update everything "except" foo in the above manifest?

I've tried to come up with satisfying semantics for that and failed.

To update bar, you kind of have to update foo first, since manifest-rev for the bar repository is whatever foo says it is at whatever commit happens to be HEAD in foo 's main branch.

That's why west update bar will fail with such a manifest. A plain west update will actually update foo first, then update bar. And west update foo is easy enough. But west update bar seems meaningless to me.

[mbolivar-nordic]

This should let users issue pull requests "as usual". Would this work? Any better way?

That looks like it would work to me.

[aborisovich]

I've resolved this issue with @nashif help.
Using in SOF manifest

import:
  name-whitelist:
    <list_of_projects_without_sof>

causes to update only whitelisted project names (this affects derived manifests too so zephyr/west.yml does not clone all dependencies).
Moreover, there are many ways to achieve this - you can also do

import:
  blacklist-path:
    <path_to_SOF>

And put a path to SOF there, works too.
The third option is quite clumsy but also resolved the issue:
in SOF manifest add

projects:
  - name: sof
  - path: .
  - revision: main

This will cause Zephyr's west.yml to:

• recignize that sof project had been already defined (project entry with similar -name label value)
• do not update it any further

[marc-hb]

Here's a more detailed use case: zephyrproject-rtos/west#543