Add self-hosted github actions checks, linting and docs #2777

gastmaier · 2025-05-12T16:28:17Z

PR Description

Experimental use of self-hosted runners to speed-up CI checking by the use of caching git history between runs and versioned containerfiles with well defined tool versions.
This version focuses on providing code checking and linting, and not building artifacts.

The first commit adds a base documentation at docs for documentation non-upstream content, that is, continuous integrations and non upstream drivers.
Overall, non-upstream content is stored at .github, ci and docs, with the exception of devicetrees and defconfigs.
This is used by the workflows themself to skip jobs or checking non-relevant files.

Code checking

For checking, the following features are implemented:

per commit:
- checkpatch: per commit
touched files:
- dt-binding-check
- coccicheck
- cppcheck
- kconfig: infer enabled driver by touched (*requirements must be enabled)
- compile devicetree
- compile kernel sparse
- compile kernel smatch
- compile gcc_fanalyzer
- assert touched *.c were compiled
multiarch and llvm+gcc

Minimal defconfigs are added to enable only the driver of interest.

Workflow

cron: updates mirror branches with ci
- on forks, when setting the default branch as cron with the provided template, it self-updates alongside the mirrors.
kernel: build the linux kernel, run the checks.

Containerfile

gcc14 instead of gcc15 due to defaulting to -std=gnu23 and not respecting gnu11 flag
Can be used locally for development
- previously I investigated using act to run all workflows locally, but I found not necessay/getting in the way, instead, when logging in the image, the steps are listed:

$ pdr adi/linux:latest
Set git commit range as ($base_sha..$head_sha)
530ac1a63a7d96dd2f41..d5c4ec65295941ec1acd
Adjust variables range as needed.

Sourced methods from ci/build.sh:
check_checkpatch check_dt_binding_check check_coccicheck check_cppcheck compile_devicetree compile_kernel compile_kernel_sparse compile_kernel_smatch compile_gcc_fanalyzer assert_compiled apply_prerun touch_files auto_set_kconfig set_step_warn set_step_fail
#

Sample run

sample cached run: https://github.com/gastmaier/adi-linux/actions/runs/14976764241/job/42072683004
sample cold start run: https://github.com/gastmaier/adi-linux/actions/runs/14976764241/attempts/1

Next steps

The intention is to run these along-side the existing azure ci, then migrate all missing functionality if better results.
When merged to main, pushes to main and PRs will run the kernel.yml worklow, and once a day the mirror branches will be updated.

PR Type

Bug fix (a change that fixes an issue)
New feature (a change that adds new functionality)
Breaking change (a change that affects other repos or cause CIs to fail)

PR Checklist

I have conducted a self-review of my own code changes
I have tested the changes on the relevant hardware
I have updated the documentation outside this repo accordingly (if there is the case)

Signed-off-by: Jorge Marques <jorge.marques@analog.com>

nunojsa

Just some minimal comments from me... I plan not being to picky and just let you experiment with all of this :)

Looks very promising!

docs/ci.rst

nunojsa · 2025-05-13T14:07:04Z

arch/x86/configs/adi_ci_defconfig

+CONFIG_IIO_TRIGGERED_BUFFER=y
+CONFIG_IIO_SW_DEVICE=y
+CONFIG_IIO_SW_TRIGGER=y
+CONFIG_IIO_TRIGGERED_EVENT=y


How are our drivers being enabled? Are we relying on defaults? Just wondering how will this grow? We also have some fair amount of HWMON drivers (and media now with all the gsml stuff)

The idea is to have the DEPENDS and if Kconfig blocks enabled on this defconfig, and auto_set_kconfig tries to set it.

I will add more defaults to the defconfigs, since HWMOM and others are missing.
on special cases the defconfig can be manipulated by a commit, which may either be relevant only for the pr or be merged.

It is hard to discover which configs to enable based on the touched drivers, since when it is hidden by a DEPENDS or if block, there is no tool to enable/resolve it.
but:

scripts/config -e $conf appends $conf to .config

make yes2modconfig, resolves the .config, removing from .config if hidden by DEPENDS or if
so an assert step after yes2modconfig could be inserted, and on failure, then just all ADI drivers as before.

The idea is to have the DEPENDS and if Kconfig blocks enabled on this defconfig, and auto_set_kconfig tries to set it.

I see, but that might be hard to track long term

I took a look at yocto's symbol_why.py and gave a second try, resulting in ci/symbols_depend.py

it is aware of if blocks, and source and depends methods.
the complex boolean logic comes from architecture related symbols (ARCH_, CPU_, X86) and these are set by the base defconfig adi_ci_defconfig, and therefore filtered out.

if any touched .c file is not compiled the CI fails, so if the script is not good enough the assertion will catch.

this build is intended for checking/linting, and forgetting to properly set the depends on a driver is also a common mistake

.github/workflows.mirror/cron.yml

ci/Containerfile

nunojsa · 2025-05-13T14:15:32Z

.github/workflows/checks.yml

+        source ci/build.sh
+        check_coccicheck
+
+    - name: DT binding


did not went to deep but this bindings are running against which kernel tree? This is very sensible so I would be careful about that.

Right now we use the dt/next tree IIRC but I have been thinking about switching to Linus master branch as that is what Rob (DT maintainer) is using AFAIU and the current tree already gave us some issues.

all checks are run at the checked out tree.
your suggestion is to checkout Linux master and checkout only the files related to the dt bindings?
in the scenario of using jic23/iio/testing, would this still be necessary?
I can Linux master to the list of fetched remotes so this can be done, (checkout linus/master -> checkout individual files -> dt_binding_check -> checkout trunk back)
I would make this a feature of a subsequent pr.

all checks are run at the checked out tree.

That might be hard to match dt tools against the checked tree... For instance, validating a binding file in our tree triggers a bunch of unrelated errors in other files.

your suggestion is to checkout Linux master and checkout only the files related to the dt bindings?

That's what we do today

in the scenario of using jic23/iio/testing, would this still be necessary?

We can try that one... But linus/master should be more reliable

Yes it does raise a lot of unrelated errors.

I will give it a try then, but overall opening a pr against the mirrors are also an option.

Docs intended to document non-upstream drivers and general code such as continuous integration. Signed-off-by: Jorge Marques <jorge.marques@analog.com>

Minimal defconfig to have only IIO drivers of interest enabled for CI contenxt. Signed-off-by: Jorge Marques <jorge.marques@analog.com>

Add workflow to build the documentation. Signed-off-by: Jorge Marques <jorge.marques@analog.com>

The cron jobs synchronize mirrors, applying the continuous integration on top. The alternative workflows.mirror files are provided to enable synchronizing forks also. It is necessary in the fork, however to prepare the default cron branch with the same steps as in self.yml, so then the subsequent runs are functional. Signed-off-by: Jorge Marques <jorge.marques@analog.com>

Include a Containerfile that contains the common tools to compiling and checking the Linux Kernel across x86, arm and aarch64 architectures. The Container image can be run: * Interactively: login into and call build methods directly * Self-hosted run: host a github runner * Cluster run: host multiple github runners And is versioned, so runner instances can be progressively deployed. Signed-off-by: Jorge Marques <jorge.marques@analog.com>

The ci/build.sh contains checking and building methods aware of the pushed/wants-to-merge commits. To show the warnings and errors on the GitHub GUI, each command is filtered and piped as GitHub annotations, tuned for the verbosity and format of each command, since there is no standardized output. The methods can also be called individually locally using the container, easing development and debugging. Signed-off-by: Jorge Marques <jorge.marques@analog.com>

These workflows call the methods from ci/build.sh to check and compile the Linux Kernel in x86, aarch64, and arm architectures. The results of each job are collected and then checked against kernel.yml/assert. If any warning/error under 'fail' is logged, the CI fails; however, warning/error under 'warn' do not fail the CI. The reason is that some warnings are acceptable and shouldn't prevent CIs downstream from triggering. Signed-off-by: Jorge Marques <jorge.marques@analog.com>

Add a Python script to find dependencies of a symbol recursively. It is aware of 'source', 'if', 'config', and 'menuconfig' methods. At the 'depends' boolean logic, symbols '!' and '=n' are discarded; parentheses are not considered. A basic filter removes architecture symbols ('ARCH_*', 'CPU_*', ...) which are involved in most complex boolean expressions. Signed-off-by: Jorge Marques <jorge.marques@analog.com>

Interpret branches that are not mirrors are far behind or diverged from upstream. In this case, run the dt-bindings check checking out next/linux-next/master. It is limited to checking out only .yaml, so header changes have no effect and may cause the check to fail. Still, it is better to apply those changes closer to upstream, that means, targetting the mirror branches instead. Signed-off-by: Jorge Marques <jorge.marques@analog.com>

Use GitHub Actions for CIs

8df085a

Signed-off-by: Jorge Marques <jorge.marques@analog.com>

gastmaier marked this pull request as ready for review May 13, 2025 13:40

gastmaier requested review from nunojsa, mhennerich, dbogdan, btogorean, chegbeli, buha, amiclaus, rbolboac, danmois, CiprianRegus and machschmitt as code owners May 13, 2025 13:40

mhennerich approved these changes May 13, 2025

View reviewed changes

nunojsa reviewed May 13, 2025

View reviewed changes

gastmaier added 5 commits May 14, 2025 11:05

docs: Add sphinx docs structure and ci page

030f27c

Docs intended to document non-upstream drivers and general code such as continuous integration. Signed-off-by: Jorge Marques <jorge.marques@analog.com>

arch: x86: configs: add adi_ci_defconfig

acf7138

Minimal defconfig to have only IIO drivers of interest enabled for CI contenxt. Signed-off-by: Jorge Marques <jorge.marques@analog.com>

arch: arm64: configs: add adi_ci_defconfig

2106ec7

Minimal defconfig to have only IIO drivers of interest enabled for CI contenxt. Signed-off-by: Jorge Marques <jorge.marques@analog.com>

arch: arm: configs: add adi_ci_defconfig

617f515

Minimal defconfig to have only IIO drivers of interest enabled for CI contenxt. Signed-off-by: Jorge Marques <jorge.marques@analog.com>

.github: Documentation workflow

bf554d2

Add workflow to build the documentation. Signed-off-by: Jorge Marques <jorge.marques@analog.com>

gastmaier force-pushed the main-gh-self-host branch from 06952b8 to 9220796 Compare May 15, 2025 07:03

gastmaier marked this pull request as draft May 15, 2025 07:10

gastmaier marked this pull request as ready for review May 15, 2025 08:17

gastmaier force-pushed the main-gh-self-host branch from 9220796 to 754d4de Compare May 15, 2025 15:29

gastmaier added 6 commits May 15, 2025 17:41

gastmaier force-pushed the main-gh-self-host branch from 754d4de to 1b9a495 Compare May 15, 2025 15:45

gastmaier requested a review from nunojsa May 16, 2025 07:36

nunojsa approved these changes May 16, 2025

View reviewed changes

mhennerich merged commit ec60624 into main May 16, 2025
20 of 22 checks passed

mhennerich deleted the main-gh-self-host branch May 16, 2025 13:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add self-hosted github actions checks, linting and docs #2777

Add self-hosted github actions checks, linting and docs #2777

Uh oh!

gastmaier commented May 12, 2025 •

edited

Loading

Uh oh!

nunojsa left a comment

Uh oh!

Uh oh!

Uh oh!

nunojsa May 13, 2025

Uh oh!

gastmaier May 14, 2025

Uh oh!

nunojsa May 15, 2025

Uh oh!

gastmaier May 15, 2025

Uh oh!

Uh oh!

Uh oh!

nunojsa May 13, 2025

Uh oh!

gastmaier May 14, 2025

Uh oh!

nunojsa May 15, 2025

Uh oh!

gastmaier May 15, 2025

Uh oh!

Uh oh!

Uh oh!

Add self-hosted github actions checks, linting and docs #2777

Add self-hosted github actions checks, linting and docs #2777

Uh oh!

Conversation

gastmaier commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Description

Code checking

Workflow

Containerfile

Sample run

Next steps

PR Type

PR Checklist

Uh oh!

nunojsa left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gastmaier commented May 12, 2025 •

edited

Loading