Compatibilty with running the Debian Salsa CI locally

[ottok]

This is halfway between a bug report and a feature request. Since the latest update to https://salsa.debian.org/salsa-ci-team/pipeline#running-the-pipeline-locally this gitlab-ci-local is now advertised as a potential tool for running the Salsa CI pipeline locally. Salsa CI is Debian's custom GitLab CI pipeline for testing Debian packaging git repositories before uploading Debian source packages to Debian archives.

I am however not able to fully run the pipeline locally, as the RELEASE variable does not propagate properly.

Steps to reproduce on a Debian/Ubuntu host:

# Install git-buildpackage
sudo apt install git-buildpackage --no-install-recommends
# Clone Debian packaging sources of an example package
gbp clone --verbose --add-upstream-vcs https://salsa.debian.org/debian/entr.git
cd entr
# Preview seems to work correctly
gitlab-ci-local --file debian/salsa-ci.yml --preview
# The list seems mostly correct, but some test jobs are missing
gitlab-ci-local --file debian/salsa-ci.yml --list
name                              description  stage                       when        allow_failure  needs
extract-source                                 provisioning                on_success  false      
build                                          build                       on_success  false      
build armel                                    build                       on_success  false      
build armhf                                    build                       on_success  false      
build arm64                                    build                       on_success  false      
build riscv64                                  build                       on_success  false      
build source                                   build                       on_success  false      
upgrade from bookworm                          upgrade package and distro  on_success  false      
upgrade from bullseye                          upgrade package and distro  on_success  false      
upgrade from buster                            upgrade package and distro  on_success  false      
upgrade from stretch                           upgrade package and distro  on_success  false
# Run the entire pipeline locally
gitlab-ci-local --file debian/salsa-ci.yml

The first job extract-sources passes correctly, but the second job build fails on:

parsing and downloads finished in 318 ms.
json schema validated in 133 ms
build starting registry.salsa.debian.org/salsa-ci-team/pipeline/base: (build)
Error: Command failed with exit code 125: docker pull registry.salsa.debian.org/salsa-ci-team/pipeline/base:
Error: invalid reference format
    at makeError (/snapshot/firecow-gitlab-ci-local/node_modules/execa/lib/error.js:60:11)
    at <anonymous> (/snapshot/firecow-gitlab-ci-local/node_modules/execa/index.js:118:26)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async i (/snapshot/firecow-gitlab-ci-local/src/job.ts:1003:13)
    at async Ut.pullImage (/snapshot/firecow-gitlab-ci-local/src/job.ts:1016:13)
    at async Ut.start (/snapshot/firecow-gitlab-ci-local/src/job.ts:531:13)
    at async <anonymous> (/snapshot/firecow-gitlab-ci-local/node_modules/p-map/index.js:57:22)

This error is from Podman/Docker as the image is missing the tag. I can get past this by defining the RELEASE manually:

gitlab-ci-local --file debian/salsa-ci.yml --variable RELEASE=unstable build

However, this command will complain about /builds and /gcl-builds not existing and fail:

build $ chmod o-w /builds
build > chmod: cannot access '/builds': No such file or directory
build finished in 14 s  FAIL 1 
build Running after script...

With the steps above to repeat this, could you please advice if you have any tips to make Salsa CI more compatible with gitlab-ci-local?
Or is there something in gitlab-ci-local that could be improved to be able to run Salsa CI as-is?

Thanks for maintaining this tool!

[firecow]

I'll give it a go, one of these days 😄 Maybe there is a bug somewhere.

[firecow]

What does you .gitlab-ci.yml look like @ottok ?

[firecow]

@ottok Also add a link to a gitlab.com repository, where we can see your work in action.
It makes it a lot easier to debug

[ottok]

The steps above have all info. Open with browser https://salsa.debian.org/debian/entr to browse pipelines in action.

…

On Fri., Jan. 31, 2025, 11:49 Mads Jon Nielsen, ***@***.***> wrote: @ottok <https://github.com/ottok> Also add a link to a gitlab.com repository, where we can see your work in action. It makes it a lot easier to debug — Reply to this email directly, view it on GitHub <#1473 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFDINDAYZBIOWLHSIZPHOT2NNIMPAVCNFSM6AAAAABVRVOBGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMRWHEYTCMBSGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ottok]

The pipeline file is debian/salsa-ci.yml

…

On Fri., Jan. 31, 2025, 12:39 Otto Kekäläinen, ***@***.***> wrote: The steps above have all info. Open with browser https://salsa.debian.org/debian/entr to browse pipelines in action. On Fri., Jan. 31, 2025, 11:49 Mads Jon Nielsen, ***@***.***> wrote: > @ottok <https://github.com/ottok> Also add a link to a gitlab.com > repository, where we can see your work in action. > It makes it a lot easier to debug > > — > Reply to this email directly, view it on GitHub > <#1473 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAFDINDAYZBIOWLHSIZPHOT2NNIMPAVCNFSM6AAAAABVRVOBGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMRWHEYTCMBSGU> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[ANGkeith]

@ottok

However, this command will complain about /builds and /gcl-builds not existing and fail:

regarding ^,

https://salsa.debian.org/salsa-ci-team/pipeline/-/blob/master/salsa-ci.yml?ref_type=heads#L433
update chmod o-w /builds to chmod o-w $CI_BUILDS_DIR

---

ps: currently CI_BUILDS_DIR is undefined in gcl
#1489

[ottok]

Thanks for spotting that, I filed now MR to fix it: https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/577

I also tested that the build indeed proceeds past this point now. Next thing it fails on this:

$ git show
commit d7b92dea3bb3e41ac69c3572ac37f0956c831c97 (HEAD -> debian/latest)
--- a/debian/salsa-ci.yml
+++ b/debian/salsa-ci.yml
@@ -1,5 +1,6 @@
 include:
-  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml
+  - https://salsa.debian.org/otto/salsa-ci-pipeline/-/raw/fix-hard-coded-builds-path/salsa-ci.yml
+  - https://salsa.debian.org/otto/salsa-ci-pipeline/-/raw/fix-hard-coded-builds-path/pipeline-jobs.yml
 
new file mode 100644
index 0000000..aef8eaf
--- /dev/null
+++ b/salsa.env
@@ -0,0 +1 @@
+RELEASE_FROM_CHANGELOG=unstable

$ gitlab-ci-local --file debian/salsa-ci.yml --variable RELEASE=unstable "build source"
...
build source > Updating symlinks in /usr/lib/ccache ...
build source $ PATH="/usr/lib/ccache/:${PATH}"
build source $ ccache -z
build source > Statistics zeroed
build source $ chmod o-w $CI_BUILDS_DIR
build source > chmod: missing operand after 'o-w'
build source > Try 'chmod --help' for more information.
build source finished in 24 s  FAIL 1 
build source Running after script...
build source imported cache 'build-amd64_' in 140 ms
build source imported artifacts in 78 ms
build source > time="2025-02-16T22:59:38Z" level=warning msg="The cgroupv2 manager is set to systemd but there is no systemd user session available"
build source > time="2025-02-16T22:59:38Z" level=warning msg="For using systemd, you may need to login using an user session"
build source > time="2025-02-16T22:59:38Z" level=warning msg="Alternatively, you can enable lingering with: `loginctl enable-linger 1000` (possibly as root)"
build source > time="2025-02-16T22:59:38Z" level=warning msg="Falling back to --cgroup-manager=cgroupfs"
build source $ CCACHE_DIR=${CCACHE_TMP_DIR} ccache -s
build source > Local storage:
build source >   Cache size (GiB): 0.0 / 5.0 ( 0.00%)
build source $ mv -v ${CCACHE_TMP_DIR} ${CCACHE_WORK_DIR}
build source > mv: inter-device move failed: '/gcl-builds/../.entr.ccache' to '/gcl-builds/.ccache/.entr.ccache'; unable to remove target: Directory not empty
build source finished in 25 s  WARN 1  after_script
build source exported artifacts in 523 ms
build source copied artifacts to cwd in 91 ms
 WARN  build source  after_script
 FAIL  build source
  > Local storage:
  >   Cache size (GiB): 0.0 / 5.0 ( 0.00%)
  > mv: inter-device move failed: '/gcl-builds/../.entr.ccache' to '/gcl-builds/.ccache/.entr.ccache'; unable to remove target: Directory not empty

Because of..

build source $ chmod o-w $CI_BUILDS_DIR
build source > chmod: missing operand after 'o-w'

..I also tried defining if manually:

$ gitlab-ci-local --file debian/salsa-ci.yml --variable RELEASE=unstable --variable CI_BUILDS_DIR=/builds "build source"
...
build source $ chmod o-w $CI_BUILDS_DIR
build source > chmod: cannot access '/builds': No such file or directory
build source finished in 18 s  FAIL 1 
build source Running after script...
build source imported cache 'build-amd64_' in 137 ms
build source imported artifacts in 75 ms
build source > time="2025-02-16T23:04:22Z" level=warning msg="The cgroupv2 manager is set to systemd but there is no systemd user session available"
build source > time="2025-02-16T23:04:22Z" level=warning msg="For using systemd, you may need to login using an user session"
build source > time="2025-02-16T23:04:22Z" level=warning msg="Alternatively, you can enable lingering with: `loginctl enable-linger 1000` (possibly as root)"
build source > time="2025-02-16T23:04:22Z" level=warning msg="Falling back to --cgroup-manager=cgroupfs"
build source $ CCACHE_DIR=${CCACHE_TMP_DIR} ccache -s
build source > Local storage:
build source >   Cache size (GiB): 0.0 / 5.0 ( 0.00%)
build source $ mv -v ${CCACHE_TMP_DIR} ${CCACHE_WORK_DIR}
build source > mv: inter-device move failed: '/gcl-builds/../.entr.ccache' to '/gcl-builds/.ccache/.entr.ccache'; unable to remove target: Directory not empty
build source finished in 18 s  WARN 1  after_script
build source exported artifacts in 519 ms
build source copied artifacts to cwd in 91 ms
 WARN  build source  after_script
 FAIL  build source
  > Local storage:
  >   Cache size (GiB): 0.0 / 5.0 ( 0.00%)
  > mv: inter-device move failed: '/gcl-builds/../.entr.ccache' to '/gcl-builds/.ccache/.entr.ccache'; unable to remove target: Directory not empty

$ ls -la /builds/
drwxrwxrwx 2 admin admin 4096 Jan 20 21:46 .
drwxr-xr-x 7 admin root  4096 Jan 20 21:46 ..

I have at least one patch in Docker and I am a fairly experienced open source contributor. I might be able to debug this myself if you give some pointers on what the architecture/process here is, as I am unable to figure it our just by skimming your docs/code.

Thanks for maintaining gitlab-ci-local!

[ANGkeith]

 gitlab-ci-local --file debian/salsa-ci.yml --variable RELEASE=unstable --variable CI_BUILDS_DIR=/builds "build source"

can you try to use --variable CI_BUILDS_DIR=/gcl-builds instead ? the CI_BUILDS_DIR is currently hard coded to /gcl-builds

[ANGkeith]

I might be able to debug this myself if you give some pointers on what the architecture/process here is, as I am unable to figure it our just by skimming your docs/code.

hmm, not too sure, about this,

but probably trying to build from source should be a good starting point?

npm ci             # install dependency
npm run dev   # should run gitlab-ci-local with hot re-loading enabled
npm run test   # for the regression test 

[ottok]

Thanks, indeed, with gitlab-ci-local --file debian/salsa-ci.yml --variable RELEASE=unstable --variable CI_BUILDS_DIR=/gcl-builds I get the builds to progress further now. In fact, when I removed the non-amd64 build jobs I got the whole pipeline to pass 🎉

$ git log -p
commit a74d6aa29f10ba8ae9a9801a9296a883c6aa905a (HEAD -> debian/latest)
Author: Otto <otto@debian.org>
Date:   Tue Feb 18 02:45:45 2025 +0000

    WIP: Disable non-amd64 builds

diff --git a/debian/salsa-ci.yml b/debian/salsa-ci.yml
index 9e21e05..8bdb04f 100644
--- a/debian/salsa-ci.yml
+++ b/debian/salsa-ci.yml
@@ -9,31 +9,6 @@ stages:
   - upgrade package and distro
   - publish  # Stage referenced by Salsa CI template aptly stanza, so must exist even though not used
 
-# Additional builds on arm64 only when in main project as only it has runners
-# available in https://salsa.debian.org/groups/debian/-/runners
-# Don't use the SALSA_CI_DISABLE_BUILD_PACKAGE_ARM64 & co tags, as they don't
-# have this logic for testing group name and will end up making all pipelines
-# in forks stuck forever.
-build arm64:
-  extends: .build-package-arm64
-  rules:
-    - if: $CI_PROJECT_ROOT_NAMESPACE  == "debian"
-
-build armel:
-  extends: .build-package-armel
-  rules:
-    - if: $CI_PROJECT_ROOT_NAMESPACE  == "debian"
-
-build armhf:
-  extends: .build-package-armhf
-  rules:
-    - if: $CI_PROJECT_ROOT_NAMESPACE  == "debian"
-
-build riscv64:
-  extends: .build-package-riscv64
-  rules:
-    - if: $CI_PROJECT_ROOT_NAMESPACE  == "debian"
-
 .test-prepare-container: &test-prepare-container |
   # Change directory to ./debian/output where build artifacts can be found
   cd ${WORKING_DIR}

commit e2b540858a0f70f93bbe1fe2b63eadf0decd000d
Author: Otto <otto@debian.org>
Date:   Sun Feb 16 22:54:11 2025 +0000

    WIP: Compatibility with gitlab-ci-local

diff --git a/debian/salsa-ci.yml b/debian/salsa-ci.yml
index 865f4f5..9e21e05 100644
--- a/debian/salsa-ci.yml
+++ b/debian/salsa-ci.yml
@@ -1,5 +1,6 @@
 include:
-  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml
+  - https://salsa.debian.org/otto/salsa-ci-pipeline/-/raw/fix-hard-coded-builds-path/salsa-ci.yml
+  - https://salsa.debian.org/otto/salsa-ci-pipeline/-/raw/fix-hard-coded-builds-path/pipeline-jobs.yml
 

gitlab-ci-local --file debian/salsa-ci.yml --variable RELEASE=unstable --variable CI_BUILDS_DIR=/gcl-builds
...
 PASS  extract-source                  
 PASS  build                           
 PASS  build source                    
 PASS  upgrade from bookworm           
 PASS  upgrade from bullseye           
 PASS  upgrade from buster             

[ottok]

Another example:

gbp clone --verbose https://salsa.debian.org/debian/trend.git
cd trend
nano debian/salsa-ci.yml
  include:
  -  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml
  +  - https://salsa.debian.org/otto/salsa-ci-pipeline/-/raw/fix-hard-coded-builds-path/salsa-ci.yml
  +  - https://salsa.debian.org/otto/salsa-ci-pipeline/-/raw/fix-hard-coded-builds-path/pipeline-jobs.yml
gitlab-ci-local --file debian/salsa-ci.yml --variable RELEASE=unstable --variable CI_BUILDS_DIR=/gcl-builds
...
 PASS  extract-source
 PASS  build
 PASS  build source
pipeline finished in 43 s

Once https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/577 is merged any typical debian/salsa-ci.yml will run out-of-the-box.

Once #1489 is merged I assume there will be no more need to pass --variable CI_BUILDS_DIR=/gcl-builds to the command.

[ottok]

Next thing I am trying to debug is why it only runs the 'provisioning' and 'build' stages, but not the 'test' stage. Compare the above jobs completed to full pipeline at https://salsa.debian.org/debian/trend/-/pipelines/817877