Nightly release CI action is broken

[TommyMurphyTM1234]

See here:

• https://github.com/riscv-collab/riscv-gnu-toolchain/releases/tag/2024.10.23
• https://github.com/riscv-collab/riscv-gnu-toolchain/actions/runs/11470734480

Nightly Release
Error when evaluating 'strategy' for job 'upload-assets'. .github/workflows/nightly-release.yaml (Line: 199, Col: 15): Matrix must define at least one vector

Only source bundles generated, no binary toolchains.

[TommyMurphyTM1234]

This seems to be the culprit but I don't really understand it yet...

• riscv-gnu-toolchain/.github/workflows/nightly-release.yaml

  Lines 198 to 199 in 935b263

  	strategy:
  	matrix: ${{ fromJson( needs.create-release.outputs.asset_matrix ) }}

[TommyMurphyTM1234]

The last successful nightly release was 3rd September 2024:

• https://github.com/riscv-collab/riscv-gnu-toolchain/releases
  • https://github.com/riscv-collab/riscv-gnu-toolchain/releases/tag/2024.09.03

so I presume that one of the commits since that date caused the problem?

• https://github.com/riscv-collab/riscv-gnu-toolchain/commits/master/.github/workflows/nightly-release.yaml

I hope it's not one of mine! :-)

[cmuellner]

I created a PR that should address this issue: #1592

[TommyMurphyTM1234]

I created a PR that should address this issue: #1592

Thanks @cmuellner. 👍

[TommyMurphyTM1234]

Any idea why the nightly build still doesn't seem to be working or, at least, hasn't completed and uploaded a complete set of built artifacts yet?

[cmuellner]

I was waiting for a review of my PRs that address issues in the CI/CD scripts (#1582 and #1592).
I just merged them without receiving a review.

[TommyMurphyTM1234]

Still something wrong I guess? Only sources in the latest release again.

• https://github.com/riscv-collab/riscv-gnu-toolchain/releases/tag/2024.10.26
• https://github.com/riscv-collab/riscv-gnu-toolchain/actions/runs/11527279579

Edit: oh, out of disk space? Even though it's supposed to clean up after itself as far as I can see?

• riscv-gnu-toolchain/.github/workflows/nightly-release.yaml

  Line 62 in 7aa6d5e

  - name: Remove unneeded frameworks to recover disk space

• riscv-gnu-toolchain/.github/workflows/nightly-release.yaml

  Line 92 in 7aa6d5e

  - name: recover space

Does it maybe need to do more to clean up?
Do older release artifacts need to be deleted?
Are the changes to enable additional Linux musl and uClibc builds exceeding the available resources?

[jordancarlin]

It looks like it is the "create release" job that is running out of space. It downloads all of the artifacts from previous steps, which take up 25 GB but the runner only has 21 GB available. Each job is run on a separate runner, so the space needs to be cleaned up in this job too.

[cmuellner]

The CI seems to be regularly (I've observed this multiple times since we added the musl builds into the CI/CD) broken because of git/musl issues:

error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504
fatal: expected 'packfile'
fatal: clone of 'https://git.musl-libc.org/git/musl' into submodule path '/home/runner/work/riscv-gnu-toolchain/riscv-gnu-toolchain/musl' failed
Failed to clone 'musl' a second time, aborting

I'm not sure what the best way to move forward here is.

[TommyMurphyTM1234]

https://git.musl-libc.org/git/musl

That's the wrong URL as far as I can see:

• https://musl.libc.org/
  • https://git.musl-libc.org/cgit/musl

Edit: ah - sorry - ignore that...

• https://git.musl-libc.org/cgit/musl

[mickflemm]

Maybe we are hitting an issue with HTTP (e.g. http.postBuffer is not enough to hold the pack file), does doing a shallow clone solve this ? Yocto at some point used this mirror: https://github.com/kraj/musl/tree/master it seems to be up to date.

Maybe @richfelker can help.

[richfelker]

Can you provide a minimal test case to reproduce the failure to git-clone? I just cloned successfully.

[richfelker]

FWIW if you're re-cloning on every CI job, the polite thing to do is make it a shallow clone. The more polite thing to do would be to cache git clones. But I don't think this is related to the problem.