Skip to content

Split artifact uploads into hostapp and testing artifacts #638

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jun 12, 2025

Conversation

klutchell
Copy link
Contributor

@klutchell klutchell commented May 22, 2025

This will improve the download time for each, and allow
reduction of required hostapp files without breaking tests.

Moving the creation of deflate files to the build job enables us to upload S3 artifacts in both the deploy-S3 and the deploy-hostapp steps (as webresources).

See: https://balena.fibery.io/Work/Improvement/Prepare-deflate-files-before-hostapp-deploy-2913

@klutchell klutchell temporarily deployed to balena-staging.com May 22, 2025 18:01 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to balena-staging.com May 22, 2025 18:01 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to balena-staging.com May 22, 2025 18:33 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to balena-cloud.com May 22, 2025 18:33 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to balena-cloud.com May 22, 2025 18:33 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to balena-staging.com May 22, 2025 18:49 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to balena-staging.com May 22, 2025 19:42 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to balena-staging.com May 22, 2025 19:42 — with GitHub Actions Inactive
@klutchell klutchell changed the title Prepare deflate files as part of hostapp-deploy Separate hostapp artifacts from leviathan artifacts May 22, 2025
@klutchell klutchell force-pushed the kyle/web-resources branch from 51fac7e to 9e15137 Compare May 22, 2025 19:57
@klutchell klutchell temporarily deployed to balena-staging.com May 22, 2025 19:58 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to balena-staging.com May 22, 2025 19:58 — with GitHub Actions Inactive
@klutchell klutchell had a problem deploying to balena-staging.com May 22, 2025 20:44 — with GitHub Actions Failure
@klutchell klutchell force-pushed the kyle/web-resources branch from 9e15137 to 402542e Compare May 22, 2025 23:03
@klutchell klutchell temporarily deployed to balena-staging.com May 22, 2025 23:03 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to balena-staging.com May 22, 2025 23:03 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to balena-cloud.com May 22, 2025 23:37 — with GitHub Actions Inactive
@klutchell klutchell had a problem deploying to balena-staging.com May 22, 2025 23:37 — with GitHub Actions Failure
@klutchell klutchell temporarily deployed to balena-cloud.com May 22, 2025 23:37 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to balena-cloud.com May 22, 2025 23:37 — with GitHub Actions Inactive
@klutchell klutchell force-pushed the kyle/web-resources branch from 402542e to c8293d2 Compare May 23, 2025 16:30
@klutchell klutchell temporarily deployed to balena-staging.com May 23, 2025 16:30 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to balena-staging.com May 23, 2025 16:30 — with GitHub Actions Inactive
@klutchell klutchell changed the title Separate hostapp artifacts from leviathan artifacts Separate hostapp artifacts from testing artifacts May 23, 2025
This environment variable provides a structured path for deploying artifacts that includes the device slug and OS version, which will be used to organize S3 uploads and downloads.

Change-type: patch
Signed-off-by: Kyle Harding <kyle@balena.io>
This marks a potential future optimization to move the OS contract build step closer to where it's actually used.

Signed-off-by: Kyle Harding <kyle@balena.io>
- Move Docker Hub login earlier in workflow to be available for subsequent steps

- Update artifact preparation to use S3_DEPLOY_PATH and disable compression

- Add deflate file preparation step for S3/webresource deployment

- Update encryption to work with individual files instead of tarball

Change-type: minor
Signed-off-by: Kyle Harding <kyle@balena.io>
- Replace single 'Upload artifacts' with separate uploads for hostapp and testing

- Hostapp artifacts include deflate files, manifests, and metadata for S3/webresource deployment

- Testing artifacts include images and docker containers needed by Leviathan test suites

- Use specific file patterns instead of tarball for better separation of concerns

Change-type: minor
Signed-off-by: Kyle Harding <kyle@balena.io>
- Update hostapp-deploy, s3-deploy, ami-deploy, and leviathan jobs to use new hostapp/testing artifacts

- Replace tarball-based artifact handling with individual file decryption and decompression

- Update artifact download paths and decryption logic across all deployment jobs

- Add validation for Docker image loading and improve error handling

- Clean up environment variable assignments and remove duplicate Docker Hub login steps

Change-type: minor
Signed-off-by: Kyle Harding <kyle@balena.io>
We haven't supported a deploy type of docker-image for a few years
now so we can remove these conditions.

Change-type: patch
Signed-off-by: Kyle Harding <kyle@balena.io>
Signed-off-by: Kyle Harding <kyle@balena.io>
This makes it easier to see what we are encrypting vs what
we are uploading as artifacts.

Signed-off-by: Kyle Harding <kyle@balena.io>
@klutchell klutchell force-pushed the kyle/web-resources branch from 6f31fb6 to 2f878b6 Compare June 12, 2025 01:51
@klutchell klutchell marked this pull request as ready for review June 12, 2025 01:51
@klutchell klutchell temporarily deployed to balena-staging.com June 12, 2025 01:52 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to balena-staging.com June 12, 2025 01:52 — with GitHub Actions Inactive
@flowzone-app flowzone-app bot enabled auto-merge June 12, 2025 01:56
@klutchell klutchell temporarily deployed to balena-cloud.com June 12, 2025 02:34 — with GitHub Actions Inactive
@rcooke-warwick rcooke-warwick self-requested a review June 12, 2025 14:10
@klutchell klutchell temporarily deployed to balena-cloud.com June 12, 2025 15:51 — with GitHub Actions Inactive
@klutchell klutchell temporarily deployed to balena-cloud.com June 12, 2025 15:51 — with GitHub Actions Inactive
@flowzone-app flowzone-app bot merged commit 42fac1b into master Jun 12, 2025
81 of 87 checks passed
@flowzone-app flowzone-app bot deleted the kyle/web-resources branch June 12, 2025 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants