Support artifact uploads from lava jobs

[sjoerdsimons]

for some lava tests it would be really nice to also attach artifacts; Think of e.g. seperate (bigger log files), image captures being made or really anything else that's not suitable to stick into just plain text output.

Lava does not have inbuild functionality for this; For reasonable reasons; The suggested solution in the lava documentation is to have a seperate service running and upload to it using curl with some token; Which is not a bad suggestion per se, but does require a seperate service while we already have gitlab being able to do artifact management :).

So a suggested solution for this would be to have the gitlab runner also be a http service that can receive such files, caches them locally while the job is running and when finishing pushes them into the general gitlab artifacts.

To both authenticate and identify which job an upload is for the runner could generate a random dynamic endpoint for each job to receive uploads which is only valid during a job run. This endpoint can be communicate to lava via the templating mechanism we already have available, leaving it up to job writer on how to cooperate this into the yaml job.

Implementation detail wise; we could simply integrate axum as a http server (to stay nicely in the hyper universe) and for simplicity we can assume there is a frontend server that does SSL termination so there is no need to manage ssl certificates. Similarly the base URL for where to reach a given runner should be provided by runner configuration.

[emanueleaina]

From an operational point of view exposing a http enpoint is quite annoying.

The runner at the moment is a headless service, so on k8s no need to deal with services, ingresses, certificates, etc. That alone would be annoying but not a big deal. What I am actually worried about is how to map the incoming upload to the correct runner replica when there are multiple instances. At the moment we run the runner with replicas > 1 deployment, this may require some sort of custom proxy that forwards a request for /lavarunner/6f95bcdd66-4mdlx to the lava-gitlab-runner-6f95bcdd66-4mdlx pod, as I am not aware of any native kubernetes solution.

The other annoyance is that at the moment the DUT only need to talk to the dispatcher, while this would likely require some more global network routing.

[obbardc]

Perhaps the mechanism to upload files from DUT could be handled in the lava job itself & uploaded to the lava job ? Then you can retrieve it from LAVA server using the gitlab-runner ?

[sjoerdsimons]

Perhaps the mechanism to upload files from DUT could be handled in the lava job itself & uploaded to the lava job ? Then you can retrieve it from LAVA server using the gitlab-runner ?

Lava doesn't have a mechanism for dealing with that; as per https://docs.lavasoftware.org/lava/publishing-artifacts.html the suggested method for publishing files externally is to use curl to do a POST to an external service

Having it go to the lava server is annoying as well because then we have to deal with more GC and artifacts there. So it just ends up complicating things more

[obbardc]

Perhaps the mechanism to upload files from DUT could be handled in the lava job itself & uploaded to the lava job ? Then you can retrieve it from LAVA server using the gitlab-runner ?

Lava doesn't have a mechanism for dealing with that; as per https://docs.lavasoftware.org/lava/publishing-artifacts.html the suggested method for publishing files externally is to use curl to do a POST to an external service

My point is we could change that in lava. But it was only a suggestion.

[sjoerdsimons]

From an operational point of view exposing a http enpoint is quite annoying.

The runner at the moment is a headless service, so on k8s no need to deal with services, ingresses, certificates, etc. That alone would be annoying but not a big deal. What I am actually worried about is how to map the incoming upload to the correct runner replica when there are multiple instances. At the moment we run the runner with replicas > 1 deployment, this may require some sort of custom proxy that forwards a request for /lavarunner/6f95bcdd66-4mdlx to the lava-gitlab-runner-6f95bcdd66-4mdlx pod, as I am not aware of any native kubernetes solution.

Right for the simplest setup we'd have to move the k8s deployment to be non-replica based; I don't think that's the end of the world. if we do that then we can have e.g. upload.badgers/runner-0 and upload.badgers/runner-1.

To be able to do replica based we'd indeed have to encode a link to a given pod replica in the URL such that the ingress proxy can route it to the right pod. Note that there is prior art for session affinity in k8s with various ingress controllers, though that's mostly either based on client IP (not wanted here), cookies or headers (adding complixity for the test writers)..

All in all i'd be in favor for keeping the actual runner implementation being relatively simple as outlined in the issue and handle improving the k8s deployment story with that feature seperately (e.g. if we'd do a some type of proxy in front of the "backend" runner pods, that will be transparent to the runners)

The other annoyance is that at the moment the DUT only need to talk to the dispatcher, while this would likely require some more global network routing.

Mind the DUT never talks to the dispatchers, the dispatcher only drives the DUT.. As part of at least the Collabora lab setup internet connectivity is guaranteed already for DUTs and we do have a fair amount of tests taking advantage of that already

[sjoerdsimons]

Perhaps the mechanism to upload files from DUT could be handled in the lava job itself & uploaded to the lava job ? Then you can retrieve it from LAVA server using the gitlab-runner ?

Lava doesn't have a mechanism for dealing with that; as per https://docs.lavasoftware.org/lava/publishing-artifacts.html the suggested method for publishing files externally is to use curl to do a POST to an external service

My point is we could change that in lava. But it was only a suggestion.

Oh sure; And i appreciate the suggestion!

[sjoerdsimons]

For reference @eds-collabora did a first pass already a few years back in #17, while that unfortunately stalled it's a good starting point