idea: shared, dedicated builder

[howardjohn]

Currently in presubmit we have 15 jobs that build all images and then run them. These are the same exact images across all 15 jobs.

Building these images can take up to 5 minutes, depending on co-located jobs and cache efficiency. This build represents the peak CPU usage of the job; typically after the initial build CPU is fairly low for the remainder of the test, so we are over provisioned.

An alternative idea is to create a dedicated builder. A sketch design:

• Create a deployment. For cache efficiency, prefer 1 replica on a large machine (can test where we get diminishing returns, 32-64 core sounds great though)
• In each job, call curl builder/prepare/COMMIT_SHA.
  • builder will do a standard git clone, following the same semantics as jobs (this may require more than just COMMIT_SHA, tbd). git work-tree may optimize cloning since we will be cloning the same repo repeatedly.
  • builder will build all of our binaries and docker images. Note: we have some WIP work to allow this without any docker daemon use ggcr, so its even simpler to run this builder
• later in jobs, we use builder as the docker registry and COMMIT_SHA as the tag. So we would end up with an image like docker pull builder/pilot:COMMIT_SHA. The builder should already have this image preload due to the prepare call, but if not it will build it on-demand. Once it has the image it sends it back.

One downside is when we run go test, it will still need to compile every test. This defeats much of the benefits of not building in the jobs, as the tests have a lot of shared dependencies. We could, similarly, build the test binaries on this shared builder as well. We don't have quite as elegant of a method as the docker image thing but its not too hard. I think Go has some remote cloud-run builder and taaa is a google thing that might be usable here