feat: Dockerfile for faster CI/CD pipelines #1380

belt · 2024-05-26T01:05:09Z

Some pipeline require repository cloning. Provide an alternative to git within such workflows

Byron

Thanks a lot! I love the idea of being able to use gix clone in pipelines.

I left some comments, and one more request: there should be a new job that validates the dockerfiles, probably by running them. That's probably quite costly but can hopefully be done within 15 minutes or so.

Finally, I wonder if two dockerfiles are needed, or if one could just endorse one base image - after all, both base images build a binary, the result should be the same. Or does one binary not run in most places? Or asked differently, is alpine not producing the more compatible binary, so bookworm is superseded?

Thanks again for your help.

Byron · 2024-05-26T05:48:28Z

README.md

@@ -62,6 +62,31 @@ Follow linked crate name for detailed status. Please note that all crates follow

 [semver]: https://semver.org

+### Pipeline Integration
+
+Some CI/CD pipelines leverage repository cloning. Below is a copy-paste-able example to build docker images for such workflows.


Could you provide more context? Which pipelines are you talking about? What's the general idea on how to get from images to using them in a pipeline?

Think blank slate, and I am good test for that as I wouldn't be able to use the docker images from this description.

Byron · 2024-05-26T05:49:12Z

README.md

+
+#### Pipeline Integration (recommended)
+
+Build an image without a target and then copy the binaries into your local image


I don't get that - could you describe that without assuming too much docker knowledge?

Same copy-pasta command as above without the --target=pipeline

I am trying to say that I don't understand the paragraph, and that it should be adjusted to fit people who don't do that every day. Ideally, no prior knowledge about this usually works is assumed.

Byron · 2024-05-26T05:49:57Z

README.md

+Build an image without a target and then copy the binaries into your local image
+
+```dockerfile
+COPY --from gitoxide:latest /bin/gix /usr/local/bin/


This works if the gitoxide image was already built, right? That wasn't clear to me and I thought maybe it pulls it from the network. Could that be clarified in the paragraph above as well?

@Byron Do you have a docker hub account? If so, that (gitoxide:latest or perhaps Bryon/gitoxide:latest more likely, we could use paulbelt/gitoxide:latest, but you are more... committed to this project than I am [pun intended]) should be the "official" image... and docker build would automatically pull the image and COPY the binary from it.

Let's leave dockerhub out of the picture for now as I have no plans to maintain an official image for now. This would mean the gitoxide:latest name assumes a local image, which probably means the description on how to build that should come first.

Byron · 2024-05-26T05:51:33Z

README.md

+```
+
+#### Pipeline Integration (base image)
+


I think there should be text here that explains what it's doing and why - it probably all flows together with the paragraphs above, and it feels like these could be re-ordered so the user first builds an image, then uses the image in some way, and of course also gets some idea on how to use it in a pipeline. My preference would be to show how to use such image with GitHub actions if at all possible.

Byron · 2024-05-26T05:51:51Z

docker/Dockerfile.alpine

@@ -0,0 +1,69 @@
+ARG GITOXIDE_VERSION=0.36.0


Let's move these files into etc/docker/.

belt · 2024-05-26T13:08:29Z

Thanks a lot! I love the idea of being able to use gix clone in pipelines.

I left some comments, and one more request: there should be a new job that validates the dockerfiles, probably by running them. That's probably quite costly but can hopefully be done within 15 minutes or so.

Finally, I wonder if two dockerfiles are needed, or if one could just endorse one base image - after all, both base images build a binary, the result should be the same. Or does one binary not run in most places? Or asked differently, is alpine not producing the more compatible binary, so bookworm is superseded?

Thanks again for your help.

Alpine is based off the musl system library. Debian is based off libc system library. The resulting binaries are different. Though, alpine does provide a gcompat library when it's needed. For what it's worth, Ubuntu (or any other distro for that matter) isn't guaranteed to be running a binary compatable libc version as Debian, so things might need recompilation for compatibility on those systems.

I'll write up an inception build instruction set using gix clone vs git clone at some point soon.

Byron · 2024-05-26T16:51:41Z

Alpine is based off the musl system library. Debian is based off libc system library. The resulting binaries are different. Though, alpine does provide a gcompat library when it's needed.

Wouldn't this mean that the alpine version could run anywhere where there is a recent-enough kernel? That seems to mean it supersedes the debian version which only runs on libc compatible distros, which seems less compatible.
If that's true, who would want a libc version of the binary? Somebody saving every bit of bandwidth as it's probably a little smaller?

belt · 2024-05-27T12:32:22Z

Alpine is based off the musl system library. Debian is based off libc system library. The resulting binaries are different. Though, alpine does provide a gcompat library when it's needed.

Wouldn't this mean that the alpine version could run anywhere where there is a recent-enough kernel? That seems to mean it supersedes the debian version which only runs on libc compatible distros, which seems less compatible. If that's true, who would want a libc version of the binary? Somebody saving every bit of bandwidth as it's probably a little smaller?

Correct. Trade-offs. I'll call that out explicitly. Most official docker projects build multiple architectures and base distros.

Byron · 2024-05-27T12:55:11Z

Alright, let's go with just alpine then, MUSL seems to be the most compatible while keeping things simple.

Would you be able to setup a CI step to build the docker image, to assure it keeps working? I probably won't run it locally very much beyond initial testing.

Thanks again.

belt · 2024-05-27T13:32:55Z

Alright, let's go with just alpine then, MUSL seems to be the most compatible while keeping things simple.

Would you be able to setup a CI step to build the docker image, to assure it keeps working? I probably won't run it locally very much beyond initial testing.

Thanks again.

The CI step in GH actions is beyond my muscle-memory. I'm giving it a go, but no guarantees on results

... for linting purposes

- remove ci/docker step as it takes too long at 61 minutes - remove `docker.alpine` - add note that it might degrade

Byron · 2024-05-28T05:45:04Z

Thanks a lot! With the improved documentation I was able to understand what's intended, and with that it could be pushed over the finishing line.

Please note that I removed the CI integration as it was to slow, taking 1h to build with a budget of only 30 minutes - however, I noted that the dockerfile might stop working eventually so users are encouraged to fix it by PR.

Paul Belt added 3 commits May 25, 2024 21:01

doc: Update README for docker integration

0c06fc7

feat: Dockerfile for Alpine pipelines

c2707bf

feat: Dockerfile for Debian bookworm pipelines

a49fd60

Byron requested changes May 26, 2024

View reviewed changes

chore: move docker/ to etc/docker/

05101e5

doc: update docker docs assuming blank slate

0b0e871

Paul Belt and others added 2 commits May 27, 2024 09:53

feat: integrate docker build into CI pipeline

dc0ec2b

... for linting purposes

refactor

4ca7f6c

- remove ci/docker step as it takes too long at 61 minutes - remove `docker.alpine` - add note that it might degrade

Byron merged commit 226e758 into GitoxideLabs:main May 28, 2024
19 checks passed


		#### Pipeline Integration (recommended)

		Build an image without a target and then copy the binaries into your local image

Uh oh!

feat: Dockerfile for faster CI/CD pipelines #1380

feat: Dockerfile for faster CI/CD pipelines #1380

Uh oh!

Conversation

belt commented May 26, 2024

Uh oh!

Byron left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

belt May 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

belt commented May 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Byron commented May 26, 2024

Uh oh!

belt commented May 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Byron commented May 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

belt commented May 27, 2024

Uh oh!

Uh oh!

Byron commented May 28, 2024

Uh oh!

Uh oh!

belt May 26, 2024 •

edited

Loading

belt commented May 26, 2024 •

edited

Loading

belt commented May 27, 2024 •

edited

Loading

Byron commented May 27, 2024 •

edited

Loading