Hops: experimental Homebrew bottle installer and relocator

[justenstall]

Hello! I have been working on an experimental Homebrew bottle installer called Hops: https://github.com/act3-ai/hops

The goal was to use Homebrew in air gapped networks, which is enabled by a command that copies bottles and their metadata to a standalone OCI registry and an install command that uses the standalone registry. The project was written in Go, so I was able to use concurrency to improve performance, especially for downloads.

Would love to share more about the experiment if there is any interest
@MikeMcQuaid

[MikeMcQuaid]

@justenstall This sounds interesting, thanks for sharing. Could you provide more details on how this works for the hops runner for the end-user? What's your personal use-case here? Thanks!

[MikeMcQuaid]

I'd like to upstream the bottle registry changes including the self-contained bottle images and pushing latest tags so the bottle version can be determined without the API.

I like this idea but, if you don't use the API, how are you doing checksum validation here?

[MikeMcQuaid]

If there is any interest in rewriting Homebrew's CLI in Go, Hops is a good starting point for what that would look like. There are some performance, maintainability, and distribution benefits that I think would make it worth looking into. Since Homebrew's commands are already a mix of Ruby and shell scripts, it would be well suited to a progressive migration if it got to that point.

Homebrew itself could never really fully be moved to Go without breaking the Formula DSL (which heavily relies on Ruby syntax/standard library) but I agree it would be an interesting idea to explore improving the performance of Homebrew by narrowing the scope to "only works for installing bottles from JSON into the supported location" using a Hops-like approach.

[justenstall]

I like this idea but, if you don't use the API, how are you doing checksum validation here?

As long as I trust that the version tag is pointing to the correct manifest, then I know that the bottle and API JSON referenced in the tree are valid and I can verify their digests/checksums when downloading either.

Signing artifacts would guarantee a mirror of the registry couldn't change the tags maliciously (back to this issue again!)

[justenstall]

I guess I have a question about how you handle things like e.g. def post_install blocks without Homebrew installed? Are they just skipped for now?

Homebrew itself could never really fully be moved to Go without breaking the Formula DSL (which heavily relies on Ruby syntax/standard library) but I agree it would be an interesting idea to explore improving the performance of Homebrew by narrowing the scope to "only works for installing bottles from JSON into the supported location" using a Hops-like approach.

Yep, using Ruby as the Formula DSL defines the limit of what could be easily moved. If the "run the Formula DSL file" Ruby code was an isolated component, I would like to know how often a consumer would need to use it. Since it's only used for installing from source and the post_install steps, I know that my day-to-day use of Homebrew does not require that component except when I am installing from a third-party tap.

If the Formula DSL runner was its own Ruby program, an updated CLI implementation could call the Ruby program when needed. That would have the same performance benefits and most of the maintainability benefits, but it would still rely on a Ruby interpreter at times.

[MikeMcQuaid]

There's definitely some more code that could be moved to Go for more users in those cases, agreed. Unfortunately, it would require breaking far more backwards compatibility that we're willing to do for a long while. Never say never, though!

Thanks for the fruitful conversation, @justenstall.