[Bug]: `make goreleaser` fails to build a .deb file

[hax0rbana-adam]

Steps to Reproduce

1. git clone https://github.com/smallstep/cli
2. cd cli
3. sudo apt install -y golang
4. apply the patch to the Makefile from [Bug]: Compilation error during vulncheck #1403
5. make bootstrap
6. Install the goreleaser package
7. Install Google Cloud CLI (gcloud)
8. make goreleaser

Your Environment

• OS - Debian Testing (trixie)
  • This is only because of a bug preventing compilation from the latest stable version of Debian
• step CLI Version - bfab777

Expected Behavior

A .deb package would be build locally without needing to register an account with Google.

Actual Behavior

$ make goreleaser
  • by using this software you agree with its EULA, available at https://goreleaser.com/eula
  • running goreleaser v2.8.2
  • skipping validate...
  • cleaning distribution directory
  • loading environment variables
  • getting and validating git state
    • git state                                      commit=bfab7772bbd208a682936e93c16f43b3098769cb branch=master current_tag=v0.28.6 previous_tag=v0.28.5 dirty=true
    • pipe skipped or partially skipped              reason=validation is disabled
  • parsing tag
  • setting defaults
    • DEPRECATED:  archives.format_overrides.format  should not be used anymore, check https://goreleaser.com/deprecations#archivesformat_overridesformat for more info
    • DEPRECATED:  archives.builds  should not be used anymore, check https://goreleaser.com/deprecations#archivesbuilds for more info
    • DEPRECATED:  nfpms.builds  should not be used anymore, check https://goreleaser.com/deprecations#nfpmsbuilds for more info
  • partial
    • adjusting environment                          by=target target=linux_amd64 dist=dist/linux_amd64
  • snapshotting
    • building snapshot...                           version=v0.28.6-next
  • running before hooks
    • running hook                                   hook=go mod download
  • ensuring distribution directory
  • setting up metadata
  • writing release metadata
  • loading go mod information
  • build prerequisites
  • building binaries
    • partial build                                  filter=target=linux_amd64_v1 matches=linux_amd64_v1
    • building                                       binary=dist/linux_amd64/default_linux_amd64_v1/bin/step
  • setting up metadata
  • storing artifacts metadata
  • running after hooks
    • running hook                                   hook=bash scripts/package-repo-import.sh step-cli v0.28.6-next
Package: step-cli
Version: v0.28.6-next
ERROR: (gcloud.artifacts.yum.import) You do not currently have an active account selected.
Please run:

  $ gcloud auth login

to obtain new credentials.

If you have already logged in with a different account, run:

  $ gcloud config set account ACCOUNT

to select an already authenticated account to use.
  ⨯ build failed after 3s                   
    error=
    │ after hook failed: hook failed: shell: 'bash scripts/package-repo-import.sh step-cli v0.28.6-next': exit status 1: Package: step-cli
    │ Version: v0.28.6-next
    │ ERROR: (gcloud.artifacts.yum.import) You do not currently have an active account selected.
    │ Please run:
    │   $ gcloud auth login
    │ to obtain new credentials.
    │ If you have already logged in with a different account, run:
    │   $ gcloud config set account ACCOUNT
    │ to select an already authenticated account to use.
make: *** [Makefile:127: goreleaser] Error 1
user@disp7125:~/cli$ find . -name '*.deb'

Additional Context

I installed gcloud to illustrate that the .deb file can not be built even with this installed. If gcloud is not installed, the error message is as follows:

$ make goreleaser
  • by using this software you agree with its EULA, available at https://goreleaser.com/eula
  • running goreleaser v2.8.2
  • skipping validate...
  • cleaning distribution directory
  • loading environment variables
  • getting and validating git state
    • git state                                      commit=bfab7772bbd208a682936e93c16f43b3098769cb branch=master current_tag=v0.28.6 previous_tag=v0.28.5 dirty=true
    • pipe skipped or partially skipped              reason=validation is disabled
  • parsing tag
  • setting defaults
    • DEPRECATED:  archives.format_overrides.format  should not be used anymore, check https://goreleaser.com/deprecations#archivesformat_overridesformat for more info
    • DEPRECATED:  archives.builds  should not be used anymore, check https://goreleaser.com/deprecations#archivesbuilds for more info
    • DEPRECATED:  nfpms.builds  should not be used anymore, check https://goreleaser.com/deprecations#nfpmsbuilds for more info
  • partial
    • adjusting environment                          by=target target=linux_amd64 dist=dist/linux_amd64
  • snapshotting
    • building snapshot...                           version=v0.28.6-next
  • running before hooks
    • running hook                                   hook=go mod download
  • ensuring distribution directory
  • setting up metadata
  • writing release metadata
  • loading go mod information
  • build prerequisites
  • building binaries
    • partial build                                  filter=target=linux_amd64_v1 matches=linux_amd64_v1
    • building                                       binary=dist/linux_amd64/default_linux_amd64_v1/bin/step
  • setting up metadata
  • storing artifacts metadata
  • running after hooks
    • running hook                                   hook=bash scripts/package-repo-import.sh step-cli v0.28.6-next
Package: step-cli
Version: v0.28.6-next
scripts/package-repo-import.sh: line 49: gcloud: command not found
  ⨯ build failed after 1s                   
    error=
    │ after hook failed: hook failed: shell: 'bash scripts/package-repo-import.sh step-cli v0.28.6-next': exit status 127: Package: step-cli
    │ Version: v0.28.6-next
    │ scripts/package-repo-import.sh: line 49: gcloud: command not found
make: *** [Makefile:127: goreleaser] Error 1
$ find . -name '*.deb'
$

To fix this, "variables" and "check_boxes" need to be removed from .goreleaser.yml (patch below) then goreleaser release --snapshot --clean --skip sign will get further.

diff --git a/.goreleaser.yml b/.goreleaser.yml
index 5a9930ce..d536c9d4 100644
--- a/.goreleaser.yml
+++ b/.goreleaser.yml
@@ -3,19 +3,19 @@
 version: 2
 project_name: step
 
-variables:
-  packageName: step-cli
-  packageRelease: 1 # Manually update release: in the nfpm section to match this value if you change this
+#variables:
+#  packageName: step-cli
+#  packageRelease: 1 # Manually update release: in the nfpm section to match this value if you change this
 
 before:
   hooks:
     - go mod download
 
-after:
-  hooks:
-    # This script depends on IS_PRERELEASE env being set. This is set by CI in the Is Pre-release step.
-    - cmd: bash scripts/package-repo-import.sh {{ .Var.packageName }} {{ .Version }}
-      output: true
+#after:
+#  hooks:
+#    # This script depends on IS_PRERELEASE env being set. This is set by CI in the Is Pre-release step.
+#    - cmd: bash scripts/package-repo-import.sh {{ .Var.packageName }} {{ .Version }}
+#      output: true
 
 builds:
   - &BUILD
@@ -96,7 +96,7 @@ nfpms:
     id: packages
     builds:
       - nfpm
-    package_name: "{{ .Var.packageName }}"
+    package_name: "step-cli"
     release: "1"
     file_name_template: >-
       {{- trimsuffix .ConventionalFileName .ConventionalExtension -}}
@@ -160,7 +160,7 @@ publishers:
 - name: Google Cloud Artifact Registry
   ids:
   - packages
-  cmd: ./scripts/package-upload.sh {{ abs .ArtifactPath }} {{ .Var.packageName }} {{ .Version }} {{ .Var.packageRelease }}
+  cmd: ./scripts/package-upload.sh {{ abs .ArtifactPath }} step-cli {{ .Version }} 1
 
 snapshot:
   version_template: "{{ .Tag }}-next"
@@ -206,10 +206,10 @@ release:
     - 📦 [step_linux_{{ .Version }}_amd64.tar.gz](https://dl.smallstep.com/gh-release/cli/gh-release-header/{{ .Tag }}/step_linux_{{ .Version }}_amd64.tar.gz)
     - 📦 [step_linux_{{ .Version }}_arm64.tar.gz](https://dl.smallstep.com/gh-release/cli/gh-release-header/{{ .Tag }}/step_linux_{{ .Version }}_arm64.tar.gz)
     - 📦 [step_linux_{{ .Version }}_armv7.tar.gz](https://dl.smallstep.com/gh-release/cli/gh-release-header/{{ .Tag }}/step_linux_{{ .Version }}_armv7.tar.gz)
-    - 📦 [step-cli_{{ replace .Version "-" "." }}-{{ .Var.packageRelease }}_amd64.deb](https://dl.smallstep.com/gh-release/cli/gh-release-header/{{ .Tag }}/step-cli_{{ replace .Version "-" "." }}-{{ .Var.packageRelease }}_amd64.deb)
-    - 📦 [step-cli-{{ replace .Version "-" "." }}-{{ .Var.packageRelease }}.x86_64.rpm](https://dl.smallstep.com/gh-release/cli/gh-release-header/{{ .Tag }}/step-cli-{{ replace .Version "-" "." }}-{{ .Var.packageRelease }}.x86_64.rpm)
-    - 📦 [step-cli_{{ replace .Version "-" "." }}-{{ .Var.packageRelease }}_arm64.deb](https://dl.smallstep.com/gh-release/cli/gh-release-header/{{ .Tag }}/step-cli_{{ replace .Version "-" "." }}-{{ .Var.packageRelease }}_arm64.deb)
-    - 📦 [step-cli-{{ replace .Version "-" "." }}-{{ .Var.packageRelease }}.aarch64.rpm](https://dl.smallstep.com/gh-release/cli/gh-release-header/{{ .Tag }}/step-cli-{{ replace .Version "-" "." }}-{{ .Var.packageRelease }}.aarch64.rpm)
+    - 📦 [step-cli_{{ replace .Version "-" "." }}-1_amd64.deb](https://dl.smallstep.com/gh-release/cli/gh-release-header/{{ .Tag }}/step-cli_{{ replace .Version "-" "." }}-1_amd64.deb)
+    - 📦 [step-cli-{{ replace .Version "-" "." }}-1.x86_64.rpm](https://dl.smallstep.com/gh-release/cli/gh-release-header/{{ .Tag }}/step-cli-{{ replace .Version "-" "." }}-1.x86_64.rpm)
+    - 📦 [step-cli_{{ replace .Version "-" "." }}-1_arm64.deb](https://dl.smallstep.com/gh-release/cli/gh-release-header/{{ .Tag }}/step-cli_{{ replace .Version "-" "." }}-{{ .Var.packageRelease }}_arm64.deb)
+    - 📦 [step-cli-{{ replace .Version "-" "." }}-1.aarch64.rpm](https://dl.smallstep.com/gh-release/cli/gh-release-header/{{ .Tag }}/step-cli-{{ replace .Version "-" "." }}-1.aarch64.rpm)
     - see `Assets` below for more builds
 
     #### macOS Darwin
@@ -407,7 +407,7 @@ winget:
       pull_request:
         # Whether to enable it or not.
         enabled: true
-        check_boxes: true
+        #check_boxes: true
         # Whether to open the PR as a draft or not.
         #
         # Default: false

I'm not sure why the .goreleaser.yml file needs the above modifications, but goreleaser release will refuse to run as long as those are there. Mainly I just wanted to provide a workaround that would allow people to build their own .deb package locally until this can be fixed properly.

After the workaround is applied, the .deb files are there:

$ find . -name '*.deb'
./dist/step-cli_0.28.6~next-1_amd64.deb
./dist/step-cli_amd64.deb
./dist/step-cli_0.28.6~next-1_arm64.deb
./dist/step-cli_arm64.deb

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[hslatman]

Hey @hax0rbana-adam, may I ask why you're trying to build a .deb locally? Is the one available in our releases not OK, e.g.: https://github.com/smallstep/cli/releases/tag/v0.28.6?

[hslatman]

To fix this, "variables" and "check_boxes" need to be removed from .goreleaser.yml (patch below) then goreleaser release --snapshot --clean --skip sign will get further.

I don't see how these would be relevant to it breaking on the gcloud part, and the changes look too drastic to me. The part that calls gcloud could be made optional, but that's actually the part where an artifact gets pushed to a GCP storage location. This is part of our release process.

[hax0rbana-adam]

The gcloud is only a barrier to using the Makefile to build deb packages, however packaging also fails when trying to run goreleaser release --snapshot --clean --skip sign.

Making gcloud optional would address half of the issue while still allowing your release process to work unchanged.

The other half of the fix are the changes to .goreleaser.yml, as they are required to build is even after the gcloud dependency is removed. This was verified using v1.24 of the go compiler (the latest available in Debian Trixie).

As to the question on why I was trying to build a .deb package locally, it started because the the installation page indicates that the only packages available are AMD64 despite binaries being available for arm64. So I expected that I'd need to package it up myself. It was only later that I found the arm64 .deb package. If the instructions were updated to explain that there are arm64 packages, or if #660 were addressed and the documentation was updated to instruct people to use the 3rd party apt repo, this confusion would be eliminated.

However, one of the major benefits of using open source projects is that people can build & package the code them themselves. If the project is ever abandoned, which is extremely common, it's vital that users be able to build & package the code. The same is true if people want to do end-to-end testing when they make a code change before submitting their patch upstream.

This is, of course, a philosophical argument with which people may not agree. I've managed to build the .deb files locally and done my best to offer the patches upstream to offer this capability to others. I don't like having to get rid of using variables, so I hope that someone will build on my improvements and preserve the ability to use them before merging this into the default branch. Maybe there's just some compiler flag that is needed or something?

[hslatman]

There was work in flight updating the docs for getting the repositories setup. The page is now updated: https://smallstep.com/docs/step-cli/installation/#linux-packages-amd64.

We'll need to keep the gcloud hooks in by default, because that's what powers uploading to GCP, and is where the packages are served from.

We depend on Goreleaser Pro, which is not something just anyone would do or is willing to pay for. That is why you needed some more changes to the Makefile, as those are not supported in the normal version. We might update some of the Makefile to make it simple to run goreleaser locally, but it's not an explicit goal for others to be able to create releases in the exact same way as we do at this time. Also, technically the binary is just a go build away, after all.

[hax0rbana-adam]

The installation page still says the Linux Packages are AMD64, with no mention of ARM in that heading nor section. Here's what it looks like for me:

Thanks for filling me in on why I had to make changes to the yml file in order to get goreleaser to work. That all makes sense.

I. Would you be willing to accept a contribution of an additional .yml file that people could use to build a .deb package themselves? If so, I'm willing to submit a pull request with that change and documentation to explain how to build it.

II. Would you be willing to accept a modification to the Makefile add a goreleaser-local target, which would not require gcloud? I'd also include documentation on how to do so. My proposed change is to leave the roreleaser target exactly the same so your release process would not be affected.

[hslatman]

The installation page still says the Linux Packages are AMD64, with no mention of ARM in that heading nor section. Here's what it looks like for me:

If I understood my colleagues correctly, it should also support ARM64.

Thanks for filling me in on why I had to make changes to the yml file in order to get goreleaser to work. That all makes sense.

I. Would you be willing to accept a contribution of an additional .yml file that people could use to build a .deb package themselves? If so, I'm willing to submit a pull request with that change and documentation to explain how to build it.

That would be nice. But see remark below; might need to go after that.

II. Would you be willing to accept a modification to the Makefile add a goreleaser-local target, which would not require gcloud? I'd also include documentation on how to do so. My proposed change is to leave the roreleaser target exactly the same so your release process would not be affected.

This is on my plate.

[hslatman]

WIP: #1408

Could you give that a try, @hax0rbana-adam? More changes may be necessary, but I wanted to keep the changes minimal to not incur a lot more maintenance burden.

[hax0rbana-adam]

That fixed the ability to make goreleaser and make goreleaser locally with GoReleaser OSS, however it only builds bin/step. It does not produce a .deb package. If that was the intended scope of this change, then this is good to go.

FYI: make bootstrap is still failing because it has the wrong URL to the GoReleaser Pro download. That doesn't really affect me personally since I need the OSS version anyway, but I wanted to make sure it was noted for the people who use make bootstrap to download the paid version. I guess maybe that should be its own ticket?

[hslatman]

That fixed the ability to make goreleaser and make goreleaser locally with GoReleaser OSS, however it only builds bin/step. It does not produce a .deb package. If that was the intended scope of this change, then this is good to go.

For most purposes the current PR should be sufficient. The goal of the new target is not to make it full parity with our CI releases; just to make it easier to run the release process locally while debugging and updating things etc. You did point out some failures with our make bootstrap and the release process that crept in over time, so it made sense to improve things. I'm not sure how much the (local) creation of a .deb package brings us, considering the above, and the fact that the packages are available in a repo now.

FYI: make bootstrap is still failing because it has the wrong URL to the GoReleaser Pro download. That doesn't really affect me personally since I need the OSS version anyway, but I wanted to make sure it was noted for the people who use make bootstrap to download the paid version. I guess maybe that should be its own ticket?

I've pushed some additional commits, incl. a fix for that download URL. When I used Pro locally without a license (afaik), it succeeds building the binary when using make goreleaser-local. I believe GoReleaser might be doing some checks to only require a license for specific functionalities, and thus working just like OSS if the config does not use Pro features.

There are some additional changes in the Makefile to help with an edge case affecting the behavior of OSS vs. Pro (which might in fact be a bug in GoReleaser). It is thus possible to use both OSS and Pro now, but make bootstrap will always download the Pro version, which will overwrite other versions of the goreleaser binary in $GOPATH/bin. We might change things in the future be depending more on the go tools directive if it makes sense.

[hax0rbana-adam]

According to the documentation, the Smallstep apt repo does not include any .deb files for ARM64. I don't know if that's an oversight in spinning up the apt repo, or an oversight in updating the documentation. step-cli docs and step-ca docs

If the AMD64 & ARM64 packages are both available from the Smallstep apt repo, I agree that it is much less important for people to be able to build their own .deb package.

However, there are many situation where it is still useful:

• deploying custom patches that are not [yet] accepted upstream (using one's own apt repo)
• reliability: in case the Smallstep releases and apt repo goes away, organizations could continue on their own
• security: so organizations can build packages from source and self-host them on their own apt repo
• legal requirements: it's common in highly regulated industries such as critical infrastructure to have those security & reliability requirements imposed by regulators (or lawyers)

The title of this issue still holds. Using the makefile, one still can not create a .deb file. However, with the improvements in #1408, it's extremely close to being able to create a .deb file locally using an undocumented build process. The following was tested on Debian Testing (trixie).

git clone https://github.com/smallstep/cli
cd cli
sudo apt install -y golang
git checkout herman/goreleaser-improvements
make bootstrap

# Work around gcloud requirement
export PATH=~/go/bin:$PATH
printf '#!/bin/sh\necho $@\n' > ~/go/bin/gcloud
chmod +x ~/go/bin/gcloud 

GPG_PRIVATE_KEY_FILE="" goreleaser release --snapshot --clean --skip sign
find . -name '*.deb'

Without the environment variable, goreleaser fails with the following error (despite the the --skip sign command line option being used):

  • linux packages
  ⨯ release failed after 6m39s               error=template: failed to apply "{{ .Env.GPG_PRIVATE_KEY_FILE }}": map has no entry for key "GPG_PRIVATE_KEY_FILE"

If build instructions are added which include building .deb files, I'd consider this ticket to be resolved even though the process is not as easy as running make goreleaser.

The real intent of this ticket was to make sure people could create their own .deb files. As long as there's a documented way to do so, that takes care of my concern. Even just publishing the build instructions above would suffice, even with the peculiarities of making a script called gcloud and specifying an empty environment variable.

[hslatman]

According to the documentation, the Smallstep apt repo does not include any .deb files for ARM64. I don't know if that's an oversight in spinning up the apt repo, or an oversight in updating the documentation. step-cli docs and step-ca docs

If the AMD64 & ARM64 packages are both available from the Smallstep apt repo, I agree that it is much less important for people to be able to build their own .deb package.

It should be there. At least that's my understanding of talking to my colleagues. Should thus just be a simple change to the docs. I'm not daily driving Linux, nor am I installing step from our releases generally. It's just a go run, go build or go install for me.

However, there are many situation where it is still useful:

* deploying custom patches that are not [yet] accepted upstream (using one's own apt repo)

* reliability: in case the Smallstep releases and apt repo goes away, organizations could continue on their own

* security: so organizations can build packages from source and self-host them on their own apt repo

* legal requirements: it's common in highly regulated industries such as critical infrastructure to have those security & reliability requirements imposed by regulators (or lawyers)

These all sound plausible, but in my opinion don't necessarily warrant maintaining support for the release process to be performed outside of environments that we control. The first is acceptable, but in the end if you want to test a patch, a binary can be created with go build and then tested. There's no real need to make a full .deb package out of that, imo. We release regularly, and we can create releases ad-hoc if necessary, so a release doesn't need to take long. For the second one: it's a fair point, but it's a matter of fixing it now, or fixing it in the case our releases go away. We prefer spending time on our releases not going away instead of maintaining the alternative option(s). For the last two, in these situations there are likely already resources available to maintain the internal builds and repos, so should we really take that burden onto us, let alone for free? I'd like to add that we already maintain separate packages for certain projects with patches applied for paid customers, some of which are subject to the security and legal requirements you're referring to.

The title of this issue still holds. Using the makefile, one still can not create a .deb file. However, with the improvements in #1408, it's extremely close to being able to create a .deb file locally using an undocumented build process. The following was tested on Debian Testing (trixie).

git clone https://github.com/smallstep/cli
cd cli
sudo apt install -y golang
git checkout herman/goreleaser-improvements
make bootstrap

Work around gcloud requirement

export PATH=~/go/bin:$PATH
printf '#!/bin/sh\necho $@\n' > ~/go/bin/gcloud
chmod +x ~/go/bin/gcloud

GPG_PRIVATE_KEY_FILE="" goreleaser release --snapshot --clean --skip sign
find . -name '*.deb'

Try adding after and/or post-hooks to your --skip, like in the updated goreleaser target, and then I think you won't need the gcloud hack.

Without the environment variable, goreleaser fails with the following error (despite the the --skip sign command line option being used):

  • linux packages
  ⨯ release failed after 6m39s               error=template: failed to apply "{{ .Env.GPG_PRIVATE_KEY_FILE }}": map has no entry for key "GPG_PRIVATE_KEY_FILE"

Instead of defining the environment variable before executing the build, it seems possible to use the envOrDefault function for the GoReleaser config template: https://goreleaser.com/customization/templates/#functions. The default value could be something like NOTSET, assuming signing the build (when not using --skip sign) would then fail with an error including that value.

If build instructions are added which include building .deb files, I'd consider this ticket to be resolved even though the process is not as easy as running make goreleaser.

The real intent of this ticket was to make sure people could create their own .deb files. As long as there's a documented way to do so, that takes care of my concern. Even just publishing the build instructions above would suffice, even with the peculiarities of making a script called gcloud and specifying an empty environment variable.

While I appreciate your concerns, the time you put into figuring out a working local release process and describing all of it at length in this issue, I believe it's a bit of a niche use case. It's not one of our primary interests to let everyone build their own releases in every format easily, let alone in environments that we don't control. The release process primarily exists for us to release software, and if there's ever a need for someone else to release our software, they can use their own release process and tooling if needed. There's more to it than just building a .deb, because people will likely want to use their own signing key (or method), push the .deb to some other place, obtain dependencies from an internal Go module proxy, etc.

With the additional changes applied as described above I think the process could be as simple as goreleaser release --snapshot --clean --skip sign,after. I don't know what a good place would be to put that at the moment.