Unable to push an image to ECR

[jammymalina]

Hi Packer Docker plugin team, I observed this peculiar issue.

Overview of the Issue

Post-processor docker-push fails when it tries to push an image to the ECR. Both build and login were successful. If I try to push the docker image manually (doing docker push myself) it succeeds.

Reproduction Steps

Run packer build

Plugin and Packer version

From packer version 1.8.3 and plugin version 1.0.8

Simplified Packer Buildfile

packer {
  required_plugins {
    docker = {
      version = "1.0.8"
      source  = "github.com/hashicorp/docker"
    }
  }
}

source "docker" "jdk" {
  image    = "public.ecr.aws/docker/library/openjdk:20-slim-bullseye"
  commit   = true
  platform = "linux/amd64"

  changes = [...]
}

build {
  name = "jdk_image"
  sources = [
    "source.docker.jdk"
  ]

  provisioner "shell" {
    inline = [...]
  }

  post-processors {
    post-processor "docker-tag" {
      repository = "${var.aws_account_id}.dkr.ecr.${var.aws_region}.amazonaws.com/${var.image_name}"
      tags       = [var.image_tag]
    }

    post-processor "docker-push" {
      ecr_login    = true
      login_server = "https://${var.aws_account_id}.dkr.ecr.${var.aws_region}.amazonaws.com"
    }
  }
}

Operating system and Environment details

macOS Monterey 12.6, Darwin 21.6.0 Darwin Kernel Version 21.6.0, Docker with Colima VM

Log Fragments and crash.log files

==> jdk_image.docker.jdk: Running post-processor:  (type docker-push)
    jdk_image.docker.jdk (docker-push): Creating temporary Docker configuration directory
    jdk_image.docker.jdk (docker-push): Fetching ECR credentials...
2022/10/27 17:23:45 packer-plugin-docker_v1.0.8_x5.0_darwin_amd64 plugin: 2022/10/27 17:23:45 Getting ECR token for account: 123456789 in eu-west-1..
2022/10/27 17:23:45 packer-plugin-docker_v1.0.8_x5.0_darwin_amd64 plugin: 2022/10/27 17:23:45 [INFO] AWS Auth provider used: "SharedCredentialsProvider"
2022/10/27 17:23:45 packer-plugin-docker_v1.0.8_x5.0_darwin_amd64 plugin: 2022/10/27 17:23:45 Found region eu-west-1
2022/10/27 17:23:45 packer-plugin-docker_v1.0.8_x5.0_darwin_amd64 plugin: 2022/10/27 17:23:45 [INFO] AWS authentication used: "SharedCredentialsProvider"
2022/10/27 17:23:45 packer-plugin-docker_v1.0.8_x5.0_darwin_amd64 plugin: 2022/10/27 17:23:45 Successfully got login for ECR: https://123456789.dkr.ecr.eu-west-1.amazonaws.com
    jdk_image.docker.jdk (docker-push): Logging in...
2022/10/27 17:23:45 packer-plugin-docker_v1.0.8_x5.0_darwin_amd64 plugin: 2022/10/27 17:23:45 Executing: /usr/local/bin/docker [--config /var/folders/d_/37637p2n7xz6nqdk5_ks2znh0000gn/T/packer2477848013 login -u AWS --password-stdin https://123456789.dkr.ecr.eu-west-1.amazonaws.com]
    jdk_image.docker.jdk (docker-push): WARNING! Your password will be stored unencrypted in /var/folders/d_/37637p2n7xz6nqdk5_ks2znh0000gn/T/packer2477848013/config.json.
    jdk_image.docker.jdk (docker-push): Configure a credential helper to remove this warning. See
    jdk_image.docker.jdk (docker-push): https://docs.docker.com/engine/reference/commandline/login/#credentials-store
    jdk_image.docker.jdk (docker-push): Login Succeeded
    jdk_image.docker.jdk (docker-push): Pushing: 123456789.dkr.ecr.eu-west-1.amazonaws.com/push-repo:latest
2022/10/27 17:23:46 packer-plugin-docker_v1.0.8_x5.0_darwin_amd64 plugin: 2022/10/27 17:23:46 Executing: /usr/local/bin/docker [--config /var/folders/d_/37637p2n7xz6nqdk5_ks2znh0000gn/T/packer2477848013 push 123456789.dkr.ecr.eu-west-1.amazonaws.com/repo:latest]
    jdk_image.docker.jdk (docker-push): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
    jdk_image.docker.jdk (docker-push): Logging out...
2022/10/27 17:23:46 packer-plugin-docker_v1.0.8_x5.0_darwin_amd64 plugin: 2022/10/27 17:23:46 Executing: /usr/local/bin/docker [--config /var/folders/d_/37637p2n7xz6nqdk5_ks2znh0000gn/T/packer2477848013 logout https://123456789.dkr.ecr.eu-west-1.amazonaws.com]
    jdk_image.docker.jdk (docker-push): Removing login credentials for 123456789.dkr.ecr.eu-west-1.amazonaws.com
    jdk_image.docker.jdk (docker-push): Removing temporary Docker configuration directory
2022/10/27 17:23:46 packer-plugin-docker_v1.0.8_x5.0_darwin_amd64 plugin: 2022/10/27 17:23:46 error: Bad exit status: 1
2022/10/27 17:23:46 [INFO] (telemetry) ending docker-push

* Post-processor failed: Bad exit status: 1

[smaddock]

Just running into this myself, although I don't think it's specific to ECR. Rather it seems to be that docker-push does not respect the Docker context set nor have a way to specify the Docker context, and only uses the default context. Colima (and other container runtimes) specify alternate contexts as can be seen by running docker context ls. Example:

❯ docker context ls
NAME       DESCRIPTION                               DOCKER ENDPOINT
colima *   colima                                    unix:///Users/smaddock/.colima/default/docker.sock
default    Current DOCKER_HOST based configuration   unix:///var/run/docker.sock

This is why docker push works manually but the post processor does not. Note that this is a different issue than running Packer on a remote host... the host is still local; the socket is just in an alternate location.

[smaddock]

Documentation: https://docs.docker.com/engine/context/working-with-contexts/

If we could set the DOCKER_CONTEXT environment variable, or the --context command-line argument like we can with run_command on the builder, that would work.

[wawrzek]

This might be a Mac specific problem. I just noticed that there is more than one context with Docker Desktop:

NAME              DESCRIPTION                               DOCKER ENDPOINT                                 ERROR
default           Current DOCKER_HOST based configuration   unix:///var/run/docker.sock
desktop-linux *   Docker Desktop                            unix:///Users/wawrzek/.docker/run/docker.sock

and Rancher Desktop has own:

rancher-desktop   Rancher Desktop moby context              unix:///Users/wawrzek/.rd/docker.sock

I'll check @smaddock your suggestion to set DOCKER_CONTEXT.

[wawrzek]

After setting the DOCKER_CONTEXT I see now a different error:

    docker.eigenlayer (docker-push): Failed to initialize: unable to resolve docker endpoint: context "desktop-linux": context not found: open /var/folders/l4/0bhk3tfj2dqg60yp93c7dtl00000gn/T/packer1747910882/contexts/meta/fe9c6bd7a66301f49ca9b6a70b217107cd1284598bfc254700c989b916da791e/meta.json: no such file or directory
    docker.eigenlayer (docker-push): Removing temporary Docker configuration directory

[wawrzek]

During further investigation, I fixed the default context in Docker Desktop. In my case it was pointing ad the socket from Rancher Desktop which was broken. I deleted '/var/run/docker.sock', disabled and then re-enabled Allow the default Docker socket to be used (requires password) option in the Docker Desktop Advance Settings. Default context (builder in Docker Desktop) was fixed, and then I can run packer normally.

I suggest anyone with the problem to check if /var/run/docker.sock is valid and enabled in Docker Desktop.

[smaddock]

Yes, I got it working by installing Docker Desktop which setup /var/run/docker.sock. That is a workaround, though. Docker Desktop should not be a requirement for the Packer docker-push post-processor, as it is not for the builder or other post-processors. docker-push should correctly ascertain and use the active Docker endpoint based on the Docker context.