18.04 CML image failing to install terraform

[risktoparkrageram]

We are running CML using AWS and from last week have been getting errors when spinning up the runners like this:

cloud-init[3069]: tpi:setup.sh: sudo mv docker-credential-gcr /usr/bin/docker-credential-gcr\n[   81.521044]
cloud-init[3069]: tpi:setup.sh: curl -fsSL https://apt.releases.hashicorp.com/gpg\n[   81.521111]
cloud-init[3069]: tpi:setup.sh: sudo apt-key add -\n[   81.556216]
cloud-init[3069]: Warning: apt-key output should not be parsed (stdout is not a terminal)\n[   81.642723]
cloud-init[3069]: OK\n[   81.647318]
cloud-init[3069]: tpi:setup.sh: lsb_release -cs\n[   81.673421]
cloud-init[3069]: tpi:setup.sh: sudo apt-add-repository deb [arch=amd64] https://apt.releases.hashicorp.com bionic main\n         Starting Update APT News...\n         Starting Update the local ESM caches...\n[   82.100306]
cloud-init[3069]: Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease\n[   82.100457]
cloud-init[3069]: Hit:2 http://eu-west-2.ec2.archive.ubuntu.com/ubuntu bionic InRelease\n[   82.100620]
cloud-init[3069]: Hit:3 http://ppa.launchpad.net/git-core/ppa/ubuntu bionic InRelease\n[   82.103283]
cloud-init[3069]: Hit:4 http://eu-west-2.ec2.archive.ubuntu.com/ubuntu bionic-updates InRelease\n[   82.105690]
cloud-init[3069]: Hit:5 http://eu-west-2.ec2.archive.ubuntu.com/ubuntu bionic-backports InRelease\n[   82.107193]
cloud-init[3069]: Hit:6 https://download.docker.com/linux/ubuntu bionic InRelease\n[\u001b[0;32m  OK  \u001b[0m]
Started Update APT News.\n[   82.368015]
cloud-init[3069]: Ign:7 https://apt.releases.hashicorp.com bionic InRelease\n[   82.627173]
cloud-init[3069]: Err:8 https://apt.releases.hashicorp.com bionic Release\n[   82.628441]
cloud-init[3069]:   404  Not Found [IP: some-ip]\n[\u001b[0;32m  OK  \u001b[0m]
Started Update the local ESM caches.\n[   85.152879]
cloud-init[3069]: Reading package lists...\n[   85.180259]
cloud-init[3069]: E: The repository 'https://apt.releases.hashicorp.com bionic Release' does not have a Release file.\n[   85.181761]
cloud-init[3069]: tpi:setup.sh: sudo apt update\n[   85.186698]
cloud-init[3069]: WARNING: apt does not have a stable CLI interface. Use with caution in scripts.\n         Starting Update APT News...\n         Starting Update the local ESM caches...\n[   85.251238]
cloud-init[3069]: Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease\n[   85.251381]
cloud-init[3069]: Hit:2 http://eu-west-2.ec2.archive.ubuntu.com/ubuntu bionic InRelease\n[   85.251510]
cloud-init[3069]: Hit:3 http://eu-west-2.ec2.archive.ubuntu.com/ubuntu bionic-updates InRelease\n[   85.254358]
cloud-init[3069]: Hit:4 http://eu-west-2.ec2.archive.ubuntu.com/ubuntu bionic-backports InRelease\n[   85.259454]
cloud-init[3069]: Hit:5 https://download.docker.com/linux/ubuntu bionic InRelease\n[   85.261302]
cloud-init[3069]: Ign:6 https://apt.releases.hashicorp.com bionic InRelease\n[   85.264582]
cloud-init[3069]: Err:7 https://apt.releases.hashicorp.com bionic Release\n[   85.264645]
cloud-init[3069]:   404  Not Found [IP: some-ip]\n[   85.294728]

With the image being set as aws_build_ubuntu_image = "*ubuntu-*-18.04-amd64-server-*", then this line points us to a APT repo that is no longer there, causing CML to fail!

The symptoms are that the EC2 runner gets spun up, but then gets terminated quite soon after that, whilst the github action eventually times out:

{"level":"info","message":"Plan: 1 to add, 0 to change, 0 to destroy."}
{"level":"info","message":"iterative_cml_runner.runner: Creating..."}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [10s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [20s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [30s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [40s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [50s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [1m0s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [1m10s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [1m20s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [1m30s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [1m40s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [1m50s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [2m0s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [2m10s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [2m20s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [2m30s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [2m40s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [2m50s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [3m0s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [3m10s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [3m20s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [3m30s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [3m40s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [3m50s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [4m0s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [4m10s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [4m20s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [4m30s elapsed]"}
{"level":"info","message":"iterative_cml_runner.runner: Still creating... [4m40s elapsed]"}
Error: The operation was canceled.

Please let me know if there is any more context I can provide, I am hopeful that the solution could be solved by updating the ubuntu version so it points to an available hashicorp repo, or allowing arbitrary ubuntu versions? Alternatively I would be very grateful for any workaround ideas you might have!

[kqf]

@risktoparkrageram sorry for pinging you, but did you find any workaround on your own? Or any alternative way to spin EC2 instances. Would be very helpful for any insight since this issue has immense impact on my work as well

[risktoparkrageram]

Nae bother @kqf - I am still hopeful for a fix to CML, but for now we have gone with removing CML and spinning up / stopping EC2 runners manually using this action https://github.com/machulav/ec2-github-runner.

The example in their readme is a good place to start.