`wget` exit code 141 for `https` requests when running `linux/amd64` on M2 for cross-platform build

[fabriciojs]

I am adding this although I am not sure this is the correct place, but I figured it's the best place to begin looking for help at least - as other people facing it is likely to gather here, I assume.

I noticed a very strange behavior with wget on a busybox based image where, if I try to build for linux/amd64 running on an Apple silicon M2, wget simply cuts off after logging that is connecting to the server on port 443, and exits with an exit code 141. This happens to any HTTPS request, while plain HTTP requests still work.

The very same Dockerfile will build properly on a truly Intel amd64 environment, as well as M2 builds it properly when building linux/arm64. The problem is on the cross-platform build.

One may think that the problem could point to Docker - have tried with both OrbStack and Docker Desktop for Mac on different M2 machines, both presenting the very same behavior. I also tried building it on a virtual Ubuntu machine - managed via OrbStack - so installed Docker amd64 properly within it, and... very same error.

So I am inclined to say that it's something within busybox wget itself on the linux/amd64 build that doesn't like to be emulated from linux/arm64, or it could be something really weird from Mac's Roseta/emulation engine.

Anyone else seen something or has clues on where to look/go next? Any help is much appreciated.

[fabriciojs]

Adding the examples here to reproduce - and of course important to mention, there's a handful of reports of issues with wget on busybox with HTTPS but all I have seen behaved differently with the output/exit code, and no mention to cross-platform, thus I think what I am seeing is a different thing.

The actual image I am trying to extend and cross-build is Prometheus:

➜  prometheus git:(main) docker run --platform=linux/arm64 --rm -it --entrypoint="" prom/prometheus:v2.45.6 sh
/prometheus $ wget https://example.com
Connecting to example.com (93.184.215.14:443)
wget: note: TLS certificate validation not implemented
saving to 'index.html'
index.html           100% |********************************************************************************************************************************************************|  1256  0:00:00 ETA
'index.html' saved
/prometheus $ 
/prometheus $ exit
➜  prometheus git:(main) 
➜  prometheus git:(main) docker run --platform=linux/amd64 --rm -it --entrypoint="" prom/prometheus:v2.45.6 sh
Unable to find image 'prom/prometheus:v2.45.6' locally
v2.45.6: Pulling from prom/prometheus
Digest: sha256:15ccbb1cec5fad2cd9f20f574ba5a4dd4160e8472213c76faac17f6481cb6a75
Status: Downloaded newer image for prom/prometheus:v2.45.6
/prometheus $ 
/prometheus $ wget https://example.com
Connecting to example.com (93.184.215.14:443)
/prometheus $ echo $?
141
/prometheus $ exit
➜  prometheus git:(main) 

But it's also reproducible with the baseline prometheus/busybox image:

➜  prometheus git:(main) docker run --platform=linux/amd64 --rm -it --entrypoint="" prom/busybox:latest sh    
Unable to find image 'prom/busybox:latest' locally
latest: Pulling from prom/busybox
aa2a8d90b84c: Pull complete 
f64cf4e3a904: Pull complete 
Digest: sha256:564adfbf59f2b684bfe8798f1dfd83d785378ce28058b3c0302e22628486277d
Status: Downloaded newer image for prom/busybox:latest
/ # 
/ # wget https://example.com
Connecting to example.com (93.184.215.14:443)
/ # echo $?
141
/ # 

And it also happens with the base image for that:

➜  prometheus git:(main) docker run --platform=linux/amd64 --rm -it --entrypoint="" busybox:uclibc sh     
Unable to find image 'busybox:uclibc' locally
uclibc: Pulling from library/busybox
205c43019ebc: Pull complete 
Digest: sha256:97d85ff9630b634ddff3e3ff69fd02bc3b69de8dba0c5002eb0ad6915d1bf4c0
Status: Downloaded newer image for busybox:uclibc
/ # 
/ # wget https://example.com
Connecting to example.com (93.184.215.14:443)
/ # echo $?
141
/ # 

We observe the same with busybox:glibc as well.

[fabriciojs]

The exit code 141 standing for Broken pipe - adding to help with SEO of this issue for others facing it.

[polarathene]

Given the below findings, I think it's not platform specific in the image but rather the native host kernel affects the success. Since the busybox wget command works fine on Alpine regardless of platform, it's likely due to how it's been compiled differently for the BusyBox image vs Alpine.

I don't know if you can expect much action from this report though since this is a mirror org. I haven't checked if any activity from upstream actually goes on here 🤔

---

Ignore this observation

prom/busybox:latest

As the wget method mentions, it wasn't implemented with TLS support so certs won't verify. Prometheus image adds that afterwards:

Doesn't seem relevant to this issue though, nor does wget seem to know of it or support referencing it.

---

Reproducing with busybox:latest (aka :stable-glibc) on Windows 11 x86_64 (AMD64) system via WSL2 I get:

# AMD64 (glibc)
$ docker run --rm -it --platform=linux/amd64 busybox
$ wget -q --no-check-certificate https://github.com

$ echo $?
0

$ uname -m
x86_64

$ /lib/ld-linux-x86-64.so.2 --list /bin/wget
        linux-vdso.so.1 (0x00007ffc9b506000)
        libm.so.6 => /lib/libm.so.6 (0x00007f868dec0000)
        libresolv.so.2 => /lib/libresolv.so.2 (0x00007f868deaf000)
        libc.so.6 => /lib/libc.so.6 (0x00007f868dcce000)
        /lib64/ld-linux-x86-64.so.2 => /lib/ld-linux-x86-64.so.2 (0x00007f868e09c000)

# ARM64 (glibc)
$ docker run --rm -it --platform=linux/arm64 busybox
$ wget -q --no-check-certificate https://github.com
wget: error getting response: Connection reset by peer

$ echo $?
1

$ uname -m
aarch64

$ /lib/ld-linux-aarch64.so.1 --list /bin/wget
         (0x0000400000810000)
        libm.so.6 => /lib/libm.so.6 (0x0000400000940000)
        libresolv.so.2 => /lib/libresolv.so.2 (0x00004000009e0000)
        libc.so.6 => /lib/libc.so.6 (0x0000400000a10000)
        /lib/ld-linux-aarch64.so.1 (0x00007f63ae93e000)

The native host platform differs, which is probably why we have flipped results here. Containers images are only user-space, and while the platform can be emulated via --platform, the kernel is still the host kernel AFAIK (for me uname -a was the same as my WSL2 kernel except for the arch, I don't think it's actually using an ARM64 version of the kernel though when using --platform).

Thus wget is probably trying to use some feature of the kernel that is arch dependent? 🤷‍♂️

I had the same issue with busybox:musl btw, which has no /lib (so I assume prometheus image is extending from that), the binary is static according to patchelf / file utilities.

FWIW, the Alpine image has no problem. Must be something to do with how BusyBox was built differently vs Alpine package (wget / busybox there does dynamically link to musl though). The Alpine version also has no TLS error (/etc/ssl/certs/ca-certificates.crt bundled), so I assume it's been built to handle the connection a bit differently which prevents failure.