APIClient.push() with no auth config results in EOF with Docker 28.3.3

[felixfontein]

I reported this to moby/moby (moby/moby#50614), but since it also affects Docker SDK for Python, I'd like to report it here as well.

Reproducer from moby/moby#50614 (comment):

1. Run on shell:

   docker run --name registry --publish 5000:5000 --detach registry:2.8.3
   docker pull hello-world:latest
   docker tag hello-world:latest localhost:5000/hello-world:latest

2. Run in Python, with Docker SDK for Python available:

   import docker
   list(docker.APIClient().push("localhost:5000/hello-world:latest", stream=True, decode=True))

This results in:

Traceback (most recent call last):
  File "/usr/lib/python3.13/http/client.py", line 579, in _get_chunk_left
    chunk_left = self._read_next_chunk_size()
  File "/usr/lib/python3.13/http/client.py", line 546, in _read_next_chunk_size
    return int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.13/http/client.py", line 597, in _read_chunked
    while (chunk_left := self._get_chunk_left()) is not None:
                         ~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/http/client.py", line 581, in _get_chunk_left
    raise IncompleteRead(b'')
http.client.IncompleteRead: IncompleteRead(0 bytes read)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/urllib3/response.py", line 779, in _error_catcher
    yield
  File "/usr/lib/python3.13/site-packages/urllib3/response.py", line 904, in _raw_read
    data = self._fp_read(amt, read1=read1) if not fp_closed else b""
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/urllib3/response.py", line 887, in _fp_read
    return self._fp.read(amt) if amt is not None else self._fp.read()
           ~~~~~~~~~~~~~^^^^^
  File "/usr/lib/python3.13/http/client.py", line 473, in read
    return self._read_chunked(amt)
           ~~~~~~~~~~~~~~~~~~^^^^^
  File "/usr/lib/python3.13/http/client.py", line 609, in _read_chunked
    raise IncompleteRead(b''.join(value)) from exc
http.client.IncompleteRead: IncompleteRead(0 bytes read)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<python-input-1>", line 1, in <module>
    list(docker.APIClient().push("localhost:5000/hello-world:latest", stream=True, decode=True))
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/docker-py/docker/api/client.py", line 360, in _stream_helper
    yield from json_stream(self._stream_helper(response, False))
  File "/path/to/docker-py/docker/utils/json_stream.py", line 60, in split_buffer
    for data in stream_as_text(stream):
                ~~~~~~~~~~~~~~^^^^^^^^
  File "/path/to/docker-py/docker/utils/json_stream.py", line 16, in stream_as_text
    for data in stream:
                ^^^^^^
  File "/path/to/docker-py/docker/api/client.py", line 365, in _stream_helper
    data = reader.read(1)
  File "/usr/lib/python3.13/site-packages/urllib3/response.py", line 980, in read
    data = self._raw_read(amt)
  File "/usr/lib/python3.13/site-packages/urllib3/response.py", line 903, in _raw_read
    with self._error_catcher():
         ~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/usr/lib/python3.13/site-packages/urllib3/response.py", line 806, in _error_catcher
    raise ProtocolError(f"Connection broken: {e!r}", e) from e
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

A simple workaround is to pass auth_config={}:

list(docker.APIClient().push("localhost:5000/hello-world:latest", stream=True, decode=True, auth_config={}))

This bug can be fixed on Docker SDK for Python's side with the following patch:

diff --git a/docker/api/image.py b/docker/api/image.py
index 85109473..6e28894e 100644
--- a/docker/api/image.py
+++ b/docker/api/image.py
@@ -484,6 +484,8 @@ class ImageApiMixin:
             header = auth.get_config_header(self, registry)
             if header:
                 headers['X-Registry-Auth'] = header
+            else:
+                headers['X-Registry-Auth'] = auth.encode_header({})
         else:
             log.debug('Sending supplied auth config')
             headers['X-Registry-Auth'] = auth.encode_header(auth_config)

[When-No-Light]

I had a similar problem, and adding auth_config={} solved it (thanks for the advice). Everything was working correctly until some operating system upgrades. Our team doesn’t know what exactly changed, but we now see this problem on Ubuntu and Fedora.

We know that it worked on Fedora (version from 06.2025) and on Ubuntu 24.04 (until 08.2025) without any problems:

client = docker.from_env()
client.images.push(
        repository=repository,
        tag=version,
    )

But on 26.08.2025 we noticed that our tests stopped working:

>       push_details = client.images.push(
            repository=remote_tsxlib_testing_package_image_docker_package.image_tag,
            tag=remote_tsxlib_testing_package_image_docker_package.version,
        )

src/python/integration_tests/running_scenarios/common/infrastructure.py:859: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/home/nightfoxer/.cache/pants/named_caches/pex_root/venvs/1/s/a448a1bf/venv/lib/python3.10/site-packages/docker/models/images.py:478: in push
    return self.client.api.push(repository, tag=tag, **kwargs)
/home/nightfoxer/.cache/pants/named_caches/pex_root/venvs/1/s/a448a1bf/venv/lib/python3.10/site-packages/docker/api/image.py:491: in push
    response = self._post_json(
/home/nightfoxer/.cache/pants/named_caches/pex_root/venvs/1/s/a448a1bf/venv/lib/python3.10/site-packages/docker/api/client.py:303: in _post_json
    return self._post(url, data=json.dumps(data2), **kwargs)
/home/nightfoxer/.cache/pants/named_caches/pex_root/venvs/1/s/a448a1bf/venv/lib/python3.10/site-packages/docker/utils/decorators.py:44: in inner
    return f(self, *args, **kwargs)
/home/nightfoxer/.cache/pants/named_caches/pex_root/venvs/1/s/a448a1bf/venv/lib/python3.10/site-packages/docker/api/client.py:242: in _post
    return self.post(url, **self._set_request_timeout(kwargs))
/home/nightfoxer/.cache/pants/named_caches/pex_root/venvs/1/s/a448a1bf/venv/lib/python3.10/site-packages/requests/sessions.py:637: in post
    return self.request("POST", url, data=data, json=json, **kwargs)
/home/nightfoxer/.cache/pants/named_caches/pex_root/venvs/1/s/a448a1bf/venv/lib/python3.10/site-packages/requests/sessions.py:589: in request
    resp = self.send(prep, **send_kwargs)
/home/nightfoxer/.cache/pants/named_caches/pex_root/venvs/1/s/a448a1bf/venv/lib/python3.10/site-packages/requests/sessions.py:746: in send
    r.content
/home/nightfoxer/.cache/pants/named_caches/pex_root/venvs/1/s/a448a1bf/venv/lib/python3.10/site-packages/requests/models.py:902: in content
    self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def generate():
        # Special case for urllib3.
        if hasattr(self.raw, "stream"):
            try:
                yield from self.raw.stream(chunk_size, decode_content=True)
            except ProtocolError as e:
>               raise ChunkedEncodingError(e)
E               requests.exceptions.ChunkedEncodingError: Response ended prematurely

/home/nightfoxer/.cache/pants/named_caches/pex_root/venvs/1/s/a448a1bf/venv/lib/python3.10/site-packages/requests/models.py:822: ChunkedEncodingError

We use a container with a Docker registry, and there were no logs when we tried pushing using the Docker SDK — but everything worked with the Docker CLI.

When I checked the Docker daemon logs, I found error messages http: panic serving @: runtime error: invalid memory address or nil pointer dereference (attached below).

docker_log.log

.

At the same time it has been working od the dev containers with the same Ubuntu and docker engine versions that was running on my computer.

Distributor ID:	Ubuntu
Description:	Ubuntu 24.04.3 LTS
Release:	24.04
Codename:	noble

dpkg-query -W libsystemd0
dpkg-query -W libc6
dpkg-query -W libcap2
libsystemd0:amd64	255.4-1ubuntu8.10
libsystemd0:i386	255.4-1ubuntu8.10
libc6:amd64	2.39-0ubuntu8.5
libc6:i386	2.39-0ubuntu8.5
libcap2:amd64	1:2.66-5ubuntu2.2
libcap2:i386	1:2.66-5ubuntu2.2

dpkg-query -W libssl3
libssl3:amd64	3.0.10-1ubuntu2.3
libssl3:i386	3.0.10-1ubuntu2.3

Libraries that uses docker are the same on my system and in my dev container, but as I sad it had been worked correctly only in dev. And the same situation we had with Fedora and dev container that was ranning (container with ubuntu).

[heitorPB]

I'm also hit by this. The issue of adding auth_config={} to client.images.push() is that it overwrites previous authentication with a registry, so it will only work in the cases that we don't need auth for registries.

Is there a reliable way to check if the client is already authenticated to a registry?

[bbezak]

I'm also hit by this. The issue of adding auth_config={} to client.images.push() is that it overwrites previous authentication with a registry, so it will only work in the cases that we don't need auth for registries.

Is there a reliable way to check if the client is already authenticated to a registry?

In openstack kolla we did like so: openstack/kolla@ddac7ca

[thaJeztah]

This was a regression in v28.3.3; a fix was merged and backported moby/moby#50738, which will be included in the v28.4 release of docker (likely to be released this week); release candidates are available in the test channel on download.docker.com; https://github.com/moby/moby/releases/tag/v28.4.0-rc.2