Why are the docker images so huge?

[thfrei]

Dear community

First off: Thank you for your effort. I really like this software!

When trying to run different versions of LocalAI on my computer, i always run into the problem, that image pull takes very long. The images are huge! Deploying it with portainer will even fail, because a 70GB image is not pulled in a minute or so. Using some tag like :latestwill often pull a new image. It is at a point, where testing is tedious, because oftentimes I cannot pull the image.

So I was wondering, if someone can explain: Why is it so huge? Are all these layers required?

Thank you and best regards
Thomas

Below you see a list of some of my images. I also added some other images for reference: 68GB, 70GB?

[thfrei]

One addition. I just checked the layers of the docker image:

tom@MACHINE:/mnt/c/Users/windo$ docker history 54f2ed26fbc6   <---- #  v2.0.0
IMAGE          CREATED       CREATED BY                                      SIZE      COMMENT
54f2ed26fbc6   5 days ago    ENTRYPOINT ["/build/entrypoint.sh"]             0B        buildkit.dockerfile.v0
<missing>      5 days ago    EXPOSE map[8080/tcp:{}]                         0B        buildkit.dockerfile.v0
<missing>      5 days ago    HEALTHCHECK &{["CMD-SHELL" "curl -f $HEALTHC…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG= BUILD_TYPE= TARGETARCH=amd64 …   0B        buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG= BUILD_TYPE= TARGETARCH=amd64 …   8.66GB    buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG= BUILD_TYPE= TARGETARCH=amd64 …   5.07GB    buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG= BUILD_TYPE= TARGETARCH=amd64 …   7.5GB     buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG= BUILD_TYPE= TARGETARCH=amd64 …   5.48GB    buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG= BUILD_TYPE= TARGETARCH=amd64 …   5.5GB     buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG= BUILD_TYPE= TARGETARCH=amd64 …   8.49GB    buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG= BUILD_TYPE= TARGETARCH=amd64 …   5.29GB    buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG= BUILD_TYPE= TARGETARCH=amd64 …   5.77GB    buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG= BUILD_TYPE= TARGETARCH=amd64 …   8.48GB    buildkit.dockerfile.v0
<missing>      5 days ago    COPY /build/backend-assets/grpc/stablediffus…   24.4MB    buildkit.dockerfile.v0
<missing>      5 days ago    COPY /build/sources/go-piper/piper/build/pi/…   37MB      buildkit.dockerfile.v0
<missing>      5 days ago    COPY /build/local-ai ./ # buildkit              379MB     buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG= BUILD_TYPE= TARGETARCH=amd64 …   1.41GB    buildkit.dockerfile.v0
<missing>      5 days ago    COPY /build/grpc ./grpc/ # buildkit             2.35GB    buildkit.dockerfile.v0
<missing>      5 days ago    COPY /build/sources ./sources/ # buildkit       1.88GB    buildkit.dockerfile.v0
<missing>      5 days ago    COPY . . # buildkit                             3.51MB    buildkit.dockerfile.v0
<missing>      5 days ago    WORKDIR /build                                  0B        buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG= BUILD_TYPE= TARGETARCH=amd64 …   0B        buildkit.dockerfile.v0
<missing>      5 days ago    ENV NVIDIA_VISIBLE_DEVICES=all                  0B        buildkit.dockerfile.v0
<missing>      5 days ago    ENV NVIDIA_REQUIRE_CUDA=cuda>=.0                0B        buildkit.dockerfile.v0
<missing>      5 days ago    ENV NVIDIA_DRIVER_CAPABILITIES=compute,utili…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    ARG CUDA_MAJOR_VERSION=11                       0B        buildkit.dockerfile.v0
<missing>      5 days ago    ENV HEALTHCHECK_ENDPOINT=http://localhost:80…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    ENV REBUILD=false                               0B        buildkit.dockerfile.v0
<missing>      5 days ago    ENV BUILD_TYPE=                                 0B        buildkit.dockerfile.v0
<missing>      5 days ago    ARG IMAGE_TYPE=extras                           0B        buildkit.dockerfile.v0
<missing>      5 days ago    ARG TARGETARCH                                  0B        buildkit.dockerfile.v0
<missing>      5 days ago    ARG BUILD_TYPE                                  0B        buildkit.dockerfile.v0
<missing>      5 days ago    ARG FFMPEG                                      0B        buildkit.dockerfile.v0
<missing>      5 days ago    RUN /bin/sh -c curl --proto '=https' --tlsv1…   1.26GB    buildkit.dockerfile.v0
......

So I guess, the lines starting with RUN |5 FFMPEG= BUILD_TYPE= TARGETARCH=amd64 are the different builds for the different versions? Maybe one can do the docker image build fresh or something? Without layering it? Core image, with size of 10GB contains fewer of these lines and they are also smaller:

tom@MACHINE:/mnt/c/Users/windo$ docker history 9cd093fd258f    # <---- v2.0.0-cublas-cuda11-ffmpeg-core
IMAGE          CREATED       CREATED BY                                      SIZE      COMMENT
9cd093fd258f   5 days ago    ENTRYPOINT ["/build/entrypoint.sh"]             0B        buildkit.dockerfile.v0
<missing>      5 days ago    EXPOSE map[8080/tcp:{}]                         0B        buildkit.dockerfile.v0
<missing>      5 days ago    HEALTHCHECK &{["CMD-SHELL" "curl -f $HEALTHC…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG=true BUILD_TYPE=cublas TARGETA…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG=true BUILD_TYPE=cublas TARGETA…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG=true BUILD_TYPE=cublas TARGETA…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG=true BUILD_TYPE=cublas TARGETA…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG=true BUILD_TYPE=cublas TARGETA…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG=true BUILD_TYPE=cublas TARGETA…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG=true BUILD_TYPE=cublas TARGETA…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG=true BUILD_TYPE=cublas TARGETA…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG=true BUILD_TYPE=cublas TARGETA…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG=true BUILD_TYPE=cublas TARGETA…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    COPY /build/backend-assets/grpc/stablediffus…   24.4MB    buildkit.dockerfile.v0
<missing>      5 days ago    COPY /build/sources/go-piper/piper/build/pi/…   37MB      buildkit.dockerfile.v0
<missing>      5 days ago    COPY /build/local-ai ./ # buildkit              392MB     buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG=true BUILD_TYPE=cublas TARGETA…   1.41GB    buildkit.dockerfile.v0
<missing>      5 days ago    COPY /build/grpc ./grpc/ # buildkit             2.35GB    buildkit.dockerfile.v0
<missing>      5 days ago    COPY /build/sources ./sources/ # buildkit       2.12GB    buildkit.dockerfile.v0
<missing>      5 days ago    COPY . . # buildkit                             3.51MB    buildkit.dockerfile.v0
<missing>      5 days ago    WORKDIR /build                                  0B        buildkit.dockerfile.v0
<missing>      5 days ago    RUN |5 FFMPEG=true BUILD_TYPE=cublas TARGETA…   93.2MB    buildkit.dockerfile.v0
<missing>      5 days ago    ENV NVIDIA_VISIBLE_DEVICES=all                  0B        buildkit.dockerfile.v0
<missing>      5 days ago    ENV NVIDIA_REQUIRE_CUDA=cuda>=11.0              0B        buildkit.dockerfile.v0
<missing>      5 days ago    ENV NVIDIA_DRIVER_CAPABILITIES=compute,utili…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    ARG CUDA_MAJOR_VERSION=11                       0B        buildkit.dockerfile.v0
<missing>      5 days ago    ENV HEALTHCHECK_ENDPOINT=http://localhost:80…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    ENV REBUILD=false                               0B        buildkit.dockerfile.v0
<missing>      5 days ago    ENV BUILD_TYPE=cublas                           0B        buildkit.dockerfile.v0
<missing>      5 days ago    ARG IMAGE_TYPE=extras                           0B        buildkit.dockerfile.v0
<missing>      5 days ago    ARG TARGETARCH                                  0B        buildkit.dockerfile.v0
<missing>      5 days ago    ARG BUILD_TYPE                                  0B        buildkit.dockerfile.v0
<missing>      5 days ago    ARG FFMPEG                                      0B        buildkit.dockerfile.v0
<missing>      5 days ago    RUN |6 BUILD_TYPE=cublas CUDA_MAJOR_VERSION=…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    WORKDIR /build                                  0B        buildkit.dockerfile.v0
<missing>      5 days ago    RUN |6 BUILD_TYPE=cublas CUDA_MAJOR_VERSION=…   28B       buildkit.dockerfile.v0
<missing>      5 days ago    RUN |6 BUILD_TYPE=cublas CUDA_MAJOR_VERSION=…   942MB     buildkit.dockerfile.v0
<missing>      5 days ago    ENV PATH=/usr/local/cuda/bin:/go/bin:/usr/lo…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    RUN |6 BUILD_TYPE=cublas CUDA_MAJOR_VERSION=…   2.55GB    buildkit.dockerfile.v0
<missing>      5 days ago    RUN |6 BUILD_TYPE=cublas CUDA_MAJOR_VERSION=…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    RUN |6 BUILD_TYPE=cublas CUDA_MAJOR_VERSION=…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    RUN |6 BUILD_TYPE=cublas CUDA_MAJOR_VERSION=…   200kB     buildkit.dockerfile.v0
<missing>      5 days ago    COPY custom-ca-certs/* /usr/local/share/ca-c…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    RUN |6 BUILD_TYPE=cublas CUDA_MAJOR_VERSION=…   130MB     buildkit.dockerfile.v0
<missing>      5 days ago    ARG GO_TAGS=stablediffusion tts                 0B        buildkit.dockerfile.v0
<missing>      5 days ago    ENV GALLERIES=[{"name":"model-gallery", "url…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    ENV EXTERNAL_GRPC_BACKENDS=huggingface-embed…   0B        buildkit.dockerfile.v0
<missing>      5 days ago    ENV BUILD_TYPE=cublas                           0B        buildkit.dockerfile.v0
<missing>      5 days ago    ARG TARGETVARIANT                               0B        buildkit.dockerfile.v0
<missing>      5 days ago    ARG TARGETARCH                                  0B        buildkit.dockerfile.v0
<missing>      5 days ago    ARG CUDA_MINOR_VERSION=7                        0B        buildkit.dockerfile.v0
<missing>      5 days ago    ARG CUDA_MAJOR_VERSION=11                       0B        buildkit.dockerfile.v0
<missing>      5 days ago    ARG BUILD_TYPE                                  0B        buildkit.dockerfile.v0
<missing>      2 weeks ago   /bin/sh -c #(nop) WORKDIR /go                   0B
<missing>      2 weeks ago   /bin/sh -c mkdir -p "$GOPATH/src" "$GOPATH/b…   0B
<missing>      2 weeks ago   /bin/sh -c #(nop)  ENV PATH=/go/bin:/usr/loc…   0B
<missing>      2 weeks ago   /bin/sh -c #(nop)  ENV GOPATH=/go               0B
<missing>      2 weeks ago   /bin/sh -c #(nop)  ENV GOTOOLCHAIN=local        0B
<missing>      2 weeks ago   /bin/sh -c set -eux;  arch="$(dpkg --print-a…   214MB
<missing>      2 weeks ago   /bin/sh -c #(nop)  ENV GOLANG_VERSION=1.21.4    0B
<missing>      2 weeks ago   /bin/sh -c #(nop)  ENV PATH=/usr/local/go/bi…   0B
<missing>      2 weeks ago   /bin/sh -c set -eux;  apt-get update;  apt-g…   228MB
<missing>      2 weeks ago   /bin/sh -c apt-get update && apt-get install…   152MB
<missing>      2 weeks ago   /bin/sh -c set -eux;  apt-get update;  apt-g…   28.6MB
<missing>      2 weeks ago   /bin/sh -c #(nop)  CMD ["bash"]                 0B
<missing>      2 weeks ago   /bin/sh -c #(nop) ADD file:71543995e4d314b0c…   124MB
tom@MACHINE:/mnt/c/Users/windo$

[mabuchan]

Thomas - I was running into the same thing. For the maintainers, here is my quick analysis of what may have happened (note, I play a developer on TV, but am doing this quickly and this may be completely inaccurate):

LocalAI:

In the images, which appear from about 1.40.0 forward, it appears that the /opt/conda/envs/*/lib directories are being duplicated across all the different conda install groupings. Each one is about 5GB - which I believe is causing the bloat, although I believe many of the dependancies are duplicated across the different extra build settings, I believe this is somewhat cascading and causing the unnecessary growth of the image:

I think while the concept was good to try to remove having a huge layer, this section from the Dockerfiles (introduced in a couple of commits), is contributing to the build size being huge:

For reference, these appear to be the two commits to Dockerfile that may be causing some of the issues, in an effort to get folks back to the potential issues fastest:

Conda Extras: f347e51
Split Extras: 622aaa9

[thiner]

Exactly the same journey! I was surprized by the huge size of image. I had to modify the Dockerfile, build from scratch. I finally got a smaller image with size around 12 GB. It's still too big, I would expect you guys can create a PR to fix the issue. Thanks in advance.

[mudler]

core images don't contain conda environments. See also https://localai.io/basics/getting_started/#container-images and https://localai.io/basics/news/#04-12-2023-__v200__

I think probably we should make core the defaults, and have images with python dependencies separately - however I think that would just make things just worse, indeed I'm more keen towards #1411 .

[egerlach]

@mudler Bringing up an old thread. Is there an appetite for creating smaller Docker images that contain fewer intermediate results and fewer layers, and are therefore smaller? We could pretty easily keep the ability to rebuild everything as an option, I think.

Even without unifying the backends, there's a lot of work that can be done here, just from a cursory glance at the Dockerfile. This is kinda my thing professionally, so please don't take offense at my taking offense to the state of the Dockerfile 😁

[egerlach]

Well, I'll take my L here.

There's a lot more use of good builder layers than I initially read. There's a few optimizations (building CMake from source can be done cleaner) but it's solid. I was building with some of the extras, so that's why I was seeing it whiz by, I think. It still baffles me why the images are so big though... I'll see if there's anything that can be done.

[mudler]

Well, I'll take my L here.

There's a lot more use of good builder layers than I initially read. There's a few optimizations (building CMake from source can be done cleaner) but it's solid. I was building with some of the extras, so that's why I was seeing it whiz by, I think. It still baffles me why the images are so big though... I'll see if there's anything that can be done.

hey - yes please any optimization is very much welcome, and it's an ongoing effort time to time which I focus too, but not entirely my primary focus.

CPU images are smaller, but extras images take much more space due to the venv that each backend brings. I've been trying lately to merge some of these to minimize space.

Please open up a PR if you have improvements and we'll pick it up from there. Thanks!

[egerlach]

Was digging through the layers, had the idea that a non-rebuildable image might be a lot smaller. Probably not something you can distribute given the current state of how LocalAI works overall, but it could be something that someone (maybe me) could trigger when building to keep their built image small, and maybe eventually it would become more generally useful?

Let me know if I should consider that in a PR as well (probably separate).