DOCS: LightGbm in Linux Container (System.DllNotFoundException: Unable to load shared library 'lib_lightgbm' or one of its dependencies.) #7454
Description
Hey!
There was an issue about this before: #4600
When using ML.NET and wanting to train a model with LightGBM, you get this error when deploying with Docker to a linux container.
I'm posting this here for posterity because I'm a total noob and spent 4-6 hours on this or something...
There are also docs at: https://github.com/dotnet/machinelearning/blob/main/docs/building/unix-instructions.md
But they are somewhat outdated...
The error is:
Unhandled exception. System.DllNotFoundException: Unable to load shared library 'lib_lightgbm' or one of its dependencies.
In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable:
liblib_lightgbm: cannot open shared object file: No such file or directory
at Microsoft.ML.Trainers.LightGbm.WrappedLightGbmInterface.DatasetCreateFromSampledColumn(IntPtr sampleValuePerColumn, IntPtr sampleIndicesPerColumn, Int32 numCol, Int32[] sampleNonZeroCntPerColumn, Int32 numSampleRow, Int32 numTotalRow, String parameters, IntPtr& ret)
The solution is to build LightGBM in the container, which then failed with:
CMake 3.28 or higher is required. You are running version 3.25.1
My final Dockerfile:
FROM mcr.microsoft.com/dotnet/aspnet:8.0 as runtime
RUN apt-get update
RUN apt-get install git clang-3.9 libunwind8 curl libomp-dev -y
RUN apt-get install python3-pip -y
RUN pip install cmake --upgrade --break-system-packages
#RUN cmake --version
RUN git clone --recursive https://github.com/microsoft/LightGBM.git
RUN cd LightGBM && mkdir build && cd build && cmake .. && make -j && make install
#RUN find / -name 'lib_lightgbm.so' 2>/dev/null
This is not great, see for example the --break-system-packages which would've been OK in the build step only. And I don't clean the stuff I no longer need at the end.
The --break-system-packages, for that ChatGPT said to do the following instead -- but I haven't tested it:
RUN apt-get update && apt-get install -y python3-venv
RUN python3 -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
RUN pip install --upgrade pip cmake
I tried to do this from a build step too.
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
# same code as above
COPY --from=build /usr/local/lib/lib_lightgbm.so /app
And maybe that could have worked if I did an extra copy of libgomp.so, (or maybe just an install of libomp-dev) but, as I said, 5 hours wasted.