Best way to host a LLM in a Java app?

[mahdix]

Hi,

I can run a LLM model (like ggml-model-f16.bin) locally from CLI via Llama.cpp: ./main -m ./models/7B/ggml-model-q4_0.bin -n 128...

How can I embed this within my Java app? For example to send multiple texts and ask questions about each one?

[bullno1]

You should use JNI to create a wrapper over the library.
The C interface doesn't help much here since Java has its own way of doing FFI.

Then re-implement main example in Java to have an idea of how the API works.

[mahdix]

Which library? I have llama.cpp locally but there is no ".so" file that I can wrap over.

[bullno1]

You can use CMake to build a dynamic library.

But for Java, you would build your own library to be loaded with Runtime.loadLibrary instead.
This would link with llama.cpp statically just like the main example.
It would have to use JNI: https://www.baeldung.com/jni

[dspasyuk]

The easiest is to use it through child process, subprocess or whatever Java is using to execute external apps, see example here, you need server.js file https://github.com/deonis1/llcui