Keep running llamacpp in background

[Fulgurance]

Hi everyone, I did yet few tests now with llama-cli. I would like to know if there is a possibility to keep the chat mode running in background of the system, and send when necessary a request when it's needed to the running modele instead of restart the modele everytime.

Because starting the modele consume a bit of time.

[dspasyuk]

you can run in a conversation mode for example: ../llama.cpp/llama-cli --model /home/denis/CODE/lama/llamaESM/models/Qwen2-7B-Instruct.Q4_0.gguf --n-gpu-layers 33 -cnv --simple-io -b 2048 --ctx_size 8192 --temp 0.3 -fa -t 6 --top_k 10 --multiline-input --chat-template chatml --log-disable
or use https://github.com/dspasyuk/llama.cui

[Fulgurance]

Oh okay !

And then from another terminal, I just need to run llama-cli again I guess ? Do I need special argument then ?

[marcingomulkiewicz]

Unsure if it's relevant, but perhaps creating FIFOs and redirecting stdout and stdin would do the trick you need?