llama-parallel is processing only one request, when run with various parameters

[skprasadu]

I cloned and built llama.cpp in my Ubuntu GPU A100 box. And why I am running this example as below

./llama-parallel --prompt "where is bangalore\nwho is lord krishna" --parallel 2 --cont-batching --sequences 1 -ntg 1024 -npp 1024,1024 -ngl 35 I am only getting response for 1st request. The second request's response is not printing. Am I missing something. Can someone, put a readme of how we do parallel processing of prompts.

[skprasadu]

I think I figured out, this is the right command,
./llama-parallel --prompt "where is bangalore\nwho is lord krishna" --parallel 2 --cont-batching --sequences 2 -ntg 1024 -npp 1024,1024 -ngl 35 actually we need to give the sequence size equal to the number of prompts that need to be processed.

Hope it will be useful to others.

[ggerganov]

The -ntg and -npp args are ignored by the llama-parallel so you can remove those