Getting GGML_ASSERT false && not implemented (LLAMA_MPI enabled build)

[NoelVictor89]

Hello All, I followed the instructions in README to enable MPI and it fails with this message. I looked at the code and found that there is an assertion.

5834 #ifdef GGML_USE_MPI
5835 // TODO: needs fix after #3228
5836 GGML_ASSERT(false && "not implemented");
5837 //ggml_mpi_eval_init(lctx.ctx_mpi, &n_tokens, &n_past, &n_threads);
5838 #endif

Let me know if I am doing anything wrong. The command line i am using is,

mpirun --hostfile machinefile -np 2 ./main -m -p -n 1024 -c 512

I would appreciate all the help. Thanks in Advance

[AutonomicPerfectionist]

MPI is disabled at the moment due to being broken awhile ago. I've been working on fixing it but had to pause development for semester finals, you can use my current implementation from one of my forks but be warned, there's a bug with synchronizing the KV cache operations

[amgowda-oci]

I also tried this and still blocked to run. @AutonomicPerfectionist any updates here how to get passed this? What is your forks branch name I can try with?

llama_new_context_with_model: KV self size = 400.00 MiB, K (f16): 200.00 MiB, V (f16): 200.00 MiB
llama_new_context_with_model: KV self size = 400.00 MiB, K (f16): 200.00 MiB, V (f16): 200.00 MiB
llama_build_graph: non-view tensors processed: 844/844
llama_new_context_with_model: compute buffer total size = 78.19 MiB
GGML_ASSERT: llama.cpp:9900: false && "not implemented"
llama_build_graph: non-view tensors processed: 844/844
llama_new_context_with_model: compute buffer total size = 78.19 MiB
GGML_ASSERT: llama.cpp:6649: false && "not implemented"
A process has executed an operation involving a call
to the fork() system call to create a child process.

[AutonomicPerfectionist]

The branch is named mpi-speculative, but don't expect everything to work perfectly. It's also not entirely up to date, Mixtral doesn't work on it for example

[amgowda-oci]

thanks for your quick reply, will try once and see how it goes. Please let us know if you plan to pick this back up full implementation again.

[stevef1uk]

Hello. Firstly, thanks for producing an amazing project. Being able to run an LLM on a RPi is awesome! I have had no issues with running it on a Rpi 5B with 8GB. using your phi-2 branch following this blog: https://github.com/ggerganov/llama.cpp/tree/gg/phi-2

Since I have a few 4Gb RPi 4Bs lying around I was trying to use the mpirun approach to run a model across a few RPIs. I was following this interesting thread: #2164

However, in the end I had to restrict myself to just one remote Pi as I had issues with dynamically linked executables not working other than on the machine they were built on. I ensured that the OS as the same on them all though. I managed to get the phi-2.Q2_K.gguf model to work on the RPi4, but only when llama.cp was compiled normally. The moment I tried the 'make CC=mpicc CXX=mpicxx LLAMA_MPI=1' approach I got the error:

_llama_new_context_with_model: KV self size = 160.00 MiB, K (f16): 80.00 MiB, V (f16): 80.00 MiB
llama_build_graph: non-view tensors processed: 774/774
llama_new_context_with_model: compute buffer total size = 113.19 MiB
GGML_ASSERT: llama.cpp:6649: false && "not implemented"
[New LWP 3282]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0x0000007fa1888724 in __GI___wait4 (pid=3288, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30 ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory.
#0 0x0000007fa1888724 in __GI___wait4 (pid=3288, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30 in ../sysdeps/unix/sysv/linux/wait4.c
#1 0x00000055570a7a8c in ggml_print_backtrace ()
#2 0x00000055570ee0e4 in llama_decode ()
#3 0x0000005557136078 in llama_init_from_gpt_params(gpt_params&) ()
#4 0x000000555709ac00 in main ()
[Inferior 1 (process 3281) detached]
[rpi400:03281] *** Process received signal ***
[rpi400:03281] Signal: Aborted (6)
[rpi400:03281] Signal code: (-6)
[rpi400:03281] [ 0] linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0x7fa20147bc]
[rpi400:03281] [ 1] /lib/aarch64-linux-gnu/libc.so.6(+0x809d0)[0x7fa18509d0]
[rpi400:03281] [ 2] /lib/aarch64-linux-gnu/libc.so.6(gsignal+0x1c)[0x7fa180a76c]
[rpi400:03281] [ 3] /lib/aarch64-linux-gnu/libc.so.6(abort+0xf0)[0x7fa17f74bc]
[rpi400:03281] [ 4] /mnt//main(+0x5e0e8)[0x55570ee0e8]
[rpi400:03281] [ 5] /mnt//main(+0xa6078)[0x5557136078]
[rpi400:03281] [ 6] /mnt//main(+0xac00)[0x555709ac00]
[rpi400:03281] [ 7] /lib/aarch64-linux-gnu/libc.so.6(+0x27780)[0x7fa17f7780]
[rpi400:03281] [ 8] /lib/aarch64-linux-gnu/libc.so.6(_libc_start_main+0x98)[0x7fa17f7858]
[rpi400:03281] [ 9] /mnt//main(+0x13e30)[0x55570a3e30]
[rpi400:03281] *** End of error message ***

I can't find the branch you suggested NoelVictor89 should try. Is there a more recent one you suggest I try?

Many thanks.

[AutonomicPerfectionist]

As for the compilation errors, those appear to be caused by examples not being updated for my changes, I've only been working with speculative so many of the others may not work correctly

[stevef1uk]

Thanks.

I am trying to run using mpirun (openmpi) across several machines and the llama_cpp main works on both machines now, b ut I am getting this error:

lama_new_context_with_model: KV self size = 256.00 MiB, K (f16): 128.00 MiB, V (f16): 128.00 MiB
llama_build_graph: non-view tensors processed: 676/676
llama_new_context_with_model: compute buffer total size = 73.82 MiB
GGML_ASSERT: llama.cpp:9449: false && "Must have same number of split percentages as devices"
llama_new_context_with_model: KV self size = 256.00 MiB, K (f16): 128.00 MiB, V (f16): 128.00 MiB
llama_build_graph: non-view tensors processed: 676/676
llama_new_context_with_model: compute buffer total size = 73.82 MiB
[New LWP 105324]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0x0000007f8e2c8724 in __GI___wait4 (pid=105326, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30 ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory.
#0 0x0000007f8e2c8724 in __GI___wait4 (pid=105326, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30 in ../sysdeps/unix/sysv/linux/wait4.c
#1 0x0000005583f1754c in ggml_print_backtrace ()
#2 0x0000005583f5b418 in llama_split_layers_weighted ()
#3 0x0000005583f0af7c in main ()
[Inferior 1 (process 105323) detached]
[rpi4-4:105323] *** Process received signal ***
[rpi4-4:105323] Signal: Aborted (6)
[rpi4-4:105323] Signal code: (-6)
[rpi4-4:105323] [ 0] linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0x7f8ea157bc]
[rpi4-4:105323] [ 1] /lib/aarch64-linux-gnu/libc.so.6(+0x809d0)[0x7f8e2909d0]
[rpi4-4:105323] [ 2] /lib/aarch64-linux-gnu/libc.so.6(gsignal+0x1c)[0x7f8e24a76c]
[rpi4-4:105323] [ 3] /lib/aarch64-linux-gnu/libc.so.6(abort+0xf0)[0x7f8e2374bc]
[rpi4-4:105323] [ 4] /var/bin/main(+0x5b41c)[0x5583f5b41c]
[rpi4-4:105323] [ 5] /var/bin/main(+0xaf7c)[0x5583f0af7c]
[rpi4-4:105323] [ 6] /lib/aarch64-linux-gnu/libc.so.6(+0x27780)[0x7f8e237780]
[rpi4-4:105323] [ 7] /lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98)[0x7f8e237858]
[rpi4-4:105323] [ 8] /var/bin/main(+0x135f0)[0x5583f135f0]
[rpi4-4:105323] *** End of error message ***

I have seen this flag:
--mpi-layer-split N percentiles to split the layers by across nodes

I have tried 50 for 2 nodes but no change.

Any thoughts?

[AutonomicPerfectionist]

Try --mpi-layer-split 0.5,0.5

I intend to re-enable the logic to auto-split when that parameter isn't given but haven't had time yet

[stevef1uk]

Yes, that moved things on. However, after these messages:

1 more process has sent help message help-pcompress.txt / unavailable
1 more process has sent help message help-pcompress.txt / unavailable

I can see main running on both machines for ages with no output c. 3mins!

When I build the original branch of llama_cpp without the MPI flag the RPI5 responds in 52s and the RPI4Bs in quite a few minutes so it could be they are just too slow for this. I was hoping adding nodes would not only make it possible to run larger models on the RPIs but also speed-up response times.

[AutonomicPerfectionist]

main doesn't do any optimizations for MPI, no. If you want speed improvements I'm researching that, but presently I can't share too much, and we believe the minimum number of nodes for our method is 3 anyway. For now, if you can split the layers such that as many as possible are on the faster nodes (change the 0.5,0.5 to 0.7,0.3 for example), that is about as good as you'll get.