MoE crash is fixed upstream in #13439

[majick]

Describe the Issue
The current concedo branch is affected by the MoE crash in upstream's ggml-org#13437. It's fixed by upstream ggml-org#13435 in 4bc8f75

Additional Information:
I've manually cherry-picked the upstream commit into my local branch of 1.91 and it now allows my MoE models to be loaded. Given that it's a crash for common-ish model types it might be a good idea to hotfix.

[LostRuins]

Good point, I might have to do a patch release then.

[LostRuins]

Seems like there are too many changes for this to be a patch release. Probably better to accumulate it into v1.92 - but people who need the fix immediately can use the CI builds for now at https://github.com/LostRuins/koboldcpp/actions

[pro7357]

FWIW, I got the same error when using "--usecublas" and partial offloading MoE model. Removing "--usecublas" fix it for me.

For record, error when MoE crash: "ggml/src/ggml-cuda/mmvq.cu:519: GGML_ASSERT(!src0->view_src) failed"

[majick]

Yep, that description is consistent with what was discussed upstream and what I ran into. It was only in the CUDA path (the .cu code). Turning off hardware acceleration avoided the bad code path.

Thanks for putting the actual failure error on record. I neglected to do it when I filed the issue!

[pro7357]

It seem that I can avoid the crash if using "--usecublas all" or specify which gpu to use "--usecublas 0".

[majick]

Cool workaround! I've already cp'ed the commit from upstream so I can't confirm it, and there's already something similar in _experimental. I can't use _experimental right now though because it randomly hangs and goes like 0.5t/s for no reason.

It's good to have it written down, though, in case someone else is running into this crash.

[LostRuins]

Hi, please try the latest version v1.92, this should be resolved.

[pro7357]

Yes, I tried the latest version. No longer crash when loading MoE model.

[majick]

Confirmed on my end as well. Thank you!