MoE of Multiple LoRa Adapter

[forrackun]

stupid question ahead

i want to do experiment with MoE but using lora adapter kind thing. My understanding is currently lora adapter is applied to layers in llama_apply_lora_from_file_internal but i want to maintain multiple adapters without fusing with base model (not apply on top of another). maybe route to best adapter + base combo bashed on prompt using some another router adapter or could be small bert that tags type of field prompt is related etc.

Might sound stupid but i want to play around these ideas any guidance on how to achieve something like that would very helpful thank you

[ianscrivener]

MoE - Mixture of experts (MoE) - https://en.wikipedia.org/wiki/Mixture_of_experts

For those of us who are not just C/C++ or ML experts...

[mikecovlee]

We have implemented a similar approach called MixLoRA: TUDB-Labs/mLoRA#136