Vertical Sharding of Models

[jaxs-ribs]

If we want to distribute shards of a larger model (say llama 70B or larger) across several machines, we can cut off the architecture and weights at the end of a specified transformer block, outputting intermediate activations, which would get fed back into the next shard.

How easy/hard would it be to generate .gguf files that don't lose performance? Is there work being done on that?

If not, I would love to help getting this to work, got it to work on tinygrad already.

[cmp-nct]

I guess gguf will need to support it, given the size limitation of Huggingface uploads and the increasing model sizes.
It's probably not a huge development, just needs to be done with care to not break mmap support which relies on the file data being 1:1 mappable to internal structures.

A temporary solution is to just use a splitting/recombination utility, so you work with one gguf file locally only