Question on recommended resources

[kaieberl]

Love this project!
I am currently only writing Python and don't really understand what you did here, can you recommend some resources for converting models to C++ and to use hardware acceleration on Mac?
Thank you in advance!

[KerfuffleV2]

Models don't get converted to C++. They just get converted to the format the project can use (GGUF). The easiest way to get started is to just download a model that's already been converted to GGUF format: https://huggingface.co/TheBloke

7B models are pretty small and should run on low memory systems. 33+B models you probably have to have at least 32GB RAM. For 65B and 70B you need 64GB RAM. You can get an idea of how much memory will be required from the file sizes: generally you'll need memory available at least equal to the size of the model file plus a few gigs extra for stuff like context.

[kaieberl]

Thanks for the clarification! I was actually more interested in what it takes to develop such a project from scratch, because I would like to make Meta Nougat run faster on my Mac. My first try was to convert the PyTorch model to a CoreML model and load this via Python, but this is still extremely slow.
For people interested in macOS development, I can recommend this blog article by Apple: https://machinelearning.apple.com/research/neural-engine-transformers

[KerfuffleV2]

I was actually more interested in what it takes to develop such a project from scratch

I'm not sure how someone could answer that, since you're talking about a project that's been developed for what, around 8 months now, has hundreds of contributors, was made by someone who had also been working on machine learning stuff for a while, etc. llama.cpp also was able to take advantage of the community interest in LLaMA having recently been released and the hype of being able run models locally.