Models for testing inference / sampling

[danemadsen]

I see there's a number of models for testing tokenization here However i would like a model for testing out my implementation of llama.cpp into my front end. Is anyone aware of such a model?

basically any model that would work with simple.cpp and is relatively small (less than 100mb so GIT LFS isn't needed)

[ggerganov]

You can use tinyllamas: https://github.com/ggerganov/llama.cpp/tree/master/examples/convert-llama2c-to-ggml

[danemadsen]

Ahh ok. They are indeed small but still too large for a test model unfortunately. I don't really care about the output, it could only output EOS tokens for example. Just that it would be good to have a model I can use in my CI to test things are working.

[ggerganov]

The stories260k is 1MB - you can hardly get smaller than that. We use it in the CI as well:

https://github.com/ggerganov/llama.cpp/blob/24affa7db3c9db148854b0ab4fd63de8bca7d898/.github/workflows/build.yml#L236-L246

[danemadsen]

Ahh OK that might be more what I'm looking for. Thanks for the advice