How to use other activation functions such as ReGLU rather than default SwiGLU in llama.cpp?
#11384
Unanswered
Raincleared-Song
asked this question in
Q&A
How to use other activation functions such as ReGLU rather than default SwiGLU in `llama.cpp`?
#11384
Replies: 1 comment
-
|
It has to be implemented: https://github.com/ggerganov/llama.cpp/blob/c07e87f38bd0c22ec6dbc852ae50aaa1c64632d4/ggml/include/ggml.h#L531-L539. Currently not available. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Is it possible for
llama.cppto support models with other activation functions such as ReGLU?Beta Was this translation helpful? Give feedback.
All reactions