Does imatrix work independent of the dtype used for calibration? #7538
Unanswered
bartowski1182
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
IE, if I convert to F32 and BF16, calculate the imatrix from F32 (since it supports GPU offloading), but then apply it to the BF16 weights during quantization, will that work as expected or will the change in dtype affect it?
@ikawrakow sorry for the ping but after searching couldn't find any way to reach out to you directly and I imagine you're the most well versed on this subject..
Similarly, if F16 for imatrix and BF16 for quantization, curious how this would work
Beta Was this translation helpful? Give feedback.
All reactions