How to keep some layers in fp32? #2368
Guitaricet
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I want to use DeepSpeed fp16 support but want to keep some of the layers (embeddings, normalizations) in full precision (fp32). Is there a recipe for that?
Beta Was this translation helpful? Give feedback.
All reactions