Wrong doc and function signature for 8-bit optim

[gau-nernst]

System Info

NA

Reproduction

Argument optim_bits and amsgrad is not used for 8-bit Adam

https://github.com/TimDettmers/bitsandbytes/blob/dada530149212d64d4b69534716202659ef37ec8/bitsandbytes/optim/adam.py#L77-L95

The document for optim_bits is also wrong in this case, because we are using 8 bits here.

https://github.com/TimDettmers/bitsandbytes/blob/dada530149212d64d4b69534716202659ef37ec8/bitsandbytes/optim/adam.py#L107-L110

I'm thinking of 2 ways to solve this:

• Remove optim_bits and amsgrad arguments (actually even 32-bit Adam also doesn't use amsgrad argument)
• If we want to keep the function signature the same, there should be a check that optim_bits == 8 and amsgrad == False

Expected behavior

NA

[nighting0le01]

? so does it always run in 8 bit?? @gau-nernst

[gau-nernst]

Yes, Adam8bit will always run in 8 bit. The optim_bits argument has no effect

[nighting0le01]

@gau-nernst i see, thanks also, did you notice any difference varying min_8bit_size? would it make sense to start with such a high value if your hidden dim ==4096 and only goes above in projection layers?

[gau-nernst]

I didn't play around with that parameter.