Any plans to implement mxfp4 and/or nvfp4?

[davidmezzetti]

Hello. I'm a big fan of bitsandbytes, torch and the general HF ecosystem!

I'm aware of torchao but I'm wondering if there are any plans to support mxfp4 and/or nvfp4? Obviously the native support for GPUs is great but a really nice thing about bitsandbytes is it has a basic fallback to cpu. That makes it much easier to implement the functionality and ensure it works, even if it's slow.

Thank you!

[matthewdouglas]

Hi @davidmezzetti

We're currently considering our options for additional quantization schemes. MXFP4 and NVFP4 are definitely options we're considering to add. I've mostly been thinking about this in terms of LLMs to be honest, but I think your use case is quite interesting too. Do you have any background you can share on how well either of those formats would potentially perform for ANN?

Also as an FYI regarding hardware support:

• bitsandbytes officially supports Intel XPU and Gaudi2/Gaudi3.
• We have AMD GPU support in an experimental phase as well - the main limitation with that will be that on the datacenter GPUs the minimum blocksize for 4bit is 128 instead of 64. It's not included in release wheels yet, but likely will be soon.
• We can also build for macOS on CPU and MPS devices, and while not optimized for performance it's also a good fallback option for dev. As with AMD, we don't ship the wheels yet, but very likely will do so in the near future.

[davidmezzetti]

I appreciate the response and the additional information on the current runtime environments.

In terms of MXFP4 / NVFP4, the main upside I'd see is support for native 4 bit matrix multiplications for vector similarity (i.e. NV Blackwell GPUs).

Storing vectors as 4-bit embeddings and a Safetensors file is probably a bit different. But long term, I'm wondering if there is some way to merge those into LLMs in some way to have a more efficient RAG (inspired by the ideas in REFRAG).