CPU/XPU tests align

[jiqing-feng]

For CUDA: pytest

CUDA: 3711 passed, 0 failed, 15 skipped, 24 xfailed, 781 warnings in 498.21s

For CPU: pytest --ignore test_optim.py --ignore test_triton.py --ignore test_cuda_setup_evaluator.py
For XPU: BNB_TEST_DEVICE="xpu" pytest --ignore test_optim.py --ignore test_triton.py --ignore test_cuda_setup_evaluator.py

CPU previous: 378 passed, 1537 failed, 1638 skipped, 197 xfailed, 153 warnings in 613.27s

CPU current: 2079 passed, 1498 skipped, 153 deselected, 9 xfailed, 59 warnings in 1192.94s

XPU previous: not enabled

XPU current: 2093 passed, 1493 skipped, 153 deselected, 63 warnings in 562.25s

Hi @matthewdouglas . The CPU and XPU can pass most cases in this PR: #1628 , but I still have some concerns about the skipped tests. Please check the following issues:

1. These tests are tagged to be deprecated in the future. Can we deprecate the tests now? (Including matmullt, has_fp16_weights, linear8bitlt, )
2. These tests are test functions where I cannot see anywhere used. Can we remove the tests now? Including quantile, spmm, coo2csc,
3. This test didn't actually test anything: bench 4bit dequant. Could we remove it now?
4. I see we skip blockwise quantization when block size != 256 in CPU/XPU, but we support any size blockwise quantization, can we also enable other sizes check?
5. We skip blockwise quantization when dtype != float32, but I already support fp16/bf16 dtype, should we add these dtypes to tests?
6. Why we run CPU ops even the device is XPU or CUDA? cpu tests in xpu
7. Why we check torch op?

[matthewdouglas]

1. These tests are tagged to be deprecated in the future. Can we deprecate the tests now? (Including matmullt, has_fp16_weights, linear8bitlt, )

I'd like to get at least one more release out (v0.46.0) with deprecation warnings for these features before we go ahead and remove them completely. This way anyone using them will have time between deprecation notice and removal. For the purposes of non-CUDA implementations, we can simply ignore. Until we do that release, we still don't want to inadvertently break one of the deprecated features.

2. These tests are test functions where I cannot see anywhere used. Can we remove the tests now? Including quantile, spmm, coo2csc,

Similar answer as #1. These features may be used by consumers of the library, and we'd want at least some time between deprecation and removal. With that said I think we could still move some of these tests over to test_deprecated.py to clean things up a bit.

3. This test didn't actually test anything: bench 4bit dequant. Could we remove it now?

Since it's marked with @pytest.mark.benchmark it shouldn't run by default. Even so, I'm good with removing this.

4. I see we skip blockwise quantization when block size != 256 in CPU/XPU, but we support any size blockwise quantization, can we also enable other sizes check?

I disabled it for CPU simply because it's quite slow. Blocksize 256 is used for the nested 4bit quantization as well as the optimizers, so the thought is that this is the most important to test. I think we could add back 4096. I just didn't want the test suite to spend too much time on it.

6. We skip blockwise quantization when dtype != float32, but I already support fp16/bf16 dtype, should we add these dtypes to tests?>

Sure, let's make sure to cover those!

6. Why we run CPU ops even the device is XPU or CUDA? cpu tests in xpu

This is a good point, I agree! I started to make a change like that yesterday in a PR here: https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1636/files#diff-659bad232c71219167252c1a5ccbc427b6f54925b78741df18613c3c49aaa4c1R148

7. Why we check torch op?

We're following advice from the PyTorch custom operators manualand have had some discussion with the PyTorch team during our refactor to use the custom ops. This is a quick smoke test to make sure we've registered our operators correctly, i.e. that they have well-defined schemas and FakeTensor implementations so that they can be traced with the torch compilation APIs.

More about that here: https://docs.pytorch.org/docs/stable/library.html#testing-custom-ops

[jiqing-feng]

Thanks for your detailed explanation! Let's continue to review this PR: #1628