Does cuSPARSELt not support SpMV operations?

[wang-qitong]

Hi

As shown in the figure, during the decoding phase, the 2:4 sparsity model is about 12% slower than the dense model, the questions are as follows:

1. Is the decode phase dominated by GEMV / small‑N GEMM operations, and does cuSPARSELt not support SpMV operations?
2. Are there any sparse kernels or recommended practices for GEMV (matrix‑vector) that can take advantage of 2:4 sparsity?

Thanks!

[j4yan]

Hi @wang-qitong

1. I'm not an expert on transformers, so I don't know what dominates the decoding. But cuSPARESLt doesn't support SpMV.
2. Not that I know of.

Could you share the problem sizes (m, n, k) and data types you are using?