seems not faster?

[ichejun]

Hi there~
I tried this library in the environment (python 3.12.8 and torch 2.6.0+cu124), but it didn't seem to speed up, but became slower?
Is it an environment problem? Or maybe the compile of the new version of torch more powerful? thks~

$python3 test.py 
Output From nn.Linear (compiled):
 tensor([[-0.2332,  0.0908, -0.0438,  ...,  0.1122,  0.0160,  0.1544],
        [ 0.0186,  0.1241, -0.0663,  ..., -0.0545, -0.0704, -0.1599],
        [-0.1022,  0.0374,  0.1137,  ..., -0.0730,  0.0739, -0.2000],
        ...,
        [ 0.1812, -0.1332, -0.2463,  ..., -0.1785,  0.0004,  0.2174],
        [ 0.0446, -0.2808, -0.0799,  ...,  0.0555, -0.1138, -0.0076],
        [ 0.0435, -0.0329, -0.0820,  ..., -0.0171, -0.0221,  0.1840]],
       device='cuda:0', dtype=torch.float16)
Output From CublasLinear:
 tensor([[-2.3328e-01,  9.1309e-02, -4.3945e-02,  ...,  1.1200e-01,
          1.6022e-02,  1.5430e-01],
        [ 1.8524e-02,  1.2421e-01, -6.6589e-02,  ..., -5.4535e-02,
         -7.0435e-02, -1.6052e-01],
        [-1.0205e-01,  3.7170e-02,  1.1316e-01,  ..., -7.3181e-02,
          7.3730e-02, -1.9958e-01],
        ...,
        [ 1.8091e-01, -1.3342e-01, -2.4609e-01,  ..., -1.7834e-01,
         -2.2888e-04,  2.1716e-01],
        [ 4.4464e-02, -2.8101e-01, -7.9956e-02,  ...,  5.5481e-02,
         -1.1414e-01, -7.6065e-03],
        [ 4.3610e-02, -3.2715e-02, -8.2092e-02,  ..., -1.6968e-02,
         -2.2156e-02,  1.8457e-01]], device='cuda:0', dtype=torch.float16)
TORCH_F16_W/_F32_ACC_(COMPILED):_ 1038.24 us 132.38 TFLOPS
CUBLAS_F16_W/_F16_ACC:_ 1328.69 us 103.44 TFLOPS

[jovan2009]

I have a GTX 1050 Ti, cublas_ops compiled on windows 11, python 3.13.3, CUDA Toolkit 12.8.1, torch 2.8.0.dev20250429+cu128
This is what I get on test.py ( I took the liberty to reduce the number of warmups and iterations because otherwise it would take forever to complete:

C:\Python313\torch-cublas-hgemm>python test.py
Output From nn.Linear (compiled):
 tensor([[ 0.0966,  0.1613,  0.0836,  ..., -0.0593, -0.0833,  0.0682],
        [ 0.1349,  0.0446, -0.1699,  ..., -0.2362,  0.1400,  0.1472],
        [-0.1113, -0.0252,  0.0383,  ..., -0.0341,  0.0079, -0.0971],
        ...,
        [-0.0740,  0.0113, -0.1131,  ...,  0.0865, -0.2639, -0.0134],
        [-0.1346,  0.1519,  0.0850,  ..., -0.0484, -0.1863, -0.0705],
        [ 0.1907,  0.1161,  0.0336,  ...,  0.1377, -0.0214,  0.1077]],
       device='cuda:0', dtype=torch.float16)
Output From CublasLinear:
 tensor([[ 0.0966,  0.1604,  0.0841,  ..., -0.0600, -0.0836,  0.0679],
        [ 0.1348,  0.0439, -0.1720,  ..., -0.2336,  0.1411,  0.1461],
        [-0.1102, -0.0253,  0.0377,  ..., -0.0344,  0.0080, -0.0961],
        ...,
        [-0.0745,  0.0100, -0.1107,  ...,  0.0874, -0.2659, -0.0139],
        [-0.1348,  0.1499,  0.0872,  ..., -0.0487, -0.1858, -0.0703],
        [ 0.1903,  0.1163,  0.0332,  ...,  0.1390, -0.0207,  0.1080]],
       device='cuda:0', dtype=torch.float16)
TORCH_F16_W/_F32_ACC_(COMPILED):_ 89945.61 us 1.53 TFLOPS
CUBLAS_F16_W/_F16_ACC:_ 3403603.01 us 0.04 TFLOPS

I'm not sure what is the problem, is the 1050 Ti sm_61 too old and not supported? 38x slower with cublas_ops, clearly not a speed up :D

[aredden]

Ah yeah @jovan2009 I guess the 10x0 gpus don't have tensor cores, so you wouldn't see any benefit.

[aredden]

@ichejun What gpu are you using?

[ichejun]

@ichejun What gpu are you using?

Hi, I am using NVIDIA H20 NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4