TensorRT-LLM released

[Dampfinchen]

https://www.tomshardware.com/news/nvidia-boosts-ai-performance-with-tensorrt

Could TensorRT-LLM be useful for CUDA acceleration? @slaren @JohannesGaessler

[slaren]

There are a few areas that I think could still improve the performance of the CUDA backend significantly, especially in prompt or batch processing:

• Matrix multiplication kernels for quantized formats using tensor cores, which requires custom kernels to support our quantization formats
• Fused attention kernels similar to flash attention or paged attention, which again will require writing custom kernels to support the way we handle attention and multiple sequences
• Storing activations as F16 may also help somewhat

I don't think that TensorRT is likely to help with these issues. Additionally, in general we try to avoid adding large dependencies to llama.cpp.

[Dampfinchen]

I see, thank you for the heads up. Do you think its possible to reduce VRAM usage more, to make it compete with MMQ in that regard? Perhaps its also possible to quantize the KV-Cache to 8 bit like Johannes has demonstrated with MMQ.