support FP8 or FP4 in UAI_NV_TRT

[animeojisan]

Thank you for the great app!
Are there any plans to support FP8 or FP4 in UAI_NV_TRT?

[hooke007]

It's just the wrapper of vs-mlrt.
If vs-mlrt would support fp4/int8, it could follow up easily.

[animeojisan]

Thanks for the reply!

[animeojisan]

May I ask a question here since it is similar? Why only FP32 in UAI_DML, is there any possibility of FP16 support? Is this also a problem on the vs-mlrt side?

[animeojisan]

Am I correct in understanding that UAI_DML is already operating at a processing speed equivalent to fp16? Is there a possibility that DirectML processing speed will improve in the future?

[hooke007]

I believe fp16 I/O would accelerate a lot.

[animeojisan]

I am very thankful!

[hooke007]

fp16 model can be used now in mlrt.(I have to update here later).

hooke007/MPV_lazy@8773309 FP16 model's support for DML was added.

[animeojisan]

Thank you very much for the update!