Are there other quantized versions of the model

[frankinstien111]

I'm assuming the inference is using FP32 or BF16 but is there a model that is quantized for FP16, INT8, or even INT4?

[frankinstien111]

ChatGpt came up with a solution, not sure if it would work?

from diffusers import DiffusionPipeline
import torch

repo_id = "stabilityai/stable-video-diffusion-img2vid-xt"

load in BF16 (default) then cast

pipe = DiffusionPipeline.from_pretrained(
repo_id,
torch_dtype=torch.bfloat16,
safety_checker=None,
use_safetensors=True,
)

cast all model sub-modules to float16

pipe.unet.to(torch.float16)
pipe.text_encoder.to(torch.float16)
pipe.vae.to(torch.float16)

if there’s a motion module or others, cast them too:

pipe.motion_module.to(torch.float16)

save your new FP16 copy

pipe.save_pretrained("stable-video-diffusion-img2vid-xt-fp16", safe_serialization=True)