Can this model be used, DFloat11/FLUX.1-Fill-dev-DF11， It can reduce VRAM by 70%. What is this open-source license?

[libai-lab]

Can this model be used, DFloat11/FLUX.1-Fill-dev-DF11， It can reduce VRAM by 70%. What is this open-source license?

[thss15fyt]

The license is Apache 2.0.

For the DFloat11 version, we will try if it is compatible with our checkpoint, and report it later. Thanks for the suggestion.

[thss15fyt]

After test, DFloat11 can be directly enaged in the inference script gradio_demo.py as follows:

1. Add after the initialization of transformer:

from dfloat11 import DFloat11Model
DFloat11Model.from_pretrained('DFloat11/FLUX.1-Fill-dev-DF11', device='cpu', bfloat16_model=transformer)

2. Modify the hacked_lora_forward() for dynamic dtype convertion:

def create_hacked_forward(module):

    def lora_forward(self, active_adapter, x, *args, **kwargs):
        input_dtype = x.dtype
        x = x.to(self.base_layer.weight.dtype)
        result = self.base_layer(x, *args, **kwargs)
        if active_adapter is not None:
            torch_result_dtype = result.dtype
            lora_A = self.lora_A[active_adapter]
            lora_B = self.lora_B[active_adapter]
            dropout = self.lora_dropout[active_adapter]
            scaling = self.scaling[active_adapter]
            x = x.to(lora_A.weight.dtype)
            result = result + lora_B(lora_A(dropout(x))) * scaling
            
        return result.to(input_dtype)
    
    def hacked_lora_forward(self, x, *args, **kwargs):
        return torch.cat((
            lora_forward(self, 'vtryon_lora', x[:1], *args, **kwargs),
            lora_forward(self, 'garment_lora', x[1:], *args, **kwargs),
        ), dim=0)
    
    return hacked_lora_forward.__get__(module, type(module))

After applying DFloat11, the maximum VRAM is decreased to be no more than 21GB.

[libai-lab]

After test, DFloat11 can be directly enaged in the inference script gradio_demo.py as follows:

1. Add after the initialization of transformer:

from dfloat11 import DFloat11Model
DFloat11Model.from_pretrained('DFloat11/FLUX.1-Fill-dev-DF11', device='cpu', bfloat16_model=transformer)

2. Modify the hacked_lora_forward() for dynamic dtype convertion:

def create_hacked_forward(module):

    def lora_forward(self, active_adapter, x, *args, **kwargs):
        input_dtype = x.dtype
        x = x.to(self.base_layer.weight.dtype)
        result = self.base_layer(x, *args, **kwargs)
        if active_adapter is not None:
            torch_result_dtype = result.dtype
            lora_A = self.lora_A[active_adapter]
            lora_B = self.lora_B[active_adapter]
            dropout = self.lora_dropout[active_adapter]
            scaling = self.scaling[active_adapter]
            x = x.to(lora_A.weight.dtype)
            result = result + lora_B(lora_A(dropout(x))) * scaling
            
        return result.to(input_dtype)
    
    def hacked_lora_forward(self, x, *args, **kwargs):
        return torch.cat((
            lora_forward(self, 'vtryon_lora', x[:1], *args, **kwargs),
            lora_forward(self, 'garment_lora', x[1:], *args, **kwargs),
        ), dim=0)
    
    return hacked_lora_forward.__get__(module, type(module))

After applying DFloat11, the maximum VRAM is decreased to be no more than 21GB.

Can we use the relevant models of qwen-image

[thss15fyt]

OmniTry is built on inpainting-based models, and can not be directly extended to editing models like qwen-image.

[thss15fyt]

Closing the issue. Feel free to re-open it if you have more questions.

[WbLi0203]

I tested the DFloat11/FLUX.1-Fill-dev-DF11 model and found that it did not reduce VRAM usage, which remained around 28G. Why is that?

[thss15fyt]

I tested the DFloat11/FLUX.1-Fill-dev-DF11 model and found that it did not reduce VRAM usage, which remained around 28G. Why is that?

Could you share the implementation details?

[WbLi0203]

I tested the DFloat11/FLUX.1-Fill-dev-DF11 model and found that it did not reduce VRAM usage, which remained around 28G. Why is that?

Could you share the implementation details?

The modified code is as follows, and the rest of the code remains consistent with your previous code.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
model = DFloat11Model.from_pretrained(
'checkpoints/FLUX.1-Fill-dev-DF11',
device='cpu'
)

如果成功，再尝试转换模型

if transformer:
model = model.to(torch.bfloat16)

hack lora forward

def create_hacked_forward(module):

def lora_forward(self, active_adapter, x, *args, **kwargs):
    input_dtype = x.dtype
    x = x.to(self.base_layer.weight.dtype)
    result = self.base_layer(x, *args, **kwargs)
    if active_adapter is not None:
        torch_result_dtype = result.dtype
        lora_A = self.lora_A[active_adapter]
        lora_B = self.lora_B[active_adapter]
        dropout = self.lora_dropout[active_adapter]
        scaling = self.scaling[active_adapter]
        x = x.to(lora_A.weight.dtype)
        result = result + lora_B(lora_A(dropout(x))) * scaling
    # return result
    return result.to(input_dtype)

def hacked_lora_forward(self, x, *args, **kwargs):
    return torch.cat((
        lora_forward(self, 'vtryon_lora', x[:1], *args, **kwargs),
        lora_forward(self, 'garment_lora', x[1:], *args, **kwargs),
    ), dim=0)

return hacked_lora_forward.__get__(module, type(module))

++++++++++++++++++++++++++++++++++++++++++++++++
Model loading process:

Loading DFloat11 safetensors: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 56/56 [00:01<00:00, 36.60it/s]
Total model size: 13.4768 GB

• Running on local URL: http://0.0.0.0:7861
  2025/09/10 08:39:41 [W] [service.go:132] login to server failed: dial tcp 44.237.78.176:7000: i/o timeout

Could not create share link. Please check your internet connection or our status page: https://status.gradio.app.

++++++++++++++++++++++++++++++++++++++++++++++++
GPU：26G

[WbLi0203]

[thss15fyt]

@WbLi0203 I noted that your code is slightly different that converting the model to bfloat16 after initialization. Below is the definition of mine:

transformer = FluxTransformer2DModel.from_pretrained(f'{args.model_root}/transformer').requires_grad_(False).to(dtype=weight_dtype)

from dfloat11 import DFloat11Model
DFloat11Model.from_pretrained('DFloat11/FLUX.1-Fill-dev-DF11', device='cpu', bfloat16_model=transformer)