flux ip-adpater only load in double stream block?

[jiang-rz]

Excuse me, why only load ipadpter in the double-stream layer

def convert_flux_ipadapter_checkpoint_to_diffusers(original_state_dict, num_layers):
    converted_state_dict = {}

    # image_proj
    ## norm
    converted_state_dict["image_proj.norm.weight"] = original_state_dict.pop("ip_adapter_proj_model.norm.weight")
    converted_state_dict["image_proj.norm.bias"] = original_state_dict.pop("ip_adapter_proj_model.norm.bias")
    ## proj
    converted_state_dict["image_proj.proj.weight"] = original_state_dict.pop("ip_adapter_proj_model.norm.weight")
    converted_state_dict["image_proj.proj.bias"] = original_state_dict.pop("ip_adapter_proj_model.norm.bias")

    # double transformer blocks
    for i in range(num_layers):
        block_prefix = f"ip_adapter.{i}."
        # to_k_ip
        converted_state_dict[f"{block_prefix}to_k_ip.bias"] = original_state_dict.pop(
            f"double_blocks.{i}.processor.ip_adapter_double_stream_k_proj.bias"
        )
        converted_state_dict[f"{block_prefix}to_k_ip.weight"] = original_state_dict.pop(
            f"double_blocks.{i}.processor.ip_adapter_double_stream_k_proj.weight"
        )
        # to_v_ip
        converted_state_dict[f"{block_prefix}to_v_ip.bias"] = original_state_dict.pop(
            f"double_blocks.{i}.processor.ip_adapter_double_stream_v_proj.bias"
        )
        converted_state_dict[f"{block_prefix}to_k_ip.weight"] = original_state_dict.pop(
            f"double_blocks.{i}.processor.ip_adapter_double_stream_v_proj.weight"
        )

    return converted_state_dict

and this code comes from

But the xlabs code does have a single-stream ipadpater
https://github.com/XLabs-AI/x-flux

[hlky]

The models only use double blocks
https://huggingface.co/XLabs-AI/flux-ip-adapter
https://huggingface.co/XLabs-AI/flux-ip-adapter-v2/tree/main

[jiang-rz]

Oh, I see. But because I trained an ipa with both single and double blocks

[hlky]

There is not much code that needs to be changed to support single blocks.

This would either move out of the if block and use a tuple attention_outputs in the block so we can add the ip attention output, or duplicate into the else case.

https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py#L2795-L2818

There is code related to above in the transformer block
https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_flux.py#L158-L163

And conversion/setting attention processor as you mentioned.

Would you be interested in opening a PR? We are happy to assist.

Is your ip adapter public? It would be a great addition to the Hub and allow us to help with testing.