Question about layer decision of LoRA

[itsnickchiu]

Hi, thank you for your great work and detailed explanation!
I’m currently working on a project based on your work and have a few questions.

I noticed a similar discussion in issue #49, but I’d like to follow up with a more specific question.
I’m curious about your decision to apply LoRA only on attn1 in each transformer block and on the convolutional layers, instead of applying it to all attention layers as typically done in the PEFT configuration.

Was this choice based on qualitative observations or some other considerations we might be overlooking?

Thanks again!

[rohitgandikota]

Hey!

We generally noticed that applying LoRAs on convolutions have a stronger effect on capturing nuanced concepts (like visual sliders)

We avoided cross attention layers as we want to avoid capturing language features and more "visual features"

So more of a qualitative choice! This should not affect your diffusers/peft implementation in my opinion

Did you check out our sliders compatibility with peft and diffusers - https://github.com/rohitgandikota/sliders/blob/main/concept_sliders_diffusers.ipynb (you dont need to convert any weights - they are directly compatible with peft now!)

[itsnickchiu]

Thanks for your prompt reply and explanation!

I’ve conducted a small-scale experiment to see whether removing LoRA from the cross-attention layers truly helps avoid capturing language features and improves performance, and in my experiments, it indeed did.

I also tried training LoRA sliders only on the self-attention layers to evaluate the performance. Interestingly, training LoRA sliders solely on self-attention layers yielded similar results compared to the original setup, which included LoRA on convolution and time-embedding projection layers.

This was at least the case in my small-scale tests, perhaps because I only experimented with textual sliders and concepts focused on human facial attributes (e.g., age and smile), using two different pre-trained SD models (Realistic Vision and Majic Realistic) with the same prompt ("image of a person, 8k, realistic").

I’m curious whether there are any examples showing that adding LoRA to the convolution and time-embedding projection layers generally improves performance for certain types of concepts—perhaps those unrelated to humans, such as background elements, or maybe it performs better for visual sliders.

Also, I noticed that the sliders are compatible with Diffusers and PEFT. Many thanks for the heads-up!