Training on Multiple Subjects (Faces) Resulting in Blended Characteristics in Images

[voidxd]

Hi. I'm trying to train 2 subjects with DreamBooth and the output images seem to have mixed characteristics from both subjects, instead of them being very clearly defined as either one subject or the other.

• I'm using 10 images for each.
  

• I have tried both training them at the same time, and one after the other. From what I've seen in another discussion, it's better to train them at the same time.

• Model: SD 1.5

• The training parameters I've used:

  • UNet_Training_Steps: 2000 (when training at the same time), and 1000 when training them one after the other
  • UNet_Learning_Rate: 5e-6
  • Text_Encoder_Training_Steps: 350
  • Text_Encoder_Learning_Rate: 1e-6

• Output Images

  1. I first trained Aemond from House of the Dragon.
     
  2. Next I trained Sandman from the Netflix show. This is where I try to generate something with the keyword Sandman2022. You can see StableDiffusion adds that eyepatch that from Aemond.
     
  3. Next, I try to see if the training also affected Aemond and it seems it has.
     

Any advice on preventing this from happening would be very much appreciated. Thank you!

[TheLastBen]

training two subjects from the same gender always leads to blending, keep the learning rate 2e-6 or below to prevent excessive overfitting, I recommend training each subject on a separate model for maximum quality. and avoid using instance names like "sandman" it is a known word and it reduces the effectiveness of the training.