Train sd3 controlnet but result is bad

[xduzhangjiayu]

Describe the bug

Hi,
I tried to train sd3 controlnet with examples\controlnet\train_controlnet_sd3.py, but it seems the validation results are really bad, and the training loss was oscillating all the time, it really confused me for a long time, could you please give some advice? Thanks a lot !

Here are some validation results and train loss:
validation output on steps 2900:

loss function in tensorboard

Reproduction

accelerate launch examples/controlnet/train_controlnet_sd3.py --pretrained_model_name_or_path=stable-diffusion-3-medium-diffusers --output_dir=output --resolution=1024 --learning_rate=1e-5 --validation_image "./conditioning_image_1.png" "./conditioning_image_2.png" --validation_prompt "red circle with blue background" "cyan circle with brown floral background" --train_batch_size=1 --gradient_checkpointing --gradient_accumulation_steps=4

Logs

No response

System Info

diffusers: 0.31.0.dev (latest)
torch: 2.4.0

Who can help?

@sayakpaul @DN6 @asomoza

[sayakpaul]

This should belong to discussions. Ccing @PromeAIpro who contributed the script.

[PromeAIpro]

I think this is normal,it is just learning how to control，The loss is also oscillating. Try to use real data for training.

[wuutiing]

seems @DavyMorgan pushed the initial commits of train-sd3-script, @PromeAIpro was flux-train-script contributor :)
look into the training dataset check if it was prepared right.

[xduzhangjiayu]

I think this is normal,it is just learning how to control，The loss is also oscillating. Try to use real data for training.

Could you explain what is "real data" ? Thanks so much!

[xduzhangjiayu]

seems @DavyMorgan pushed the initial commits of train-sd3-script, @PromeAIpro was flux-train-script contributor :) look into the training dataset check if it was prepared right.

Thanks for the comment, the train dataset I used was fill50k in huggingface, so I think it was prepared right.

[xduzhangjiayu]

Hi, thanks for the replies.
I have tried another test with batch=32
After 10000 steps, the train loss is still decrease, but the validate results seems not good as follow:

prompt: cyan circle with brown floral background

Maybe the model is overfitting? Or loss function is set incorrectly ?

[TccccD]

Hi, I have a similar problem, have you solved it?

[ningpingwang]

@TccccD Maybe the training loss has a bug. Can you take a look at https://github.com/huggingface/diffusers/discussions/10618?