About "Guidance distilled diffusion models"

[lyx0208]

According to my own understanding, these checkpoints are distilled SD models that generates with Dynamic, is this right? Moreover, can you provide more training details of such models? Thanks a lot in advance !

[quickjkee]

Hi! Thanks for your interest.
Not really. We just embed the guidance through a new layer. In more detail:

• the guidance has the following form:
  $\epsilon^{w} (x_{t}, t, c) = \epsilon_{\theta}(x_{t}, t, 0) + w (\epsilon_{\theta}(x_{t}, t, c) - \epsilon_{\theta}(x_{t}, t, 0))$
  As you can see, the neural network is called twice: $\epsilon_{\theta}(x_{t}, t, 0), \epsilon_{\theta}(x_{t}, t, c)$

• We just distill this guidance into a new layer called an embedding layer. That is, solve the following taks:
  $| \epsilon^{w} (x_{t}, t, c) - \epsilon (x_{t}, t, c, w)|$. Where $w$ is a new layer (it is implemented in the same way as $t$ embedding). In this way, we have one propagation instead of two.

• To make a dynamic guidance, after distillation, we can make $w$ dependent on $t$. We use the step function $w(t)$. That is, if $t > \tau$, then $w(t)=0$, otherwise $w(t)=w$.

[lyx0208]

Thanks for your clear answer! I understand what the distilled models work for.

[lyx0208]

So what should I do if I want to train the model with another distillation teacher, like the newly released SD3?

[lyx0208]

There seems no official implementation of the paper "On Distillation of Guided Diffusion Models"

[quickjkee]

Yes, you are right. There is no official implementation. Probably, we will release the code for guidance distillation when we have free time. However, it is not necessary to do Consistency Distillation on top of a guidance distilled model. In other words, if you want to play with SD3, you can skip the guidance distillation step and use our CD directly with the SD3 teacher.

[lyx0208]

Ok, thanks a lot for your kind response!!

[lyx0208]

Hello!I've tried training the sd1.5 based model with the script you provided, but without the "embed_guidance" flag. The model I get shows poorer performance in both reconstruction accuaracy and editability. See the image below:

[quickjkee]

Hi. Have you tried generating? How do the metrics behave during training (FID, reconstruction loss)?

[lyx0208]

The generating results also seems to be less satisfactory than the provided checkpoints.
I don't have visualized loss metrics since the machine I use can not access to wandb and tensorboard.
By the way, have you tried performing iCD without the w embedding and what's the corresponding results?

[lyx0208]

Here is the demo generating result with prompt "a cute owl with a graduation cap."

[quickjkee]

It looks like the guidance has been turned off. Do you distil the model with guidance ($w>0$?)