Content in outputs gradually disappear when Training depth conditioined model

[OrangeSodahub]

Hi, I'm trying to train the depth-conditoned model from scratch on custom data, and confused about the results:

val at step=70:

val at step=140:

val at step 210:

val at step 280:

val at step 350:

val at step 420:

As the example shows, the prediction outputs gradually becomes blurred until there is no content, I wonder if it is correct phenomenon? Or did you occurred this issue before? Thanks.

[OrangeSodahub]

@Tangshitao I'm a little bit confused that if this codebase could be trained correctly, while I'm sure that inference has no problem. I wonder if there is anyone else trained successfully.

[Tangshitao]

I feel you motion is too large, which could cause inconsistent generation. Can you try data with small motions?

[OrangeSodahub]

@Tangshitao Thanks. But with small motions this issue still exists:

Even though the motion too large causes inconsistent generation, the quality of generations shouldn't degrade. Now the generations are totally meaningless.

[YunzeMan]

@Tangshitao I observe exactly the same problem. Very curious about the reason.

[Tangshitao]

I feel there might be extrinsic and intrinsic issues. Can you try to train with Scannet data?

[OrangeSodahub]

I'm sure that cameras have no problems. I found that the method of sampling frames to form a batch data is crucial, how do you produce the key_frames_0.6.txt files?

[Tangshitao]

I compute the overlap between each frame within a video, can record frame pair with overlap larger than 0.6

[OrangeSodahub]

@Tangshitao Hi, I'm so confused. I tried your codebase, and I intergrated your training script (depth) into another diffuser pipeline then tried again, this issue always exist -- The content disappeared gradually during training.

I really want to know is there anyone trained depth version successfully? Because I can't find out the problem. Big thanks!

[Tangshitao]

Have tried to train the model with the codes released instead of intergrated my codes into another codebase?

[OrangeSodahub]

Yes, I first tried that, then I tried to integrated one. Both of them have similar results, the outputs get disappeared

[OrangeSodahub]

By the way, can I just train your depth version model but replace the SD-2-Depth with SD1.5? I'm not sure will it impact the performance of the correspondence-aware attention layer.

[yuvalH9]

I get similar issue of degrading images when training the depth model generation.
@OrangeSodahub @YunzeMan have you find any solution for this issue?

Thanks
Yuval

[OrangeSodahub]

@yuvalH9 Nah, we have not

[yuvalH9]

@OrangeSodahub I see.
Maybe you have a suggestion for other multi view diffusion depth conditioned method with working training?