Please confirm something about the error of modality_shape size in sample

[junphine]

When using unet as an argument to pre_post_transformer_enc_dec, the function maybe_transition_to_modality_decoding parse out the modality_shape，it should be multiplied by 2:

modality_shape = tuple(map(lambda x: x * 2,modality_shape))

the shape flow：
modality_shape[7,7]->nosie[7,7,4]->denoised[7,7,4]->latent modality_tensor[7,7,4]->model modality_tensor[4,4,dim]

[lucidrains]

@junphine 🤦 yes you are right

i think the modality shape being stored should be the one after the unet encoding, in the presence of unet encoding / decoding

will get this fixed, thank you!

[lucidrains]

@junphine hey, so i took a closer look and i think it is correct

to be clear, the modality_shape that the language model is learning is always the latent dimensions (so after the pre unet encoding and the post unet decoding). i should probably make that distinction somewhere, as i have the fixed_modality_shape input and researchers probably assume that's the pre-encoded shape

[lucidrains]

@junphine could you confirm the error is because you are trying to fix the shape?

[junphine]

@lucidrains
Beacuase the modality shape is always parsed from language model output,the fixed_modality_shape is not used! Maybe meta shape use modality encoder(such as vae) output shape is good idea!

[lucidrains]

@junphine it can be overridden, it is just that if you were to pass it in right now, it may be confusing as it refers to the latent modality dimensions at the unet bottleneck

[lucidrains]

@junphine let me polish that up a bit more now that you are looking at it, rename a few things