Code about ablations

[tomo120717]

Thank you for your great work and beautiful code.

I would like you to upload your code for ablations. You did a lot of ablations like visual-only and future prediction in your paper. I want to conduct them on my own.

Thanks,

[LostXine]

Hi @tomo120717 ,

Thanks again for your interest. I'll give you a quick note here to replicate the ablations.

1. Visual-only
   just comment out this block which reconstructs the low-dim states:
   

   crossway_diffusion/diffusion_policy/model/diffusion/conditional_unet1d.py

   Lines 438 to 441 in d53a4a3

   	x = mid_ld
   	for layer in self.decs[key]:
   	x = layer(x)
   	recons[key] = x

2. future prediction
   First, add the following code after this line

future_nobs = dict_apply(nobs, 
              lambda x: x[:,self.n_pred_obs_shift: self.n_pred_obs_shift + self.n_obs_steps,...].reshape(-1,*x.shape[2:])) 
# self. n_pred_obs_shift stands for how far the future you want to predict, when it is 0, that means reconstructing the current observation without future prediction.

Then replace the following block

crossway_diffusion/diffusion_policy/policy/crossway_diffusion_unet_hybrid_image_policy.py

Lines 385 to 388 in d53a4a3

	if rec.shape != this_nobs[key].shape:
	rec_target = T.Resize(rec.shape[-2:])(this_nobs[key])
	else:
	rec_target = this_nobs[key]

with

if rec.shape != future_nobs[key].shape:
    rec_target = T.Resize(rec.shape[-2:])(future_nobs[key])
else:
    rec_target = future_nobs[key]

Best,

[tomo120717]

Hi @LostXine ,

Thank you for your quick responce and good code. I understand what you did.
I'll try it.

Best,

[tomo120717]

Hi @LostXine ,

I tried your code about future predictoin, but it doesn't work well.
The variable 'future_nobs' has 0 batch size, so it is not be able to be compared.

I print the shape of this nobs and future nobs shape
key: agentview_image rec.size(): torch.Size([32, 3, 84, 84]) rec_target.size(): torch.Size([0, 3, 84, 84])

Did you change the robomimic_replay_image_dataset.py file from the diffusion policy one?

Please help me.

Best,

[LostXine]

Hi @tomo120717 ,
could you share your config file here? I want to check the value of horizon and n_pred_obs_shift,
thank you.

[tomo120717]

Hi @LostXine ,

Thank you for your reply.
horizon = 16 and n_pred_obs_shift=2

Thank you.

[LostXine]

Hi @tomo120717 ,

Please delete this line and try again.

crossway_diffusion/config/tool_hang_ph/typea.yaml

Line 137 in d53a4a3

n_obs_steps: ${n_obs_steps}

The reason behind:

crossway_diffusion/diffusion_policy/dataset/robomimic_replay_image_dataset.py

Lines 217 to 221 in d53a4a3

	# to save RAM, only return first n_obs_steps of OBS
	# since the rest will be discarded anyway.
	# when self.n_obs_steps is None
	# this slice does nothing (takes all)
	T_slice = slice(self.n_obs_steps)

Thanks,

[tomo120717]

Hi, @LostXine ,

Thank you for your replying!

The program looks working well by your advice! Additionaly, I want visualize the true future state image in wandb.
How did you do this?

Best,

[LostXine]

Hi @tomo120717 ,

To upload images to wandb, you can refer to this link: https://docs.wandb.ai/guides/track/log/media

Thanks,

[tomo120717]

Hi @LostXine ,

Sorry. My sentence was wrong.

What I wanted to say is in your code you visualize your reconstruction result and the GT. This program is for predict N=0, so you should change the program to predict future state.

This line

crossway_diffusion/diffusion_policy/workspace/train_crossway_diffusion_unet_hybrid_workspace.py

Line 254 in d53a4a3

org_img = show_imgs[key][:, t_idx]

If you would be willing to change this part of the program, I would appreciate it.

Best,