get_observations sometimes returns null values

[fred3105]

It seems like this issue has been going on for quite some time but is difficult to reproduce.

This dataset leads to a successful training but the loss is null and gradient is nan for the entire training: https://huggingface.co/datasets/LegrandFrederic/grab_food

On the contrary, with the exact same code, this dataset shows a correct loss and gradient: https://huggingface.co/datasets/LegrandFrederic/clean_desk

This problem has been going on since gr00t-n1 and is still relevant with gr00t-n1.5

I will keep exploring this issue

[fred3105]

Update: this is a recording error where a null value appears in the actions column
After removing the episode with null values, the training works no problem

[oulianov]

Interesting. We should add a filtering step in the gr00t training pipeline after cloning the huggingface repo.

The simplest way would be to block the training pipeline if an episode with null values for observation is detected and make it crash with an explicit error message. Fore example:

"Can't train because episodes (2,3,5) have null values in observation.state column"

[fred3105]

Added a check when saving episodes to mitigate the issue, also added a check in the gr00t pipeline
#197

This issue seems to appear more often on long recordings