How to load a hdf file into a hugginface dataset?

[FeryET]

Suppose I have a h5 file consisting of two datasets: array and metadata. Their names are pretty descriptive. In the metadata dataset I have all the labels and metadata for an array of the same index.

And the array is a np.float32 matrix of N * C * H * W.

h5_file = h5py.File("path/to/dataset.h5")
array_dset = h5_file['array']
metadata_dset = h5_file['metadata'] 

How can I load this into huggingface dataset? I've seen HF's dataset speeding up data processing pipelines by a considerable margin. I want to use that.

Thank you!

[mariosasko]

Hi! Currently, a dataset script is required to generate a dataset from an h5 file. Also, adding an h5 loader to `datasets is an open issue: #3113. I just haven't had time to work on it yet.