Inconsistent Parameter Saving Between .flt and .h5

[abmajith]

HI @jonwright , @jadball

In the long term, we plan to support saving a list of parameters in both .flt and .h5 formats.
Ideally, these lists of parameters should remain consistent across both formats.

Currently, I noticed a difference in how parameters are saved:

The columnfile class's writefile method saves all attributes.
The colfile_to_hdf function mostly (or only) saves title attributes.

This causes an issue during indexing, where it expects to read the wavelength attribute from the loaded columnfile (from the .h5 file), but it isn't available, while using the function

venv/lib/python3.12/site-packages/ImageD11/indexing.py:292: in indexer_from_colfile_and_ucell
    w = float(colfile.parameters.get("wavelength"))

Would it be sufficient to work around this issue by relying solely on the .flt format using writefile and readfile methods?
Or do we need to extend the .h5 saving method to ensure it stores all attributes, similar to how .flt does?

[jonwright]

Hi @abmajith

It is not very clear to me right now. A columnfile can also be coming from an mmap.

Historically we intended to save in the hdf5:

ImageD11/ImageD11/columnfile.py

Line 559 in 5ead15d

FIXME TODO - add the parameters somewhere (attributes??)

I suspect you need parameters for each column in the dataset. For example, the "raw data" columns would refer to the peaksearch threshold and frames. The sc/fc would refer to the spline file. The tth/eta/gx/gy/gz would refer to the geometry. The 'ring_id' would refer to a unit cell.

It might have been better if we had not saved parameters into colfiles, but we cannot change the past.

[loichuder]

Related to this, is the HDF5 saving/loading of ColumnFiles used somewhere ?

We are currently considering different ideas to save CF as NeXus-compliant data and we could implement this directly inside the ColumnFile class (either as a replacement or as a different HDF5 saving/loading) if you are interested ?

[jonwright]

Yes - columnfiles are frequently loaded and saved as hdf in the notebooks. We sometimes use the memory mapped columnfiles backed by hdf5 (in point by point indexing I think).

If you can make a Nexus compliant version it would be great. Can it be made compatible by just adding some attributes?

[loichuder]

Yes - columnfiles are frequently loaded and saved as hdf in the notebooks. We sometimes use the memory mapped columnfiles backed by hdf5 (in point by point indexing I think).

Noted, thanks 👍

If you can make a Nexus compliant version it would be great. Can it be made compatible by just adding some attributes?

Not sure, I haven't looked at the current implementation in details. I'll keep you informed when I do