Skip to content

Look into recovering missing record-level HDF5 Attributes in raw data files that are not cleanly closed #110

@bieryAtFnal

Description

@bieryAtFnal

In October/November 2024, there were several PDS calibration runs that did not end gracefully, and this resulted in raw data files that were not cleanly closed.

A utility was developed to recover missing file-level HDF5 Attributes in these files and thereby allow successful creation of the file-transfer metadata and successful transfer of the raw data files to offline storage. That utility is described here.

As part of reviewing that PR, Wes noticed that there may be additional recovery steps that we can take if/when we see similar un-closed raw data files in the future. We may be able to recover TriggerRecord-level HDF5 Attributes for the last TriggerRecord in un-closed files, and thereby make those records read-able offline.

For reference, the hdf5_dump.py utility was successfully run on all of the TRs in one of the sample unclosed PDS raw data files. This seems to imply that all of the data is available for the last TR in the file, and it may just be missing record-level Attributes that prevents the last TR from being examined by more sophisticated tools.

This Issue suggests that we look into all of this at some point in time.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions