Meta-Field in TbTData (#28)

Meta-Field in TbTData

Author: JoschD

CHANGELOG.md

@@ -0,0 +1,116 @@
+# Changelog
+
+All notable changes to **turn_by_turn** will be documented in this file.
+
+#### v1.0.0 – 2025-10-27
+
+This is the first major release of `turn_by_turn`, marking the transition from a pre-1.0 version to a stable API.
+This release includes introduces some small breaking changes to the API, mainly the removal of the `date` attribute from the `TbtData` dataclass, as it was not consistently populated across all datatypes and readers.
+Instead a new attribute `meta` has been added
+which is a dictionary to hold any additional metadata that might be relevant for a specific datatype or reader in the future,
+or can be used to store user-defined metadata,
+but the entries should not be relied upon to be present across all datatypes or readers.
+
+Some common meta-entries are:
+
+- `date`: The date and time of the measurement, if available from the file.
+- `file`: The path to the file the data was loaded from.
+- `source_datatype`: The datatype the data was loaded from, e.g. `lhc`, `sps`, `doros`, etc.
+- `comment`: Any comment on the measurement.
+
+**Changed:**
+
+- Removed the `date` attribute from the `TbtData` dataclass.
+- Reordered the parameters of the `TbtData` dataclass to have `matrices`, `nturns` first, as required attributes, then optinally `bunch_ids`,  and `meta`.
+- Added a `meta` attribute to the `TbtData` dataclass to hold additional metadata as a dictionary.
+- Updated all readers to populate the `meta` attribute with relevant metadata where available.
+- Restructured the `iota` module. This should be mostly transparent to the user, unless they were using internal functions or classes from the `iota` module directly.
+- Added a test for `esrf` datatype reading.
+
+
+#### v0.9.1 – 2025-07-10
+
+This patch release fixes the reading of SPS files after the technical stop in 2025, during which the format seems to have been changed. The array `MonPlanes` in the SDDS file, which before contained `1` if the BPM was vertical and `0` if it was horizontal switched to using `3` for vertical and `0` for horizontal BPMs.
+The current implementation now tries to determine the BPM plane not from this array, but from the ‘.H’ and ‘.V’ at the end of the BPM name in the file. Only if this ending is not present — you are able to deactivate it in the writer as this ending is also not present in the SPS model — it will first be checked if `3`s are present in the array and then the new format used, otherwise it will be checked if `0`s are in the array and then the new format used.
+Otherwise the reader will raise an informative error.
+If you only have vertical BPMs in the old format or only horizontal BPMs in the new format (i.e. your `MonPlanes` array will consist only of `1`s) this will also cause the reader to not be able to determine the format and raise an error.
+
+#### v0.9.0 – 2025-07-09
+
+Release `0.9.0` adds functionality for the loading of tracking simulation data from an `xtrack.Line`. A specific tracking setup and the use of `ParticleMonitor`s is necessary.
+This version introduces a new top-level function, `convert`, to handle data that already lives in-memory: the result of an `xtrack.Line` tracking and potentially data from `MAD_NG`, for now.
+
+**Added:**
+
+- A new module, `turn_by_turn.xtrack_line`, to handle loading data from an `xtrack.Line` after tracking. See the documentation for details.
+- A new function, `turn_by_turn.convert`, similar to `turn_by_turn.read` but to handle in-memory data.
+
+#### v0.8.0 – 2025-01-08
+
+In release 0.8.0:
+Added support for converting MAD-NG tracking results into turn-by-turn (“TBT”) form.
+
+#### v0.7.2 – 2024-10-11
+
+This patch release enables the capability to read also the oscillation data from DOROS and the means to switch between positions and oscillations data.
+
+**Added:**
+
+- `doros_oscillations`: Read/write data into the oscillations attributes of the doros-hdf5 file.
+- `doros_positions`: Read/write data into the positions attributes of the doros-hdf5 file.
+- The original `doros` datatype defaults now to `oscillations`.
+
+#### v0.7.1 – 2024-10-02
+
+In this patch release, the DOROS reader has been updated to handle files that have more entries on the root level than BPMs and `METADATA`.
+
+**Changed:**
+
+- Identifying BPMs in DOROS data by having the `"nbOrbitSamplesRead"` entry.
+
+#### v0.7.0 – 2024-08-20
+
+In this release, a reader and writer for DOROS BPMs in `hdf5` format has been added.
+
+**Changed:**
+
+- Added DOROS `hdf5` reader/writer
+- Clean-up of the Documentation
+
+#### v0.6.0 – 2023-12-01
+
+Release `0.6.0` adds to the SPS-module the possibility to remove the trailing planes (.H/.V) from the BPM names upon reading, and adding them back on writing. Both are enabled by default.
+This allows compatibility with the MAD-X models.
+
+**Added:**
+
+- sps-reader: `remove_trailing_bpm_plane` removes the trailing plane-suffixes (.H/.V) from the BPM names, if present
+- sps-writer: `add_trailing_bpm_plane` adds plane-suffixes (.H/.V) to the BPM names, if not already present
+Fixed:
+- ascii-reader: returns `TbtData`-object instead of the individual parts for one.
+
+#### v0.5.0 – 2023-06-05
+
+Release `0.5.0` adds functionality for the loading of tracking simulation data in the `trackone` module.
+Important: With this release the minimum supported Python version is upped to 3.8
+
+**Added:**
+
+- A new class, `TrackingData`, was added to `turn_by_turn.structures` which is similar to `TransverseData` but holds all 8 dimensions (`X`, `PX`, `Y`, `PY`, `T`, `PT`, `S`, `E`).
+- The `read_tbt` function in `turn_by_turn.trackone` has a new boolean argument, `is_tracking_data`, to specify data should be loaded with this new class. Default behavior is unchanged.
+- The `numpy_to_tbt` function in `turn_by_turn.utils`, which handles the loading, has a `dtype` argument to specify the datatype to load into. Default behavior is unchanged.
+- The `generate_average_tbtdata` function in `turn_by_turn.utils` handles the new class.
+Fixed:
+- The `fieldnames` method in `TransverseData` and `TrackingData` is now a `classmethod` and is properly called.
+
+#### v0.4.2 – 2022-09-21
+
+A patch release, that now allows the ASCII module to be accessed directly from the main read/write functionality.
+
+#### v0.4.1 – 2022-09-21
+
+This is a bugfix release.
+
+**Fixed:**
+
+- Less strict checking for ASCII-File format (only a `#` in the first line is now required)

README.md

@@ -1,9 +1,7 @@
 # Turn-By-Turn
 
 [![Cron Testing](https://github.com/pylhc/turn_by_turn/workflows/Cron%20Testing/badge.svg)](https://github.com/pylhc/turn_by_turn/actions?query=workflow%3A%22Cron+Testing%22)
-[![Code Climate coverage](https://img.shields.io/codeclimate/coverage/pylhc/turn_by_turn.svg?style=popout)](https://codeclimate.com/github/pylhc/turn_by_turn)
-[![Code Climate maintainability (percentage)](https://img.shields.io/codeclimate/maintainability-percentage/pylhc/turn_by_turn.svg?style=popout)](https://codeclimate.com/github/pylhc/turn_by_turn)
-<!-- [![GitHub last commit](https://img.shields.io/github/last-commit/pylhc/turn_by_turn.svg?style=popout)](https://github.com/pylhc/turn_by_turn/) -->
+[![Coverage](https://raw.githubusercontent.com/pylhc/turn_by_turn/python-coverage-comment-action-data/badge.svg)](https://github.com/pylhc/turn_by_turn/tree/python-coverage-comment-action-data)
 [![PyPI Version](https://img.shields.io/pypi/v/turn_by_turn?label=PyPI&logo=pypi)](https://pypi.org/project/turn_by_turn/)
 [![GitHub release](https://img.shields.io/github/v/release/pylhc/turn_by_turn?logo=github)](https://github.com/pylhc/turn_by_turn/)
 [![Conda-forge Version](https://img.shields.io/conda/vn/conda-forge/turn_by_turn?color=orange&logo=anaconda)](https://anaconda.org/conda-forge/turn_by_turn)
@@ -18,28 +16,30 @@ See the [API documentation](https://pylhc.github.io/turn_by_turn/) for details.
 ## Installing
 
 Installation is easily done via `pip`:
+
 ```bash
 python -m pip install turn_by_turn
 ```
 
 One can also install in a `conda` environment via the `conda-forge` channel with:
+
 ```bash
 conda install -c conda-forge turn_by_turn
 ```
 
 ## Example Usage
 
  The package is imported as `turn_by_turn`, and exports top-level functions for reading and writing:
+
 ```python
 import turn_by_turn as tbt
 
 # Loading a file is simple and returns a custom dataclass named TbtData
 data: tbt.TbtData = tbt.read("Beam2@BunchTurn@2018_12_02@20_08_49_739.sdds", datatype="lhc")
 
-# Easily access relevant information from the loaded data: transverse data, measurement date, 
+# Easily access relevant information from the loaded data: transverse data,
 # number of turns, bunches and IDs of the recorded bunches
 first_bunch_transverse_positions: tbt.TransverseData = data.matrices[0]
-measurement_date = data.date  # a datetime.datetime object
 
 # Transverse positions are recorded as pandas DataFrames
 first_bunch_x = first_bunch_transverse_positions.X.copy()

pyproject.toml

@@ -85,6 +85,20 @@ repository = "https://github.com/pylhc/turn_by_turn"
 documentation = "https://pylhc.github.io/turn_by_turn/"
 # changelog = "https://github.com/pylhc/turn_by_turn/blob/master/CHANGELOG.md"
 
+# ----- Testing ----- #
+
+[tool.pytest.ini_options]
+addopts = [
+    "--import-mode=importlib",
+]
+# Helpful for pytest-debugging (leave commented out on commit):
+# log_cli = true
+# log_cli_level = "DEBUG"
+# log_format = "%(levelname)7s | %(message)s | %(name)s"
+
+[tool.coverage.run]
+relative_files = true
+
 # ----- Dev Tools Configuration ----- #
 
 [tool.ruff]

tests/test_doros.py

@@ -1,5 +1,8 @@
+from __future__ import annotations
+
 from datetime import datetime
 from pathlib import Path
+from typing import TYPE_CHECKING
 
 import h5py
 import numpy as np
@@ -11,6 +14,9 @@
 from turn_by_turn.doros import DEFAULT_BUNCH_ID, DataKeys, read_tbt, write_tbt
 from turn_by_turn.structures import TbtData, TransverseData
 
+if TYPE_CHECKING:
+    from turn_by_turn.constants import MetaDict
+
 INPUTS_DIR = Path(__file__).parent / "inputs"
 
 
@@ -100,6 +106,9 @@ def _tbt_data() -> TbtData:
     """TbT data for testing. Adding random noise, so that the data is different per BPM."""
     nturns = 2000
     bpms = ["TBPM1", "TBPM2", "TBPM3", "TBPM4"]
+    meta: MetaDict = {
+        "date": datetime.now(),
+    }
 
     return TbtData(
         matrices=[
@@ -126,7 +135,7 @@ def _tbt_data() -> TbtData:
                 ),
             )
         ],
-        date=datetime.now(),
-        bunch_ids=[DEFAULT_BUNCH_ID],
         nturns=nturns,
+        bunch_ids=[DEFAULT_BUNCH_ID],
+        meta=meta,
     )

tests/test_iota.py

@@ -1,4 +1,4 @@
-from datetime import datetime
+from pathlib import Path
 
 import h5py
 import numpy as np
@@ -11,57 +11,61 @@
 from turn_by_turn.structures import TbtData, TransverseData
 
 
-def test_tbt_read_hdf5(_hdf5_file):
-    origin = _hdf5_file_content()
-    new = iota.read_tbt(_hdf5_file, hdf5_version=1)
-    compare_tbt(origin, new, no_binary=False)
+def test_tbt_read_hdf5(_hdf5_file_v1, _hdf5_file_content):
+    new = iota.read_tbt(_hdf5_file_v1, version=1)
+    compare_tbt(_hdf5_file_content, new, no_binary=False)
 
 
-def test_tbt_read_hdf5_v2(_hdf5_file_v2):
-    origin = _hdf5_file_content()
+def test_tbt_read_hdf5_v2(_hdf5_file_v2, _hdf5_file_content):
     new = iota.read_tbt(_hdf5_file_v2)
-    compare_tbt(origin, new, no_binary=False)
+    compare_tbt(_hdf5_file_content, new, no_binary=False)
 
 
-def test_tbt_raises_on_wrong_hdf5_version(_hdf5_file):
+def test_tbt_raises_on_wrong_hdf5_version(_hdf5_file_v1, _hdf5_file_v2):
     with pytest.raises(HDF5VersionError):
-        iota.read_tbt(_hdf5_file, hdf5_version=2)
+        iota.read_tbt(_hdf5_file_v1, version=2)
+
+    with pytest.raises(HDF5VersionError):
+        iota.read_tbt(_hdf5_file_v2, version=1)
+
 
 
+@pytest.fixture(scope="module")
 def _hdf5_file_content() -> TbtData:
     """TbT data as had been written out to hdf5 files (see below)."""
     return TbtData(
         matrices=[
             TransverseData(
                 X=pd.DataFrame(
                     index=["IBPMA1C", "IBPME2R"],
-                    data=create_data(np.linspace(-np.pi, np.pi, 2000, endpoint=False), 2, np.sin),
+                    data=create_data(np.linspace(-np.pi, np.pi, 2000, endpoint=False), 2, np.sin, noise=0.02),
                     dtype=float,
                 ),
                 Y=pd.DataFrame(
                     index=["IBPMA1C", "IBPME2R"],
-                    data=create_data(np.linspace(-np.pi, np.pi, 2000, endpoint=False), 2, np.cos),
+                    data=create_data(np.linspace(-np.pi, np.pi, 2000, endpoint=False), 2, np.cos, noise=0.015),
                     dtype=float,
                 ),
             )
         ],
-        date=datetime.now(),
         bunch_ids=[1],
         nturns=2000,
     )
 
 
 @pytest.fixture()
-def _hdf5_file(tmp_path) -> h5py.File:
-    """IOTA File standard."""
+def _hdf5_file_v1(tmp_path, _hdf5_file_content) -> Path:
+    """IOTA File v1 standard."""
+    content: TransverseData = _hdf5_file_content.matrices[0]
+
     with h5py.File(tmp_path / "test_file.hdf5", "w") as hd5_file:
         hd5_file.create_dataset(
             "N:IBE2RH",
-            data=create_data(np.linspace(-np.pi, np.pi, 2000, endpoint=False), 1, np.sin).flatten(),
+            data=content.X.loc["IBPME2R"].to_numpy(),
         )
         hd5_file.create_dataset(
             "N:IBE2RV",
-            data=create_data(np.linspace(-np.pi, np.pi, 2000, endpoint=False), 1, np.cos).flatten(),
+            data=content.Y.loc["IBPME2R"].to_numpy(),
         )
         hd5_file.create_dataset(
             "N:IBE2RS",
@@ -70,31 +74,33 @@ def _hdf5_file(tmp_path) -> h5py.File:
 
         hd5_file.create_dataset(
             "N:IBA1CH",
-            data=create_data(np.linspace(-np.pi, np.pi, 2000, endpoint=False), 1, np.sin).flatten(),
+            data=content.X.loc["IBPMA1C"].to_numpy(),
         )
         hd5_file.create_dataset(
             "N:IBA1CV",
-            data=create_data(np.linspace(-np.pi, np.pi, 2000, endpoint=False), 1, np.cos).flatten(),
+            data=content.Y.loc["IBPMA1C"].to_numpy(),
         )
         hd5_file.create_dataset(
             "N:IBA1CS",
             data=create_data(np.linspace(0, 20, 2000, endpoint=False), 1, np.exp).flatten(),
         )
-    yield tmp_path / "test_file.hdf5"
+    return tmp_path / "test_file.hdf5"
 
 
 @pytest.fixture()
-def _hdf5_file_v2(tmp_path) -> h5py.File:
-    """IOTA File standard."""
+def _hdf5_file_v2(tmp_path, _hdf5_file_content) -> Path:
+    """IOTA File v2 standard."""
+    content: TransverseData = _hdf5_file_content.matrices[0]
+
     with h5py.File(tmp_path / "test_file_v2.hdf5", "w") as hd5_file:
         hd5_file.create_group("A1C")
         hd5_file["A1C"].create_dataset(
             "Horizontal",
-            data=create_data(np.linspace(-np.pi, np.pi, 2000, endpoint=False), 1, np.sin).flatten(),
+            data=content.X.loc["IBPMA1C"].to_numpy(),
         )
         hd5_file["A1C"].create_dataset(
             "Vertical",
-            data=create_data(np.linspace(-np.pi, np.pi, 2000, endpoint=False), 1, np.cos).flatten(),
+            data=content.Y.loc["IBPMA1C"].to_numpy(),
         )
         hd5_file["A1C"].create_dataset(
             "Intensity",
@@ -104,14 +110,14 @@ def _hdf5_file_v2(tmp_path) -> h5py.File:
         hd5_file.create_group("E2R")
         hd5_file["E2R"].create_dataset(
             "Horizontal",
-            data=create_data(np.linspace(-np.pi, np.pi, 2000, endpoint=False), 1, np.sin).flatten(),
+            data=content.X.loc["IBPME2R"].to_numpy(),
         )
         hd5_file["E2R"].create_dataset(
             "Vertical",
-            data=create_data(np.linspace(-np.pi, np.pi, 2000, endpoint=False), 1, np.cos).flatten(),
+            data=content.Y.loc["IBPME2R"].to_numpy(),
         )
         hd5_file["E2R"].create_dataset(
             "Intensity",
             data=create_data(np.linspace(0, 20, 2000, endpoint=False), 1, np.exp).flatten(),
         )
-    yield tmp_path / "test_file_v2.hdf5"
+    return tmp_path / "test_file_v2.hdf5"

tests/test_lhc_and_general.py

@@ -78,8 +78,8 @@ def compare_tbt(
                 assert np.all(origin_mat == new_mat)
 
 
-def create_data(phases, nbpm, function, noise: float = 0) -> np.ndarray:
-    rng = np.random.default_rng()
+def create_data(phases, nbpm, function, noise: float = 0, seed: int = None) -> np.ndarray:
+    rng = np.random.default_rng(seed=seed)
     return np.ones((nbpm, len(phases))) * function(phases) + noise * rng.standard_normal(
         size=(nbpm, len(phases))
     )

tests/test_madng.py

@@ -32,7 +32,7 @@ def test_write_ng(_ng_file: Path, tmp_path: Path, example_fake_tbt: TbtData):
 
     new_tbt = read_tbt(from_tbt, datatype="madng")
     compare_tbt(written_tbt, new_tbt, no_binary=True)
-    assert written_tbt.date == new_tbt.date
+    assert written_tbt.meta["date"] == new_tbt.meta["date"]
 
 
 def test_error_ng(_error_file: Path):