Do we want to store information about how to interpolate a dynamic node along the time coordinate ?

[imbeauf]

In a recent discussion on the conversion of time bases of the pulse_schedule IDS from DDv3 to DDv4 iterorganization/IMAS-Python#21, there was a question on whether the interpolation should be made using the "previous neighbour" or "linear interpolation" method. The choice can't be generic since both methods assume a certain behaviour of the original dynamic signal, behaviour which is presently not stored in the Data Dictionary (we just store samples along its time coordinate).
For instance using linear interpolation is problematic for cases in which the systems have only discrete values (e.g. for on/off systems some intermediate power levels are not achievable). Using "previous neighbour" is safer from this point of view but linear waveforms are often used to define the pulse schedule, and if the time base step is too coarse the signal dynamics will not be rendered properly with the "previous neighbour" method.
A long time ago, there was also a suggestion to add in the DD specific nodes allowing to record the "stepwise" character of a dynamic signal https://jira.iter.org/browse/IMAS-835.
Proposals:

1. Add in the PULSE_SCHEDULE IDS nodes to record which time interpolation method to use, and modify the GET_SLICE function to read and use this information automatically
2. Do 1 for all dynamic nodes of the DD, not only the PULSE_SCHEDULE IDS
3. Take this opportunity to introduce string formulas in the PULSE_SCHEDULE IDS describing the complete dynamic behaviour of a node, for instance using the syntax proposed by the Python waveform editor under development. To use this new information and provide values, a new AL functionality must be implemented (using the plugin framework maybe ?)
4. Do 3 for all dynamic nodes of the DD, not only the PULSE_SCHEDULE IDS

[olivhoenen]

IMO we can't take into proposals 1 and 3 as they rely on a specific treatment of the data when it comes from special IDS (here pulse_schedule). At the same time doing this for ALL dynamic fields of ALL IDSs may be an overkill.

The main problem this proposal is targeting is to avoid making linear interpolations on a quantity which is typically having only discrete values. I do believe that in the vast majority of cases the user of the data shall be able to tell if a quantity is discrete or not. But the tricky part is that the same IDS may also contain continuous data and currently the access-layer API only defines slicing operation at the entire IDS structure level.

Another solution would be to allow to perform time slicing per quantity within an IDS. With this, the user can perform slicing with linear interpolation on continuous data, and slicing with previous neighbor (or even closest sample if he does not care about causality in this use case). This would be more appealing to me as it does not involve increasing the complexity for the data producer (specifying the extra metadata or formula all over the place) or in the data access layer API.

[imbeauf]

Yes, independently of whether we trace or not the nature of the signal in the DD, in the end we will need a get_slice per node for the Access Layer. In fact we are close to have it, since we already have partial GET thanks to the plugin architecture. I assume it would not be a major amount of work to replicate the partial GET plugin into a partial GET_SLICE.

[imbeauf]

Not so much feedback, so we assume that not so many people are interested in this feature.
We note also that get_slice of individual nodes with different interpolation methods is already doable manually with IMAS_PYTHON (lazy loading).
At the progress meeting of June 3rd we decided to close this discussion.