Marshal input data from several families to an algorithm

[knoepfel]

DUNE US S&C R&D item 100

Marshaling input data from several families to an algorithm

1. Data windows (processing adjacent data)
2. Algorithms that take input data from a spill and its containing subrun
3. Algorithms that take input data from a spill and from an unrelated calibration period
   • Ask DUNE for specific example of calibration data

Items 2 and 3 as described might be satisfied by running an "art" job that takes data from the Run and SubRun and uses calibration data. We're assuming that this already exists and has been done.

For the full milestone, and assuming a hierarchy of Run/SubRun/Spill/Event and one containing Calibrations (as exemplars), we need a demonstrator that can:

• run an algorithm
• put products into a "spill" Data Family (maybe called a Run) and retrieving those products in an algorithm.
• put products into a "SubRun" Data Family and retrieving those objects in an algorithm
• retrieve calibration objects for a particular "interval".

Some relevant definitions:

• Example of Data Set - set of containers of Hits from a particular detector. An entry in the set is a single container that might correspond to a particular Event.
• Example of Data Family - Event
• Example of Data Family Hierarchy - Run, SubRun, Event
• Example of a Data Family not fitting in the Hierarchy - calibration interval

The first item in the milestone description goes beyond anything "art" can do: "1. Data windows (processing adjacent data)"

How do we implement this? We would have to decide how to implement this.

• Adjacent Events duplicate the same data in the boundary region on the edge so they overlap.
• There are small separate boundary Events that include data near the boundary between Events. This data region is small and does not overlap with the main Events. When you process an Event, then you process it with the bordering regions.
• Something else?

Then we have to implement this and the code to process this. Ideally, there is a (Mock) I/O system to support this milestones as it seems more straightforward to work on an adjacency breadboard if the data has characteristics that are realistic. Alternatively, we can rely on dummy objects, but without DUNE input, there is no guarantee that these are representative.

There is nothing in the current description of the milestone that requires persistent storage of data.

This also raises additional questions that we may or may not need to satisfy for this milestone. What will be the "Data Family"'s used for this demonstrator? Is Run, SubRun, Event plus the non-hierarchical calibration interval sufficient? Do we need additional families (since the user can defined their own data families and hierarchies) or call them different names?

[sabasehrish]

Subsystems

Required

1. Data modeling
2. Task management
3. Mock I/O

Nice to have

1. I/O
2. Configuration
3. Logging

[knoepfel]

Related to:

• Create task manager for F1 demonstrators #33
• Create mock IO subsystem for F1 demonstrators #34
• Create data product management subsystem for F1 demonstrators #38

[knoepfel]

Responsible developers: @wddgit and @knoepfel