Data transfer between models

[cmutel]

We will have an API to retrieve raw data, but as we traverse along the supply chain graph we also need to provide output data from previous nodes.

Models which are simple transformations are pretty easy - e.g. a vehicle fuel combustion model could have a linear coefficient turning fuel into $CO_{2}$, and if we had multiple observations they would flow naturally as a DataFrame into downstream models. However, we will also have the following cases:

• A model is resolved in a context (technology, space, time). This context is not necessarily the same as the starting node, so needs to be propagated along the value chain.
• A model will produce outputs with specific properties, such as contaminant content, quality classes, densities or specific weights, etc. This properties can influence how downstream models do their calculations (e.g. a gas might need to be compressed or decompressed before usage in a certain system).

We need a standard for this data flow, and while it would be nice if it were the same as that provided by the generic normalized data API, it seems probable that this won't be enough.

[cmutel]

We probably need a demand object as one of the DTOs. It could have the following form:

{
  'product': glossary_reference,
  'unit': glossary_reference,
  'amount': numerical_value,
  'properties': {
    # Optional list of properties which are per-product (stored in product glossary entry)
  }
}

[simontaurus]

I would put it this way

{
  'product': {
    'type': glossary_reference,
    # Optional list of properties which are defined in the product schema
      'p1': '...'
  }
  'unit': glossary_reference,
  'amount': numerical_value,
}

[mklarmann]

We do provide to the gap-filling models always the current state of the full-calculation graph in memory. Any information we want to provide hence to a model, would be needed to enter the graph through a mutation. This could be another gap-filling module adding/modifying a node. Or could even be a user input (like in setting a parameter in a property of the node).

There would never be any direct exchange / communications of models to prevent dependencies.

This concept limits us to only communicate on "static/linear" instantiations of a graph (as allowed by the mutations). Our gap-filling models would not be able to provide a function as an output to another model.

[cmutel]

We do provide to the gap-filling models always the current state of the full-calculation graph in memory. Any information we want to provide hence to a model, would be needed to enter the graph through a mutation.

Can't decide on how I feel about this - it makes some things easier, but also makes it tough if not impossible to get outside the model of everything running sequentially on a single server. I don't really think that's possible for us, as there will almost certainly be black-box proprietary models accessible only via some interface.

It also assumes that you know the graph in advance, as opposed to determining it at run time. I would like to learn more about what the research that @michaelweinold is doing, not necessarily with the way that hybridization is run right now, but with the general concepts of how to merge top-down and bottom-up data. I feel like this would also make the graph more dynamic.

But maybe we are agreeing more than it seems - for me, a model (or product system model) is not always a node, but a subgraph, and that subgraph can be mutated through the addition of generic and ancillary models, including gap filling. So definitely here we would need such a mutation log.

[mklarmann]

External Black-box proprietary models: Would still work with the same structure, but would make things "slow" as to the whole graph would need to be serialized and passed over a wire. Yet I could think of some way of obfuscation, that would even work on the same server (thinking about the xz hack https://gynvael.coldwind.pl/?lang=en&id=782 ). But then again, "slow" might not be an issue, as any way of external black-box would make the system slow, independent of the interface.

Knowing the graph in advance: I do not understand, why that would need to be the case. The graph is assembled by the GFMs piece by piece. Which could also be async, that means with some delays.

I thought about the term "product system model", it has 3 ambiguous terms already used somewhere else. I thought we might want to call it "Modelling System Environment" oder "Modelling System - GFM", to give it a specific place. Just a thought.