Updated Storage Configuration and Connection Class Structure

[LswaN58]

Storage Config and Connectors

At present, the system for configuring and connecting to data sources (and destinations) is limited due to a good number of assumptions that held up well enough in early versions of OGD, including:

• We only read in Event data from sources (not feature data)
• Output data is always written to a file (no credentials needed)
• Data is always imported based on IDs or date ranges (not both) with no additional filtering

The ogd-common 2.0 milestone aims to overhaul this system to allow support for new expansions in OGD pipeline capabilities, such as allowing import of feature data, allow writing to credential-protected storage, and allow more complex filtering on input data. Other proposals can document filtering and feature data for interfaces. This proposal deals with how the broader config schemas and storage connection classes must change.

Class Relationship Summary

Putting together all the various classes described in the sections below, and hiding direct connections where certain configs/schemas are passed to an upper-level class through lower-level classes, the proposed refactor/redesign can be summarized as follows:

classDiagram

StorageConnector <|-- MySQLConnector
MySQLConnector <|-- MySQLInterface
Interface <|-- MySQLInterface
StorageConnector <|-- FileConnector
FileConnector <|-- FileOuterface
Outerface <|-- FileOuterface

DataStoreConfig *-- "1" CredentialConfig 
DataStoreConfig *-- "1" LocationConfig 
CredentialConfig <|-- PasswordCredentialConfig 
CredentialConfig <|-- KeyCredentialConfig 
LocationConfig <|-- HostLocationConfig 
LocationConfig <|-- FileLocationConfig 

TableSchema *-- "1" TableStructureSchema
EventTableStructureSchema <|-- TableStructureSchema
FeatureTableStructureSchema <|-- TableStructureSchema
TableSchema *-- "1" TableLocationSchema
TableLocationSchema <|-- DatabaseTableLocationSchema

GameDataConfig *-- "1" DataStoreConfig 
GameDataConfig *-- "1..2" TableSchema 
MySQLInterface *-- "1" GameDataConfig 
FileOuterface *-- "1" GameDataConfig 

Loading

Storage Connection Class Hierarchy

The Old Way

In the old Interface -> Outerface hierarchy, an Interface combined general storage connection logic with data reading logic, and an Outerface added writing logic.

classDiagram
Interface <|-- Outerface
Interface <|-- MySQLInterface
Outerface <|-- FileOuterface

Loading

The New Way

In the new hierarchy, we separate connection logic from data read, creating a common base class called StorageConnector to handle connection logic. Then Interface and Outerface classes are set up independently as mixin classes that define a set of functions for reading or writing data.
Then for any data storage medium we want to support, we write a subclass of StorageConnector as e.g. MySQLConnector to take in the corresponding config and call appropriate functions from whatever (in this case) MySQL library is in use.
To create an actual Interface or Outerface for MySQL, we would create MySQLInterface or MySQLOuterface that inherits from MySQLConnector and Interface/Outerface.

This is a bit more complicated than before, but minimizes code duplication and ensures that we can write interfaces/outerfaces independently of each other, and don't have to worry about connection logic coming out of sync between the two.

It also gives us a clearer way to think about configs/schemas.

classDiagram
StorageConnector <|-- MySQLConnector
MySQLConnector <|-- MySQLInterface
Interface <|-- MySQLInterface
StorageConnector <|-- FileConnector
FileConnector <|-- FileOuterface
Outerface <|-- FileOuterface

Loading

Storage Config Schemas

With the new hierarchy separating connection from data read/write, we can also improve how configs are set up.

The Old Way

In the old system, schemas are structured based on how ogd-core configs are set up. There, we have a series of "game sources," which each define a specific source, type, and some kind of credential and location (i.e. a project ID, database host, etc).
We then also have a "game source map" that maps games to an object consisting of "game source," database, table, and a table schema name.

As a result, the old system has the following classes:

• DataSourceSchema : Contains data from a "game source."
• TableSchema : Contains data loaded from a table schema JSON file.
• GameSourceSchema : Contains the data from an element of the "game source map":
  • A DataSourceSchema, directly, thanks in part to hacking in a data_sources dict to the FromDict(...) params
  • The name of a table schema, no direct access to dict of schemas
  • A database name
  • A table name

So the relationship looks something like:

classDiagram
class GameSourceSchema {
    str game_id
    str db_name
    str table_name
    str table_schema_name
}
GameSourceSchema *-- DataSourceSchema
Interface "1" *-- GameSourceSchema

Loading

Even if we fix it up to get the table schema in there, the relationship is like

classDiagram
class GameSourceSchema {
    str game_id
    str db_name
    str table_name
}
GameSourceSchema "1" *-- DataSourceSchema
GameSourceSchema "1" *-- TableSchema
Interface "1" *-- GameSourceSchema

Loading

The New Way

So, with the new class hierarchy, we should also have a more formal splitting up of configs and schemas, and then let core adjust to what makes sense for common.

First, we separate the "config" portion, which defines the data source and/or destination.
In particular, each StorageConnector will take a DataStoreConfig that has a location and credential to access the storage. DataStoreConfig will be a base class, though not necessarily abstract, with subclasses written to pair with individual, specific StorageConnector subclasses, which will check for the appropriate DataStoreConfig subclass.
That way, we can support configs being specific to storage type.

Credential Configs

For the "credential" element of a config schema, we'll have a CredentialConfig base class, with subclasses for PasswordCredentialConfig containing a username-password pair, as well as a KeyCredentialConfig with just a path to a key file. These can be expanded as needed.

Location Configs

For the "location" element of a config schema, we'll have a LocationConfig base class, which has just a "location" string. Subclasses include HostLocationConfig, which will add a port and a "host" prop mapped to location var; and FileLocationConfig, which will add a folder path, call "location" the file path, but treat combined path as the true "location." Or something like that.

Overview of New Storage Configs

The resulting diagram looks like:

classDiagram
DataStoreConfig *-- "1" CredentialConfig 
DataStoreConfig *-- "1" LocationConfig 
CredentialConfig <|-- PasswordCredentialConfig 
CredentialConfig <|-- KeyCredentialConfig 
LocationConfig <|-- HostLocationConfig 
LocationConfig <|-- FileLocationConfig 
StorageConnector *-- "1" DataStoreConfig 

Loading

Interface Schemas

So, if the DataStoreConfigs have the information to connect to a storage location, the actual interface/outerface classes need information to locate the necessary data within that location.
These will be "schemas" in the sense defined by #64, as they describe the structure of a resource, rather than configuring how/where to connect to the resource.
For that, we'll have a TableSchema that includes both a TableStructureSchema and a TableLocationSchema.

Table Structure Schemas

The TableStructureSchema is the thing currently called TableSchema.
It will have subclasses for EventTableStructureSchema and FeatureTableStructureSchema, which look for event-specific and feature-specific columns.

Table Location Schemas

A TableLocationSchema has whatever information is needed to locate the table within the given resource.
The main subclass would be DatabaseTableLocationSchema, which would have a database and a table.

classDiagram
TableSchema *-- "1" TableStructureSchema
TableSchema *-- "1" TableLocationSchema
EventTableStructureSchema <|-- TableStructureSchema
FeatureTableStructureSchema <|-- TableStructureSchema
TableLocationSchema <|-- DatabaseTableLocationSchema

Loading

Configuration Wrapper

All of the various configs and schemas can be wrapped in a GameDataConfig or something, which has schemas defining all the input (or all the output) info for a given game. This means up to two TableSchemas, where one is for events and the other for features.

[LswaN58]

Note on naming: Proposal #63 would rename a number of the schema classes described here to "configs."

[LswaN58]

Updated based on #63 and changes from #64