Storage

Non-relational Backends

In order to support data with an arbitrary schema, MongoDB is the primary backend storage technology. MongoDB is a non-relational (“No-SQL”) data store consisting of JSON-formatted documents identified with a unique ID, provided by the MongoDB server running within the PDS. In order to support queries against this data based on time and type of data, each piece of raw data stored in this system has a key denoting the type of data, and a timestamp denoting when the particular sample of data was taken. Given these constraints, relational backends with fixed schemas can also be supported.

Relational Backend Support

To this effect, a SQLInternalDataStore base class has been implemented with corresponding PostgresInternalDataStore and SQLiteInternalDataStore subclasses. Schemas for relational backends are specified in a backend-agnostic manner in order to support the multitude of different SQL dialects, or other query languages, such as SPARQL. A backend-agnostic schema for a table is represented as a Python dictionary with a name, a list of column tuples containing the name and data type for the column, as well as an optional mapping list for retrieving data from a field that does not match the name provided for the column. An example backend-agnostic schema definition for the CallLogProbe table is provided below:

CALL_LOG_TABLE = {
    "name": "CallLogProbe",
    "columns": [
        ("_id", "INTEGER"),
        ("name", "TEXT"),
        ("number", "TEXT"),
        ("number_type", "TEXT"),
        ("date", "BIGINT"),
        ("type", "INTEGER"),
        ("duration", "INTEGER")
    ],
    "mapping": {
        "funf": {
            "number_type": lambda d: d["numbertype"]
        }
    }
}