Replicate data from external source to internal source with a Plugin #72
Description
Is your feature request related to a problem? Please describe.
StarbaseDB instances support by default an internal database (SQLite offered by the Durable Object) as well as an optional external data source. External data sources can be powered in one of two ways, both by providing values in the wrangler.toml file of the project.
- Outerbase API Key
- Connection details of the database
Describe the solution you'd like
What would be beneficial for some use cases is the ability to bring in an external data source (e.g. a Postgres on Supabase) and have a pull mechanism where data can be brought into the internal DO SQLite so that the instance serves as a close-to-edge replica that can be queried alternatively to querying the Supabase Postgres instance.
Describe alternatives you've considered
- Considering the pull vs push mechanism. A pull mechanism seems to be a better global solution where a push mechanism would be required to live elsewhere on a per provider basis.
Additional context
- Might be beneficial for users to be able to define in the plugin what intervals data should be pulled at
- Might be beneficial to allow users to define which tables should have data pulled into it (perhaps not all tables need replicated)
- Likely need a way to know for each table what the last queried items were so you can do append-only type polling for new data. Does a user need to define a column to base this on (e.g.
idorcreated_atcolumns perhaps)?