Extract data via AWS Lambda & S3

Note that the contract with the AWS Lambda is pre-agreed and a specific concrete implementation

Add ability to extract data via an AWS Lambda

Note that the contract with the AWS Lambda is pre-agreed and a specific concrete implementation

Bug Fix

v0.1.14-beta

Merge pull request #57 from pageuppeople-opensource/fix/undelete-used…

Standardise <Foo>DataSource API Calls

<Foo>DataSource now provide 3 public methods:

• can_handle_connection_string can be used to determine whether or not to use this source for a given connection string
• get_table_info can be used to get table metadata information like the current columns at source and the change tracking information
• get_table_data_frame can be used to get table data and returns a pandas.DataFrame structure

Standardise Schema

• all tables in state-schema now have standard columns
  • GUID primary keys
  • created on and updated on timestamps
• isolate integration test execution user

make the source database user read only

Merges:

[OSC-1317] remove alter table calls to source db (#43)

• remove alter table calls to source db
• add change tracking in integration test create table scripts
• add change tracking in unit test create table scripts

Optimisations

Changes:

• bump dependencies (#38)
• remove csv support (#41)
• Print number of rows processed as an output of process command (#42)
• [OSC-1366] Skip on no changes (#40)
  • skip if there's no changes
  • check if data changed to avoid reloading
  • ensure sync versions are valid before skipping

Fixes v0.1.8-beta

Some of the schema changes in v0.1.8-beta were invalid, please rollback and upgrade now

See #37

Split up execution table

Merges [OSC-1301] Split up execution table (#35)

• creates a table similar to https://github.com/PageUpPeopleOrg/data-pipeline-orchestrator such that there is an overall summary of each execution

Summary-Log fixes

• The current multi-line log message for the execution summary is difficult to search for in logs, so switching back to single-line summary message
• The current unformatted numbers in log are hard to read, so adding a thousand separator to improve read-ablity