Code

Carlton file connector#5

Code

Draft

matt2000 wants to merge 17 commits intomastertribune/cranial-messaging:masterfrom

Carlton-file-connectortribune/cranial-messaging:Carlton-file-connectorCopy head branch name to clipboard

Commits

Commits on Jul 2, 2019

• Implemented a file connector based of local to work with S3 via the smart open package and boto3 for reading in multiple keys in a bucket.

  CarltonS13

  committed
  4d49bd5

  Copy full SHA for 4d49bd5

• Initial version of File Connector. To be used with S3

  CarltonS13

  committed
  69432f3

  Copy full SHA for 69432f3

• Modified the file messenger to rely on the newly implemented file connector

  CarltonS13

  committed
  4994551

  Copy full SHA for 4994551

Commits on Jul 3, 2019

• Implemented get_last_id in messagin.file to be used in pipe to get the

  Show description for b02e515
  CarltonS13

  committed
  b02e515

  Copy full SHA for b02e515

• Created an S3 notifier for easier modification and testing with pipe.

  Show description for 94e40b5
  CarltonS13

  committed
  94e40b5

  Copy full SHA for 94e40b5

Commits on Jul 8, 2019

• Commented out parsers code in connectors so that file could inherit it

  Show description for 06c836a
  CarltonS13

  committed
  06c836a

  Copy full SHA for 06c836a

• Removed unnecessary comments from s3

  Show description for 827cfce
  CarltonS13

  committed
  827cfce

  Copy full SHA for 827cfce

• Merge branch 'master' into file-connector

  CarltonS13

  authored
  4810c29

  Copy full SHA for 4810c29

Commits on Jul 9, 2019

• Initial work on supporting files stored under a date prefix

  CarltonS13

  committed
  6a198eb

  Copy full SHA for 6a198eb

• More work on supporting files stored under a date prefix

  CarltonS13

  committed
  ef86434

  Copy full SHA for ef86434

Commits on Jul 11, 2019

• Clean-up work.

  matt2000

  committed
  635ca33

  Copy full SHA for 635ca33

Commits on Jul 16, 2019

• Add Performance warning comment.

  matt2000

  committed
  96c494b

  Copy full SHA for 96c494b

• Refactoring & de-duplication for File Connector.

  matt2000

  committed
  8885e09

  Copy full SHA for 8885e09

Commits on Jul 23, 2019

• Merge remote-tracking branch 'origin/master' into Carlton-file-connector

  matt2000

  committed
  b002f78

  Copy full SHA for b002f78

• Deprecation warnings.

  matt2000

  committed
  2090547

  Copy full SHA for 2090547

• Pluggable executor.

  matt2000

  committed
  62c872a

  Copy full SHA for 62c872a

• Move get_last_id to Connector. WIP.

  matt2000

  committed
  07ac7df

  Copy full SHA for 07ac7df