Skip to content
This repository was archived by the owner on Mar 13, 2020. It is now read-only.

Add new commands - 'compare' and 'complete' data pipeline execution

Pre-release
Pre-release
Compare
Choose a tag to compare
@ChintanRaval ChintanRaval released this 24 Jan 01:10
· 134 commits to master since this release
5f1ee98

New commands:

  • compare: Compares & persists SHA256-hashed checksums of the given models against those of the last successful execution. Returns comma-separated string of changed model names. Parameters required:
    • execution-id: a GUID identifier of an existing data pipeline execution as returned by the init command.
    • model-type: type of models being processed e.g.: load, transform, etc. this model-type is used to group the model checksums by and used to find and compare older ones.
    • base-path: absolute or relative path to the models e.g.: ./load, /home/local/load, C:/path/to/load
    • model-patterns: path-based patterns (relative to base-path) to different models with extensions. models within a model-type must be named uniquely regardless of their file extension. e.g.: *.txt, **/*.txt, ./relative/path/to/some_models/**/*.csv, relative/path/to/some/more/related/models/**/*.sql
  • complete: Marks the completion of an existing execution by updating a record for the same in the given database. Returns nothing unless there's an error. Parameter required:
    • execution-id: a GUID identifier of an existing data pipeline execution as returned by the init command.