This repository was archived by the owner on Mar 13, 2020. It is now read-only.
Add new commands - 'compare' and 'complete' data pipeline execution
Pre-release
Pre-release
·
134 commits
to master
since this release
New commands:
compare
: Compares & persists SHA256-hashed checksums of the given models against those of the last successful execution. Returns comma-separated string of changed model names. Parameters required:execution-id
: a GUID identifier of an existing data pipeline execution as returned by theinit
command.model-type
: type of models being processed e.g.:load
,transform
, etc. thismodel-type
is used to group the model checksums by and used to find and compare older ones.base-path
: absolute or relative path to the models e.g.:./load
,/home/local/load
,C:/path/to/load
model-patterns
: path-based patterns (relative tobase-path
) to different models with extensions. models within a model-type must be named uniquely regardless of their file extension. e.g.:*.txt
,**/*.txt
,./relative/path/to/some_models/**/*.csv
,relative/path/to/some/more/related/models/**/*.sql
complete
: Marks the completion of an existing execution by updating a record for the same in the given database. Returns nothing unless there's an error. Parameter required:execution-id
: a GUID identifier of an existing data pipeline execution as returned by theinit
command.