v1.0.1

[antoniivanov]

Major features include:

Secrets Service Helm Chart installation

Vault integration configuration for storing Data Job Secrets has been added to the Helm chart:

secrets:
    vault:
        enabled: false
        uri: "http://localhost:8200"
        
        externalSecretName: ""
        ## Alternatively provide the uri and Approle Settings here. externalSecretName takes precedence if both are set.
        approle:
            roleid: foo
            secretid: foo

        sizeLimitBytes: "1048576"

VDK Secrets CLI

Job secrets are used to store credentials/tokens/sensitive data securely. They can be updated using vdk-control-cli now:

Install vdk-control-cli if needed (it comes pre-installed in quickstart-vdk)

pip install vdk-control-cli 

vdk secrets --help

For example:

     # Set single secret with key "my-key" and value "my-value". If no value is passed you'll get prompted so it's not printed on the screen.
     vdk secrets --set my-key "my-value"

     # Update multiple secrets at once.
     vdk secrets --set "key1" "value1" --set "key2" "value2" --set "secret1" --set "secret2"

Convert Directory-style To Notebook-style Data Job

With the introduction of Notebook-style data jobs, the user has the option to Convert Directory-style to Notebook-style Data Job.

VDK Menu	Pop-up window	Notebook

VDK Jupyter Extension published in PyPi

Users can now install the Jupyter extension with VDK in their own Python and jupyter environment with a single line :

pip install vdk-jupyterlab-extension

Then start Jupyter lab as usual:

jupyter lab

Users can now see the notebook:

Notebook

New plugin: vdk-smarter

VDK Smarter introduces proof of concept (pre-alpha) integration with OpenAI.

In the POC it does a review of all SQL queries managed by VDK.

For more details see the plugin home page

What's Changed

• control-service: Add helm chart entries for Vault Configuation by @dakodakov in control-service: Add helm chart entries for Vault Configuation #2418
• control-service: Update contributing.md with correct java requirements by @danail-georgiev in control-service: Update contributing.md with correct java requirements #2430
• control-service: add configurable smtp host property by @mrMoZ1 in control-service: add configurable smtp host property #2411
• control-service: add helm template for alertmanager by @mrMoZ1 in control-service: add helm template for alertmanager #2326
• control-service: add timestamps to helm chart by @DeltaMichael in control-service: add timestamps to helm chart #2344
• control-service: better error logging for failed test by @murphp15 in control-service: better error logging for failed test #2374
• control-service: fix helm chart by @dakodakov in control-service: fix helm chart #2449
• control-service: fix publish-job-base-image script by @mivanov1988 in control-service: fix publish-job-base-image script #2473
• control-service: fix typo in helm chart read only root filesystem property by @mrMoZ1 in control-service: fix typo in helm chart read only root filesystem property #2476
• control-service: install necessary dependencies to job builder secure by @mivanov1988 in control-service: install necessary dependencies to job builder secure #2472
• control-service: job-builder using kaniko fix by @tozka in control-service: job-builder using kaniko fix #2429
• control-service: job-builder-secure using kaniko fix by @tozka in control-service: job-builder-secure using kaniko fix #2447
• control-service: logs endpoint doesn't hang by @murphp15 in control-service: logs endpoint doesn't hang #2370
• control-service: prevent integer translation in helm chart by @dakodakov in control-service: prevent integer translation in helm chart #2470
• control-service: push to multiple registries by @tozka in control-service: push to multiple registries #2381
• control-service: release job builder in 2 repos by @tozka in control-service: release job builder in 2 repos #2413
• control-service: remove default vault token by @dakodakov in control-service: remove default vault token #2475
• control-service: remove unused dependency influxdb by @tozka in control-service: remove unused dependency influxdb #2388
• control-service: run integration tests on multiple namespace. by @murphp15 in control-service: run integration tests on multiple namespace. #2446
• control-service: set Execution and JobQuery APIs to stable by @tozka in control-service: set Execution and JobQuery APIs to stable #2417
• control-service: split build job base image CI/CD step by @mivanov1988 in control-service: split build job base image CI/CD step #2348
• control-service: switch to Approle Vault authentication by @dakodakov in control-service: switch to Approle Vault authentication #2435
• control-service: use full url for heartbeat tests and heartbeat tests run in multiple namespaces by @murphp15 in control-service: use full url for heartbeat tests and heartbeat tests run in multiple namespaces #2295
• frontend: Fix navigation in Data Jobs by @gorankokin in frontend: Fix navigation in Data Jobs #2356
• frontend: Fix router event handling in base class by @gorankokin in frontend: Fix router event handling in base class #2375
• frontend: bump toolchain versions in frontend build docker image by @DeltaMichael in frontend: bump toolchain versions in frontend build docker image #2358
• frontend: enable stable tagging by @DeltaMichael in frontend: enable stable tagging #2378
• frontend: fix data-pipelines build scripts by @DeltaMichael in frontend: fix data-pipelines build scripts #2389
• frontend: push docker images to both repos by @tozka in frontend: push docker images to both repos #2390
• frontend: quickstart-vdk operability tests using cypress by @DeltaMichael in frontend: quickstart-vdk operability tests using cypress #2359
• frontend: remove e2e tests restrictions by @DeltaMichael in frontend: remove e2e tests restrictions #2386
• support: slack notification on pipeline failure by @DeltaMichael in support: slack notification on pipeline failure #2338
• vdk-control-cli: add vdk secrets command by @dakodakov in vdk-control-cli: add vdk secrets command #2342
• vdk-control-cli: add vdk secrets command by @dakodakov in vdk-control-cli: add vdk secrets command #2357
• vdk-control-cli: remove set-secret for properties by @dakodakov in vdk-control-cli: remove set-secret for properties #2409
• vdk-core: Allow different python versions for vdk docker images by @doks5 in vdk-core: Allow different python versions for vdk docker images #2346
• vdk-core: Set sender when checking if email exists by @doks5 in vdk-core: Set sender when checking if email exists #2376
• vdk-core: [Hot Fix] Stop throwing exceptions if config.ini not present by @doks5 in vdk-core: [Hot Fix] Stop throwing exceptions if config.ini not present #2367
• vdk-heartbeat: cover requirements.txt automatic installs by @tozka in vdk-heartbeat: cover requirements.txt automatic installs #2393
• vdk-impala: Truncate table before inserting data by @sbuldeev in vdk-impala: Truncate table before inserting data #2369
• vdk-impala: Update README.md for vdk-impala by @sbuldeev in vdk-impala: Update README.md for vdk-impala #2355
• vdk-impala: support also pydantic 1.0 by @tozka in vdk-impala: support also pydantic 1.0 #2368
• vdk-impala: upgrade code to support pydantic 2.0 by @tozka in vdk-impala: upgrade code to support pydantic 2.0 #2362
• vdk-ipython: README.md fix by @duyguHsnHsn in vdk-ipython: README.md fix #2345
• vdk-jupyter: fix server error in jupyter ui and remove unneeded code by @duyguHsnHsn in vdk-jupyter: fix server error in jupyter ui and remove unneeded code #2361
• vdk-jupyter: Add a message describing how to contact the Jupyter devs by @gageorgiev in vdk-jupyter: Add a message describing how to contact the Jupyter devs #2414
• vdk-jupyter: Create init cell when opening new notebook by @gageorgiev in vdk-jupyter: Create init cell when opening new notebook #2352
• vdk-jupyter: Sample job notebook step by @gageorgiev in vdk-jupyter: Sample job notebook step #2364
• vdk-jupyter: add Convert Job To Notebook UI button by @yonitoo in vdk-jupyter: add Convert Job To Notebook UI button #2329
• vdk-jupyter: convert job operation by @duyguHsnHsn in vdk-jupyter: convert job operation #2406
• vdk-jupyter: publish image to pip registry by @murphp15 in vdk-jupyter: publish image to pip registry #2407
• vdk-jupyter: remove delete operation by @duyguHsnHsn in vdk-jupyter: remove delete operation #2428
• vdk-plugin-control-cli: add secrets command by @dakodakov in vdk-plugin-control-cli: add secrets command #2387
• vdk-plugins: fix build of multiple plugins by @tozka in vdk-plugins: fix build of multiple plugins #2445
• vdk-plugins: include Ingestion hooks documentation by @tozka in vdk-plugins: include Ingestion hooks documentation #2416
• vdk-server: url in swagger ui is correct by @murphp15 in vdk-server: url in swagger ui is correct #2380
• vdk-smarter: introduce simple open ai plugin by @tozka in vdk-smarter: introduce simple open ai plugin #2351
• versatile-data-kit: clarify structure of VDK repo by @tozka in versatile-data-kit: clarify structure of VDK repo #2432
• versatile-data-kit: remove gitlint pre-commit hook by @tozka in versatile-data-kit: remove gitlint pre-commit hook #2360

New Contributors

• @danail-georgiev made their first contribution in control-service: Update contributing.md with correct java requirements #2430

Full Changelog: v1.0...v1.0.1

---

This discussion was created from the release v1.0.1.