Virtual Data Warehouse v.1.6.5.3

Updated DataWarehouseAutomation library so that the {{stringwrap}} function can be used in the templating engine.

Virtual Data Warehouse v.1.6.5.2

Bug fix, containing reported issues. No material changes, just tidying up.

Virtual Data Warehouse v.1.6.5.1

Bug fix release to support ongoing projects.

The main improvement is the complete removal of custom Handlebars functions so that everything is used from the DataWarehouseAutomation classlibrary.

Also, all patterns have been tested (and updated) to adjust to templating engine changes.

All libraries have been updated to the latest version.

Virtual Data Warehouse v.1.6.5

This is a minor bug-fix release to align with TEAM v.1.6.3, primarily to include the updated examples and very slight pattern fixes for the Staging Layer.

It also installs in a more convenient directory ;).

Virtual Data Warehouse v.1.6.4

The Virtual Data Warehouse v.1.6.4. release contains some more quality-of-life fixes such as delays on the metadata refresh (this can consume the application when large amounts of metadata are refreshed), better logging and exception handling.

Also, some of the examples have been updated and the installer has been improved. Some of the libraries that were causing issues for some users are now deployed automatically and all references have been updated to the latest versions.

Virtual Data Warehouse v.1.6.3

Virtual Data Warehouse v.1.6.3. is effectively a bug-fix release that adds better exception handling in various places, especially when metadata (Json) files do not match the expected format.

This is now handled gracefully and warnings or errors are visible in the event log. Also, defaults are put in place to allow code generation for metadata that does confirm to the Data Warehouse Automation schema but is missing some details that VDW needs to build the user interface (e.g. mapping name, classification).

Lastly, a new feature was added to monitor the input directory (where the metadata files reside) on changes and update the user interface when metadata is modified, added or removed.

Virtual Data Warehouse v.1.6.2

The Virtual Data Warehouse (VDW) release 1.6.2. is a major revision / upgrade which marks the first true 'database-less' release. No database or repository is needed to start generating code!

Because of a greater adoption of patterns, the existing built-in screens have been removed. This simplifies the menu to only file management and help. Everything else is dynamically generated from the available patterns.

Changes for this release:

• Upgraded to be compatible with version 1.2 of the schema for Data Warehouse Automation
• All patterns updated for compliance with the above
• Removal of Test Data, Referential Integrity and Point-in-Time (delivery) screens. These have been replaced by patterns / templates.
• Quality of life improvements such as tooltips, better auto aligning of grids and sizing of GUI / screen controls. It just looks and works better and easier.
• Sample library 'examples' containing the sample 'profiler' data warehouse provided upon installation. No need to first use TEAM to generate examples.
• Added a few custom functions to the templating engine: stringcompare, stringdiff, now, randomstring, randomnumber, randomdate.
• Feature to reuse existing TEAM connection files, and displaying the connection key on each pattern screen. Connections can be assigned to patterns.
• Event log and screen to view it.
• Feature to select, change and load TEAM environments and automatically upload the associated settings.

VDW v.1.6.1

The most recent release for Virtual Data Warehouse this is 1.6.1.

These updates have fully decoupled the management of the (source-to-target) mapping metadata with the code generation, making it easier to use other tools for some of the functions if desired and by virtue of this support a bigger ecosystem for open source Data Warehouse Automation.

TEAM now saves the design metadata as Json files that conform to the generic schema for Data Warehouse automation. The Virtual Data Warehouse tool can now be configured to read all (Json) files from a designated directory and apply the templates (patterns) to these files using the provided templating engine.

This means it is possible to use TEAM without VDW and vice-versa. It is also possible to incorporate the schema validation functionality available in the Data Warehouse Automation class library to make sure all files are conform the standard. Last, but not least it makes it easier to create your own patterns using the available metadata without software constraints.

The TEAM software is still geared towards Data Vault use-cases, but due to the ability to tweak the patterns without code changes it can now be better adapted for other applications also. For example, this release was used to generate an Persistent Staging Area for Azure Data Factory!

Because of the pattern engine, VDW is completely agnostic of design approaches.

This approach also makes it easier to integrate the metadata into CI/CD pipelines and version control, because the key design artefacts are now all text based – making it easier to commit changes (differentials).

The TEAM metadata remain the key artefacts to version control because these contain the true design Intellectual Property explaining which data elements go where – mapping and lineage. However, the Json files that are created as part of the ‘activation’ routine can be consumed in a DevOps pipeline for code generation.

Alternatively, complex output from the pattern code generation can be written back into the Json structure as (source) transformations.

Since the Json files conform to the generic schema for Data Warehouse Automation they can also be manually tweaked or even created. This may be useful in case there is no interest in using TEAM, but still intent to use the VDW code generation and patterns, or if custom complex transformation need to be added.

In addition to the above there is a long list of minor usability improvements including, but not limited to:

• Easier interaction with the data grids for data entry. Various context menus have been created to simplify adding, exporting and removing rows.
• Improved validation mechanisms that perform pre-checks before any generation work is started.
• Masking of passwords in the software (note that passwords are still visible in the text files).
• Better exception handling, logging and reporting.
• Usage of external files for many operations, making the tools easier to configure and upgrade. For example – many SQL statements, patterns and lists are now stored in script and Json files and are installed as part of the software.
• Better handling of schemas and tables, for example tables with the same name but different schemas do not cause errors anymore.
• Usability improvements in the GUI, for example updating values when paths change etc.

VDW v1.6.0

Virtual Data Warehouse release v.1.6.0

• All-round improvement of UI and exception handling
• Incorporation of Handlebars templating engine
• Incorporation of Load Patterns for Data Vault
• Custom Tab Page to automatically create UI based on pattern definition
• Compatible with TEAM repository version 1.6, including fixes in RI screen, test data, PIT etc.