Designed by Agile Lab, Witboost is a versatile platform that addresses a wide range of sophisticated data engineering challenges. It enables businesses to discover, enhance, and productize their data, fostering the creation of automated data platforms that adhere to the highest standards of data governance. Want to know more about Witboost? Check it out here or contact us!
This repository is part of our Starter Kit meant to showcase Witboost's integration capabilities and provide a "batteries-included" product.
Use this template to create a Storage Area on a BigQuery instance. The Storage Area is implemented as a single table, used to store data generated or consumed by the System's workloads.
Refer to the Witboost Starter Kit repository for information on the Tech Adapter that can be used to deploy components created with this template.
A Template is a tool that helps create components inside a Data Mesh. Templates help establish a standard across the organization. This standard leads to easier understanding, management and maintenance of components. Templates provide a predefined structure so that developers don't have to start from scratch each time, which leads to faster development and allows them to focus on other aspects, such as testing and business logic.
For more information, please refer to the official documentation.
This template uses Skeleton Entities. To instantiate it, a version of Witboost greater or equal to 2.3 is required.
Skeleton Entities provide a more dynamic and user-friendly approach to define systems and components. They seamlessly integrate with tools like the Editor Wizard and the Reverse Provisioning Wizard, allowing for easier entity management.
For more information, please refer to the official documentation.
A Storage Area is what the components of a System use to persist data that is created or used by the various Workloads or served by Output Ports.
BigQuery is a fully-managed, serverless cloud data warehouse offered by Google Cloud, designed to enable fast SQL queries and interactive analysis of large datasets. It abstracts infrastructure concerns, providing scalability, security, and ease of use for analysts and engineers alike. Key features include:
- Serverless and Fully Managed: BigQuery automatically handles infrastructure, performance tuning, and resource provisioning, allowing users to focus on data and insights rather than operations.
- Separation of Storage and Compute: Like modern cloud warehouses, BigQuery separates compute and storage, enabling flexible scaling and cost optimization based on actual usage.
- Standard SQL Support: It supports ANSI SQL, making it accessible to users familiar with datasets.
- High Performance and Scalability: BigQuery can process petabytes of data quickly using Dremel-based execution and distributed computing under the hood.
- Integrated Machine Learning (BigQuery ML): Users can create and execute machine learning models directly within BigQuery using SQL, reducing the need to move data across services.
- Security and Access Control: Provides fine-grained access control via IAM, support for column-level security, and data encryption both at rest and in transit.
- Streaming and Real-Time Analytics: BigQuery allows streaming data inserts, enabling near real-time analytics without complex ingestion pipelines.
- Time Travel: Enables querying of historical data (up to 7 days) for rollback, recovery, or auditing purposes.
- Data Sharing and Federation: Allows secure data sharing across projects and organizations, and querying external data sources like Google Sheets, Cloud Storage, and Bigtable without ingestion.
Learn more about it on the official website.
To get information on how to use this template, refer to this document.
To verify the component before deploying it along with the System, the component needs to be tested against a CUE Policy defined for BigQuery Storage Area. This policy needs to be defined inside the Governance section of the Witboost Platform.
For more information, please refer to the official documentation.
This project is available under the Apache License, Version 2.0; see LICENSE for full details.
Witboost is a cutting-edge Data Experience platform, that streamlines complex data projects across various platforms, enabling seamless system production and consumption. This unified approach empowers you to fully utilize your data without platform-specific hurdles, fostering smoother collaboration across teams.
It seamlessly blends business-relevant information, data governance processes, and IT delivery, ensuring technically sound data projects aligned with strategic objectives. Witboost facilitates data-driven decision-making while maintaining data security, ethics, and regulatory compliance.
Moreover, Witboost maximizes data potential through automation, freeing resources for strategic initiatives. Apply your data for growth, innovation and competitive advantage.
Contact us or follow us on: