Skip to content

Dev #106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 31 commits into from
Jan 8, 2025
Merged

Dev #106

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
00acf01
Add custom properties in roles
simonharrer Oct 20, 2024
180217d
Merge pull request #94 from bitol-io/dev-customProperties-in-roles
jgperrin Oct 21, 2024
7e6a312
Status is not related to dev/prod
simonharrer Nov 4, 2024
43f43fc
Merge pull request #96 from bitol-io/dev-status
pflooky Nov 4, 2024
b2876a7
Add three new fields:
simonharrer Nov 11, 2024
88611d5
Add to full example
simonharrer Nov 11, 2024
d447868
Introduce examples for status
simonharrer Nov 11, 2024
7966e0b
Merge pull request #98 from bitol-io/dev-authoriativedefs
simonharrer Nov 11, 2024
8616984
Update docs/README.md
simonharrer Nov 12, 2024
67bad52
Update odcs-json-schema-latest.json
simonharrer Nov 12, 2024
5893a67
Merge branch 'dev' into dev-betterstatus
simonharrer Nov 12, 2024
e5bb855
Merge pull request #99 from bitol-io/dev-betterstatus
simonharrer Nov 12, 2024
2ebe21b
Kafka schema examples
dirkvandepoel Dec 5, 2024
12c28ab
Add Meta Analysis to list of vendors
msorel-meta Dec 20, 2024
ecd40ad
Update vendors.md - Meta Analysis Link
msorel-meta Dec 20, 2024
51a73ac
Update vendors.md - formating
msorel-meta Dec 20, 2024
0818280
Merge pull request #101 from bitol-io/dev-kafkaexamples
pflooky Dec 22, 2024
5d93d5d
Create new JSON schema for v3.0.1, update changelog, update examples …
pflooky Dec 22, 2024
f60b9d3
Run github action check on pull request
pflooky Dec 22, 2024
ccd12d8
Merge pull request #103 from msorel-meta/main
pflooky Dec 22, 2024
5886348
Update validate examples script to use version v3.0.1
pflooky Dec 22, 2024
57ff39e
Update full example to use array of authoritativeDefinitions
pflooky Dec 22, 2024
c07474a
add sample tags in contract readme examples
dccakes Dec 23, 2024
708dc46
add tags to psql example
dccakes Dec 23, 2024
e6c89c7
add tags to README & update tags description
dccakes Dec 23, 2024
4b45841
Merge branch 'dev' into issue-add-tags-to-readme
pflooky Dec 23, 2024
aa68783
update tag description to schema
dccakes Dec 23, 2024
58d932e
Merge pull request #105 from bitol-io/issue-add-tags-to-readme
pflooky Dec 27, 2024
ce76d0b
Merge branch 'dev' into prepare-release-v3.0.1
pflooky Dec 27, 2024
e375f66
Updated changelog to reflect changes to JSON schema, moved tags updat…
pflooky Jan 3, 2025
e645ae9
Merge pull request #104 from pflooky/prepare-release-v3.0.1
pflooky Jan 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/validate-examples.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
name: validate-examples
on:
pull_request:
branches: ["*"]
push:
branches: ["*"]

Expand Down
10 changes: 10 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,16 @@ image: "https://raw.githubusercontent.com/bitol-io/artwork/main/horizontal/color

This document tracks the history and evolution of the **Open Data Contract Standard**.

# v3.0.1 - 2024-12-22 - APPROVED

* Added field `authoritativeDefinitions` into JSON schema
* Added field `description.customProperties` into JSON schema
* Added field `description.authoritativeDefinitions` into JSON schema
* Added field `role.customProperties` into JSON schema
* Updated `status` field to include examples
* Updated `authoritativeDefinitions` description to be vendor agnostic
* Updated `tags` description and included examples

# v3.0.0 - 2024-10-21 - APPROVED

* **New section**: Support & communication channels.
Expand Down
65 changes: 35 additions & 30 deletions docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,13 +43,13 @@ This section contains general information about the contract.
### Example

```YAML
apiVersion: v3.0.0 # Standard version
apiVersion: v3.0.1 # Standard version
kind: DataContract

id: 53581432-6c55-4ba2-a65f-72344a91553a
name: seller_payments_v1
version: 1.1.0 # Data Contract Version
status: production
status: active
domain: seller
dataProduct: payments
tenant: ClimateQuantumInc
Expand All @@ -59,26 +59,30 @@ description:
limitations: null
usage: null

tags: null
tags: ['finance']
```

### Definitions

| Key | UX label | Required | Description |
|-------------------------|------------------|----------|------------------------------------------------------------------------------------------------|
| apiVersion | Standard version | Yes | Version of the standard used to build data contract. Default value is `v3.0.0`. |
| kind | Kind | Yes | The kind of file this is. Valid value is `DataContract`. |
| id | ID | Yes | A unique identifier used to reduce the risk of dataset name collisions, such as a UUID. |
| name | Name | No | Name of the data contract. |
| version | Version | Yes | Current version of the data contract. |
| status | Status | Yes | Current status of the data contract. Valid values are `production`, `test`, or `development`. |
| tenant | Tenant | No | Indicates the property the data is primarily associated with. Value is case insensitive. |
| domain | Domain | No | Name of the logical data domain. |
| dataProduct | Data Product | No | Name of the data product. |
| description | Description | No | Object containing the descriptions. |
| description.purpose | Purpose | No | Intended purpose for the provided data. |
| description.limitations | Limitations | No | Technical, compliance, and legal limitations for data use. |
| description.usage | Usage | No | Recommended usage of the data. |
| Key | UX label | Required | Description |
|--------------------------------------|---------------------------|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| apiVersion | Standard version | Yes | Version of the standard used to build data contract. Default value is `v3.0.1`. |
| kind | Kind | Yes | The kind of file this is. Valid value is `DataContract`. |
| id | ID | Yes | A unique identifier used to reduce the risk of dataset name collisions, such as a UUID. |
| name | Name | No | Name of the data contract. |
| version | Version | Yes | Current version of the data contract. |
| status | Status | Yes | Current status of the data contract. Examples are "proposed", "draft", "active", "deprecated", "retired". |
| tenant | Tenant | No | Indicates the property the data is primarily associated with. Value is case insensitive. |
| domain | Domain | No | Name of the logical data domain. |
| dataProduct | Data Product | No | Name of the data product. |
| authoritativeDefinitions | Authoritative Definitions | No | List of links to sources that provide more details on the data contract. |
| description | Description | No | Object containing the descriptions. |
| description.purpose | Purpose | No | Intended purpose for the provided data. |
| description.limitations | Limitations | No | Technical, compliance, and legal limitations for data use. |
| description.usage | Usage | No | Recommended usage of the data. |
| description.authoritativeDefinitions | Authoritative Definitions | No | List of links to sources that provide more details on the dataset; examples would be a link to privacy statement, terms and conditions, license agreements, data catalog, or another tool. |
| description.customProperties | Custom Properties | No | Custom properties that are not part of the standard. |



## Schema
Expand Down Expand Up @@ -112,7 +116,7 @@ schema:
type: businessDefinition
- url: https://youtu.be/jbY1BKFj9ec
type: videoTutorial
tags: null
tags: ['finance']
dataGranularityDescription: Aggregation on columns txn_ref_dt, pmt_txn_id
properties:
- name: txn_ref_dt
Expand Down Expand Up @@ -214,16 +218,16 @@ schema:

#### Applicable to Elements (either Objects or Properties)

| Key | UX label | Required | Description |
|--------------------------|------------------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| name | Name | Yes | Name of the element. |
| physicalName | Physical Name | No | Physical name. |
| description | Description | No | Description of the element. |
| businessName | Business Name | No | The business name of the element. |
| authoritativeDefinitions | Authoritative Definitions | No | List of links to sources that provide more details on the table; examples would be a link to an external definition, a training video, a GitHub repo, Collibra, or another tool. See `authoritativeDefinitions` below. |
| quality | Quality | No | List of data quality attributes. |
| tags | Tags | No | A list of tags that may be assigned to the elements (object or property); the tags keyword may appear at any level. |
| customProperties | Custom Properties | No | Custom properties that are not part of the standard. |
| Key | UX label | Required | Description |
|--------------------------|------------------------------|----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| name | Name | Yes | Name of the element. |
| physicalName | Physical Name | No | Physical name. |
| description | Description | No | Description of the element. |
| businessName | Business Name | No | The business name of the element. |
| authoritativeDefinitions | Authoritative Definitions | No | List of links to sources that provide more details on the element; examples would be a link to privacy statement, terms and conditions, license agreements, data catalog, or another tool. |
| quality | Quality | No | List of data quality attributes. |
| tags | Tags | No | A list of tags that may be assigned to the elements (object or property); the tags keyword may appear at any level. Tags may be used to better categorize an element. For example, `finance`, `sensitive`, `employee_record`. |
| customProperties | Custom Properties | No | Custom properties that are not part of the standard. |

#### Applicable to Objects

Expand Down Expand Up @@ -643,6 +647,7 @@ roles:
| roles.access | Access | No | The type of access provided by the IAM role. |
| roles.firstLevelApprovers | 1st Level Approvers | No | The name(s) of the first-level approver(s) of the role. |
| roles.secondLevelApprovers | 2nd Level Approvers | No | The name(s) of the second-level approver(s) of the role. |
| roles.customProperties | Custom Properties | No | Any custom properties. |


## Service-Level Agreement (SLA)
Expand Down Expand Up @@ -733,7 +738,7 @@ servers:
- **description**: A description of the server.
- **environment**: The environment where the server operates (e.g., `prod`, `dev`, `uat`). There are no set values.
- **roles**: An optional array of roles that have access to the server.
- **customProperties**: Any additional custom properties specific to the server.
- **customProperties**: Any additional custom properties specific to the server that are not part of the standard.

### Specific Server Properties

Expand Down
11 changes: 7 additions & 4 deletions docs/examples/all/full-example.odcs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,21 @@
domain: seller # Domain
dataProduct: my quantum # Data product name
version: 1.1.0 # Version (follows semantic versioning)
status: current
status: active
id: 53581432-6c55-4ba2-a65f-72344a91553a

# Lots of information
description:
purpose: Views built on top of the seller tables.
limitations: Data based on seller perspective, no buyer information
usage: Predict sales over time
authoritativeDefinitions:
- type: privacy-statement
url: https://example.com/gdpr.pdf
tenant: ClimateQuantumInc

kind: DataContract
apiVersion: v3.0.0 # Standard version (follows semantic versioning)
apiVersion: v3.0.1 # Standard version (follows semantic versioning)

# Infrastructure & servers
servers:
Expand All @@ -35,7 +38,7 @@ schema:
type: businessDefinition
- url: https://youtu.be/jbY1BKFj9ec
type: videoTutorial
tags: [ ]
tags: [ 'finance', 'payments']
dataGranularityDescription: Aggregation on columns txn_ref_dt, pmt_txn_id
properties:
- name: txn_ref_dt
Expand Down Expand Up @@ -74,7 +77,7 @@ schema:
partitioned: false
partitionKeyPosition: -1
criticalDataElement: false
tags: [ ]
tags: [ 'uid' ]
classification: restricted
- name: rcvr_cntry_code
primaryKey: false
Expand Down
6 changes: 4 additions & 2 deletions docs/examples/all/postgresql-adventureworks-contract.odcs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
version: "1.0.0"
apiVersion: "v3.0.0"
id: "6aeafdc1-ed62-4c8f-bf0a-da1061c98cdb"
status: "Development"
status: active
kind: "DataContract"
description: {}
schema:
schema:
- name: "department"
physicalName: "department"
physicalType: "table"
Expand Down Expand Up @@ -82,6 +82,7 @@ schema:
criticalDataElement: false
primaryKey: false
required: false
tags: ['sensitive']
- name: "loginid"
logicalType: "string"
physicalType: "varchar[256]"
Expand Down Expand Up @@ -115,6 +116,7 @@ schema:
criticalDataElement: false
primaryKey: false
required: false
tags: ['sensitive']
- name: "maritalstatus"
logicalType: "string"
physicalType: "bpchar"
Expand Down
4 changes: 2 additions & 2 deletions docs/examples/data-types/all-data-types.odcs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
version: 1.0.0
kind: DataContract
id: 53581432-6c55-4ba2-a65f-72344a91553a
status: current
status: active
name: my_table
dataProduct: my_quantum
apiVersion: v3.0.0
apiVersion: v3.0.1
schema:
- name: transactions_tbl
description: Provides core payment metrics
Expand Down
4 changes: 2 additions & 2 deletions docs/examples/fundamentals/table-column-description.odcs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
version: 1.0.0
kind: DataContract
id: 53581432-6c55-4ba2-a65f-72344a91553a
status: current
status: active
name: my_quantum
apiVersion: v3.0.0
apiVersion: v3.0.1
schema:
- name: tbl
description: Provides core payment metrics
Expand Down
4 changes: 2 additions & 2 deletions docs/examples/quality/column-accuracy.odcs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
version: 1.0.0
kind: DataContract
id: 53581432-6c55-4ba2-a65f-72344a91553a
status: current
status: active
name: my_table
dataProduct: my_quantum
apiVersion: v3.0.0
apiVersion: v3.0.1
schema:
- name: Air_Quality
description: Air quality of the city of New York
Expand Down
7 changes: 4 additions & 3 deletions docs/examples/quality/column-completeness.odcs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
version: 1.0.0
apiVersion: v3.0.0
apiVersion: v3.0.1
kind: DataContract
id: 53581432-6c55-4ba2-a65f-72344a91553a
status: current
status: active
name: my_table
dataProduct: my_quantum
schema:
Expand All @@ -13,6 +13,7 @@ schema:
type: Reference definition on Data.gov
dataGranularityDescription: Raw records
physicalType: table
tags: ['nyc', 'regional']
properties:
- name: UniqueID
primaryKey: true
Expand All @@ -24,4 +25,4 @@ schema:
dimension: completeness
severity: error
rule: nullCheck
businessImpact: operational
businessImpact: operational
4 changes: 2 additions & 2 deletions docs/examples/quality/column-custom.odcs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
version: 1.0.0
apiVersion: v3.0.0
apiVersion: v3.0.1
kind: DataContract
id: 53581432-6c55-4ba2-a65f-72344a91553a
status: current
status: active
name: my_table
dataProduct: my_quantum
schema:
Expand Down
4 changes: 2 additions & 2 deletions docs/examples/quality/column-validity.odcs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
version: 1.0.0
apiVersion: v3.0.0
apiVersion: v3.0.1
kind: DataContract
id: 53581432-6c55-4ba2-a65f-72344a91553a
status: current
status: active
name: my_table
dataProduct: my_quantum
schema:
Expand Down
Loading
Loading