Skip to content

Add additional fields/stricter requirements to dbt config files #879

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
May 9, 2024
Merged

Add additional fields/stricter requirements to dbt config files #879

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
650aea5
WIP
rlh1994 May 8, 2024
8c259ee
add exclude file?
rlh1994 May 8, 2024
0ba41f9
try this
rlh1994 May 8, 2024
afebbbf
fix only top versions
rlh1994 May 8, 2024
d43b7df
make date required for attribution
rlh1994 May 8, 2024
61a5d92
Add utils release
rlh1994 May 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/validate_schema.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@ jobs:
uses: GrantBirki/json-yaml-validate@v2.6.0
with:
comment: "true" # enable comment mode
json_schema: ./src/components/JsonSchemaValidator/dbt-variables-meta-schema # validate JSON files against the schema
json_schema: ./src/components/JsonSchemaValidator/dbt-variables-meta-schema.json # validate JSON files against the schema
base_dir: ./src/components/JsonSchemaValidator/Schemas/
json_schema_version: "draft-07"
ajv_strict_mode: false
exclude_file: src/components/JsonSchemaValidator/Schemas/exclude_from_checks.txt
2 changes: 1 addition & 1 deletion ...odeling-your-data/modeling-your-data-with-dbt/dbt-configuration/utils/index.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ export const printYamlVariables = (data) => {

export const MyMdxComponent = () => {
const schemaName = 'dbtUtils';
const versions = ['0.16.4', '0.16.3', '0.16.2', '0.16.1', '0.16.0', '0.15.2'];
const versions = ['0.16.5', '0.16.4', '0.16.3', '0.16.2', '0.16.1', '0.16.0', '0.15.2'];
const label = 'dbt utils version';
const output = printYamlVariables;
return (
Expand Down
2 changes: 1 addition & 1 deletion src/componentVersions.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ export const versions = {
dbtSnowplowUnified: '0.4.0',
dbtSnowplowWeb: '1.0.0',
dbtSnowplowMobile: '1.0.0',
dbtSnowplowUtils: '0.16.4',
dbtSnowplowUtils: '0.16.5',
dbtSnowplowMediaPlayer: '0.7.2',
dbtSnowplowNormalize: '0.3.5',
dbtSnowplowFractribution: '0.3.6',
Expand Down
2 changes: 1 addition & 1 deletion src/components/JsonSchemaValidator/Schemas/dbtAttribution_0.2.0.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -323,7 +323,7 @@
"snowplow__attribution_start_date": {
"recommendFullRefresh": true,
"order": 7,
"consoleGroup": "basic",
"consoleGroup": "required",
"type": "string",
"format": "date",
"title": "Attribution Start Date",
Expand Down
34 changes: 27 additions & 7 deletions src/components/JsonSchemaValidator/Schemas/dbtEcommerce_0.8.2.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,15 @@
"title": "SQL & Alias",
"properties": {
"sql": {
"type": "string"
"type": "string",
"title": "Field SQL",
"format": "sql",
"order": 1
},
"alias": {
"type": "string"
"type": "string",
"title": "Field Alias",
"order": 2
}
},
"required": [
Expand Down Expand Up @@ -358,11 +363,15 @@
"properties": {
"schema": {
"type": "string",
"title": "(JSON) schema name for the field",
"order": 1,
"description": "The schema name of your events table, atomic in most use cases, alternatively for sdes/contexts this should instead be the name of the field itself"
},
"field": {
"type": "string",
"description": "The name of the field to use as session identifier, alternatively, in case of sdes/contexts it is the name of the element that refers to the field to be extracted"
"order": 2,
"title": "Field name",
"description": "The name of the field to use as user identifier, alternatively, in case of sdes/contexts it is the name of the element that refers to the field to be extracted"
}
},
"required": [
Expand Down Expand Up @@ -410,10 +419,14 @@
"properties": {
"schema": {
"type": "string",
"title": "(JSON) schema name for the field",
"order": 1,
"description": "The schema name of your events table, atomic in most use cases, alternatively for sdes/contexts this should instead be the name of the field itself"
},
"field": {
"type": "string",
"title": "Field name",
"order": 2,
"description": "The name of the field to use as user identifier, alternatively, in case of sdes/contexts it is the name of the element that refers to the field to be extracted"
}
},
Expand Down Expand Up @@ -514,19 +527,26 @@
"properties": {
"schema": {
"type": "string",
"description": "Table name"
"title": "Table name",
"description": "Table name",
"order": 1
},
"prefix": {
"type": "string",
"description": "Prefix to add to columns"
"title": "Column prefix",
"description": "Prefix to add to columns",
"order": 2
},
"alias": {
"type": "string",
"description": "Table alias for the subquery"
"title": "CTE Alias",
"description": "Table alias for the subquery",
"order": 3
},
"single_entity": {
"type": "boolean",
"title": "Is single entity?"
"title": "Is single entity?",
"order": 4
}
},
"required": [
Expand Down
28 changes: 22 additions & 6 deletions src/components/JsonSchemaValidator/Schemas/dbtMediaPlayer_0.7.2.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -230,6 +230,7 @@
},
"surrogate_key_treat_nulls_as_empty_strings": {
"type": "boolean",
"order": 999,
"title": "Surrogate Key Treat Nulls as Empty Strings",
"group": "Operation and Logic",
"longDescription": "Passed through to `dbt_utils` to match legacy surrogate key behavior.",
Expand All @@ -253,11 +254,15 @@
"properties": {
"schema": {
"type": "string",
"title": "(JSON) schema name for the field",
"order": 1,
"description": "The schema name of your events table, atomic in most use cases, alternatively for sdes/contexts this should instead be the name of the field itself"
},
"field": {
"type": "string",
"description": "The name of the field to use as user identifier, alternatively, in case of sdes/contexts it is the name of the element that refers to the field to be extracted"
"order": 2,
"title": "Field name",
"description": "The name of the field to use as session identifier, alternatively, in case of sdes/contexts it is the name of the element that refers to the field to be extracted"
}
},
"required": [
Expand Down Expand Up @@ -305,10 +310,14 @@
"properties": {
"schema": {
"type": "string",
"title": "(JSON) schema name for the field",
"order": 1,
"description": "The schema name of your events table, atomic in most use cases, alternatively for sdes/contexts this should instead be the name of the field itself"
},
"field": {
"type": "string",
"order": 2,
"title": "Field name",
"description": "The name of the field to use as user identifier, alternatively, in case of sdes/contexts it is the name of the element that refers to the field to be extracted"
}
},
Expand Down Expand Up @@ -512,22 +521,29 @@
"items": {
"type": "object",
"title": "Entity or SDE",
"properties": {
"properties": {
"schema": {
"type": "string",
"description": "Table name"
"title": "Table name",
"description": "Table name",
"order": 1
},
"prefix": {
"type": "string",
"description": "Prefix to add to columns"
"title": "Column prefix",
"description": "Prefix to add to columns",
"order": 2
},
"alias": {
"type": "string",
"description": "Table alias for the subquery"
"title": "CTE Alias",
"description": "Table alias for the subquery",
"order": 3
},
"single_entity": {
"type": "boolean",
"title": "Is single entity?"
"title": "Is single entity?",
"order": 4
}
},
"required": [
Expand Down
55 changes: 43 additions & 12 deletions src/components/JsonSchemaValidator/Schemas/dbtUnified_0.2.0.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,15 @@
"title": "SQL & Alias",
"properties": {
"sql": {
"type": "string"
"type": "string",
"title": "Field SQL",
"format": "sql",
"order": 1
},
"alias": {
"type": "string"
"type": "string",
"title": "Field Alias",
"order": 2
}
},
"required": [
Expand Down Expand Up @@ -159,26 +164,33 @@
"name": {
"type": "string",
"title": "Name",
"description": "Name of your conversion type"
"description": "Name of your conversion type",
"order": 1
},
"condition": {
"type": "string",
"title": "Condition",
"description": "SQL condition e.g. event_name = 'page_view'"
"description": "SQL condition e.g. event_name = 'page_view'",
"format": "sql",
"order": 2
},
"value": {
"type": "string",
"title": "Value",
"description": "SQL value e.g. tr_total_base"
"description": "SQL value e.g. tr_total_base",
"format": "sql",
"order": 3
},
"default_value": {
"type": "number",
"title": "Default value",
"description": "Default value e.g. 0"
"description": "Default value e.g. 0",
"order": 4
},
"list_events": {
"type": "boolean",
"title": "List all event ids?"
"title": "List all event ids?",
"order": 5
}
}
},
Expand Down Expand Up @@ -313,7 +325,8 @@
"packageDefault": "[ ] (no filter applied)",
"group": "Contexts, Filters, and Logs",
"items": {
"type": "string"
"type": "string",
"title": "App ID"
}
},
"snowplow__enable_consent": {
Expand Down Expand Up @@ -467,10 +480,14 @@
"properties": {
"schema": {
"type": "string",
"title": "(JSON) schema name for the field",
"order": 1,
"description": "The schema name of your events table, atomic in most use cases, alternatively for sdes/contexts this should instead be the name of the field itself"
},
"field": {
"type": "string",
"order": 2,
"title": "Field name",
"description": "The name of the field to use as user identifier, alternatively, in case of sdes/contexts it is the name of the element that refers to the field to be extracted"
}
},
Expand All @@ -487,6 +504,7 @@
"order": 47,
"consoleGroup": "advanced",
"type": "string",
"format": "sql",
"title": "SQL for your session identifier",
"longDescription": "This allows you to override the `session_identifiers` SQL, to define completely custom SQL in order to build out a session identifier for your events. If you are interested in using this instead of providing identifiers through the `session_identifiers` variable, please see the [Custom Identifiers](docs/modeling-your-data/modeling-your-data-with-dbt/package-features/customer-identifiers/) section for more details on how to do that.",
"packageDefault": "",
Expand Down Expand Up @@ -519,10 +537,14 @@
"properties": {
"schema": {
"type": "string",
"title": "(JSON) schema name for the field",
"order": 1,
"description": "The schema name of your events table, atomic in most use cases, alternatively for sdes/contexts this should instead be the name of the field itself"
},
"field": {
"type": "string",
"title": "Field name",
"order": 2,
"description": "The name of the field to use as user identifier, alternatively, in case of sdes/contexts it is the name of the element that refers to the field to be extracted"
}
},
Expand All @@ -539,6 +561,7 @@
"order": 49,
"consoleGroup": "advanced",
"type": "string",
"format": "sql",
"title": "SQL for your user identifier",
"longDescription": "This allows you to override the `user_identifiers` SQL, to define completely custom SQL in order to build out a user identifier for your events. If you are interested in using this instead of providing identifiers through the `user_identifiers` variable, please see the [Custom Identifiers](docs/modeling-your-data/modeling-your-data-with-dbt/package-features/customer-identifiers/) section for more details on how to do that.",
"packageDefault": "",
Expand All @@ -549,6 +572,7 @@
"order": 50,
"consoleGroup": "advanced",
"type": "string",
"format": "sql",
"title": "Field used when stitching together users",
"longDescription": "This is the user_id you want to stitch to sessions (and/or page views) with matching domain_userids. It supports raw `sql` expressions.",
"packageDefault": "user_id",
Expand Down Expand Up @@ -622,19 +646,26 @@
"properties": {
"schema": {
"type": "string",
"description": "Table name"
"title": "Table name",
"description": "Table name",
"order": 1
},
"prefix": {
"type": "string",
"description": "Prefix to add to columns"
"title": "Column prefix",
"description": "Prefix to add to columns",
"order": 2
},
"alias": {
"type": "string",
"description": "Table alias for the subquery"
"title": "CTE Alias",
"description": "Table alias for the subquery",
"order": 3
},
"single_entity": {
"type": "boolean",
"title": "Is single entity?"
"title": "Is single entity?",
"order": 4
}
},
"required": [
Expand Down
Loading