-
Notifications
You must be signed in to change notification settings - Fork 16
Open
Description
Problem:
When having a model with multiple versions, the code will disregard this fact, and verify columns against the base schema, not the version specific. This will result in false positive whenever one has for example a base model with a certain amount of columns, and then version(s) defined by excluding certain columns.
Example:
- name: production_time
description: |
A table containing stuff
latest_version: 1
config:
contract: {enforced: true}
meta:
owner: data-enablement
columns:
- name: last_assignment_id_in_chain
description: The last assignmentId of chain `chain_id` (currently this can change).
data_type: integer
- name: last_completed
description: The date the last assignement in the assignment chain was completed
data_type: datetime
- name: number_of_points
description: Number of points
data_type: integer
- name: number_of_3d_cube_properties
description: Number of 3D Cube properties
data_type: integer
- name: number_of_polygon_properties
description: Number of polygon properties
data_type: integer
versions:
- v: 1
- v: 2
tests:
- test_versions_equal:
old_version: ref('production_time', v='1')
temporal_dimension: last_completed
columns:
- include: all
exclude: [
number_of_3d_cube_properties,
number_of_polygon_properties
]
results in:
13:58:26 Encountered an error while running operation: Compilation Error in macro required_docs (macros/required_docs.sql)
The following columns are missing from the model yml:
- production_time.number_of_3d_cube_properties
- production_time.number_of_polygon_properties
> in macro default__format_raise_error (macros/utils/formatters/format_raise_error.sql)
> called by macro format_raise_error (macros/utils/formatters/format_raise_error.sql)
> called by macro default__required_docs (macros/required_docs.sql)
> called by macro required_docs (macros/required_docs.sql)
> called by macro required_docs (macros/required_docs.sql)
I suspect changing this line:
{% set model_columns = adapter.get_columns_in_relation(ref(model.package_name, model.name)) |
with
{% set model_columns = adapter.get_columns_in_relation(ref(model.package_name, model.name, version=model.version))
could solve the problem.
It shouldn't (to be verified) cause any back-compatibility problems with dbt-core<1.5
as the extra kwarg should (to be verified) ignored by ref()
Thanks a lot for the good work!
Metadata
Metadata
Assignees
Labels
No labels