Skip to content

Faulty behaviour when testing documentation for versioned models #28

@matteoannotell

Description

@matteoannotell

Problem:

When having a model with multiple versions, the code will disregard this fact, and verify columns against the base schema, not the version specific. This will result in false positive whenever one has for example a base model with a certain amount of columns, and then version(s) defined by excluding certain columns.
Example:

 - name: production_time
    description: |
      A table containing stuff
    latest_version: 1
    config:
      contract: {enforced: true}
    meta:
      owner: data-enablement
    columns:
      - name: last_assignment_id_in_chain
        description: The last assignmentId of chain `chain_id` (currently this can change).
        data_type: integer
      - name: last_completed
        description: The date the last assignement in the assignment chain was completed
        data_type: datetime
      - name: number_of_points
        description: Number of points
        data_type: integer
      - name: number_of_3d_cube_properties
        description: Number of 3D Cube properties
        data_type: integer
      - name: number_of_polygon_properties
        description: Number of polygon properties
        data_type: integer
    versions:
      - v: 1
      - v: 2
        tests:
          - test_versions_equal:
              old_version: ref('production_time', v='1')
              temporal_dimension: last_completed
        columns:
          - include: all
            exclude: [
              number_of_3d_cube_properties,
              number_of_polygon_properties
            ]

results in:

13:58:26  Encountered an error while running operation: Compilation Error in macro required_docs (macros/required_docs.sql)
  The following columns are missing from the model yml:
   - production_time.number_of_3d_cube_properties
   - production_time.number_of_polygon_properties
  
  > in macro default__format_raise_error (macros/utils/formatters/format_raise_error.sql)
  > called by macro format_raise_error (macros/utils/formatters/format_raise_error.sql)
  > called by macro default__required_docs (macros/required_docs.sql)
  > called by macro required_docs (macros/required_docs.sql)
  > called by macro required_docs (macros/required_docs.sql)

I suspect changing this line:

{% set model_columns = adapter.get_columns_in_relation(ref(model.package_name, model.name))

with

 {% set model_columns = adapter.get_columns_in_relation(ref(model.package_name, model.name, version=model.version))

could solve the problem.
It shouldn't (to be verified) cause any back-compatibility problems with dbt-core<1.5 as the extra kwarg should (to be verified) ignored by ref()

Thanks a lot for the good work!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions