Deprecation Warnings (coming soon in v1.10)

[graciegoheen]

Historically, the inputs to your dbt project have been largely unconstrained. You can “configure” just about anything you want.

As one example of this, dbt currently allows you to set any custom yaml properties:

models:
  - name: my_model
    description: A model in my project.
    grace_is_cool: true # a custom property

dbt sees that you’ve defined a custom property (grace_is_cool) that it does not recognize, and it silently continues.

Without a set of strictly defined inputs, it’s much trickier to determine if a configuration in your dbt project is valid. This has created issues such as:

• silently ignoring misspelled properties and configurations (desciption: instead of description:; materialization= instead of materialized=)
• unintended collisions with user code when we (dbt) want to introduce a new “reserved” property or configuration

If you joined us for dbt developer day, you’ll know that starting in v1.10 we will begin to emit deprecation warnings for future-invalid dbt code. This includes:

• unrecognized resource properties, configurations, and top-level-keys
• duplicate yaml keys in the same file
• {% endmacro %} tag without a corresponding {% macro %} tag
• and more

See the Github EPIC for more information.

How will we know what code is valid? (drum roll please...)

Better jsonschema validation for dbt yaml files

What is it?

Improvements to our jsonschemas which can statically analyze dbt yaml files and determine if they are valid. The new jsonschemas are:

• more complete
• inclusive of these stricter validation rules
• and generated from code (the new engine for dbt)

Who’s it for?

• We (dbt Labs) use these schemas to provide real-time feedback while users edit yaml files in the dbt Cloud IDE, and, coming soon, the official dbt VSCode extension (one more plug for dbt developer day).
• Partners, plugins, and other custom tools can use these if they generate dbt yaml code and want to check that it is formatted correctly, without needing to actually install/run dbt-core.

Where can I check out these new & improved jsonschemas today?

For now, you can take a look at what we’re working on here:

• https://public.cdn.getdbt.com/fs/schemas/fs-schema-dbt-yaml-files-v0.0.85.json
• https://public.cdn.getdbt.com/fs/schemas/fs-schema-dbt-project-v0.0.85.json

When we’re confident that they’re ready for primetime, we’ll update the existing repository: dbt-labs/dbt-jsonschema

What’s Happening When?

We will start raising these warnings in dbt-core v1.10, but making these changes will not be prerequisite to use v1.10. We know it will take existing users time to migrate their projects, and it is not our goal to break anyone overnight. The goal is to enable dbt users to work with more safety, feedback, and confidence going forward.

What does this mean for you?

1. If your project (or dbt package) hits a new deprecation warning in v1.10, you should plan to update your invalid code soon! Even though it’s just a warning for now, in a future version of dbt, we will be enforcing stricter validation to the inputs of your dbt project. (@b-per is working on a tool you can use to autofix a lot of these).
2. The meta config will be the only place to put custom user-defined attributes. Everything else will will be strongly typed and strictly validated. If you have an extra attribute you want to include in your project, or a model config you want to access in a custom materialization, you must nest it under meta moving forward.
3. ⚠️ If you are using the —-warn-error flag (or --warn-error-options '{"error": "all"}') to promote all warnings to errors, this will include new deprecation warnings coming to Core. If you don’t want these to be promoted to errors, the --warn-error-options flag gives you more granular control over exactly which types of warnings are treated as errors - you can set "warn": ["Deprecations"] (new as of 1.10) to continue treating the deprecation warnings as warnings.

By being stricter about what inputs are really used by dbt, and which ones are your own custom metadata, we can alert you about invalid inputs (like spelling errors) and prevent breaking changes to your projects!

[joellabes]

There's a more specific list of deprecated behaviours in the tracking issue: #11335