v1.10.0a1

[benc-db]

What's Changed (Pre-release)

Features

• Support databricks OAuth M2M auth type. Updated OAuth readme doc with instructions.(827)
• Introduced use_materialization_v2 flag for gating materialization revamps. (844)
• Introduce Tables V2, including safe_table_create which will not change the production table unless new data can safely be ingested (927)
• Views V2, including renaming safe_table_create to use_safer_relation_operations, and adding view_update_via_alter, to allow updating views by altering, instead of recreating (954)

Under the Hood

• Update pinned python SDK version from 0.17.0 to 0.41.0. (827)
• Implement new constraint logic for use_materialization_v2 flag (846), (876)
• Streamlining debug logging to make it more usable (946)
• Upgrading Databricks SQL Connector to V4 (962)
• Validation of sample mode (961)

---

This discussion was created from the release v1.10.0a1.

How to use the new Features

Materialization V2

In 1.10.0 we are introducing new versions of most materializations that are hidden behind a behavior flag. This is to limit the impact of these structural changes while we gather feedback and bug reports from users that want to pro-actively adopt them. To opt-in to the new materializations, you will need to set the following behavior flag in your dbt_project file:

flags:
   use_materialization_v2: true

To see the full impact of this flag, see the flow diagrams in the docs folder. In general, flipping this flag to true will opt you into materializations that separate creation of tables from the insertion of data into those tables. While this breaks the atomicity of a 'CREATE TABLE AS SELECT', or CTAS, approach, it provides a major benefit: specifying column features at create time that are incompatible with CTAS. This clears the way for many things in the future, but in this release, the biggest change is that now we can set comments on columns at create time. In the past, comments were applied one at a time via ALTER, and the performance of these operations was lackluster - seemingly scaling with the size of the table. With the new flag, comments are applied prior to inserting data where possible, which can provide a significant performance improvement.

Another change with this flag is how we implement constraints. We are deprecating our homespun solution (persist_constraints) to more fully adopt the dbt constraint framework. This change also means that with the use of this flag, all constraints that can be applied at create time are; the one exception are CHECK constraints, which due to Databricks limitations can only be applied via alter. In contrast to existing constraint behavior though, we now ensure that all constraints are on the destination table prior to inserting data. This means that data that violates a CHECK constraint will not make it into the destination table, whereas before we could only halt the materialization for a CHECK constraint at the point we applied the constraint, at which point we might already have invalid data in the table.

Note: for now, there are no significant changes to MV/ST materializations due to limitations in the ALTER API.

Additional Materialization Options

There are two other model-level configurations that can be used with use_materialization_v2 set to true: use_safer_relation_operations and view_update_via_alter.

Safer Relation Operations

Setting use_safer_relation_operations to true on a model configuration (or in your dbt_project file using the + syntax) will prefer operations that leave the final table/view intact until we have completed all the setup on a staging relation. At this point we will swap the new version with the old version and drop the old version. This approach may have mildly worse performance, but with the benefit that a failed multi-step materialization no longer applies any changes to your final table. This simulates the behavior of being able to rollback on failure, but also leaves the staging table until your next run, so that you can potentially debug what about the process failed.

Updating Views In Place

The other new model config is view_update_via_alter. This fixes a longstanding issue when working with views that replacement leads to loss of history, and breaks in Unity Catalog-related features. Setting this to true (along with use_materialization_v2) for a view will cause us to prefer using alter statements (and in some cases doing nothing if there are no changes) to make an existing view match what is configured in the dbt project. One caveat is that Databricks currently has a hole where we cannot alter comments on views. As such, if the description of a view changes, we will need to create a new view; however, most users that hit the issue we're solving here are just trying to run dbt run without changes to their project, in which case we should not need to replace the view.

[samuelberntzen]

Thanks! Been looking forward to this version. I am testing it now for constraint generation, and it significantly increases build time when persisting docs!

In our case, v1.10.0a1 spent 14 seconds building the same model that v1.9.7 did in 90 seconds when using Materialization V2🎉

However, it did break the constraints containing column names with special characters (æ,ø,å in our case).

See #965 for more details.

[benc-db]

Going to release another alpha today with your fix.

[Zurbste]

Upgraded from v1.9.7 to v1.10.0a2 to test the performance, and the runtime decreased from 352 seconds to 12 seconds (240 columns). So, it's looking good right now!

Thank you, Ben 👍

[Zurbste]

Edit: It doesn't seem to work if the type of a column is a complex struct. The DESCRIBE TABLE command cuts off the last part of the type definition and adds a hint like ,... 9 more fields.

array_of_structs_column array<struct<[...] ,... 9 more fields>> COMMENT 'Dummy comment.',

[benc-db]

Can you try setting this flag? use_info_schema_for_columns: true. Let me know if the error persists.

[Zurbste]

Added this flag to my dbt_project.yml, but it still doesn't work

[benc-db]

do you mind emailing me your dbt.log? ben.cassell@databricks.com

[benc-db]

Will be releasing a new alpha today to set the default for use_safer_relation_operations to false everywhere (was accidentally left as False in one place and True in two places).

[benc-db]

Hi all, rc1 was released yesterday. Two issues have been brought to my attention, so final will not release today. Probably next week, but depends on when I can get the issues solved.

[benc-db]

rc2 out today. Main thing is improving behavior of incremental comment change detection in the V2 branch.

[kmarq]

@benc-db I've been following the v2 changes, I need to test some things out but will be interested to see how this works. I just saw that in the 2025.15 SQL release Databricks now supports doing an ALTER on multiple columns. For cases where the table can't be created in full with comments, leveraging this new behavior to apply the comments and other constraints in a single alter statement would likely be very beneficial.