Decorate macros based on their side effects

[jtcohen6]

Adapted from #6278 (comment)

We enable a small subset of configurations to accept macros whose return values are "late-rendered" at runtime, instead of just being resolved at parse time. The two are pre-hook and post-hook, and while I love them dearly, they are also the banes of my existence: https://docs.getdbt.com/docs/building-a-dbt-project/dont-nest-your-curlies#an-exception

We need a more generally sensible rule about what dbt can expect to know when. We've talked a bit about "decorators" for macros that would indicate whether they're:

• "Pure" / "immutable" macros: Static inputs + outputs. Same arguments yield same results, always. These can be resolved at parse time and used for configuration — we could even look to support these in .yml files!
• "Stable" / "introspective" macros: Runs an introspective query that returns database state, without changing that state. Will always return the same results, given same inputs (arguments + db state). These should be supported at compile time, to template out model SQL. For certain configurations, we could explicitly support "re-rendering" at compile time, to gain access to database state as an input. This has been requested for configs that just modify materialization logic, e.g. partitions and merge_update_columns.
• "Dirty" / "volatile" / "state change" macros: Has side effects in the database that changes state, e.g. creating or altering tables. If we do our jobs right, these should exist exclusively within materialization logic, and never within model logic — hence, it's an anti-pattern to see post hooks containing truncate or delete!

More on pure macros

I know lots of folks have been interested in support for "pure" macros in .yml files, to DRY up the same config that they currently have to copy-paste around a bunch of different files. Think: conditional database for sources (#4753), where for tests (#6236 / #5940 / #4103), ... We also discussed this over in #6170, as a potential future avenue for vars that might look & feel more like snippets.

In a similar vein to #4999, I'm not sure if we can do this with actual Jinja, or if we'd need our own compilation / templating engine. I sure would like the idea of moving away from Jinja-in-YAML...

[jtcohen6]

Just remembered that I first wrote a proposal in this vein as a way of building toward "dry run" materializations (#4456). In this version, I'd actually envisioned four types of macros. Copying my notes from several months ago:

We establish a system for differentiating between macros/statements that introspect the database (e.g. get_columns_in_relation), and macros/statements that mutate the database (). This could look like an additional argument that gets passed through to run_query / call statement / adapter.execute... > or some sort of macro decorator????

When --dry-run is enabled, we do different things for introspective vs. mutative macros.

This sort of differentiation between macros is a thing we’ve long wanted, for purposes of the dbt language, as a prerequisite to reasonably supporting macros within configuration code. You could imagine four types of macros:

• static/pure: doesn’t need a database connection, doesn’t mess with ref/source/config
• parse-dirty: doesn’t need a database connection, does mess with ref/source/config
• introspective: needs a database connection, leaves the database unchanged
• mutative: needs a database connection, changes the state of the database after running

We could already differentiate between some of those, based on whether the macro depends on / includes calls to other built-in macros (ref, source, config, statement). At some point, a query-running macro needs to say whether it aims to introspect or mutate the database. The good news is, end users don’t write many mutative queries—they leave that to materialization logic. This is one of the most powerful abstractions in dbt—transformation (model) SQL is the output of introspective SQL, and > the input to materialization SQL. The exceptions:

• Users who have written / reimplemented their own materializations. I think this is okay, as long as we clearly document how they could extend --dry-run support
• Users who have a bad habit of sneaking DML (delete / truncate / etc) into post-hooks... oh well :)