You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This table holds billions of records from the last few years. Currently, cat_id only has one distinct value - 1. We are about to update the ref table to have cat_id: 2 with years of historical data. Of course, without full refresh this model will not have cat_id 2 with older data than this week. So I am thinking, how can I refresh old data by a specific column value without recreating the table?
I was thinking of adding something like:
{%- if var('cat_id_var', None) %}
WHERE cat_id = {{ var('cat_id_var') }}
{%- endif %}
and call dbt with cat_id_var only when I need the full refresh, but, of course, doing full refresh will cause a table rebuild. Hope it is clear, would love any feedback/ideas.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
-
We have the following model in dbt-redshift:
This table holds billions of records from the last few years. Currently,
cat_id
only has one distinct value -1
. We are about to update the ref table to havecat_id
:2
with years of historical data. Of course, without full refresh this model will not have cat_id2
with older data than this week. So I am thinking, how can I refresh old data by a specific column value without recreating the table?I was thinking of adding something like:
and call dbt with
cat_id_var
only when I need the full refresh, but, of course, doing full refresh will cause a table rebuild. Hope it is clear, would love any feedback/ideas.Beta Was this translation helpful? Give feedback.
All reactions