BigQuery Compilation Overrides

[peoute]

I currently have a compilation override in BigQuery for the schema suffix "_prod".

What is the best way to note that in my local repository (or just this plug-in?) in a way that it won't be picked up by BigQuery and end up with "_prod_prod" when the override runs?

Right now the plugin can't find the tables in BigQuery because it's looking for the suffix-less version, I think.

I tried to find the best practice for this in the documentation. Apologies if I missed it!

[ashish10alex]

Hi @peoute , if i understand your question correctly. You can configure the schema suffix or table suffix temporarily in workflow_settings.yaml by using the following

• datasetSuffix for dataset prefix
• namePrefix for table prefix

more info on the above documentation can be found here

Please see an example usage in the screenshot below

[peoute]

Yes, that's great. Thank you so much for the screenshots. It works. Maybe my question really ends up being a git question. I was concerned if I accidentally commit this version of workflow_settings.yaml, BigQuery will append the suffix twice, but hopefully it truly acts as an override and would just replace "prod" with "prod". I'll test it and find out. Thanks!

[ashish10alex]

thanks for confirming !.
I would not commit these overrides and use it for development only. At my work we control all the overrides from where we invoke the Dataform pipelines. We have code reviews which can catch these potential accidental commits. You might want to come up with a workflow that works for you though 🙂