mock_outputs won't use mock_outputs's k:v for undefined outputs in dependency

[ajoga]

Describe the bug

Given a deployed stack with two modules base and dependant, if I add a new input into dependant that takes a new output from base, I can't run a terragrunt run --all plan, even if I use mock_outputs.

This may be a desired behavior ("mock_outputs only work for defined outputs of the module on which we depend), but I can't find documentation that would go in this direction, or I do not understand it that way.

Steps To Reproduce

$ git clone https://github.com/ajoga/terragrunt-mrc.git
$ cd tg/terragrunt-mrc/tg/
$ git checkout 3d9511904ee39c9722bbcfd94c0d9a3468f79428 #initial commit
$ terragrunt run --all apply --source $PWD/../
$ # Apply is successful
$ git checkout ad56311c7c91b283635f3fade1745aae9857efb5 # There is only one commit added onto main that (1) adds an output in the terraform module (2) sets this output as an input in the `dependant` unit and an associated mock_outputs
$ terragrunt run --all plan --source $PWD/../

Here's my output for the plan:

terragrunt run --all plan --source $PWD/../
*10:23:21.026 INFO   The runner at . will be processed in the following order for command plan:
Group 1
- Unit ./live/base

Group 2
- Unit ./live/dependant


10:23:21.101 INFO   [live/base] Downloading Terraform configurations from .. into ./live/base/.terragrunt-cache/U8e_ZGH6XGekoxVFXW5jDJ30Rb4/duZE5ZFiQzLcbmVqiV25dv_Fqoc
10:23:21.164 INFO   [live/base] terraform: Initializing the backend...
10:23:21.174 INFO   [live/base] terraform: Initializing provider plugins...
10:23:21.174 INFO   [live/base] terraform: - Reusing previous version of hashicorp/local from the dependency lock file
10:23:21.326 INFO   [live/base] terraform: - Using previously-installed hashicorp/local v2.5.3
10:23:21.327 INFO   [live/base] terraform: Terraform has been successfully initialized!
10:23:21.327 INFO   [live/base] terraform: 
10:23:21.327 INFO   [live/base] terraform: You may now begin working with Terraform. Try running "terraform plan" to see
10:23:21.327 INFO   [live/base] terraform: any changes that are required for your infrastructure. All Terraform commands
10:23:21.327 INFO   [live/base] terraform: should now work.
10:23:21.327 INFO   [live/base] terraform: If you ever set or change modules or backend configuration for Terraform,
10:23:21.327 INFO   [live/base] terraform: rerun this command to reinitialize your working directory. If you forget, other
10:23:21.327 INFO   [live/base] terraform: commands will detect it and remind you to do so if necessary.
10:23:21.486 STDOUT [live/base] terraform: local_file.foo: Refreshing state... [id=4bf3e335199107182c6f7638efaad377acc7f452]
10:23:21.498 STDOUT [live/base] terraform: Changes to Outputs:
10:23:21.498 STDOUT [live/base] terraform:   + demo  = "0777"
10:23:21.498 STDOUT [live/base] terraform: You can apply this plan to save these new output values to the Terraform
10:23:21.498 STDOUT [live/base] terraform: state, without changing any real infrastructure.
10:23:21.498 STDOUT [live/base] terraform: 
10:23:21.498 STDOUT [live/base] terraform: ─────────────────────────────────────────────────────────────────────────────
10:23:21.498 STDOUT [live/base] terraform: Note: You didn't use the -out option to save this plan, so Terraform can't
10:23:21.499 STDOUT [live/base] terraform: guarantee to take exactly these actions if you run "terraform apply" now.
10:23:21.559 ERROR  [live/dependant] Error: Unsupported attribute
10:23:21.560 ERROR  [live/dependant]   on ./live/dependant/terragrunt.hcl line 17:
10:23:21.560 ERROR  [live/dependant]   17:     foo = dependency.base.outputs.demo
10:23:21.560 ERROR  [live/dependant]   18: }
10:23:21.560 ERROR  [live/dependant] This object does not have an attribute named "demo".
10:23:21.560 ERROR  [live/dependant] Unit ./live/dependant has finished with an error
10:23:21.560 ERROR  Run failed: error occurred:

* ./live/dependant/terragrunt.hcl:17,32-37: Unsupported attribute; This object does not have an attribute named "demo".

Expected behavior

I would expect either:

• the plan to success with my mock_outputs taken as an input.
• the documentation to state that mock_outputs will only override values of keys that exist in the plan of the dependency we're defining

Workaround

One can go in the unit on which you depend, apply that unit so that the tfstate will gets the new output populated. The subsequent terragrunt run --all plan will work.
Note that this workaround doesn't work for the MRC code I provided here, I suppose because of where the dummy file is written or maybe the local provider, sorry

Versions

• Terragrunt version: 0.83.0
• OpenTofu/Terraform version: Terraform v1.12.2
• Environment details (Ubuntu 20.04, Windows 10, etc.): 6.6.87.2-microsoft-standard-WSL2

Additional context

This issue seems very similar to this previously auto-closed issue: #2405 (comment)

[yhakbar]

@ajoga

What you're looking to leverage is the mock_outputs_merge_strategy_with_state attribute:
https://terragrunt.gruntwork.io/docs/reference/config-blocks-and-attributes/#dependency

By default, Terragrunt won't mix mocked outputs with real outputs as that can confuse users and lead to unexpected results. Users have to explicitly tell Terragrunt that they expect state to exist in a dependency, but that it might need to be partially mocked.

Please close this issue if using this attribute solves your issue.

By the way, I really appreciate how easy you made it to reproduce your issue. Thank you so much for that.

[ajoga]

Hey @yhakbar

You are absolutely correct, reading again the documentation and setting mock_outputs_merge_strategy_with_state = "shallow" solved the issue of the mocked new-input-from-unapplied-output.

I have submitted a PR for your consideration to steer the reader into a better understanding of that option in #4529.

I will close this issue as it is solved, we can continue the documentation-related discussion in the PR.

Thank you for your quick help on this!