rant: run --all and stack features are unusable in CI environments

[dennislapchenko]

I am aware this is a long standing issue of terragrunt and CI enviroments (like Atlantis). Issue of not being able to apply with plan file units containing inter-dependencies. Without run --all at least units can be slowly applied one by one (which is also a pain, but at least it works, one commit per unit). Testing out stacks just now I was hopeful that it has solved the CI issues. Sadly it is still exactly the same story as with run --all.

Key problem:
In CI environments like Atlantis, teams need deterministic plan files with correct outputs for each inter-dependency. Having mock outputs getting applied once and then re-running the apply seems like a clunky hack. Running applies without plan files is also dangerous.

What I am trying to understand, and perhaps this helps some other developers' sanity, are two things:

1. Are run --all and stack commands/infrastructures really intended to be always run manually? It is hard to believe so I am open to learn some key aspects that I might be missing. Perhaps gruntworks' internal pipeline platform has this solved and is a paid offering?
2. If first assumption is correct, perhaps there are some community solutions which enable flows like those with Atlantis to work reliably with stack feature?

[yhakbar]

Yes, the Gruntwork Pipelines product does have native support for Terragrunt Stacks:
https://www.gruntwork.io/platform/pipelines

Unfortunately, the way that OpenTofu and Terraform currently work, there is no way to signal to OpenTofu that some variables are unknown, and might need to be recalculated, even when a plan is saved. I've documented a workaround here in the past.

We also champion this issue to support this behavior natively in OpenTofu better. The OpenTofu engineer working on behalf of Gruntwork, @diofeher, has done some good work exploring this problem space, and might have an RFC to address it in OpenTofu in the near future.