How can I refer to an output in the remote state?

[AnonymousWP]

I'm using version 2.50.0 of the DigitalOcean Terraform Provider. I also finally got remote states working. However, the challenging thing now is to refer to an output within a remote state file. For example, I got:

resource "digitalocean_project" "staging" {
  name        = "POC"
  description = "Proof-of-Concept project for DigitalOcean and Terraform"
  environment = var.environment
  resources   = [data.terraform_remote_state.s3.outputs.terraform_state_storage_bucket]
}

To use that, I will have to refer to 'create' a data source so I can refer to it. I'm used to AWS, so I tried:

data "terraform_remote_state" "s3" {
  backend = "s3"
  config = {
    bucket = "techlab-staging-terraform-state-storage-staging"
    region = var.region
    key    = "s3.tfstate"
  }
}

Variables:

variable "region" {
  type = string
}

variable "environment" {
  type = string
}

variable "company" {
  type = string
}

variable "DIGITALOCEAN_TOKEN" {
  type      = string
  sensitive = true
}

variable "AWS_ACCESS_KEY_ID" {
  type      = string
  sensitive = true
}

variable "AWS_SECRET_ACCESS_KEY" {
  type      = string
  sensitive = true
}

region = "ams3"

environment = "staging"

company = "REDACTED"

However, when executing a terraform plan, I get:

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Missing region value
│
│ with data.terraform_remote_state.s3,
│ on data.tf line 1, in data "terraform_remote_state" "s3":
│ 1: data "terraform_remote_state" "s3" {
│
│ The "region" attribute or the "AWS_REGION" or "AWS_DEFAULT_REGION" environment variables must be set.

So it expects an AWS region, while I'm obviously using DigitalOcean's provider. I found out that this is the way for AWS, so I looked for a DO kind of way here: https://developer.hashicorp.com/terraform/language/state/remote-state-data. There's nothing related to it.

How would I then refer to an output in another stack (see the resource digitalocean_project example)? It needs to go through the remote state if I'm correct, as there's no local state.

[danaelhe]

Hi!

Thanks for the write up. I'm able to reproduce this as well. Still investigating...

[danaelhe]

Can you see if this works for you...

terraform {
  required_providers {
    digitalocean = {
      source  = "digitalocean/digitalocean"
      version = "2.25.2"
    }
  }

# This stores the terraform state on a shared s3 bucket or digital oceans space
  backend "s3" {
    endpoint = "https://do-api-eng-demo.sfo3.digitaloceanspaces.com"
    bucket   = "do-api-eng-demo"
    region   = "us-east-1"  # Must be us-east-1, true Spaces region found in endpoint
    key      = "terraform.tfstate" 
  }
}

[danaelhe]

Still looking for validation on my end, but I actually think your set up is fine, you may just have to keep region = "us-east-1" and the true spaces region will be picked up in the endpoint. Which is not at all intuitive and should be documented.

[AnonymousWP]

Thanks for your comment. 🙂 I believe I tried this too, and it also doesn't work. If I change the backend's region to an AWS' one (e.g. us-east-1), I'm getting the same data source error. If I also change the data source, it complains about STS, which is (I believe) AWS-specific:

Error: validating provider credentials: retrieving caller identity from STS: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: , api error InvalidClientTokenId: The security token included in the request is invalid.

Have you gotten any luck? My question is: is this a bug, user-error or a missing feature?

[AnonymousWP]

Hey @danaelhe, any updates on this? Is this lacking in the Terraform Provider for DigitalOcean or is this a misconfiguration from my end? If the former, want me to create an issue for this? Am I the only one using remote states? Because they're very essential when working in a team (if the statefile is locally, which is the default, then it'll be tricky to work in a team as everyone has a different statefile).

[andrewsomething]

Is this lacking in the Terraform Provider for DigitalOcean or is this a misconfiguration from my end? I

Remote state management is a feature of Terraform itself. It is not implemented by individual providers. The terraform_remote_state data source is a built-in provider provided by terraform.io/builtin/terraform. Unfortunately this can sometime make it difficult to use with S3 alternatives as they can use AWS-specific functionality by default.

The S3 remote state offers options to disable those for compatibility with S3 alternatives like DigitalOcean Spaces. You can find documentation on those here:

https://docs.digitalocean.com/products/spaces/reference/terraform-backend/

The config for a terraform_remote_state accepts all the same options as the remote state. So you want to use something like:

data "terraform_remote_state" "remote" {
  backend = "s3"
  config = {
    endpoints = {
      s3 = "https://<region>.digitaloceanspaces.com"
    }
    bucket     = "<bucket name>"
    key        = "terraform.tfstate"
    region     = "us-east-1"
    encrypt    = true
    skip_region_validation      = true
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    skip_requesting_account_id  = true
    skip_s3_checksum            = true
  }
}

[AnonymousWP]

Yeah my bad, I know it's a native Terraform feature, so perhaps I should have created an issue/Discussion at the Terraform repository. I did read that page about those data sources, but that's only for a handful of providers. Looks like it's PEBKAC.😄

The S3 remote state offers options to disable those for compatibility with S3 alternatives like DigitalOcean Spaces. You can find documentation on those here: https://docs.digitalocean.com/products/spaces/reference/terraform-backend/

Huh, I did follow the instructions on that page but I never got it to work due to different errors (see my initial post), but perhaps they're related to just the region. I remember changing the region too though.

Anyway, your config works, thanks! Only difference I see is that the data source is added. Is it mandatory to have, apart from the backend s3 part? What I thought is that with the backend you actually define the backend, and with the data source you can use/call an existing (not create) a remote state.

Nevertheless, I think documentation should be improved and some stuff should be cleared up.