Type inference error when using `default_cluster_policies_override` with mixed object schemas

[pandvan]

When attempting to use the default_cluster_policies_override variable with cluster policy objects that have different schemas, Terraform fails with a type inference error, even though the variable is declared as any type.

The given value is not suitable for
module.databricks_runtime.var.default_cluster_policies_override declared at
.terraform/modules/databricks_runtime/variables.tf:230,1-45: cannot find a
common base type for all elements.

Example

locals {
  job_cluster_override = {
    autotermination_minutes = {
      type         = "range"
      maxValue     = 60
      defaultValue = 10
      isOptional   = true
    }

    "aws_attributes.availability" = {
      type   = "fixed"
      value  = "SPOT_WITH_FALLBACK"
      hidden = true
    }

    "aws_attributes.first_on_demand" = {
      type   = "fixed"
      value  = 0
      hidden = true
    }

    dbus_per_hour = {
      type     = "range"
      maxValue = 1
      hidden   = true
    }

    node_type_id = {
      type         = "allowlist"
      values       = ["m5d.large", "m5d.xlarge"]
      defaultValue = "m5d.large"
    }

    use_ml_runtime = {
      type   = "fixed"
      value  = false
      hidden = true
    }
  }
}

module "databricks_runtime" {
  source  = "data-platform-hq/runtime/databricks"
  version = "1.0.3"
  
  default_cluster_policies_override = local.job_cluster_override
  # ... other configuration
}

The issue seems occurs because Terraform attempts to infer the type of the locals block before passing values to the module variable. Even though default_cluster_policies_override is declared as any, the local objects have inconsistent schemas:

Some objects contain: type, maxValue, defaultValue, isOptional
Others contain: type, value, hidden
Others contain: type, values, defaultValue

As a workaround, at the moment, i remove jsonencode from https://github.com/data-platform-hq/terraform-databricks-runtime/blob/main/cluster.tf#L157 and pass the object as a string using jsonencode. If i don't remove from module, it will be encoded two times.