unexpected end of JSON input

[tekollt]

Terraform Version

Terraform v0.11.8

Affected Resource(s)

external

Terraform Configuration Files

data "external" "webhook" {
  program = ["az", "webapp", "deployment", "list-publishing-profiles", "--name", "fa-we-shared-pvs-prod", "--resource-group", "rg-we-shared-prod", "--query", "\"[].{PWD:userPWD}\"", "--output", "json"]  
}

Expected Behavior

being able to reference the password, for an azure webapp webhook and use that with GitLab provider to define integrations.

Actual Behavior

* module.function_nesk.data.external.hooks: data.external.hooks: command "az" produced invalid JSON: unexpected end of JSON input

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply

Additional Context

Running the command in Azure Cli, gives the following output, with Json encoding.

{
  "PWD": "mnvQcfGqw3F3fpj0wtw12345678901234567891023456987xxxxxxxxxxxxxxxxx"
}

or by changing the outputs in cli --output to tsv gives single string, but this also gives unexpected end of JSON input with terarform

mnvQcfGqw3F3fpj0wtw12345678901234567891023456987xxxxxxxxxxxxxxxxx

References

#13

[mildwonkey]

Hi @thoregilK ,
I'm sorry you've run across this behavior. Could you share the debug log (removing any sensitive information, such as passwords) from a terraform run as a gist? It should show us some more information about the format of the az command that's being run.

[tekollt]

hi @mildwonkey

I was hoping that debug would have more information but it does not seems to be the case.

2018/09/07 07:04:12 [INFO] command: backend <nil> is not enhanced, wrapping in local
2018/09/07 07:04:12 [INFO] backend/local: starting Apply operation
2018/09/07 07:04:14 [DEBUG] New state was assigned lineage "b5172850-e4b7-bd1c-6f5d-a97004f38d88"
2018/09/07 07:04:15 [INFO] terraform: building graph: GraphTypeInput
2018/09/07 07:04:15 [DEBUG] Resource state not found for "data.external.webhook": data.external.webhook
2018/09/07 07:04:15 [DEBUG] pruning unused provider provider.azurerm.development
2018/09/07 07:04:15 [DEBUG] ReferenceTransformer: "provider.external" references: []
2018/09/07 07:04:15 [DEBUG] ReferenceTransformer: "data.external.webhook" references: []
2018/09/07 07:04:15 [TRACE] Graph after step *terraform.ReferenceTransformer:

data.external.webhook - *terraform.NodeAbstractResource
 provider.external - *terraform.NodeApplyableProvider
provider.external - *terraform.NodeApplyableProvider
2018/09/07 07:04:15 [DEBUG] Starting graph walk: walkInput
2018-09-07T07:04:15.193+0200 [DEBUG] plugin: starting plugin: path=C:\GitHub\terraform-azure\test\.terraform\plugins\windows_amd64\terraform-provider-external_v1.0.0_x4.exe args=[C:\GitHub\terraform-azure\test\.terraform\plugins\windows_amd64\terraform-provider-external_v1.0.0_x4.exe]
2018-09-07T07:04:15.196+0200 [DEBUG] plugin: waiting for RPC address: path=C:\GitHub\terraform-azure\test\.terraform\plugins\windows_amd64\terraform-provider-external_v1.0.0_x4.exe
2018-09-07T07:04:15.209+0200 [DEBUG] plugin.terraform-provider-external_v1.0.0_x4.exe: 2018/09/07 07:04:15 [DEBUG] plugin: plugin address: tcp 127.0.0.1:10000
2018/09/07 07:04:15 [INFO] terraform: building graph: GraphTypeValidate
2018/09/07 07:04:15 [DEBUG] Resource state not found for "data.external.webhook": data.external.webhook
2018/09/07 07:04:15 [DEBUG] adding missing provider: external
2018/09/07 07:04:15 [DEBUG] resource data.external.webhook using provider provider.external
2018/09/07 07:04:15 [DEBUG] pruning unused provider provider.azurerm.development
2018/09/07 07:04:15 [TRACE] Graph after step *terraform.PruneProviderTransformer:

data.external.webhook - *terraform.NodeValidatableResource
 provider.external - *terraform.NodeApplyableProvider
provider.external - *terraform.NodeApplyableProvider
2018/09/07 07:04:15 [DEBUG] ReferenceTransformer: "data.external.webhook" references: []
2018/09/07 07:04:15 [DEBUG] ReferenceTransformer: "provider.external" references: []
2018/09/07 07:04:15 [DEBUG] Starting graph walk: walkValidate
2018/09/07 07:04:15 [DEBUG] Resource state not found for "data.external.webhook": data.external.webhook
2018/09/07 07:04:15 [DEBUG] ReferenceTransformer: "data.external.webhook" references: []
2018/09/07 07:04:15 [INFO] backend/local: apply calling Refresh
2018/09/07 07:04:15 [INFO] terraform: building graph: GraphTypeRefresh
2018/09/07 07:04:15 [DEBUG] Resource state not found for "data.external.webhook": data.external.webhook
2018/09/07 07:04:15 [DEBUG] adding missing provider: external
2018/09/07 07:04:15 [DEBUG] ReferenceTransformer: "provider.external" references: []
2018/09/07 07:04:15 [DEBUG] ReferenceTransformer: "data.external.webhook" references: []
2018/09/07 07:04:15 [TRACE] Graph after step *terraform.ReferenceTransformer:

data.external.webhook - *terraform.NodeRefreshableDataResource
 provider.external - *terraform.NodeApplyableProvider
provider.external - *terraform.NodeApplyableProvider
2018/09/07 07:04:15 [DEBUG] Starting graph walk: walkRefresh
2018/09/07 07:04:15 [DEBUG] Resource state not found for "data.external.webhook": data.external.webhook
2018/09/07 07:04:15 [DEBUG] ReferenceTransformer: "data.external.webhook" references: []
2018/09/07 07:04:15 [TRACE] Graph after step *terraform.ReferenceTransformer:

data.external.webhook - *terraform.NodeRefreshableDataResourceInstance
2018/09/07 07:04:18 [ERROR] root: eval: *terraform.EvalReadDataApply, err: data.external.webhook: command "az" produced invalid JSON: unexpected end of JSON input
2018/09/07 07:04:18 [ERROR] root: eval: *terraform.EvalSequence, err: data.external.webhook: command "az" produced invalid JSON: unexpected end of JSON input
2018/09/07 07:04:18 [TRACE] [walkRefresh] Exiting eval tree: data.external.webhook
2018/09/07 07:04:18 [DEBUG] plugin: waiting for all plugin processes to complete...
2018-09-07T07:04:18.517+0200 [WARN ] plugin: error closing client during Kill: err="unexpected EOF"
2018-09-07T07:04:18.525+0200 [DEBUG] plugin: plugin process exited: path=C:\GitHub\terraform-azure\test\.terraform\plugins\windows_amd64\terraform-provider-external_v1.0.0_x4.exe

As far as i am aware, the output running the same command from cli produces a valid json object.

C:\GitHub\terraform-azure\test>az webapp deployment list-publishing-profiles --name {app_name} --resource-group {resource_group} --query "[0].{PWD:userPWD}"
{
  "PWD": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}

C:\GitHub\terraform-azure\test>

[mclamb]

Seeing the same behavior. Output is valid JSON

[omar]

Funny to see multiple people trying to work around current bugs in Azure Service Environment APIs blocking automation of the deployment.

FWIW, I got this to work by using jq to flatten the result of the Azure CLI:

# As of writing, the ASE ARM deployment don't return the IP address of the ILB
# ASE. This workaround querys Azure's API to get the values we need for use
# elsewhere in the script. 
# See this https://stackoverflow.com/a/49436100
data "external" "app_service_environment_ilb_ase_ip_address" {
  # This calls the Azure CLI then passes the value to jq to return JSON as a single
  # string so that external provider can parse it properly. Otherwise you get an
  # error. See this bug https://github.com/terraform-providers/terraform-provider-external/issues/23
  program = ["bash", "-c", "az resource show --ids ${local.app_service_environment_id}/capacities/virtualip --query '{internalIpAddress: internalIpAddress}' | jq -c"]

  # Explicit dependency on the ASE ARM deployment because this command will fail
  # if that resource isn't built yet.
  depends_on = [azurerm_template_deployment.ase]
}

[ericmacfarland]

I've run into a similar issue, but in my case I'm fetching a JSON Web Token from Azure (access_token in the below examples).
As near as I can tell, it's due to some of the response values being integers instead of strings.

When the API call returns the below response, my Terraform works as expected.

{
    "token_type": "Bearer",
    "expires_in": "3600",
    "ext_expires_in": "3600",
    "expires_on": "1572712041",
    "not_before": "1572708141",
    "resource": "https://graph.microsoft.com",
    "access_token": "<some jwt>"
}

When the API call returns the below response, I receive the error:
command "bash" produced invalid JSON: unexpected end of JSON input

{
    "token_type": "Bearer",
    "expires_in": 3599,
    "ext_expires_in": 3599,
    "access_token": "<some jwt>"
}

[nfroidure]

I have the same kind of error but I'm unable to debug it since I have no access to the real output : https://stackoverflow.com/questions/66490152/how-to-debug-terraform-external-providers-with-concurrency-issues

When I log the output of the external program, I get valid JSON, I can't manage to find out why it fails and it do not reproduces all the time 🤷‍♂.

Would be nice to have an option that allow to output the received data in debugging mode with the command line run.

[bflad]

Hi folks 👋 Sorry for the frustrating behavior here. 😖

This error tends to indicate that the output of the program was truncated in some fashion. Since the issue was opened, the external data source should now provide TRACE level logging of program output starting with JSON output: in the logs. This type of logging in Terraform can be enabled by setting the TF_LOG environment variable, e.g. TF_LOG=TRACE terraform apply.

However, properly handling non-string values in JSON documents is likely a valid issue since the data source currently expects to unmarshal the document as a map[string]string. There may be potential ways to mitigate this with custom unmarshaling logic, however a more comprehensive solution would be implement #76, likely as part of #81, to instead allow the data source to handle any valid JSON return value.

[lapkritinis]

I made some "workaround". with jq -R wrapping response into "legit" response and then parse it with jsondecode. Here is my actual code:

data "aws_region" "current" {}

data "external" "this" {
  // wrapping response into json as it fails to parse complex json
  program = ["bash", "-c", "aws eks describe-addon-configuration --region ${data.aws_region.current.name} --addon-name ${var.addon_config.name} --addon-version ${var.addon_config.version} --query 'configurationSchema' --output json | jq -R -c '{response: .}'"]
}

output "this" {
  value = jsondecode(jsondecode(data.external.this.result.response)) // double encoded
}

variable "addon_config" {
  type = object({
    name    = string
    version = string
  })
}