Getting Null Values, instead of parsed values

[MV45716]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Use Case: We are designing a pipeline to ingest a json object that will output/post a multiline string to an api.

We are able to create the string and add it successfully to a json object however when we try to replace the json document entirely with the string the variables are returning null values.

VRL Playground link: Link for Playground code

Vector transformation config: refer config section.

console output

# TYPE  gauge
# HELP   gauge
{sloId="",status="APPROVED"} 
# EOF

Expected output:

# TYPE availability_slo_total_count gauge
# HELP availability_slo_total_count availability slo total count gauge
availability_slo_total_count{sloId="033df0fd9c4040b98d94a5ec325fd3e9",status="APPROVED"} 76.8
# EOF

Configuration

transforms:
  parse_dd_api_data:
    type: remap
    inputs: 
      - dd_api_data
    source: |-
      .metricName, err= to_string(.metricname)
      if err != null {
        log("Error on metricname: " + err, level: "error")
      } 
      .sloID, err= to_string(.sloid)
      if err != null {
        log("Error on serviceid: " + err, level: "error")
      } 
      .status="APPROVED"
      .metricV= .series[0].pointlist[0]
      .metricValue, err=to_string(.metricV[1])
      if err != null {
        log("Error on pointlist value: " + err, level: "error")
      } 
      .="# TYPE "+.metricName+" gauge
      # HELP "+.metricName+" "+replace(.metricName, "_", " ")+" gauge
      "+.metricName+"{sloId=\""+.sloID+"\",status=\""+.status+"\"} "+.metricValue+"
      # EOF
     "
sinks:
  console_output:
    type: console
    inputs:
      - parse_dd_api_data
    encoding:
      codec: text

Version

vector-0.30.0-1.x86_64.rpm

Debug Output

No response

Example Data

{
	"status": "ok",
	"res_type": "time_series",
	"resp_version": 1,
	"query": "sum:Pipeline_Event_Count{service.environment:test}.as_count()",
	"from_date": 1683064680000,
	"to_date": 1683064980000,
	"series": [
		{
			"unit": null,
			"query_index": 0,
			"aggr": "sum",
			"metric": "Pipeline_Event_Count",
			"tag_set": [],
			"expression": "sum:Pipeline_Event_Count{service.environment:test}.as_count()",
			"scope": "service.environment:test",
			"interval": 2,
			"length": 1,
			"start": 1683064700000,
			"end": 1683064701000,
			"pointlist": [
				[
					1683064700000,
					76.8
				]
			],
			"display_name": "Pipeline_Event_Count",
			"attributes": {}
		}
	],
	"values": [],
	"times": [],
	"message": "",
	"group_by": [],
	"sloid": "033df0fd9c4040b98d94a5ec325fd3e9",
	"metricname": "availability_slo_total_count"
}

Additional Context

No response

References

No response

[neuronull]

Hello,

The VRL playground event data provides a means to give the VRL program anything you want.

When you get to an actual vector pipeline though, the event data is probably different from what you provided in that example, because we process the raw incoming data in the source.

Did you try piping directly to the console (without the transform), and inspecting the result?

Probably what is going on here is the fields you are looking for are behind the .message , if you are ingesting log events.