Extract Azure Flowlog content with information from different level

[difanzhao]

Hi expert,
I am new to Vector and I am from the networking background so the question maybe basic.

I am asked to parse Azure Flowlog, which is a json file. I am attaching one here for reference:
flows_github.json

The most important part is the Flowtuple which is like "1745265567019,40.87.176.0,10.1.1.69,23456,22932,6,I,E,NX,1,128,0,0". I have splitted them and save each one in the key-value pair like this:
{ "action":"E", "bytesSentDestToSource":"0", "bytesSentSourceToDest":"128", "destIp":"10.1.1.69", "destPort":"22932", "direction":"I", "flowstate":"NX", "packetsDestToSource":"0", "packetsSourceToDest":"1", "protocol":"6", "srcIp":"40.87.176.0", "srcPort":"23456", "unixtimestamp":"1745265567019" }

The question is that I also wanted to add the information from the higher level in the json file into the tuple. They are "macAddress", "aclID" and "rule", all from different levels. The final result would look like this:
{ "action":"E", "bytesSentDestToSource":"0", "bytesSentSourceToDest":"128", "destIp":"10.1.1.69", "destPort":"22932", "direction":"I", "flowstate":"NX", "packetsDestToSource":"0", "packetsSourceToDest":"1", "protocol":"6", "srcIp":"40.87.176.0", "srcPort":"23456", "unixtimestamp":"1745265567019","macAddress": "000D3AC65D84","aclID":"xxx","rule":"PlatformRule"}

How do I achieve this? Here is my current vector.yaml config file.
vector_github.yaml.txt

Please let me know what function or plugin to use. If there are already similar examples, a link would be much appreciated! Thanks!

[pront]

Hi @difanzhao, I see you are already using the remap transform, which is the goto transform for manipulating log events. I highly recommend plugging a sample event in the https://playground.vrl.dev/ and iterating on your VRL program.

Another small recommendation, you don't need really multiple remaps unless you want to fanout and feed different components down the pipeline.

[difanzhao]

Hi Pront, thanks for your response.

Regarding to the multiple remap part. I have tried but couldn't callapse them into one... As soon as I use the "unnest", it splits the input into multiple events. After that, I can't manipulate each field. For example, let's say my simplied original data is like this:

{
  "records": [
    {"targetResourceID": "/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Network/virtualNetworks/spoke11"},
    {"targetResourceID": "/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Network/virtualNetworks/spoke12"}
  ]
}

I want to extract the the last part which is called "spoke11" and "spoke12" and I want to save them to a new variable "vnet". Eventually, I want my output to be like

{"vnet": "spoke11"}
{"vnet": "spoke12"}

I used this config to parse it and I would get the "undefine error"

transforms:
  extract_records:
    type: remap
    inputs: 
      - json_files
    source: |
      . = parse_json!(.message)
      . = .records
      . = unnest!(.)
      .vnet = split!(.targetResourceID, "/")[-1] <--- error: this expression resolves to the exact type undefined

But, if I added another remap section, it will work fine

  extract_flows:
    type: remap
    inputs: 
      - extract_records
    source: |
      .vnet = split!(.targetResourceID, "/")[-1]

How do I solve this? Also, does multiple remap causes more CPU or Memory usage on the server?
Thanks!

[pront]

Hi @difanzhao, does this example help? It fixes the split!(.targetResourceID, "/")[-1] line.

Also, does multiple remap causes more CPU or Memory usage on the server?

As you add more components to the pipeline CPU and memory usage can increase. But this is not necessarily a bottleneck.