High utilization for lua transform

[amanmahajan26]

Hi,

I have a lua transform in my vector pipeline. When the traffic goes up, the utilization metric for this transform goes up to 100% (value of 1) and flattens. I want to understand a couple of things related to this event:

1. What impact does it have on the pipeline? Will it result in dropping of some events or just slow the processing?
2. How can I improve the performance for this particular transform?

Details about the transform:
In the init hook it reads a yaml configuration and creates a map.
In the process hook, it just uses the map to output the value for a given key.

Vector version: 0.20.1

Infra:
It is running in K8s on dedicated nodes (2 * c5.2xlarge) with 8vCPU each.

[jszwedko]

Hi @amanmahajan26 !

The lua transform is unfortunately usually a bottleneck when introduced as:

• It is not paralleizable by Vector and so only once instance runs (as opposed to other transforms like filter or remap)
• A single execution is significantly more expensive than the native transforms that would be surrounding it

However, it is definitely very flexible and is useful as a "trapdoor" for functionality lacking in Vector, for that reason.

There isn't much you can do to improve the performance of the transform, but if you wanted to share what you are using the transform for, we might be able to suggest alternatives that avoid the lua transform. We have some ideas for parallelizing the transform to improve its performance, but this is a ways off.

For impact, it will apply back-pressure back up to all of its inputs. This back-pressure will likely propagate all the way up to the sources where they will start consuming more slowly.

[amanmahajan26]

Thanks for your reply @jszwedko. The lua transform is being used to parse a config file which looks like this:

virtual_channels:
  <application_1>:
    <virtual_channel_1>:
    - <channel_1>
    - <channel_2>
    - <channel_3>
    <virtual_channel_2>:
    - <channel_4>
    - <channel_5>
  <application_2>:
    <virtual_channel_3>:
    - <channel_6>
    - <channel_7>

Based on the application, I need to map the channel to its corresponding virtual channel.

Is there any alternative to do this instead of using lua transform?

[jszwedko]

Thanks @amanmahajan26 ! I see. We do have an enrichment feature which can read from CSVs (https://github.com/vectordotdev/vector-demos/tree/main/csv-enrichment). If it is possible to structure your config file as a CSV file you could use enrichment tables instead. Another alternative would be to generate a VRL program to use with remap from the config file like:

if application = "1" {
  if virtual_channel = "2" {
  } else if virtual_channel = "3"
}

Maybe one of those might work?

[amanmahajan26]

Unfortunately, both of these solutions won't work for us. The config file is a nested map, so it's not possible to create a csv out of it. And since we keep adding new defintions in the config file, it would be difficult to maintain the same in the VRL program. I think the best approach for us right now is to use lua transform only.

[jszwedko]

Gotcha, yeah that makes sense 👍 Hopefully we are able to improve the lua transform performance in the future.

[JeanMertz]

We have recently landed iteration support to VRL, which would allow you to iterate over collections. Would that help migrate your transformation step away from Lua to VRL?

How I could see this work is:

• Have a pre-processor (outside of vector) that converts your config file into a valid VRL object upon deployment,
• inject that into the VRL program using Vector's environment variable pre-processing feature,
• then iterate over the configuration object, and do whatever you need to do to modify the actual event being processed.

Alternatively, if you don't want to do any preprocessing outside of vector, you could still load the config file in an environment variable, use VRL's get_env_var function to load the config as a string, and then manipulate the string until you have something that can be used to pull data from (f.e. morph it into a valid JSON object, and use decode_json).

There are many possible solutions here, I concur that none of them are as simple as loading a file in Lua, but the reason we don't allow that (and instead delegate to enrichment tables) in VRL, is because of performance reasons (which is one of the reasons why you're running into that issue in the existing Lua solution).