S3 Batching with VRL Template syntax Key_prefix

[satellite-no]

Quick question on the expected behavior of batching to s3. I have the below s3 config for my sink which is accepting a lot of different data sources. Based on this config would my assumption of multiple 30MB files being written to s3 OR 1 file a every 60 seconds be correct? We are seeing hundreds of thousands of small 10kb files being written in reality. The running theory we are testing is that the VRL Syntax is causing the files to be closed and written to s3 before they hit either condition.

For an example we have a stream of two data sources with different .unmapped.dl_partition_class_name so sense the stream is alternation the variable its causing the files to be written premature.

s3:

type: aws_s3
inputs:
  - 'ocsf_*'
bucket: "dest_bucket_for_ocsf"
region: "us-east-1"
key_prefix: "ocsf/{{.unmapped.dl_partition_class_name}}/{{.unmapped.dl_partition_vendor_name}}/%Y%m%d/"
compression: zstd

healthcheck:
  enabled: true
  timeout_secs: 15

batch:
  max_bytes: 30973400 # 30 MB
  timeout_secs: 60

encoding:
  codec: json

[pront]

When batching is configured like this:

batch:
  max_bytes: 30973400  # 30MB
  timeout_secs: 60     # 60s

Vector should flush a batch to S3 when either:

• The batch reaches 30MB
• 60 seconds pass since the batch started
• Or per when https://vector.dev/docs/reference/configuration/sinks/aws_s3/#batch.max_events are accumulated

However, a separate batch is maintained for each unique S3 object key, and in your case, the key is dynamically derived from:

ocsf/{{.unmapped.dl_partition_class_name}}/{{.unmapped.dl_partition_vendor_name}}/%Y%m%d/

If .unmapped.dl_partition_class_name or .unmapped.dl_partition_vendor_name alternates frequently (e.g. for every event), then Vector will never accumulate a large batch, because each event is routed to a different S3 object key.

[satellite-no]

Hmm, trying to understand this and have a couple of questions.

1. you linked the vector sink, Im using the s3 sink. vector sinks says 500 events default for max_events but s3 doesnt list a default. Is it also 500 events and I need to increase that or was that just an example? I really like it based on file size so negating the max_events is what I thought was happening.
2. It seems like you start off saying that I should be expecting larger files but then i seems to drift to because we have dynamic prefix and its alternating we wouldnt have large files so im a little confused there.

Scenario 1:
Say its just key_prefix: ocsf/{{.unmapped.dl_partition_class_name}}/ and .unmapped.dl_partition_class_name has two possible values that are streaming GB of data a minute. Should I expect many 30MB files or will they be small files because of it alternating.

My confusion in what you say is if it has a separate batch maintained for each unique S3 object key, then there are two batches being maintained which should route the events accordingly but if its maintaining 1 batch then sees the swap its writes that batch and starts a new one.

Note: in my usecase {{.unmapped.dl_partition_class_name}}/{{.unmapped.dl_partition_vendor_name}} does not have a lot of possible values were talking maybe 30 batches total.

[pront]

1. you linked the vector sink, Im using the s3 sink. vector sinks says 500 events default for max_events but s3 doesnt list a default. Is it also 500 events and I need to increase that or was that just an example? I really like it based on file size so negating the max_events is what I thought was happening.

Fixed the link. Unless specified it should use the default. In this case, it actually is None. So this setting doesn't really affect batching unless specified in the config.

2. It seems like you start off saying that I should be expecting larger files but then i seems to drift to because we have dynamic prefix and its alternating we wouldnt have large files so im a little confused there.

Scenario 1: Say its just key_prefix: ocsf/{{.unmapped.dl_partition_class_name}}/ and .unmapped.dl_partition_class_name has two possible values that are streaming GB of data a minute. Should I expect many 30MB files or will they be small files because of it alternating.

Very roughly, in the simplest of cases, only one distinct key, I would expect something close to 1GB/30MB files.

My confusion in what you say is if it has a separate batch maintained for each unique S3 object key, then there are two batches being maintained which should route the events accordingly but if its maintaining 1 batch then sees the swap its writes that batch and starts a new one.

For sinks the partitioner should determine how many batches are active at any given time. Without diving into code to confirm, I believe the key_prefix defines the aws_s3 partitions. So the sink should be building a batch per partition.

Edit: If the above doesn't answer your question, I will try to find some time to test this scenario myself. But cannot promise an ETA.

[satellite-no]

Thanks after doing some testing and validating of different settings I see we are hitting the issue with max_bytes being unserialized/precompressed and not this. making that settings for batch roughly 1000x the desired file size helps a bunch for us.