Performance issues

[satellite-no]

Background: I have a large Vector deployment Im running into performance issues with. Im seeing around 50MB/s average on my source & sink with 16vCPU nodes. Based on the capacity planning guide I'd expect this to be more like 160MB/s. Now with that being said our transform rate IS around 160MB/s. Note CPU on these instances if low <50% but RAM is almost always maxed out (disk buffer in use).

Question:

1. What metric is best to use to compare the throughput rate with the estimated?
2. How do I know where my bottle neck is? I'm assuming its a sink but im only sending data to s3 & Clickhouse based on some testing I think s3 is the issue but I'd assume I could write to s3 from an AWS VPC very quickly.

Any ideas to get these systems performing as expected would be appreciated.

[jszwedko]

Hi @satellite-no ,

Can you share your configuration? That might help us help you identify bottlenecks.

To identify bottlenecks in the topology, I usually look at the utilizationinternal metric. This metric is an indication of the ratio of time a given component spends processing vs. waiting for input. Typically the first component in a given topology path that has a utilization close to 1 is the bottleneck as it is never waiting for input.

[satellite-no]

I have a complex config so Im not certain i can share the entire thing but below is my s3 configuration and a generalized flow of data.

type: aws_s3
inputs:
  - 'route_ocsf_to_s3.all_vendors'
bucket: "bucket_name"
region: "us-east-1"
key_prefix: "ocsf/{{.unmapped.dl_partition_class_name}}/{{.unmapped.dl_partition_vendor_name}}/%Y%m%d/"
compression: zstd

healthcheck:
  enabled: true
  timeout_secs: 15

batch:
  max_bytes: 5368709000 # 5GB
  timeout_secs: 60

encoding:
  codec: json
  
buffer:
  type: "disk"
  max_size: 200000000000 # 200GB
  when_full: "block"

source (s3) -> remap -> sink (vector) -> source (vector) -> exclusive_route -> exclusive_route -> remap -> remap -> exclusive route) -> sink

My utilization percents are normally less than 50% across all sources, sinks, and transforms.

[satellite-no]

the other think i noticed is if I increase my batch timeout or the max_bytes from here my system will hard crash and need rebooted. I think this is an OOM issue but its very stange sense a large instance and uses a disk buffer.

[jszwedko]

Gotcha, I wonder if it is the bottleneck is the aws_s3 sink uploading batches since, currently, batches are built in memory and uploaded all at once (#3829 is an issue to improve that with multi-part uploads). To verify if that is the bottleneck, you could try replacing the sink with a dummy sink like blackhole and observing if you see higher throughput.

[satellite-no]

S3 sink is my assumption as well. I think the batching deff needs some help to keep up with large volumes of data.

I did have a n s3 source (new) that when I initially enabled it was at about 1.5Gb/s before i connected it to the s3 sink and then it went down to 50MB/s. Currently I have roughly 30TB of daily ingest maybe, I cant say for sure because my 40 Vector nodes are to unstable to know a true ingest volume right now.

Do you have any recommendations for a sink that would perform better in getting data to s3? I was debating on the AWS Firehose sink but I see it batches too so was scared it would have the same issue.