Vector with aws s3/sqs as Source and Splunk hec as Sink - help required to fine tune Vector and fix throttle issues

[ankycampy]

Hi All, Vector is deployed in a single dedicated server with 16vCPU and 32GB RAM.
Aws_s3 source with sqs is configured as a source, this is a very huge volume source.
Sink is Splunk HEC.

When started the vector with required config, i see that vector cpu utilization seems stuck at 30% only and not increasing. i believe vector is pulling limited logs per batch and hence less logs are sent to destination as compared to available at source.
As of now, i have reverted back the config to original architecture which does not involve vector. we wanted to see how vector performs in default configuration, now we want to tune it and then retest.

we have single s3 bucket which receives a huge volume of data, how to maximize vector resource utilization to achieve its full potential. We are not sure if we can have multiple s3 buckets.
we can vertically scale vector as per requirement. but first we are looking for config examples which can help tuning vector to run it with all available server resources, should we focus on increasing source specific throttle config, an example or reference will be useful if someone has worked on similar setup.

Thanks

[pront]

Hi @ankycampy,

Could you share the following?

• What log formats/types are in S3?
• Typical S3 object size (many small files vs few large ones)?
• Your current Splunk HEC batch settings or limits?
• What's the expected/desired throughput (events/sec or MB/s)?

Some ideas:

1. S3 source: tweak/increase https://vector.dev/docs/reference/configuration/sources/aws_s3/#sqs.client_concurrency

2. Splunk HEC sink: allow larger, concurrent batches

batch.max_bytes    = 5_000_000
batch.max_events   = 5_000

3. Observe internal_metrics to see real-time throughput (docs). See our guide for details: https://vector.dev/guides/developer/debugging/

[ankycampy]

Hi @pront,

Thanks for the response, below are requested details:

• S3 contains aws vpcflow logs and format is version 2, we have customized to suit our needs.
• s3 objects are all large
• I am using default Splunk HEC settings but will try what you suggested
• I will check what current through i am seeing then will reply again.

Thanks

[ankycampy]

Regarding sqs.client_concurrency, i see below in vector docs:

Defaults to the number of available CPUs on the system

But i still tried to set it to 12, but it does not make any difference. Also, i saw the instance's cpu utilization was not going beyond 30%.

[ankycampy]

@pront Here is the vector top which i captured:

i believe splunk_hec_vpc sink is processing fine what it is receiving from vpc_flow_pipe4.

vpcflow logs are going through various pipes as shown in the snap, what i think is vector should have pulled a lot more data but it may be waiting for next pipe or queue if it is full. I have not touched any queue specific config in vector yet. Any references for queues config like defining a queue size or if it can use RAM or Disk as a queue space ?
How to check if queue is full or which transform queue is full due to which processing is slowed down ?

i tried 2 ec2 instance types c5.4xlarge abd c6i.4xlarge and seeing same behaviour, cpu utilization is not going above 30%.