S3 Ingest Performance

[satellite-no]

I have large (16 cpu) AWS instances ingesting data from s3 using the sqs polling. The nodes dont seem to be doing much at around 20% CPU but I still get a backlog of SQS messages when I reduce the number of nodes polling. Currenty im using default settings for sqs. But I've tried different combinations of poll_sec, client_concurrancy, and max_number_of_messages. But total ingest seems to stick at around 50MB/s.

Any thoughts?

File sizes in s3 range a lot but are generally around 20MB.

s3 Config:

type: aws_s3
compression: auto
region: <region>
sqs:
  queue_url: https://sqs.us-east-1.amazonaws.com/<account>/<queue_name>
framing:
  method: newline_delimited
  newline_delimited:
    max_length: 1048576