When pushing to AWS S3 sink, files are never bigger than 256Mb

[cduverne]

Hello,

We're setting up a new vector sink towards AWS S3, after reading data from Kafka.
Unfortunately, eventhough there are Gb of data read from Kafka, the output files are always capped at 256Mb.

We've setup disk buffer, with a large max buffer size, without success.

Here is our config :

  type = "aws_s3"
  inputs = ["kafka_topic"]
  bucket = "{{ outputBucketName }}"
  endpoint = "{{ s3Endpoint }}"
  key_prefix = "file_{{ dateFormat }}"
  filename_extension = "{{ outputFileFormat }}"
  filename_time_format = ""
  filename_append_uuid = false
  compression = "none"

  # Encoding
  encoding.codec = "{{ outputFileFormat }}"
  encoding.csv.delimiter = "{{ delimiter }}"
  encoding.csv.quote_style = "{{ quoteStyle }}"
  encoding.csv.fields = ["md_version", "created_on", "topic", "id", "payload"]
  # Maximum size of internal buffer for writing CSV - 1GB in bytes
  encoding.csv.capacity = 1073741824

  # Healthcheck
  healthcheck.enabled = true
  # Bufferring events on the disk
  buffer.type = "disk"
  # Maximum 5Gb in the buffer
  buffer.max_size = 5368709120
  # If buffer is full wait for free space
  buffer.when_full = "block"

  # Every hour (in seconds)
  batch.timeout_secs = 3600
  # Maximum size of single batch - 1GB in bytes
  batch.max_bytes = 1073741824```
  

  Any help would be much appreciated.

[AlexandrePieroux]

Hello, I experience the same issue on my workbench! Any help with this ?

[ignacyklimont]

I have noticed the same issue. For me, the maximum output file from Vector to S3 sink is always capped at around 275.000.000 bytes, or roughly 260MB. I've set both buffer, batch, max CSV size etc. to very large values (>1GB, batch timeout at two hours), yet it still caps the output at no more than 256-265 MB.

I can see that this leads to missing events in S3, so clearly the Vector struggles with flushing larger output files to S3 (in our case it is CSV). When we set the timeout to low value, such as 300 seconds, it works correctly and flushes all events, meaning all events from Kafka for that day are available in S3. Seems like a problem in Vector S3 sink with flushing larger CSV files.

[pront]

Hello, and thank you for providing details here. Since there's an open issue for this problem, let's continue the discussion there: #22866