Backup for broken chunks occupying too much disk space

[rustlingwind]

Hi guys,

I just found our AWS EKS (Fargate) pods frequently raised disk space issue and I noticed that the size of /tmp directory of each pod increased quickly.

5 days before:
/# df -lh
Filesystem Size Used Avail Use% Mounted on
overlay 30G 17G 12G 59% /

5 days after:
/# df -lh
Filesystem Size Used Avail Use% Mounted on
overlay 30G 24G 4.8G 83% /

The space was ate by fluentd tmp backup files, the pods age is 6 days, the ate speed is about 1Gi per day.
/tmp/fluent/backup/worker0/out_cloudwatch_logs_containers# du -h --max-depth=1
6.7G .
/tmp/fluent/backup/worker0/out_cloudwatch_logs_containers# ls -l | grep "^-" | wc -l
4948

Is there a way to disable or limit the fluentd backup for broken chunks? Any impact in case disable or limit it?
Or is there a rotation for these backup of broken chunks? So that these tmp files can be cleaned up automatically when size or duration reach a threshold.

Much appreciated if any advice.

P.S. Regarding the tmp backup for broken chunks:
#1952
https://www.fluentd.org/blog/fluentd-v1.2.0-has-been-released

[rustlingwind]

Just found the feature already existing: #2117
Will have a test and update to here later.

[daipom]

Yes! You can use disable_chunk_backup!

https://docs.fluentd.org/buffer#handling-unrecoverable-errors