You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I want to limit the file size. I searched and found that vector has no relevant function and suggested using logrotate to implement it. Therefore, logrotate is used to rotate according to the file size, and the configuration is as follows:
# test the specified file
/fuse/logfs/vector/ban.ban_read_flink_log/vector-aggregator-0/ban.ban_read_flink_log-2025-04-17.log {
daily
dateext
dateformat -%Y%m%d%H%M%S
copytruncate
maxsize 500M
rotate -1
missingok
notifempty
}
After testing, it was found that Logrotate Successful, original file goes back to original size
Verbose Output of Logrotate
copying /fuse/logfs/vector/ban.ban_read_flink_log/vector-aggregator-0/ban.ban_read_flink_log-2025-04-17.log to /fuse/logfs/vector/ban.ban_read_flink_log/vector-aggregator-0/ban.ban_read_flink_log-2025-04-17.log-20250417102809
truncating /fuse/logfs/vector/ban.ban_read_flink_log/vector-aggregator-0/ban.ban_read_flink_log-2025-04-17.log
Log file after truncate happens
-rw-r--r-- 1 root root 0 Apr 17 10:28 ban.ban_read_flink_log-2025-04-17.log
-rw-r--r-- 1 root root 632M Apr 17 10:28 ban.ban_read_flink_log-2025-04-17.log-20250417102809
Literally Seconds Later
-rw-r--r-- 1 root root 634M Apr 17 10:28 ban.ban_read_flink_log-2025-04-17.log
-rw-r--r-- 1 root root 632M Apr 17 10:28 ban.ban_read_flink_log-2025-04-17.log-20250417102809
od -c ban.ban_read_flink_log-2025-04-17.log | head -n 10
0000000 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
*
4736315260 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 S U C
4736315300 C E S S ( S c a n B y I P U A N
4736315320 U R L D o S ( 2 2 2 / 6 0 0 .
4736315340 0 ) ) , p a y l o a d : { "
4736315360 r u l e _ t y p e " : " s c a
4736315400 n _ c o u n t " , " i p " :
4736315420 " 1 1 0 . 1 6 6 . 2 1 3 . 1 4 4
4736315440 " , " u s e r _ a g e n t " :
But strangely, when starting local vector -c vector.yaml, a file of about 1G size is created through dd, and then logs are generated through demo_logs and continuously written to the file. After using the same logrotate, no abnormality is reproduced
It is suspected that the written oftest has not been reset.ref The file system uses juicefs, which is mounted in the vector running in K8S through juicefs-csi-driver. Logrotate runs in a machine that also mounts the same file system. Is it related to this configuration? How should I make it work properly, or can the file sink provide a corresponding file size limit processing method ?
I will continue to try to see if there is any easy way to reproduce it.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Use helm to deploy vector-aggregator in k8s, and its configuration is as follows:
I want to limit the file size. I searched and found that vector has no relevant function and suggested using logrotate to implement it. Therefore, logrotate is used to rotate according to the file size, and the configuration is as follows:
After testing, it was found that Logrotate Successful, original file goes back to original size
But strangely, when starting local vector -c vector.yaml, a file of about 1G size is created through dd, and then logs are generated through demo_logs and continuously written to the file. After using the same logrotate, no abnormality is reproduced
It is suspected that the written oftest has not been reset.ref The file system uses juicefs, which is mounted in the vector running in K8S through juicefs-csi-driver. Logrotate runs in a machine that also mounts the same file system. Is it related to this configuration? How should I make it work properly, or can the file sink provide a corresponding file size limit processing method ?
I will continue to try to see if there is any easy way to reproduce it.
Beta Was this translation helpful? Give feedback.
All reactions