when logs are big, do not following all new logs #4173
Replies: 2 comments 4 replies
-
|
Without some degree of reproducibility, it is hard to investigate and judge if this is a bug of Fluentd. So I moved this issue to discussions (the contributing guidelines). |
Beta Was this translation helpful? Give feedback.
-
|
Does it reproduce every time when there is some large file? I am concerned about the next message. Has the machine been running abnormally slow? Weren't resources such as CPU and memory exhausted? |
Beta Was this translation helpful? Give feedback.
-
|
yes, it run as a daemonset in a k8s cluster. The above scenario occurs on some heavily loaded hosts which include run other heavily loaded apps. |
Beta Was this translation helpful? Give feedback.
-
|
Here is what we know from the information so far.
It is a specification that if Fluentd cannot find a file, it cannot be collected. We do not know the detailed cause of the failure to find files in the half-collected state, but we can at least assume that it is due to a lack of resources. Please try to resolve the lack of resources problem so that this warning message does not occur.
Also, please try updating Fluentd(td-agent) to the latest version. |
Beta Was this translation helpful? Give feedback.
-
|
Yesterday I increased the requests of cpu (fluentd pod's cpu requests from 0.1 to 0.5 ) and it worked fine this morning(also has log |
Beta Was this translation helpful? Give feedback.
-
|
These days worked fine. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Describe the bug
One of my apps writes more logs every morning. Each log size is 512 MB. Two log files from each app are kept in the log directory and the rest are migrated to nas. I found that some log files are not collected. fluentd's logs are as follows:
From above logs,
2023-05-11.1.app-9.loghad no following log and there are two warn logs:the start time of
2023-05-11.1.app-9.logis2023-05-11 01:27:21.277and the end time is2023-05-11 01:52:04.232. So I think from2023-05-11 01:27:21.277to2023-05-11 03:36:13(Be moved to nas) ,the log2023-05-11.1.app-9.logwas not being following. But the posititon file has this log entry:Because the logs are collected in clickhouse, the log content of this log file is not found in clickhouse either.
To Reproduce
Collect larger log files to clickhouse
Expected behavior
Don't lose logs
Your Environment
Your Configuration
<source> @type tail path /data/logs/*.log exclude_path ["/data/logs/console.log"] pos_file /data/td-agent-data/position/app.log.pos path_key log_path read_from_head true follow_inodes true enable_stat_watcher false rotate_wait 10 tag app refresh_interval 2 limit_recently_modified 86400 read_lines_limit 3000 multiline_flush_interval 1s emit_unmatched_lines false <parse> @type multiline format_firstline /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3}/ format1 /^Matching Rules$/ </parse> </source>Your Error Log
Beta Was this translation helpful? Give feedback.
All reactions