Vector Agent - Kubernetes source fails to follow configs "ignore_older_secs: 86400" and "read_from: end"

[EdN-terascope]

In Vector Agent the kubernetes source is configured to not replay old log events, but old logs are still being processed.

using vector 0.27.0 with helm chart 0.16.3 on kubernetes v1.22.13

Here is the source stanza from the helm chart - is this configured properly? to pick up logs that are a day old or less and follow events added to the end of the file.

         sources:
             kubernetes_logs:
               type: kubernetes_logs
               ignore_older_secs: 86400
               read_from: end
               self_node_name: ${VECTOR_SELF_NODE_NAME}

This should prevent older logs from being sourced, right? Currently prior events, in some cases going back many months, are being processed. Any suggestions are appreciated

Also using journald source with "since_now: true" and getting expected results - thanks

[jszwedko]

One note that the ignore_older_secs just applies to the file modification time rather than the timestamp on the logs themselves. It seems unlikely, but is it possible the log files are new, but the events in them are old?

[godber]

Assuming that's the case, wouldn't read_from: end prevent the reading of old events from a file with a recent modification stamp?

[jszwedko]

Yeah, that is the case as I understand it: ignore_older_secs just looks at the mtimes on the files.

[EdN-terascope]

Confirming the 0.log file is new, and always close to current, but contains old logs.

So, not expecting the ignore_older_secs config to stop the processing of the old logs. But I was expecting the read_from to tail the log file and not replay existing, old, log events.

Is there a reason the kubernetes_logs source has read_from instead of the config since_now used by the journald source?

[jszwedko]

The read_from option is inherited from the file source. It should cause Vector to start reading at the end of any new files it discovers.

since_now on the journald source is actually passed directly through to journalctl as --since=now to have it filter out old entries.