ignore_older_secs clarification

[ktamal]

Hello.

As per vector docs for ignore_older_secs Vector should ignore files with a modification date older than the specified time.

But when we restart the vector application we see that vector starts processing the same logs that were already processed before the restart. After some research we found out that during vector restart it using the ignore_older_secs to determine where to resume from.

The calculate ignore before and the set state for the checkpointer seems to clearly indicate our findings. The ignore_older_secs is being used by the checkpoints.

Please can we get some some clarifications on exactly how to use ignore_older_secs. Our main problem here is that when we restart vector, we want to resume processing from where vector last left off. I am not finding anything specific about the checkpoints in this discussion thread

[bruceg]

For clarity, are you seeing that vector re-reads logs messages that were already handled before the restart, or that it continues reading log files that it had open after the restart? If it is the former, it is a bug, as that should automatically be handled by the checkpoint logic.

[almarcotte]

For clarity, are you seeing that vector re-reads logs messages that were already handled before the restart, or that it continues reading log files that it had open after the restart? If it is the former, it is a bug, as that should automatically be handled by the checkpoint logic.

It looks like ignore_older_secs is used to expire checkpoints too, which causes problems when used with read_from: beginning, see:

vector/lib/file-source/src/file_server.rs

Line 100 in 32a935b

checkpointer.read_checkpoints(self.ignore_before);

To reproduce

1. Use read_from: beginning and ignore_older_secs: 60
2. Start Vector
3. Write to my_file.log
4. The checkpoints.json file is updated with proper checkpoint values
5. Stop Vector
6. Continue writing to my_file.log for 1-2 minute (any time longer than ignore_older_secs)
7. Restart Vector

The logs will show that Vector found a new file (instead of resuming), checkpoints.json will be empty and my_file.log will be reprocessed entirely.

[bruceg]

Ah yes, that does sound like a real soundness hole, and does agree with my reading of the code. I have opened issue #17208 to track this.

[ktamal]

thanks!