monitoring file completion status in file-to-Kafka pipelines

[whyjp]

When using Vector for a file-to-Kafka pipeline, is there a way to configure a dashboard or check metrics to confirm that file uploads are complete/done?

Due to the complexity of various rotation/rolling environments and real-time tailing issues, the pairing of filename:offset may not match the metrics:offset of transmitted logs.

Based on the current "file EOF reached" metrics content, it's difficult to make an accurate determination.

Is there any good method to precisely determine when a file is done - monitoring stopped - and no longer being written to?

[pront]

Hi @whyjp, the file source emits the open_files gauge metric (which is unfortunately not documented). Is that what you are looking for?

[whyjp]

I know that -open_files
it is just file monitor begin but I want to know rich to eof of files each files

I can see that it started with the current metric, but it's hard to know that it's been handled properly and completely to the end.