Why was fsnotify removed in favor of polling, especially in large log directories?

[cobolbaby]

Hi mtail team,

I noticed that fsnotify support was removed and replaced with a polling-based file watching mechanism. I’d like to understand more about this decision, especially since we’re using mtail in a production environment where:

• A single log directory may contain at least 1,000 files.
• New log files may be created frequently (e.g., per session or per pod).
• Efficiency and timely detection of new files are both critical.

I’m wondering:

• What were the main issues with fsnotify that led to its deprecation?

  • Was it due to reliability, resource usage, OS compatibility, or race conditions?

• In high-scale environments, polling every few seconds can result in significant IO overhead—was this trade-off considered?

• Are there any best practices or tuning parameters when using mtail in directories with large numbers of log files?

• Would you consider reintroducing fsnotify (even optionally) for users who want more efficient notification mechanisms?

Thanks again for maintaining mtail. It’s been very helpful in extracting metrics from application and database logs.

Best regards,