File source latency

[ahti]

I've been exploring using Vector to ship logs from a PHP(-FPM)/nginx application to Loki.

I started out using a syslog source, but after discovering the PHP syslog support to be somewhat broken (at least with 7.4.33) I switched to a file source reading ephemeral files inside the container. In switching, I noticed that the file source (feeding straight into a console sink, same for the syslog source) had much higher latency from clicking a link in the web app to the corresponding logs appearing in console.

When using the syslog source, there wasn't really any discernible latency, the same as when letting the web server output to stdout directly. When using a file source instead, the latency is much more variable but can get as high as ~10 seconds.

For reference, I tried tail -qc +0 --follow=name /var/log/{nginx,fpm}/*.log, matching the files Vector is monitoring. This also resulted in increased latency, but it seemed to stay roughly under 1 second.

Is there a way to tune the file source towards lower latency? I wouldn't actually mind that much, but I was hoping to avoid parsing the different date formats from the various kinds of logs the app produces (plus PHP will often log warnings with no timestamp at all), but the latency also affects the default timestamp Vector generates for the file source, and I would prefer to have timestamps more accurate than ~10s ^^

[jszwedko]

Hi @ahti ! You could try tweaking some of the file source settings. For example, setting max_read_bytes to a higher value might result in higher throughput (at the cost of fairness).