Injest delay on specific API Key after upgrade to 2025.1.14109 #2431
-
|
We've been using Seq for many years so we're very experienced with it. We just upgraded to 2025.1 over the weekend and now we're seeing an injest delay of about 30 minutes on our busiest API Key. Other API Keys are injesting fine and I can see them showing up in Tail mode, but one specific API Key is lagging behind, sometimes by several hours. How should I diagnose this issue? |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 1 reply
-
|
It's running in a Docker container on Fedora CoreOS 41.20241122.3.0. 98 GB disk free. CPU utilization seems reasonable. The Seq process is using 5% of the memory with 35 GB RAM free. |
Beta Was this translation helpful? Give feedback.
-
|
I wrote a test logger that uses the same API Key, and its logs are showing up immediately, so this seems specifically be a combination of a high-volume logger and Seq 2025.1. Did 2025.1 introduce some sort of throttling behavior? |
Beta Was this translation helpful? Give feedback.
-
|
I've isolated the issue to this error: "Rejecting CLEF payload due to excessive message template expansion". What is the "limit" on message template expansion? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @korggy, thanks for dropping us a line. This was a change late in the 2024.3 release series: The message here is a little misleading; all this means is that the payload size was below the limit, but after expansion it exceeded the limit. Increasing the limit should address this, but because the default limit is already very high, it's probably better to either set a smaller limit at the source (if you're using a client logging library that supports it; e.g. Let me know if this helps, |
Beta Was this translation helpful? Give feedback.
Hi @korggy, thanks for dropping us a line. This was a change late in the 2024.3 release series:
#2365
The message here is a little misleading; all this means is that the payload size was below the limit, but after expansion it exceeded the limit.
Increasing the limit should address this, but because the default limit is already very high, it's probably better to either set a smaller limit at the source (if you're using a client logging library that supports it; e.g.
eventBodyLimitBytesin Serilog'sWriteTo.Seq()) or update your application to truncate/rewrite very large values such as e.g. request and response payloads before logging them.Let me know if this helps,
Nick