LL-HLS falling behind, "Failed to respond to pending request." #1893
Replies: 11 comments
-
|
If WebRTC is working fine but LLHLS is buffering a lot, maybe take a look at your HTTP cache server Also, since this just started happening today, did anything change recently? Like an OME version upgrade or any config tweaks? |
Beta Was this translation helpful? Give feedback.
-
|
Just a heads-up—OME handles everything in memory except for recording output. But if your recordings (or LLHLS dumps) are going to a slow HDD, that could still cause delays and impact playback. |
Beta Was this translation helpful? Give feedback.
-
|
Somewhat higher load than normal today. I don't believe it's the cache
server - ome logs show it failing to respond to requests so I'm pretty sure
the problem is there
Also the llhls playback was extremely delayed, up to an hour or more for
some streams. Seems very unusual and I'm not sure how that would happen
with the cache server
That said, I'll check the nginx cache is working properly
…On Fri, Jul 11, 2025 at 22:20 Jeheon Han ***@***.***> wrote:
*getroot* left a comment (AirenSoft/OvenMediaEngine#1873)
<#1873 (comment)>
If WebRTC is working fine but LLHLS is buffering a lot, maybe take a look
at your HTTP cache server
Also, since this just started happening today, did anything change
recently? Like an OME version upgrade or any config tweaks?
—
Reply to this email directly, view it on GitHub
<#1873 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABV4BWJ5C5LOC2HA3BUVJP33IBWIVAVCNFSM6AAAAACBLDBXEKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTANRUGUZDCNRRHA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Recording dumps being slow can affect playback?
…On Fri, Jul 11, 2025 at 22:23 Jeheon Han ***@***.***> wrote:
*getroot* left a comment (AirenSoft/OvenMediaEngine#1873)
<#1873 (comment)>
Just a heads-up—OME handles everything in memory except for recording
output. But if your recordings (or LLHLS dumps) are going to a slow HDD,
that could still cause delays and impact playback.
—
Reply to this email directly, view it on GitHub
<#1873 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABV4BWO4VURK57ETBIJCSRD3IBWQ5AVCNFSM6AAAAACBLDBXEKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTANRUGUZDIMRSGA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Yes, since OME dumps directly from the LLHLS Packager thread, if the dump is slow, it will affect the packaging as well. When I looked into the logs, there were a lot of logs like the following, and this points exactly to that. |
Beta Was this translation helpful? Give feedback.
-
|
Is the performance of creating a recording vs llhls dumps different?
We've had corrupt recordings so we simultaneously dump llhls to then
process into recordings, would only dumping llhls be reasonable?
…On Mon, Jul 14, 2025 at 03:31 Jeheon Han ***@***.***> wrote:
*getroot* left a comment (AirenSoft/OvenMediaEngine#1873)
<#1873 (comment)>
Yes, since OME dumps directly from the LLHLS Packager thread, if the dump
is slow, it will affect the packaging as well.
When I looked into the logs, there were a lot of logs like the following,
and this points exactly to that.
�[33m[2025-07-12 00:39:39.262] W [OutboundWorker:68] ManagedQueue | managed_queue.h:586 | [149] mngq:v=#default#live:p=pub:n=aw-llhls1 has exceeded the threshold and increased peak. size: 5507392, threshold: 500, peak: 5507392�[0m
—
Reply to this email directly, view it on GitHub
<#1873 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABV4BWLTY6SOREO3IRRY5QL3INMDVAVCNFSM6AAAAACBLDBXEKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTANRYGE2DSMZVHA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
The Recording function is performed in a separate thread, and the LLHLS Dump function is performed in the same thread as the LLHLS Packager. In other words, it is recommended to use only the Recording function in order not to affect LLHLS streaming. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah we'd love to just use recordings but as reported before, we get too
many corrupt recordings.
…On Mon, Jul 14, 2025 at 06:36 Jeheon Han ***@***.***> wrote:
*getroot* left a comment (AirenSoft/OvenMediaEngine#1873)
<#1873 (comment)>
The Recording function is performed in a separate thread, and the LLHLS
Dump function is performed in the same thread as the LLHLS Packager. In
other words, it is recommended to use only the Recording function in order
not to affect LLHLS streaming.
—
Reply to this email directly, view it on GitHub
<#1873 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABV4BWJ3ABV2XGGKBZEMZIT3IOB2RAVCNFSM6AAAAACBLDBXEKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTANRYHA4DSMBVGU>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
How about using the continuous recording function that records once every 4 seconds, like LLHLS Dump? |
Beta Was this translation helpful? Give feedback.
-
|
I'm not familiar with what you're suggesting, do you mean configuring recordings to split every 4 seconds? https://docs.ovenmediaengine.com/recording#split-recording Or is there a different feature? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, that's correct. It outputs the file once every specified time (e.g. 4 seconds), so in the worst case you only lose the last 4 seconds. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Describe the bug
Our OME instance today is very very behind on HLS playback, but WebRTC is fine. Nothing in particular in the logs other than lines like
[2025-07-12 00:51:39.754] W [SW-LLHLS:1641033] LLHLS Publisher | llhls_session.cpp:497 | #default#live/streamid/llhls.m3u8 Failed to respond to pending request.
We're creating LLHLS dumps to our hard drive (regular spinning disk), as well as generating standard OME recordings to the same place.
We've got a very fast SSD for the main drive and I think everything but the recording stuff, though I'm not sure if OME uses any others for anything?
Memory pressure is low, as is CPU pressure. Not sure what else to check.
Logs
https://files.catbox.moe/d9q86a.gz
Beta Was this translation helpful? Give feedback.
All reactions