LL-HLS falling behind, "Failed to respond to pending request."

[naanlizard]

Describe the bug
Our OME instance today is very very behind on HLS playback, but WebRTC is fine. Nothing in particular in the logs other than lines like

[2025-07-12 00:51:39.754] W [SW-LLHLS:1641033] LLHLS Publisher | llhls_session.cpp:497 | #default#live/streamid/llhls.m3u8 Failed to respond to pending request.

We're creating LLHLS dumps to our hard drive (regular spinning disk), as well as generating standard OME recordings to the same place.

We've got a very fast SSD for the main drive and I think everything but the recording stuff, though I'm not sure if OME uses any others for anything?

Memory pressure is low, as is CPU pressure. Not sure what else to check.

Logs

https://files.catbox.moe/d9q86a.gz

[getroot]

If WebRTC is working fine but LLHLS is buffering a lot, maybe take a look at your HTTP cache server

Also, since this just started happening today, did anything change recently? Like an OME version upgrade or any config tweaks?

[getroot]

Just a heads-up—OME handles everything in memory except for recording output. But if your recordings (or LLHLS dumps) are going to a slow HDD, that could still cause delays and impact playback.

[naanlizard]

Somewhat higher load than normal today. I don't believe it's the cache server - ome logs show it failing to respond to requests so I'm pretty sure the problem is there Also the llhls playback was extremely delayed, up to an hour or more for some streams. Seems very unusual and I'm not sure how that would happen with the cache server That said, I'll check the nginx cache is working properly

…

On Fri, Jul 11, 2025 at 22:20 Jeheon Han ***@***.***> wrote: *getroot* left a comment (AirenSoft/OvenMediaEngine#1873) <#1873 (comment)> If WebRTC is working fine but LLHLS is buffering a lot, maybe take a look at your HTTP cache server Also, since this just started happening today, did anything change recently? Like an OME version upgrade or any config tweaks? — Reply to this email directly, view it on GitHub <#1873 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABV4BWJ5C5LOC2HA3BUVJP33IBWIVAVCNFSM6AAAAACBLDBXEKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTANRUGUZDCNRRHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[naanlizard]

Recording dumps being slow can affect playback?

…

On Fri, Jul 11, 2025 at 22:23 Jeheon Han ***@***.***> wrote: *getroot* left a comment (AirenSoft/OvenMediaEngine#1873) <#1873 (comment)> Just a heads-up—OME handles everything in memory except for recording output. But if your recordings (or LLHLS dumps) are going to a slow HDD, that could still cause delays and impact playback. — Reply to this email directly, view it on GitHub <#1873 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABV4BWO4VURK57ETBIJCSRD3IBWQ5AVCNFSM6AAAAACBLDBXEKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTANRUGUZDIMRSGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[getroot]

Yes, since OME dumps directly from the LLHLS Packager thread, if the dump is slow, it will affect the packaging as well.

When I looked into the logs, there were a lot of logs like the following, and this points exactly to that.

�[33m[2025-07-12 00:39:39.262] W [OutboundWorker:68] ManagedQueue | managed_queue.h:586  | [149] mngq:v=#default#live:p=pub:n=aw-llhls1 has exceeded the threshold and increased peak. size: 5507392, threshold: 500, peak: 5507392�[0m

[naanlizard]

Is the performance of creating a recording vs llhls dumps different? We've had corrupt recordings so we simultaneously dump llhls to then process into recordings, would only dumping llhls be reasonable?

…

On Mon, Jul 14, 2025 at 03:31 Jeheon Han ***@***.***> wrote: *getroot* left a comment (AirenSoft/OvenMediaEngine#1873) <#1873 (comment)> Yes, since OME dumps directly from the LLHLS Packager thread, if the dump is slow, it will affect the packaging as well. When I looked into the logs, there were a lot of logs like the following, and this points exactly to that. �[33m[2025-07-12 00:39:39.262] W [OutboundWorker:68] ManagedQueue | managed_queue.h:586 | [149] mngq:v=#default#live:p=pub:n=aw-llhls1 has exceeded the threshold and increased peak. size: 5507392, threshold: 500, peak: 5507392�[0m — Reply to this email directly, view it on GitHub <#1873 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABV4BWLTY6SOREO3IRRY5QL3INMDVAVCNFSM6AAAAACBLDBXEKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTANRYGE2DSMZVHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[getroot]

The Recording function is performed in a separate thread, and the LLHLS Dump function is performed in the same thread as the LLHLS Packager. In other words, it is recommended to use only the Recording function in order not to affect LLHLS streaming.

[naanlizard]

Yeah we'd love to just use recordings but as reported before, we get too many corrupt recordings.

…

On Mon, Jul 14, 2025 at 06:36 Jeheon Han ***@***.***> wrote: *getroot* left a comment (AirenSoft/OvenMediaEngine#1873) <#1873 (comment)> The Recording function is performed in a separate thread, and the LLHLS Dump function is performed in the same thread as the LLHLS Packager. In other words, it is recommended to use only the Recording function in order not to affect LLHLS streaming. — Reply to this email directly, view it on GitHub <#1873 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABV4BWJ3ABV2XGGKBZEMZIT3IOB2RAVCNFSM6AAAAACBLDBXEKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTANRYHA4DSMBVGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[getroot]

How about using the continuous recording function that records once every 4 seconds, like LLHLS Dump?

[naanlizard]

I'm not familiar with what you're suggesting, do you mean configuring recordings to split every 4 seconds?

https://docs.ovenmediaengine.com/recording#split-recording

Or is there a different feature?

[getroot]

Yes, that's correct. It outputs the file once every specified time (e.g. 4 seconds), so in the worst case you only lose the last 4 seconds.