Possibility to maintain consistent LLHLS playlist between edges in OriginMode

[fatchan]

Hi, I've got multiple edges setup pulling from an origin. RTMP ingest, LLHLS publisher. All edges are using OriginMode true.

When using LLHLS and OriginMode, I noticed that the playlists are slightly different between the edges because they might not start pulling from the origin at exactly the same time. If for example we use round robin from our CDN to the multple edges, I'm hit with errors/warnings like this:

xxxxxxxxx  | [2025-04-05 03:27:44.706] W [SW-LLHLS:120] LLHLS Publisher | llhls_stream.cpp:922  | Could not find partial segment for track_id = 0, segment = 2416, partial = 1 (last_segment = 2432, last_partial = 6)
xxxxxxxxx  | [2025-04-05 03:27:44.706] W [SW-LLHLS:126] LLHLS Publisher | llhls_stream.cpp:922  | Could not find partial segment for track_id = 0, segment = 2416, partial = 0 (last_segment = 2432, last_partial = 6)

We could use sticky backend routing from our CDN, perhaps by IP hash or a cookie, etc to hit a consistent edge. We've tested that and it works. However we also have to consider that the CDN will health check each edge and if one goes down viewers will then get rerouted to a different oven edge. Or if we want to dynamically scale the number of edges up/down while streams are live, which creates complications with rebalancing load if sticky routing is used. Both cases could lead to mismatched playlists and cause playback stuttering/errors.

In our case, every edge is "warmed up" by a script that fetches the stream names from the redis used by OriginMapStore and makes a HEAD request to the playlist url on an interval less than UnusedStreamDeletionTimeout. But this doesn't happen at exactly the same moment for all edges and thus the playlists aren't consistent.

So, I'm wondering if it's possible to make the edge playlists consistent across multiple edges in OriginMode? Perhaps redis that oven already uses for OriginMapStore could be used to share more information between origin+edges.

Thanks!

[getroot]

Since OME's edge does not receive HLS files from the origin but does HLS packaging itself, segment numbering can vary depending on timing.

OME Enterprise supports ServerTimeBasedSegmentNumbering, but it is not perfect either. (This should only work for streams that can be managed precisely.)

I recommend using an HTTP Proxy like nginx instead of using OME's edge for HLS.

[fatchan]

Hey,

Thanks for the answer. Sorry to revisit this late, I've been busy.

This is actually for LL-HLS. I'm using multiple edges e.g. one in North America, one in Europe because if I only reverse proxy with nginx there would be considerable latency to OME origin. The chunks don't actually live on the nginx server, they would have to be fetched from the origin. With nginx only, a user in Europe visiting a North American edge might struggle to download LL-HLS chunks quickly enough causing stuttering in the stream. (And this does not solve the need to scale to multiple OME edges, nginx reverse proxy will face the same problem)

e.g. this is not ideal:
User (NA) -(low latency)-> CDN (NA) -(low latency)-> Nginx (NA) -(higher latency)-> OME Origin

But this is:
User (NA) -(low latency)-> CDN (NA) -(low latency)-> OME Edge (NA) <-(latency doesn't matter)- OME Origin

Am I misunderstanding the meaning of this?:

I recommend using an HTTP Proxy like nginx instead of using OME's edge for HLS.

There is 1 origin, and N edges (1 edge per region), and I have my own CDN layer on top. The idea is that my CDN routes to the closest edge and caches the chunks, to improve performance of low latency streams. This works great so far and streams are smooth with low latency between the caching CDN and the "origin" which is actually a nearby edge. However if I wanted to load balance multiple edges in 1 region for capacity, this issue would occur. Or if an edge becomes unavailable, my CDN will route to another edge, and again the issue occurs.

If there's no solution right now that's fine. I was just hoping given that I'm using OriginMode and OriginMapStore redis, it's feasible that edges could use redis to share some metadata and be synced to start at the same time/frame, so the playlists and chunks would be identical across all edges.

I will also put up $200 bounty (BTC/ETH/XMR crypto) if somebody can write a solution for this. I know your time is probably worth more than this, unfortunately I'm not a rich man 🙇🏻

Cheers.