API for sending LLHLS ID3v2 Timed Metadata

[getroot]

An API has been released that allows you to insert events into streams. Right now events only support the ID3v2 format and only the LLHLS publisher handles it. Events delivered to LLHLS Publisher are inserted as emsg boxes within the m4s container.

POST /v1/vhosts/{vhost name}/apps/{app name}/streams/{stream name}:sendEvent

POST /v1/vhosts/default/apps/app/streams/stream:sendEvent
{
  "eventFormat": "id3v2",
  "eventType": "video",
  "events":[
      {
        "frameType": "TXXX",
        "info": "AirenSoft",
        "data": "OvenMediaEngine"
      },
      {
        "frameType": "TIT2",
        "data": "OvenMediaEngine 123"
      }
  ]
}

eventFormat
 Currently only id3v2 is supported.

eventType (Optional, Default : event)
  Select one of event, video, and audio. event inserts an event into every track. video inserts events only on tracks of type video. audio inserts events only on tracks of audio type.

events
  It accepts only Json array format and can contain multiple events.

 frameType
    Currently, only TXXX and T??? (Text Information Frames, e.g. TIT2) are supported.
 info
   This field is used only in TXXX and is entered in the Description field of TXXX.
 data
   If the frameType is TXXX, it is entered in the Value field, and if the frameType is "T???", it is entered in the Information field.

How to listen for events in OvenPlayer

Most HLS players support ID3v2 Timed Metadata events. In the case of hls.js, events can be received with Hls.Events.FRAG_PARSING_METADATA. Example code is below.

// Initialize OvenPlayer
const player = OvenPlayer.create('player', {
    sources: [
        {
            type: 'hls',
            file: 'https://xxxx/xxxx/xxxx.m3u8'

        }
    ]
});

this.player.on('hlsPrepared', function (hls) {
                        hls.on(Hls.Events.FRAG_PARSING_METADATA, function (data, meta) {
                            console.log(data, meta);
                            if (meta && meta.samples) {
                                meta.samples.forEach(function (sample) {
                                    console.log(new TextDecoder("utf-8").decode(sample.data))
                                })
                            }

                        });
                    });

[stevespaw]

Is this a path to SCTE35 insertion? :-)

[Alazaren]

Could you please clarify, is there a limit on the number of events that can be sent per second?
From my observation, it seems like there could be a limit of about two events per second. Even though the content of each push is different, the data in the stream appears duplicated, with only two unique events being processed per second. But the amount of event is correct.

[getroot]

There is no limit to how much you can send per second. If you think this is a bug, please post a detailed description of how to reproduce it on the issue board.

[Alazaren]

Hi getroot,
Yes, you are right; there is no limit. Thank you for answering this. I had some bugs in the JS, so was confused with idea there can be a limit. Could you please help me with the next question?

How is ID3v2 Timed Data synchronized with the stream? What I see now is that they are just pushed in real time as fast as they can be added. I am pushing events to the stream, but due to processing time, they are slightly out of sync and delayed.

Could you please suggest any techniques to synchronize it better? Maybe by delaying LLHLS processing or using another approach? I tried shifting on the frontend, but it seems unreliable.

To clarify what I want to achieve: I want to have an LLHLS stream running in parallel with frame processing via OpenCV and enrich the stream with metadata. Currently, I am using LLHLS and a thumbnail stream. The results from OpenCV are then pushed as ID3v2 metadata, but there is no way to properly align them.
I am also think about replacing the thumbnail stream with WebRTC, with hope that I can extract timing information from both sources. If that works, I might give up on ID3v2 and use WebSockets to stream metadata instead. The main challenge is achieving precise timing synchronization.

[getroot]

The sendEvent API inserts events based on the current time.

Please confirm if I understood your question correctly.

1. You have an LLHLS stream in your OME.
2. You extract metadata using OpenCV from the thumbnails provided by OME.
3. After extracting metadata in step (2), you insert ID3 events into the stream using the sendEvent API.
4. Since step (2) takes some time, your metadata is inserted with a delay compared to the main stream.

If this is correct, you want to intentionally delay the stream in step (1) by the amount of time required for step (2). Is that right?

To achieve your goal, you need two mechanisms:

1. Add a delay to the publisher.

This feature is officially provided in Enterprise, but there is relevant code in the Open Source version that may work. (However, it is not guaranteed.)
https://ovenmediaengine-enterprise.gitbook.io/guide/enterprise-grade-features/advanced-management-tool/publisher-extension#add-delay-to-the-stream

2. Insert events into the past instead of the current time using the sendEvent API.

This feature is not provided, but you can implement it yourself.
By using these two mechanisms, you can buffer the stream with delay using the first feature (otherwise, in LLHLS, segments would already be created and output), and then insert events at -1000ms using the second feature.

Since I think this is a very specific use case, modifying the open-source code directly might be the fastest solution for you.

[Alazaren]

Hi getroot,
Your understanding is correct. Thank you for the detailed response. I did some tests on the approach. DelayBufferTimeMs doesn’t seem to be working as expected and forking is something I would like to avoid. Eventually, I figured out how to shift this in HLS.js, even though I was forced to patch it.
In conclusion, many thanks for this great tool. My personal thoughts: if it were possible to have some timing between publishers, e.g webrtc and LLHSL so wen can use one for CV one for consumer. OME could be something like Nvidia DeepStream - but for people )). With its clear configs and easy setup, it could be a powerful solution.