Add C++ plugins framework

[ggarber]

Feature Request (more like a feedback request)

Protoype: ggarber@14a3ba0

Some use cases require custom processing of Media and Data packets and because of the performance implications it is not always possible to forward to those packets in real time from C++ to JS and then back from JS to C++ using DirectTransport capability.

Some examples of those use cases could be:

• Store RTP packets for all the producers in some pcap like format like Janus MJR format.
• Implement a custom datachannels protocol where the destination of each message is included in the message itself.

Nowadays the solution in these cases is to use an additional server and forward those messages from mediasoup to this server but that adds some extra complexity in terms of orchestration and also maintaining that additional server. The other solution today is to use Rust instead of JS library but that's not an option for all the teams.

The alternative is to provide support to extend mediasoup C++ layer with some custom code (let's call it plugins for now) so that developers can extend the standard capabilities of mediasoup with their use case specific logic.

The main components of that approach would be:
A) The API exposed by mediasoup to those plugins.
B) How those plugins are loaded.

API exposed to those plugins

It needs to be a mix of methods and events.

Methods: Initially all the public interface of mediasoup objects (Transports, Consumers, Producers... ) could be used from plugins but some interfaces could be extracted (TransportApi, ConsumerApi, ProducerApi...) so that plugins don't see the whole interface.

Events: mediasoup needs to expose a mechanism for plugins to listen for events. This can be in the form of events that plugins can be attached for (see prototype).

Loading of plugins

There are different alternatives:

• Fork mediasoup and add plugins to the repository. This is far from ideal but a potential solution for the short term and initial development and testing of the API.
• Use mediasoup as a library and generate your own mediasoup worker binary with that library and their own plugins. This looks too complex for most of developers but maybe it is reasonable in some cases.
• Add some simple dynamic loading capabilities. Mediasoup JS API could add a method worker.loadPlugins('xxxx.so', 'yyyy.so').

[jmillan]

I like the approach. Something that worries me is the synchronous nature of the implementation. Plugins processing impacts directly on worker performance (not in this specific 'ping' example but potentially in any others). I would prefer an async framework where the communication between the plugin and the worker is made by libuv communication means. Ie: the same way Rust communicates with the worker.

NOTE: I know how making this async would complicate things.., so I wonder if it would be acceptable to let the plugin user assume the performance cost of the plugin.

NOTE2: Another worrying point is the fact that everything runs in the same process, meaning a crash in a plugin would crash the worker.

[ggarber]

Thx a lot for taking the time to look at this.

NOTE1:
Makes sense. I was considering that any plugin that could affect the performance of the main event loop would create its own threads (f.e. a plugin doing audio decoding). So maybe the plugin API doesn't need to be more complex if most of plugins would be simple and rest are able to create their own processing threads. I'm not sure.

There is a feature that it is hard to achieve with asynchronous processing that is having plugins that intercept the processing of a packet so that it doesn't continue normally. For example a plugin throttling the RTP packets received so that nobody can send more than X pkts/sec to avoid DoS attacks.

NOTE2:
Correct, but I think that's common to most of the plugins systems, right? (nginx, apache, janus, gstreamer...). Given that the developer would decide what plugins to use I think it could be reasonable.

[vpalmisano]

Maybe the plugin itself could handle the events in an asynchronous way if the processing affects the main loop performance.

[ibc]

I cannot comment much about this yet since I don't know anything about how a native plug-in can be "installed", but from architectural point of view I think that every plug-in should have:

• An interface for mediasoup node to dynamically enable/disable it for specific producerIds and/or dataProducerIds.
• The plug-in should receive the whole payload (RtpPacket or SCTP message) that the worker receives on those proeucerIds or dataProducerIds and the worker should then DISCARS those messages, this is, if the payload is passed to a plug-in then the worker should drop those RtpPackets or SCTP messages instead of performing normal routing with them (unless the plug-in discards that message, see last bullet).
• The plug-in processes the message and, later, it may decide to tell the worker "ey send this RtpPacket or this SCTP message to those consumers ids or dataConsumer ids". This is, an absurd example: a plug-in receives a RtpPacket from producerId 1111 and tells the Worker to send a SCTP message to dataConsumerIds 2222 and 3333.
• In case multiple plugins are enabled for the same Producer or DataProducer, we need to run them serially, so first plug-in receives the message first from the worker and decides to handle it or ignore it. The message from the worker to the plug-in contains the producer id and the whole payload and it's a request, and the response from the plugin (true/false) tells the worker whether the plug-in will take care of the message or not. If not, the worker passes the payload to next plug-in or just routes it normally if no plug-in handles the payload. And this is SUPER problematic because it means that received RtpPackets and SCTP messages must remain in memory until that async decision is taken, which means that the C++ Producer or DataProducer class instance cannot call "delete packet" immediately after passing the packet to the Router as we do know. So smart pointers or whatever needed here.

[nazar-pc]

Worker was supposed to be hidden behind TypeScript/JavaScript library, not sure I like the idea of adding C++ plugins. You should really consider Rust if you want performance, then you can still treat worker as opaque implementation, yet achieve higher performance.

[ggarber]

Thank you for your time, see inline.

I don't know anything about how a native plug-in can be "installed"

Check the three alternatives mentioned in the description (Loading of plugins)

• An interface for mediasoup node to dynamically enable/disable it for specific producerIds and/or dataProducerIds.

Consider the option of allowing plugins to decide what producers/routers/workers they are interested in, like in the prototype. Potentially a plugin could be used for other logic that is not related to producers or consumers IMO.

• The plug-in should receive the whole payload (RtpPacket or SCTP message) that the worker receives on those proeucerIds or dataProducerIds and the worker should then DISCARS those messages, this is, if the payload is passed to a plug-in then the worker should drop those RtpPackets or SCTP messages instead of performing normal routing with them (unless the plug-in discards that message, see last bullet).

The way I have implemented this in other SFUs is with the plugins returning a flag (CONTINUE, BREAK...) to tell the next plugin or logic in the chain if the processing should continue or not and the default is to continue with normal processing.

• The plug-in processes the message and, later, it may decide to tell the worker "ey send this RtpPacket or this SCTP message to those consumers ids or dataConsumer ids". This is, an absurd example: a plug-in receives a RtpPacket from producerId 1111 and tells the Worker to send a SCTP message to dataConsumerIds 2222 and 3333.

This flow is covered in the prototype, right?

• In case multiple plugins are enabled for the same Producer or DataProducer, we need to run them serially, so first plug-in receives the message first from the worker and decides to handle it or ignore it. The message from the worker to the plug-in contains the producer id and the whole payload and it's a request, and the response from the plugin (true/false) tells the worker whether the plug-in will take care of the message or not. If not, the worker passes the payload to next plug-in or just routes it normally if no plug-in handles the payload. And this is SUPER problematic because it means that received RtpPackets and SCTP messages must remain in memory until that async decision is taken, which means that the C++ Producer or DataProducer class instance cannot call "delete packet" immediately after passing the packet to the Router as we do know. So smart pointers or whatever needed here.

This could be another reason to follow the sync approach instead of the async one, right?

[ggarber]

Worker was supposed to be hidden behind TypeScript/JavaScript library, not sure I like the idea of adding C++ plugins. You should really consider Rust if you want performance, then you can still treat worker as opaque implementation, yet achieve higher performance.

I agree it is a paradigm change where the mediasoup API is not only the JS API but also the C++ API. I like that and ideally would allow me to run mediasoup in my own C++ process without any Javascript :) but I understand it is a big change and makes the API surface much bigger.

I included Rust as an option in the Issue description and I think it is the best approach in many cases. My "problem" is that for many teams rust is not the most convenience language and is an entry barrier if to create a simple 100 lines plugin they need to rewrite 20.000 lines of javascript code.

[nazar-pc]

For context I was hoping we can eventually have more Rust in mediasoup and use https://github.com/neon-bindings/neon or something similar to create a Node.js module. Having C++ plugins support would mean more breaking changes to downstream users or more work for us to continue supporting it.

Not saying it is going to happen any time soon, but I'm hopeful that it might happen eventually.

[ibc]

Check the three alternatives mentioned in the description (Loading of plugins)

Of course I only consider the "add some simple dynamic loading capabilities" approach. Others are not plug-ins at all and we need something that should work with npm i mediasoup installation approach. So unless I miss something let's please ignore any other alternative.

Can your prototype be used as follows?

• You have /Users/vox/src/myMediasoupFooPlugin.cpp.
• I assume you must compile it and get /Users/vox/src/myMediasoupFooPlugin.so file.
• Then you have /Users/vox/src/myMediaServer Node project in which you install mediasoup npm i mediasoup as usual.
• In myMediaServer/src/index.ts you use the mediasoup Node api to create a Worker and enable your plugin on it:

  import { createWorker } from 'mediasoup';
  
  const worker = await createWorker({
    plugins: [
      { id: 'foo', path: '/Users/vox/src/myMediasoupFooPlugin.so' }
    ],
    etc
  });

• Then you create a Router in which you want to enable your plugin:

  const router = await worker.createRouter({
   plugins: [ 'foo' ],
   etc
  });

• Then you create a Router in which you want to enable the Router related stuff your plugin:

  const router = await worker.createRouter({
   plugins: [ 'foo' ],
   etc
  });

• Then you create a WebRtcTransport in which you want to enable the Transport related stuff your plugin:

  const transport = await router.createWebRtcTransport({
   plugins: [ 'foo' ],
   etc
  });

• Then you dynamically enable/disable your plugin:

  transport.enablePlugin('foo');
  transport.disablePlugin('foo');

• Then you launch your app and things just work without having to do anything else.

  node /Users/vox/lib/myMediaServer/index.js
  

• An interface for mediasoup node to dynamically enable/disable it for specific producerIds and/or dataProducerIds.

Consider the option of allowing plugins to decide what producers/routers/workers they are interested in, like in the prototype. Potentially a plugin could be used for other logic that is not related to producers or consumers IMO.

Not against this.

• The plug-in should receive the whole payload (RtpPacket or SCTP message) that the worker receives on those proeucerIds or dataProducerIds and the worker should then DISCARS those messages, this is, if the payload is passed to a plug-in then the worker should drop those RtpPackets or SCTP messages instead of performing normal routing with them (unless the plug-in discards that message, see last bullet).

The way I have implemented this in other SFUs is with the plugins returning a flag (CONTINUE, BREAK...) to tell the next plugin or logic in the chain if the processing should continue or not and the default is to continue with normal processing.

Yes, that's basically what I meant.

• The plug-in processes the message and, later, it may decide to tell the worker "ey send this RtpPacket or this SCTP message to those consumers ids or dataConsumer ids". This is, an absurd example: a plug-in receives a RtpPacket from producerId 1111 and tells the Worker to send a SCTP message to dataConsumerIds 2222 and 3333.

This flow is covered in the prototype, right?

Yes.

• In case multiple plugins are enabled for the same Producer or DataProducer, we need to run them serially, so first plug-in receives the message first from the worker and decides to handle it or ignore it. The message from the worker to the plug-in contains the producer id and the whole payload and it's a request, and the response from the plugin (true/false) tells the worker whether the plug-in will take care of the message or not. If not, the worker passes the payload to next plug-in or just routes it normally if no plug-in handles the payload. And this is SUPER problematic because it means that received RtpPackets and SCTP messages must remain in memory until that async decision is taken, which means that the C++ Producer or DataProducer class instance cannot call "delete packet" immediately after passing the packet to the Router as we do know. So smart pointers or whatever needed here.

This could be another reason to follow the sync approach instead of the async one, right?

Imagine you want to apply a lot of IA into every RTP packet. This is probably async stuff. I wouldn't assume sync comm.

[ggarber]

Imagine you want to apply a lot of IA into every RTP packet. This is probably async stuff. I wouldn't assume sync comm.

My example was decoding audio and it is similar to yours. In that case the plugin can create any threads that it needs to process the rtp packets and not block the mediasoup event loop.

[ggarber]

For context I was hoping we can eventually have more Rust in mediasoup and use https://github.com/neon-bindings/neon or something similar to create a Node.js module. Having C++ plugins support would mean more breaking changes to downstream users or more work for us to continue supporting it.

I agree it would mean more breaking changes. I think at least initially the plugins feature should be used only for very advanced use cases / developers willing to pay the extra effort and risk of breaking changes.

It could potentially be enabled during builds (#ifdef PLUGINS_SUPPORT) and only people really needing it builds its own version of mediasou pwith that support enabled.

[jmillan]

Something that worries me about the approach in your example is that there are no boundaries about what a plugin can do as it receives the core instances. It could, for example, call transport.CloseProducersAndConsumers() and leave the Node and Rust side in an inconsistent state.

We already have a communication framework between the core (Worker) and Node and Rust. Such framework provides a control and data message format (JSON in v3, flatbuffers very soon in v3) as well as the means to handle those in a secure way.

There are currently 2 ways we exchange those control and data messages (defined in ChannelSocket):

1. UnixSocket in Node
2. As a function call, via FFI in Rust using uv_async_send()

IMHO we should use this very same framework (reusing the same message formats) by adding a third way to exchange these messages between the core and, now, C++:

3. As a direct function that the worker uses to send the message to the plugin manager, sync.

If there was the need to send async messages from plugins to the core it could use uv_async_send() since, obviously it cannot just run stuff out of the loop.

[*]: New messages would be needed to be created as for example notifications to tell the plugin manager that a new transport, [data]consumer, [data]producer, etc has been created, etc.