applyConstraints() could be more useful

[eladalon1983]

applyConstraints() returns a promise.

• If the track is "ended", the promise is resolved.
• Otherwise, the promise is either...
  • ...rejected if there is a problem.
  • ...resolved if no problem is detected.

While this is somewhat useful, it might be more useful to specify that the promise is only resolved after the new set of constraints has taken effect, and all new frames emitted by the track are guaranteed to be subject to the new set of constraints.

[youennf]

Looking at w3c/mediacapture-transform#115, it seems difficult to provide such guarantees.

[eladalon1983]

Where does the difficulty lie?

Note that the Region Capture spec does something similar:

Call the track's state before this method invocation PRE-STATE, and after this method invocation POST-STATE. The user agent MUST resolve p when it is guaranteed that no more frames cropped (or uncropped) according to PRE-STATE will be delivered to the application, and that any additional frames delivered to the application will therefore be cropped (or uncropped) according to either POST-STATE or a later state.

Note:

The timing of the cropTo promise resolution and the timing of the actual cropping of video frames is observable to JavaScript through MediaStreamTrack transforms. It is expected that the first newly cropped video frame will be enqueued on the MediaStreamTrack ReadableStream just after the cropTo promise is resolved.

[youennf]

applyConstraints promise resolution will typically be done via the main thread but will receive frames off the main thread, at least in WebKit.
rfvc/video element videoWidth and so on will get updated based on off the main thread information so will not be synchronised tightly with applyConstraints resolution.

With MSTP, there is a tighter coordination since both applyConstraints resolution and frame reception happen in the worker thread.
But the two paths to get this information may be different (they are in WebKit).

When discussed at the WebRTC WG meeting, it was thought that mandating tight synchronisation in the worker thread would require a good justification to bother implementing this (and we would need to select synchronisation with which API)

Looking at region capture spec, it specifically mentions MSTP so maybe we should move this issue to media capture transform, or dupe it to w3c/mediacapture-transform#115.

I agree cropTo() and applyConstraints() should probably behave the same.

[eladalon1983]

Note that the requirements might not be as tight as they initially appear. The promise should be resolved after it is guaranteed that subsequent frames will not be associated with the old configuration. But there might be arbitrarily many "new" frames issued before the promise resolves. That is, resolving the promise indicates to the app "if you really needed to wait until things were safe, now is safe; but it might have already been safe for quite a few frames, possibly even entire seconds or minutes." With that in mind, a simple message posted between processes/threads should suffice, I think...?

[youennf]

But there might be arbitrarily many "new" frames issued before the promise resolves.

That is less tight. Does it make it sufficiently useful to implement it?

Also, the current region capture timing is tighter than what you say:
It is expected that the first newly cropped video frame will be enqueued on the MediaStreamTrack ReadableStream just after the cropTo promise is resolved.

Do we have feedback from devs using cropTo + MSTP to help us?
Can we move this issue to mediacapture-transform?

[eladalon1983]

That is less tight. Does it make it sufficiently useful to implement it?

I believe it's useful enough. I expect a Web app calling cropTo()/applyConstraints() would only connect the track to the sink after the promise resolves. Alternatively, apps disconnect the track from the sink, apply changes and then reconnect the track. Such apps do not typically care how many frames are missed, so long as things are reasonably quick.

Also, the current region capture timing is tighter than what you say:

That much is a note that begins with "expected"; it is not a guarantee.

Do we have feedback from devs using cropTo + MSTP to help us?

I am aware of products that basically obtains the track, crop it, and only start using it after the promise returned by cropTo() is resolved.

[youennf]

I think we can rediscuss this topic at a WebRTC WG meeting given this new information.

The proposal could be something like:

• When applyConstraints promise gets resolved, only samples generated for the new configuration are submitted to track sinks. It is ok for some samples of the new configuration to have already been submitted.
• When configurationchange event fires, only samples generated for the newly changed configuration are submitted to track sinks. It is ok for some samples of the newly changed configuration to have already been submitted.
• Similarly, after the onmute event fires, no sample is submitted to track sinks as long as the source stays muted. As of the unmute event, I would be tempted to not require synchronization until we have a usecase.
• Testing done with MSTP.

[eladalon1983]

Mostly agreed, with one exception - on unmute, my gut feeling is that it'd be more correct to require (loose) synchronization. For example, you wouldn't want audio to start flowing while app-level UX still claims "mute". In practice, this should usually be a very short delay, imperceptible to users, but there might be a long tail there with some users experiencing a perceptible delay. Maybe a SHOULD could soften the requirement, tacitly acknowledging that UAs are written by limited teams with time and prioritization constraints.

[youennf]

I think it is ok for an implementation to provide frames before unmute as long as the sink follows the spec guidelines (render black/silence). Ideally, it should be an implementation detail and what we are talking for mute/unmute is additional implementor's notes.

For MSTP, we should probably add a synchronous check for the enabled slot before enqueuing. We could also check the muted slot at the same time or add an assert that muted is false. That will make it sure MSTP sink is following the black guidelines.

Implementation-wise, at least in WebKit, this seems ok for video.
For audio, I'll need to check further.

[eladalon1983]

I think it is ok for an implementation to provide frames before unmute as long as the sink follows the spec guidelines

Aren't the two equivalent? I think both of us are suggesting, in different ways, that new frames should not be exposed to either the user or the app until the unmute event is fired.

[youennf]

From a user/web app perspective, it should be equivalent.

From a spec perspective, I was initially thinking that where the check happens may not really matter.
With MSTP, it actually does: either MSTP checks the muted slot or it asserts that the muted slot is false.
Given there is a step to enqueue a task to call MSTP maybeReadFrame, I would tend to do a check there.

Similarly to onmute, when onended fires, no frame should be delivered, but frames may be in transit.
MSTP maybeReadFrame should probably check the track ready state slot.