Should we revisit and document getUserMedia() track order?

[jan-ivar]

The spec says: "The tracks of a MediaStream are stored in a track set. The track set MUST contain the MediaStreamTrack objects that correspond to the tracks of the stream. The relative order of the tracks in the set is User Agent defined and the API will never put any requirements on the order. The proper way to find a specific MediaStreamTrack object in the set is to look it up by its id."

But #1020 is concerned about deterministic event order between track clones, looking at using track creation order.

If we're chasing determinism at this level, might getUserMedia need to be clearer about whether it creates its audio or video track first?

I suspect audio comes first, given "2. Let requestedMediaTypes be the set of media types in constraints with either a dictionary value or a value of true... 11.3 For each media type kind in requestedMediaTypes ...", and the fact that dictionary members are sorted lexographically.

While use of the word "set" in the sentences above do not link to set, they might be understood to mean the same thing. It reads:

Thankfully, an implementation check reveals the same order in all browsers: audio, video. Is it time to document this?

[bradisbell]

@jan-ivar Is there ever a time when a MediaStream would be returned from getUserMedia() with tracks added later? For example, an audio device is usually available seconds before a video device initializes and starts returning video data. I suppose the opposite could happen. Just thinking ahead, wondering if specifying track order might hurt optimizations in the future that return available tracks immediately and add the other requested track(s) later.

[jan-ivar]

getUserMedia({video: true, audio: true}) is guaranteed to return two tracks. The algorithm ensures that.

[fippo]

See webrtcHacks/adapter#969 and the related discussion.
From what I recall this has continued to confuse "web developers".

I am not sure if lexographically order of tracks sorted by the id is meaningful to anyone, sorting by kind and then by id seems more natural.

I'd suggest checking how getTracks + addTrack behaves and look forward to the WPT!

edit: done, see this fiddle:

• Chrome returns audio, video
• Firefox returns audio, video
• Safari returns in order of id
  If this gets you confused enough think about the average web developer.

[fippo]

Note: chromium returns audiotracks-then-videotracks from getTracks(), see here, changed in 2014

Which makes sense for getUserMedia but not so much when you consider manually added tracks (empty stream + video + audio => [audio, video] or clones.

[fippo]

[audio,video] + video clone + audio clone: https://jsfiddle.net/fippo/w7z91b8k/1/ show link
Current state:

Browser	order
"make everyone unhappy"	audio, video, video, audio
Chrome	audio, audio, video, video
Firefox	audio, video, video, audio
Safari	track-id-order as we learned

I do not think this this is an improvement.

We also need to be careful about remote tracks as shown by this which puts video first, then audio in the SDP:
https://jsfiddle.net/fippo/ec1puh6w/
The risk of causing breakage in how this translates to SDP is too big.
Multiple tracks per kind and then using that with a peer connection does not seem like a big use-case either.

I'm happy to send a PR updating https://w3c.github.io/webrtc-pc/#legacy-interface-extensions but the biggest compat-win would be Safari stopping to randomize the order. (and note that adapter does not interfere with MediaStream.getTracks(), only with legacy addStream order)

[jan-ivar]

Working like Firefox is not "breakage". There's no risk here as far as I'm concerned. Only upside.

We also need to be careful about remote tracks as shown by this which puts video first, then audio in the SDP:
https://jsfiddle.net/fippo/ec1puh6w/
The risk of causing breakage in how this translates to SDP is too big.

SDP is garbage in garbage out. E.g. put the m=audio line first instead and audio track is surfaced first. Same on input.

It seems clear we would have heard about it by now if this were problematic.

Knowing there's no breakage, it's disappointing we cannot align around the spec which has the most natural and logical behavior, which happens to be how Firefox already works in this case.

[fippo]

As an exercise let me walk you through https://www.chromium.org/blink/launching-features/#behavior-changes:

• Can the change cause user-visible or functional breakage? YES, changes the order of SDP m-lines
  • Is it possible that existing code relies on the current behavior? YES, see Safari/Freeswitch or Asterisk interop woes
  • What would that coding pattern look like? How likely it is that this coding pattern is used in the wild? see specification-shown example of stream.getTracks().forEach(track => pc.addTrack(track, stream)
• What are other browser engines doing? VARIES
  • Would this change align Chromium's behaviour with other vendors? YES, matches Firefox. Safari orders by UUID
  • Or would Chromium be the first to roll-out this behavior change? NO
• What is the usage of the feature being changed (typically measured with a UseCounter? No use counter
• How likely is breakage in sites that only support Chromium browsers? some webrtc services do not support Firefox so might not have encountered the new behavior and are not expecting it

Since we do not have a use-counter I'll make a guess:

• 99.8% of usage is going straight from GUM to addTrack
• 0.0063% of usage is stream.addTrack with a screensharing track and a microphone one. For extra fun 40% of those adds the mic to the screensharing stream, 60% the other way round. I bet this is where the new behavior would break.

Should we ask some Chromium API owners if they would sign this off as a "web-facing change"?
Even ignoring the risk (I am all for it, moar content!) is "most natural and logical behavior" sufficient motivation to justify the effort?

And keep in mind that this affects SDP... you know why Firefox continues to put version 99 into its SDP? Does not seem logical to me...

[jan-ivar]

I see no evidence that changing Chrome’s track order would cause breakage. Other browsers (Firefox and Safari) have always had different track orders with no ill effect.

Any site depending on a particular browser’s order is already non-interoperable, since the spec explicitly allows any order.

Unifying behavior reduces confusion and helps developers. If there’s usage data showing a real dependency, that would be valuable, but absent that, fears of “breakage” seem unfounded.

• 99.8% of usage is going straight from GUM to addTrack

IOW no issue for Chromium whose gUM order would remain unchanged: [audio, video].

• 0.0063% of usage is stream.addTrack with a screensharing track and a microphone one. For extra fun 40% of those adds the mic to the screensharing stream, 60% the other way round. I bet this is where the new behavior would break.

And keep in mind that this affects SDP...

There's no evidence websites rely on this order or that anything would break. There's evidence they don't (interoperable websites).

If usage data for ordering assumptions exists, it would be good to see it. Absent that, we should not assume a hidden dependency that has somehow gone unreported despite existing browser differences.

[guidou]

I see no evidence that changing Chrome’s track order would cause breakage. Other browsers (Firefox and Safari) have always had different track orders with no ill effect.

Absence of evidence is very different from evidence of absence.

Any site depending on a particular browser’s order is already non-interoperable, since the spec explicitly allows any order.

We don't want to break those sites either.

Unifying behavior reduces confusion and helps developers. If there’s usage data showing a real dependency, that would be valuable, but absent that, fears of “breakage” seem unfounded.

Let's unify then around Chrome's behavior. If we want to guarantee order for getTracks() we should also do it for getAudioTracks() and getVideoTracks().

We can specify it as internally having two ordered sets, one for getAudioTracks() and one for getVideoTracks(). The order would be the order in which tracks are added to the corresponding set.
getTracks() would return the concatenation of the audio set and the video set.

[jan-ivar]

It seems incumbent on those making unsubstantiated claims of "breakage" to provide evidence, and explain how breakage would manifest. I've provided plenty of evidence of absence: years of different browser orders without bug reports.

Any site depending on a particular browser’s order is already non-interoperable, since the spec explicitly allows any order.

We don't want to break those sites either.

Which sites are those?

Let's unify then around Chrome's behavior.

Unifying on Chrome’s historic ordering seems arbitrary. If we truly suspect real-world reliance, a usage counter could confirm that. Otherwise, there is no evidence of harm in converging on a single, simpler order that aligns track set with https://infra.spec.whatwg.org/#sets

If we want to guarantee order for getTracks() we should also do it for getAudioTracks() and getVideoTracks().

Agreed. Those already appear expressed as the track set filtered on kind, so we'd just need to remove "The conversion from the track set to the sequence is User Agent defined and the order does not have to be stable between calls."

[guidou]

It seems incumbent on those making unsubstantiated claims of "breakage" to provide evidence, and explain how breakage would manifest.

What's unsubstantiated is claiming that this change is risk-free.

I've provided plenty of evidence of absence: years of different browser orders without bug reports.

That's only evidence that the status quo does not cause problems to anyone, which is strong evidence in favor of maintaining the status quo. It is not evidence of absence of harm after we deploy the change.

Any site depending on a particular browser’s order is already non-interoperable, since the spec explicitly allows any order.

We don't want to break those sites either.

Which sites are those?

I don't want to find out.

Let's unify then around Chrome's behavior.

Unifying on Chrome’s historic ordering seems arbitrary.

No more arbitrary than any other ordering.

If we truly suspect real-world reliance, a usage counter could confirm that. Otherwise, there is no evidence of harm in converging on a single, simpler order that aligns track set with https://infra.spec.whatwg.org/#sets

In the past, behavior changes to established APIs like this one have caused breakage for Chrome users.
The benefit of making this change (aligning the spec with a new infra guideline) does not outweigh this risk.
Given this, the only change I find acceptable is aligning the spec with Chrome's ordering. It has the following benefits:

• Aligns the spec with the infra guideline, which is the motivation for the proposal.
• Trivial to implement for any UA that implements getAudioTracks() and getVideoTracks() using addition order (for which we seem to have agreement).
• Zero risk for users of Chromium-based browsers. Any other ordering fails to address this concern.
• Zero risk for Firefox users, as per Mozilla's analysis.

But we still need to wait for WebKit's assessment.

If we want to guarantee order for getTracks() we should also do it for getAudioTracks() and getVideoTracks().

Agreed. Those already appear expressed as the track set filtered on kind, so we'd just need to remove "The conversion from the track set to the sequence is User Agent defined and the order does not have to be stable between calls."

I support proceeding with this one if no one objects.

[jan-ivar]

It’s contradictory to cite “no bug reports” as evidence supporting the status quo while also dismissing the same “no bug reports” as no evidence supporting change. If lack of bug reports proves anything, it’s that different orders have coexisted for years without incident—i.e. the absence of bug reports is actually evidence that changes like these pose little risk.

[guidou]

It’s contradictory to cite “no bug reports” as evidence supporting the status quo while also dismissing the same “no bug reports” as no evidence supporting change. If lack of bug reports proves anything, it’s that different orders have coexisted for years without incident—i.e. the absence of bug reports is actually evidence that changes like these pose little risk.

An outcome completely consistent with the evidence presented so far is that the browser that changes its behavior starts to receive bug reports, and the browsers that keep their behavior continue to receive no bug reports.
Keeping the status quo where no one changes its behavior is likely to continue to result in no bugs.

But I have already presented our concerns and a compromise solution that seems to address all concerns presented so far.

I think we should now wait for WebKit's assessment and continue from there.

[fippo]

There are four separate technical issues here.

1/ getUserMedia track order (what the title says) and how it affects peerconnection

• is the same for Chromium and Firefox
• sorted by random id in Safari. As pointed out above this caused some issues in the past. I do not think a webkit bug was filed, web developers moved on.

2/ how does getTracks behave if a video track and a audio track are added to a GUM track and then used in a peerconnection

• There are difference in this case.
• How often developers do this is unknown
  Since "more than a single audio and video track" is not well supported in other APIs like the HTMLMediaElement or MediaRecorder (which surprisingly does not throw on multiple video tracks) I do not think this is actually relevant.

3/ how does getTracks behave if an audio track from getUserMedia is added to a video track from getDisplayMedia and then used in a peerconnection

• There are difference in this case.
• How often developers do this is unknown but since audio-only+screensharing is common
  I believe this use-case is common enough that we would have heard if the differences cause any issue.
  My guess is that audio-screen is the more common order here since JS has already acquired your microphone when you start sharing.

4/ SDP

• https://jsfiddle.net/fippo/6t7pa0d3/1/ shows the ontrack order on the remote end matches the addition order on the receiving end.
• I could not find a WPT.

Of those four problems I have only seen evidence of breakage for (1, link above) and of course (4) is a headache.
I have not heard of any "web developer" asking for changes in (2) or (3) and my rolodex is quite extensive.

Speaking as a web developer please @youennf consider (1)

[alvestrand]

I explained on the call the history of why track order is implemented as "group by type": The original specification had only "getAudioTracks" and "getVideoTracks"; "getTracks" was added later. Thus, a natural implementation was to return the concatenation of getAudioTracks and getVideoTracks, and since we had explicit language stating that the order of tracks was not guaranteed, this was a safe implementation choice vs the spec at the time.