getUserMedia – What happens when there’s missing media sources?

Deprecated: pathinfo(): Passing null to parameter #1 ($path) of type string is deprecated in /nas/content/live/webrtchacks/wp-content/plugins/urvanov-syntax-highlighter/class-urvanov-syntax-highlighter-langs.php on line 86

As discussed in previous posts, the mission of the W3C WebRTC WG is to define client-side APIs to enable Real-Time Communications in Web-browsers. At a very high-level overview, there are three main steps to be taken when setting up a WebRTC session:

• Obtain local media – provides access to local media input devices such as webcams and microphones
• Establish a connection between the two browsers – peer-to-peer media session between two endpoints, including any relaying necessary, allowing two users to communicate directly
• Exchange Media or Data – allows the web application to send and receive media or data over the established connection

The getUserMedia() method is generally used to obtain access to local devices and it requires user permission before accessing the device. In this post, John McLaughlin, Eamonn Power and Miguel Ponce de Leon from openRMC will be looking more closely at the getUserMedia() method, and how to deal with its outputs in order to give some meaningful feedback to the developer, and ultimately the end user. More concretely and quoting their own words:

We wanted to know what should and would happen, from the perspective of a JavaScript app, if a browser makes a getUserMedia request on a system where either the camera, mic or both are inactive or missing.

{“intro-by”, “victor”}

---

WebRTC getUserMedia()

The getUserMedia() method is used to access media streams from media input devices such as webcams or microphones. The stream obtained can then either be used locally by passing it to a HTML <audio/> or <video/> tag, lending itself to many creative and fun applications such as photobooth, facial recognition, image processing etc. Additionally it can be attached to an RTCPeerConnection object and used to establish communications with another user. Figure 1 shows possible usage scenarios:

As an overview, getUserMedia() is called with up to three parameters:

• mediaConstraints – this is an object specifying the type and optionally the quality of the media streams required. Full details can be found in the getUserMedia() method but for the purposes of this post we’ll concentrate on the boolean attributes audio, and video, which specify whether audio and/or video streams are required respectively.
• successCallback (optional) – this is called with the MediaStream. This is illustrated in figure 2. object encapsulating the media streams requested by the developer
• errorCallback (optional) – this is called with an error object, which should give an indication as to why the call failed

Altough the callbacks are optional in the spec, in practice at the very least the successCallback is required, as this is the only way to access the MediaStream object. As we shall see, providing the error callback is also beneficial

What happens when there’s missing media source

So what happens if you call getUserMedia() and ask for a media stream but there’s no device present to support it? Shouldn’t the error callback get called with a nice informative error telling you what happened so you can pass it on to the end user? Well, yes to the first part, but the error information bit leaves a lot to be desired.

From the official perspective, at the time of writing, this is what the spec says should be passed into the error callback:

[NoInterfaceObject] interface NavigatorUserMediaError : DOMError { readonly attribute DOMString? constraintName; }; 1 2 3 4 [NoInterfaceObject] interface NavigatorUserMediaError : DOMError {     readonly    attribute DOMString? constraintName; };

This is subject to constant change (and getUserMedia is generally considered the most stable of the APIs!), as can be seen by prior versions of the spec:

NavigatorUserMediaError 07/04/2013

[NoInterfaceObject] interface NavigatorUserMediaError { readonly attribute DOMString name; readonly attribute DOMString? message; readonly attribute DOMString? constraintName; }; 1 2 3 4 5 6 [NoInterfaceObject] interface NavigatorUserMediaError {     readonly    attribute DOMString  name;     readonly    attribute DOMString? message;     readonly    attribute DOMString? constraintName; };

NavigatorUserMediaError 12/12/2012

[NoInterfaceObject] interface NavigatorUserMediaError { const unsigned short PERMISSION_DENIED = 1; readonly attribute unsigned short code; }; 1 2 3 4 5 [NoInterfaceObject] interface NavigatorUserMediaError {     const unsigned short PERMISSION_DENIED = 1;     readonly attribute unsigned short code; };

So let’s look at what actually happens in various browsers when we remove or disable the audio and video input devices.

Chrome 28 error

err: { code: 1, PERMISSION_DENIED: 1 } 1 2 3 4 err: {         code: 1,         PERMISSION_DENIED: 1 }

Chrome 29 error

err: { constraintName: "", message: "", name: "PERMISSION_DENIED" } 1 2 3 4 5 err: {         constraintName: "",         message: "",         name: "PERMISSION_DENIED" }

To be fair to Chrome, the samples above do conform to the earlier specs, and would reflect what was current when that version of Chrome was in its Canary incarnation. In order to keep up to date on such changes it’s worth subscribing to the Discuss-WebRTC mailing list as posts like this give you a heads up on API changes.

Firefox 22 error

err: "HARDWARE_UNAVAILABLE" 1 err: "HARDWARE_UNAVAILABLE"

Firefox 23 error

err: "NO_DEVICES_FOUND" 1 err: "NO_DEVICES_FOUND"

Informative in a sense, though what device(s) weren’t found? Nor does it bear the slightest resemblance to any version of the spec.

Making sense of it all

Fortunately depsite the lack of granularity in the error responses returned by getUserMedia(), with a little forensic work it is possible to derive a better picture of the WebRTC media capabilities of the host browser.

We make use of the adapter.js shim from the Google WebRTC reference app. We have made a modification to this as the latest version at the time of writing, r4259, didn’t have Firefox support for the getAudioTracks() and getVideoTracks() methods of MediaStream (Firefox has only gained support for these methods as of version 23)

We have wrapped the native getUserMedia() call in an API openrmc.webrtc.api.getUserMedia(spec) where spec has the following attributes:

{ audio : true|false, // Default true video : true|false, // Default false success : function(cxt) {}, // Success callback failure : function(cxt) {} // Failure callback } 1 2 3 4 5 6 {     audio : true|false, // Default true     video : true|false, // Default false     success : function(cxt) {}, // Success callback     failure : function(cxt) {} // Failure callback }

The success and failure callbacks both take a parameter of type openrmc.webrtc.Context, of which more later.

Firstly what sort of errors are we interested in? Since we primarily want to establish the lack of presence of some media device, we came up with the following error set:

Error codes

openrmc.webrtc.Errors = { NOT_SUPPORTED : 'NOT_SUPPORTED', CONSTRAINTS_REQUIRED : 'CONSTRAINTS_REQUIRED', AUDIO_NOT_AVAILABLE : 'AUDIO_NOT_AVAILABLE', VIDEO_NOT_AVAILABLE : 'VIDEO_NOT_AVAILABLE', AV_NOT_AVAILABLE : 'AV_NOT_AVAILABLE' } ; 1 2 3 4 5 6 7 openrmc.webrtc.Errors = {     NOT_SUPPORTED : 'NOT_SUPPORTED',     CONSTRAINTS_REQUIRED : 'CONSTRAINTS_REQUIRED',     AUDIO_NOT_AVAILABLE : 'AUDIO_NOT_AVAILABLE',     VIDEO_NOT_AVAILABLE : 'VIDEO_NOT_AVAILABLE',     AV_NOT_AVAILABLE : 'AV_NOT_AVAILABLE' } ;

The first two are straightforward. NOT_SUPPORTED is raised when getUserMedia() is called on a browser that doesn’t support WebRTC. CONSTRAINTS_REQUIRED will be raised if getUserMedia() would be called with both audio, and video attributes set to false. This would result in an error anyway, and isn’t allowed.

AUDIO_NOT_AVAILABLE, VIDEO_NOT_AVAILABLE, and AV_NOT_AVAILABLE are fairly self-explanatory and are raised if any stream of a certain type has been requested, but isn’t available. Which one is determined by checking the returned error and what resources are requested. A getError() helper method is defined based on the detected browser.

These are hard errors and are raised as a result of the native getUserMedia() failing, i.e. none of the requested media constraints could be satisfied. The native getUserMedia() will succeed if say audio and video are requested, but only audio is available. In this case, the returned openrmc.webrtc.Context can give more information to the application developer which can be used to inform the end user.

The openrmc.webrtc.Context object provides the following methods:

• isrtcavailable() – Returns true if the native getUserMedia() call resulted in a valid MediaStream, false otherwise.
• isrtcaudioavailable() – Returns true if at least one audio track is available, false otherwise.
• isrtcvideoavailable() – Returns true if at least one video track is available, false otherwise.
• getError()– Returns an error, if any

The presence of audio and video tracks is determined by the getAudioTracks() and getVideoTracks() methods on MediaStream – in both cases true is returned if and only if the MediaStream object is present, and the relevant getXXXTracks() method returns an array with one or more entries.

Trying it

We have made a simple web application available where you can try it for out yourself here. To try it, go to the page and click the "Call getUserMedia" button. Try various combinations with the webcam and/or mic disabled. An error or stream status will be reported when the call returns.

The source can also be viewed and downloaded from here. It does need to be served by a web server of some sort – simply accessing the page by a file:// URL won’t work.

Note While on most modern OS’s it’s usually quite straightforward to disable a webcam if it’s not built in (just unplug it), it can be quite tricky to disable audio input. Typically ther is some sort of virtual driver which will quite happily allow getUserMedia() to return successfully with no supporting hardware. Attempts to disable this require a dance of disabling device drivers and possible reboots (especially on Windows). If you’re going down this path to test, make sure you know how to re-enable said drivers again!

Summary

It’s important to give the end user an informative message when things go wrong when initialising a WebRTC session and hopefully you can see that this approach along with the code snippets gives a developer the possibility to derive a better picture of the WebRTC media capabilities of the host browser.

The results here should be taken as a snapshot reflecting the state of play for getUserMedia() at the time of writing. As can be seen above, the API is still fluid, and could change several times before being set in stone. It is also possible that Microsoft and Apple will come to the party and add support of some sort to Internet Explorer and Safari, which will change the picture again.

The snippets presented in the post should therefore be taken as what they are – an illustration. The code on GitHub WebRTC Context and the sample app should be kept up to date though, and should therefore be taken as the reference.

The code presented here is also reasonably simplified, as it doesn’t take into account the more complex forms of mediaConstraints that can be passed to getUserMedia(). Adapting the code to take account of those should be quite straightforward however, and for the purposes of example simpler is often better.

{“authors”: [John McLaughlin”,  “Eamonn Power”,  “Miguel Ponce de Leon”]}