Newer note: February 2016 update here.

Note: Behavior has changed with latest versions of Chrome (v35+). Please see my update to this post here.

{“editor”, “chad“}

---

I have a confession to make about my WebRTC Motion Detecting Baby Monitor – the video quality was inconsistent and poor on the baby side of my original demo video, so I swapped out my old HTC Thunderbolt for another laptop in the 2^nd half of the video. The stream and motion processing consumes a full core on my 2 GHz Intel i7 laptop processor. Fortunately I have more cores there, but this would clearly be a problem for a lot of devices – both in terms of having enough CPU to meet the application’s demands and on battery life. There are also bandwidth concerns in the real world. I was am running everything over WiFi, so I was not too concerned about bandwidth, but there is no reason why one should not be able to run this on a LTE or 3G network where there are bandwidth constraints and data plan usage concerns.

This got me experimenting with getUserMedia() constraints to see what affect adjusting the source resolution and framerate has on CPU utilization and bandwidth consumption. I discovered this is not as straight forward as one would think, so I am posting my observations here.

 

getUserMedia() constraints

As covered in the What happens when there’s missing media sources? post, getUserMedia() takes a constraints object. The object, as defined in the getUserMedia() W3C specification looks like this:

The spec allows mandatory and optional constraints for both minimum and maximums on:

Object	What it does	Value options
Height	Specifies the video source height	Min and Max of an integer
Width	Specifies the video source width	Min and Max of an integer
FrameRate	specify how many frames to send per second (usually 60 for HD, 30 for SD)	Min and Max of an integer
aspectRatio	height divided by width – usually 4/3 (1.33333333333) or 16/9 (1.7777777778)	Min and Max of an decimal
facingMode	Select the front/user facing camera or the rear/environment facing camera if available	Which camera to choose – currently user, environment, left, or right

In addition to turning audio and video on/off all together, in practice minHeight, minWidth, maxHeight, and maxWidth variables are the most widely supported:

I have also seen reference to others that could theoretically be passed down to the libjingle media engine that powers WebRTC. These are out of scope for my camera resolution project, so I did not experiment with them.

Initial Experiments

Adjusting the video resolution seems simple enough using the minHeight, minWidth, maxHeight, and maxWidth constraints. I started by setting various constraints and then using the .videoWidth and .videoHeight properties of my video object to validate the actual values:

Set constraints

Call getUserMedia() & assign to a video object on success

Assign an event listener and report the results

Non-obvious Findings

Let’s start out with Chrome (I’m using 30). One would think that requiring an exact width and height would either result in:

• a stream with that exact width and height, or
• a failure (call to the error callback – onError in my code above)

Well, that’s not what happened. 1024×768 gave me an error. 1980×1080 gave me 1280×720. Going to 640×480 next gave me 1064×800. This did not make any sense.

To make sure it was not just my code, I checked out Sam Dutton’s simpl.info getUserMedia constraints demo. The VGA button uses 640×360 as max values for constraints – it returned 320×180:

So much for mandatory.

Mandatory constraints are really suggestions

I did some research and came across this statement by Justin Uberti at Google:

So I was not crazy. It turns out Chrome uses several fixed resolutions:

Chrome will return something that is close to the resolution requested among these values. Sometimes it will return something that is outside of your constraints. See this message on the Chromium board for more details.

Kill the stream before applying new constraints

I also noticed it is very important to stop and nullify the stream, if you call getUserMedia() again with new constraints without refreshing in between. If you do not kill the stream, the new constraints are ignored and the previously provided constraints are used again. You can try this here: http://src.chromium.org/svn/trunk/src/chrome/test/data/webrtc/manual/constraints.html

Set 800×600 as a min value and I get 960×720:

Move 800×600 to a max value and hit getUserMedia again and you get the same value:

Fortunately killing the stream is easy and fixes this problem:

Use https for a better user experience

Chrome forces the user to ask for permission every time getUserMedia() is called if you server your page using http. Fortunately Chrome desktop remembers your permissions if you use https:

Source: https://support.google.com/chrome/answer/2693767?hl=en

In most situations dropping the stream and then the user for permission repeatedly will not generate a good experience. Fortunately the https option allows constraints to be changed more seamlessly, at least in the desktop version. Chrome for Andriod requires permissions for every getUserMedia() call, even on https.

Don’t bother changing constraints in Firefox today

FireFox as of 24.0 does not support the video resolution constraints. You can however manually set the default parameters in the about:config preference selections under media:

This is helpful for testing but has limits the kinds of applications where you could make use of changing video resolutions.

 

Auto-identifying Camera Resolutions

An API for discovering possible camera resolutions sure would be nice. Looking through the W3C mailing list, it looks like this was suggested but dropped due to concerns about using this information to help further fingerprint users based on their hardware. For more on what this fingerprint could look like I suggest checking out this Electronic Frontier Foundation site.

Fortunately using the techniques above it is possible to determine what camera resolutions are available in Chrome by simply trying all six resolution options and seeing what is returned. You can check it out yourself here: https://gumcameraresolutions.azurewebsites.net/ and see the code here: https://github.com/webrtcHacks/WebRTC-Camera-Resolution

In this quick demo I just used the 6 resolutions identified above. Here are the results across my various Chrome supporting devices:

I did not receive an error back when requesting any of these resolutions, but interestingly in some cases the resolution returned was different than what was requested. It turned out to be less of a problem than I thought, but the results are interesting nonetheless. If image size is important to your application then this kind of test may be needed.

Since both the standard and implementation are in flux, I wanted to verify other resolutions than the 6 above were available. To do this I also created a version that runs through every possible 16:9 and 4:3 resolution combo between 100×56 to 1280×960 here: https://gumcameraresolutions.azurewebsites.net/fullscan.html. All of the constraints that were not the 6 above returned an error and the rest of the results were exactly the same.

These are still early days

If I was a professional developer instead of an inquiring hobbyist then this kind of problem would probably get me upset. In 13% of my test cases I was given a different resolution than I asked for. There was no easy way to discover what resolutions are available. The 6 resolutions provided seem somewhat arbitrary. This implementation may change, but who knows to what. Firefox has no support today. All these problems and unresolved issues and getUserMedia() is the most mature of the three WebRTC API’s!

It is clearly still the early days for WebRTC. Still, 1280×720, 640×480, and 320×240 seemed to work consistently well in all the tests. I would stick to those resolutions if I had to based on my limited dataset.

{“author”, “chad“}