Back in October 2013, the relative early days of WebRTC, I set out to get a better understanding of the getUserMedia API and camera constraints in one of my first and most popular posts. I discovered that working with getUserMedia constraints was not all that straight forward. A year later I gave an update after the situation with Chrome was greatly improved, but Firefox at the time effectively only supported a single resolution so constraints were not much help. Specifically, I am interested in understanding what happens when you ask for a specific resolution. You might want to have a specific resolution returned by getUserMedia if you want to match the camera resolution to a specific video area to have a 1 to 1 pixel correlation, in a computer vision application where each pixel represents a distance, or if you are dealing with non-standard video devices.
Now it has been another fifteen months and a lot has changed since that last post. getUserMedia, WebRTC’s most mature API is supposedly nearing standardization. The spec gone through several updates and we have full support from Firefox and Edge. I was hoping this post would be a quick code update and simple data addition to my last post to reflect these updates. It ended up being a lengthly process process that resulted in some confusing and nonintuitive results.
tl;dr summary
- Don’t require specific resolutions in your app – use a range if you can instead and try the new ideal constraint if you really want, but don’t require something specific
- If you do require specific resolutions in your app, make sure to check the video element to see if the returned video is the same size as what you asked for and only ask for standard sizes, like 1280×720, if you want your app to work will multiple browsers.
- No 4K with getUserMedia – 1080p is the maximum resolution (in 16:9 aspect ratio) you can get today
- Chrome 49 on desktop does a good job scaling to what ever you ask it
- Unlike previous versions Chrome 47 on Android does not always give you what you ask for – this might be a bug
- Use adapter.js to simplify the many differences that exist for getUserMedia and its constraints between browsers (the same is true for PeerConnection too)
- Firefox only works within a set of fixed resolutions
- Edge only works within a set of fixed resolutions
- Device orientation matters when you are requesting fixed resolutions on a mobile
- Asking for a specific square resolution will not help you much on Firefox or Edge, but (mostly) works on Chrome
- Some of the results returned here are someone confusing and nonintuitive
Much has Changed
A lot has changed since my last post:
- The WebRTC API specs were updated to support promises (as mentioned here)
- FireFox added support for getUserMedia constraints in v38
- Microsoft Edge was introduced with getUserMedia
- srcObject was added as a mandatory attribute for assigning a getUserMedia stream to video element
- MediaStream.stop() was deprecated in Chrome in favor of the new MediaStreamTrack object with MediaStreamTrack.stop()
- MediaStreamTrack.getSources() was deprecated in Chrome in favor of MediaDevices.enumerateDevices()
- The getUserMedia constraints object spec changed, with FireFox supporting the latest spec and Chrome still on the old one
That left me with 2 new browsers to support and a bunch of spec changes.
Updating my code
I fixed all the items above. I started by taking copying the latest references from https://webrtc.github.io/samples/, specifically the Choose camera resolution and Choose camera, microphone and speaker samples.
I recommend taking a look at those samples if you are looking for an up-to-date guide on using the getUserMedia API.
See my latest source on github here.
New getUserMedia constraints
Since the scanner am tests to see which specific resolutions work, and which don’t, I need getUserMedia to check one resolution per call. The old way of doing this was just to set mandatory maximum and minimum constraints to the same value:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
var constraints = { audio: false, video: { mandatory: { sourceId: device.id, minWidth: {candidate.width, minHeight: candidate.height, maxWidth: candidate.width, maxHeight: candidate.height } } }; |
The new way to do this is to use the exact object:
|
1 2 3 4 5 6 7 8 9 |
var constraints = { audio: false, video: { deviceId: device.id ? {exact: device.id} : undefined, width: {exact: candidate.width}, //new syntax height: {exact: candidate.height} //new syntax } } }; |
You will also notice that sourceId also changed to deviceId for specifying which device to choose. Note that addition to exact the new gUM constraint specs also allow ideal and advanced, but that is the topic for another post.
adapter.js
The biggest change by far was the addition of FireFox and Edge into my test set. Fortunately adapter.js mostly worked (more on that next). If you are not familiar with adapter.js, it is a brilliant piece of code that adapts for the various browsers and variances with the spec, so you can write your code to the latest spec and in theory it should work across Chrome, FireFox, and Edge (see more here). At the advice of frequent adapter.js contributor Fippo, I pointed to the latest version of adapter JS here: https://webrtc.github.io/adapter/adapter-latest.js
I found a bug
The program has 2 options:
- Quickscan that sorts through a bunch of predefined, common resolutions
- Fullscan to check every 16:9 and 4:3 ratio resolution between an entered range
I killed a couple of hours trying to figure out why the quickscan option worked but the full scan always returned a constraint error. After some debugging I discovered my full scan code was using toFixed() to figure out the width. toFixed converts the value to a string. adapter.js has a bug in Chrome where it requires the value to be number. Fippo filed a pull request to fix this, but later determined it is difficult to fix and closed it. So my code change will have to do for the time being.
Mobile improvements
I made a few other small improvements to help with usability on mobile devices, including using bootstrap to make it responsive and adding some hyperlinks to jump to the end of the table for long scans.
Also, with services like Instagram defaulting to square photos to avoid device orientations issues I decided to add 1:1, ratio test as part of the full resolution scan to see what happens.
Methodology
| Vertical Resolutions scanned | 1 to 1080 |
| Aspect ratios included | 16:9, 4:3, 1:1 |
| Browsers | Chrome 37 Android, Chrome 49, Firefox 44, Firefox Beta 45 Android, Edge 10586 |
| Operating systems | Windows 10, OS X, Android 5 |
1080p is WebRTC’s max
I did some quickscan tests of 4K (2160) resolutions in all scenarios – none of them worked. I scanned through the official webrtc.org source code, I could not find any references above 1080 in the code. 1080 appears to the max unless someone modifies the source to add higher resolutions.
[EDIT]
As Mihály indicates in the comments below, it does appear that Chrome and Opera both allow 4K resolutions on their desktop versions.
Mapping results/errors to categories
Like last time, results can fall into one of 4 categories:
- Pass – the resulting stream output has the same resolution of what was asked for in the mandatory constraints
- Mismatch – this is my programs own error which is tagged whenever the output resolution is different from what was asked for in the mandatory constraints. I see this as the most problematic result since you are getting something different than you asked for without any indication from the API.
- Device Error – For whatever reason, the browser returned a device error. For example, Chrome would sometimes return a DevicesNotFoundError and Edge would give a SourceUnavailableError. I did not get errors like this in Firefox.
- Over Constrained – I used this value whenever the browser returned an explicit error that the constraint was not satisfied. Chrome gives a ConstraintNotSatisfiedError, Firefox gives a OverconstrainedError, and Edge gives a NotFoundError.
Results
I was not able to get a condensed summary table of 45,360 datapoints to fit in this post with any reasonable readability, so I am going to provide some highlights below. I suggest you grab my summary table pdf or just check out the source Excel file.
Note that my Microsoft LifeCam is buggy on Mac, so I only used it on my Windows 10 machine.
Chrome
Desktop
Like last time, Chrome scales the returned image to whatever you ask it, up the the maximum resolution for the camera after which as ConstraintNotSatisfiedError was returned. Chrome 49 behaved consistently for me on OS X and Windows 10.
A 4:3 dimension maxed out at 1280×960 and asking for 1:1 dimension maxed out at 1024×1024 on my old Logitech Webcam 500 even though the max resolution in 16:9 dimension is 1280×720.
Chrome OS
I did a last minute scan on my Chromebook and it showed similar results to the normal desktop above. The only difference is the camera is pretty lower power so it topped out at 360p for 16:9 and 640×480 for 4:3 and 1:1 resolutions. (Note I have not added these results to the pdf table yet).
Mobile
The results on my Samsung Galaxy S5 were not very straight forward compared to desktop.
Mismatches
In my August 2013 scan Chrome 37 scaled the returned resolution down to any resolution I asked for under 1080. For example, when I asked for 1038×584 I got 2×2 as a result. I checked many of these values independently to make sure there was not some bug in my code and I got the same result.
| Browser-OS-Camera | Device Max (HD) | Pass | Mismatch |
|---|---|---|---|
| chrome 47 on Android 5.0 | |||
| camera 0, facing back | |||
| 16:9 | 2160 | 199-360; 406-540, 595-720, 811-1080 | 1-198; 361-405; 541-594; 721-810 |
| 4:3 | 2160 | 265-720 | 1-264; 721-1080 |
| 1:1 | 2160 | 2; 289-720 | 1; 2-288; 721-1080 |
| camera 1, facing front | |||
| 16:9 | 1080 | 1-180; 199-360; 406-540; 595-1080 | 180-198; 361-405; 541-594 |
| 4:3 | 1080 | 1-240; 265-540; 595-1080 | 241-264; 541-594; |
| 1:1 | 1080 | 1-240; 289-1080 | 241-288 |
As you can see in the table above, there seems to be a few seemingly random ranges that return an unusable video stream. These ranges also differ between the front and rear camera.
This seems to be a step back from the very predictable results I got last time and worthy of a bug report.
Device Orientation
Like last time, device orientation makes a difference. Rotating my GS5 vertically cases the returned resolutions to mismatch what I asked for.
The rotation not only swapped the length and width, but also adjusted the image to be more square in many cases.
Firefox
Firefox had made a ton of improvements since last year.
Desktop
Firefox sends out a dock bounce alert on OS X after a successful getUserMedia capture if the window is not in focus. Whenever this happens it appears Firefox pauses Javascript execution until you click on Firefox again to make it the focus. This makes it tough to run the scanner in the background.
Also make sure to set permissions to “Allow” instead of “Always Ask”:
Firefox acts a lot like Chrome did in my original tests in October 2013 – a fixed set of resolutions are supported and if you ask for anything else you get an over-constrained error. Firefox 45 on OSX gave me a lot more resolutions than I got out of Windows 10 with the same camera.
| Logitech Webcam 500 | ||
| Aspect Ratio | Win10 FF44 | OSX FF45 |
|---|---|---|
| 16:9 | 90, 180, 360, 450, 720 | 504, 576, 648, 720, 792, 864, 936, 1080 |
| 4:3 | 120, 240, 480, 600, 720 | 30, 60, 90, 120, 150, 180, 210, 240, 270, 300, 330, 360, 390, 420, 450, 480 |
| 1:1 | nothing | nothing |
No 1:1 aspect ratio resolutions worked.
Mobile
The permissions scheme on Firefox mobile does not cache permissions, meaning you have to explicitly approve camera sharing every time. This does not work well when you are are calling getUserMedia 3240 times.
Fortunately there is a flag you can set to allow permission by default. Just type in about:config in the url bar, then set media.navigator.permission.disabled to true.
Much like Chrome, rotating the phone to portrait mode causes returned video element area to be reversed in cases where the – a mismatch in my methodology.
| Browser | Device | Res Name | Ratio | Ask | Actual |
|---|---|---|---|---|---|
| firefox 44 | Camera 0, Facing back, Orientation 90 | 1080p(FHD) | 16:9 | 1920×1080 | 1080×1920 |
| firefox 44 | Camera 0, Facing back, Orientation 90 | 720p(HD) | 16:9 | 1280×720 | 720×1280 |
| firefox 44 | Camera 0, Facing back, Orientation 90 | VGA | 4:3 | 640×480 | 480×640 |
| firefox 44 | Camera 0, Facing back, Orientation 90 | CIF | 4:3 | 352×288 | 480×640 |
| firefox 44 | Camera 0, Facing back, Orientation 90 | QVGA | 4:3 | 320×240 | 240×320 |
| firefox 44 | Camera 0, Facing back, Orientation 90 | QCIF | 4:3 | 176×144 | 144×176 |
Edge
As the new kid on the WebRTC block, how did Microsoft’s new Edge browser fare? Like Firefox, Edge acts a lot like Chrome did in the early days with only a few pre-defined resolutions working and everything else returning a NotFoundError.
| Camera | Pass | NotFoundError |
|---|---|---|
| edge 10586 on Windows 10 | ||
| camera 1 | ||
| 16:9 | 1080 | 360, 720, 1080 |
| 4:3 | 1080 | 120, 240, 480, 600 |
| 1:1 | 1080 | |
| camera 2 | ||
| 16:9 | 720 | 180;360;720 |
| 4:3 | 720 | 120, 240, 480, 600, 720 |
| 1:1 | 720 | |
Camera 1 was my LifeCam and Camera2 was my Logitech Webcam 500. Oddly, the Logitech Webcam returned a SourceUnavailableError on 160×90 and 800×450.
With no Windows Mobile WebRTC support, this was the only test I could run.
[edit: addition below]
Try it Yourself
It would be great add to this dataset. Try the scan yourself here: https://webrtchacks.github.io/WebRTC-Camera-Resolution/
{“author”, “chad hart“}
Chad,
Great post! In so far as the webcams are concerned there are a few additional things that you might consider.
1 – You need some newer webcams. The MS Lifecam is a quirky beast that relies heavily upon a Windows device driver for decent performance. Logitech’s C500 is ancient. Neither are capable of 1080p.
The Logitech C920 and C930e are really the current benchmarks. Logitech seems to have rebranded the C930e as the HD Pro Webcam.
2 – Consider frame rates, especially at higher resolutions. There are a few webcams that deliver 1920 x 1080 pixel frames, but may not deliver them at rates that you might consider useful. If the application doesn’t leverage UVC1.1 (MJPEG) or UVC1.5 (H264, VP8?, SVC?) to request compressed frames the frames will be uncompressed. Thus the maximum frame rate rate will be limited by the USB 2.0 bus speed.
3 – Does getusermedia() allow the use of such flags for compression?
Since I have several webcams hereabouts a simple experiment or two has answered some of my own questions:
With respect to the Microsoft Lifecam Studio, when using the sample resolution select page that you cited, the camera can be asked for “Full HD” and returns a stream that is that dimension. However, I believe that the camera is actually delivering 720p that’s being scaled up. The result is simply not as sharp as selecting Full HD from a C920.
Further, the C920 delivers a proper 1080p stream. I suspect that GUM is using the UVC1.1 capability to deliver MJPEG encoded frame. In infer this by observing that there’s minimal latency to the video.
When I’ve used software (vMix) that has explicit control of the webcam encoding I can use the UVC1.5 capability to request H264. This always returns a stream that’s delayed around a second.
Michael – thanks for the great comments!
1. I am definitely in need of a new camera and will use your suggestions. I was actually considering getting a 4k camera which is what prompted me to check to see the max resolution support in the source code (where I discovered 4K would be a waste at this point).
2. I have played around with framerates in similar test as this. I ran out of time to work on that as part of this post.
3. Not that I know of or have seen anywhere. It certainly is not in the spec.
A few years ago, seeing that they were destined to become ever more important, I started to explore the state of the art in webcams. It sad really, how little progress has been made in the realm. It’s basically stagnant.
4K “webcams” are a rarity. USB 3.0 capable webcams are nearly as rare. These fact are related.
There’s more to image quality that the sheer number of pixels. Noise, performance in varying lighting and color depth are all things that could be improved, even at 720p or 1080p.
I suspect that 4K webcams won’t matter at all for another couple of years, even as 4K displays become commonplace.
If you want to pick a fight, how about we charge manufacturers with installing built-in webcams that can compete with their aged external counterparts? That’s be a noble effort.
I forgot to mention you can run your own scans here: https://webrtchacks.github.io/WebRTC-Camera-Resolution/
I added a section about this in the post.
Chad,
Great post! It is highly appreciated!
1. -=1080p max =-
You mentioned in the port that you couldn’t find any 1080p max limit in chrome source.
In JSEP I found a 1080p default max value may this is what you are looking for:
https://code.google.com/p/chromium/codesearch#chromium/src/third_party/libjingle/source/talk/app/webrtc/jsepsessiondescription.cc&q=kMaxVideoCodecWidth&sq=package:chromium&type=cs&l=71
2. -=4k & WebRTC=-
January 2015 I had received a great opportunity from PSNC, and tested webrtc in 4k environment.
Let me share in nutshell here my experiences.
That time my experience was that I could use and grab 4K input.
My setup was the following
* PC (Core i7 850 3.07Ghz CPU, 6 GB RAM, 1 TB storage)
* Blackmagic Decklink 4K Extreme card
* Blackmagic Production Camera 4K UHD (3840 × 2160) @ 30FPS
I could successfully grab with getusermedia 4k UHD
* chrome (Windows 7):
Canary 42.0.2280.2
Dev 41.0.2272.3
Beta: 40.0.2214.85
Stable 39.0.2171.99
* Opera(Windows 7)
Stable:26.0.1656.60
Failed to grep input with mozilla firefox (balck screen)
* Firefox
Stable 35
Nightly 38
Please see some images about the successful tests:
https://goo.gl/photos/6GT8r66sUcFnW7AE9
I used also your handy camera/resolution finder tool during the tests. It worked perfectly, and so it is appreciated very much your hours you had spent developing it. Many thanks again!!
I also tested peerconnection.
To setup a peerconnection with UHD 4k media I tested with apprtc.
You could see a screenshoot about the webrtc internals page:
* googFrameWidthInput ~4k
* googFrameHeightInput ~2K
but encoded and sent video was only 2K.
* googFrameWidthSent ~2k
* googFrameHeightSent ~1k
I could grab 4k video with getusermedia, but chrome encoded and transmitted only 2K.
All in all according my experiences 4k was experimental at the beginning of 2015. Had some promising signs, partially working getusermedia, but also had many limitations.
I look forward to your next great post..
Wow. That is really interesting.
On #1, I meant that I could not find reference to a resolution greater than 1920×1080 in the source code anywhere (thanks for sharing the search link). Your results indicate that the Chrome and Opera team must have changed this – in the desktop versions at least.
I was hoping to see getUserMedia return a 4K resolution on my Galaxy S5, but it did not. That is the only 4K camera I have. I would be curious to see if other people are able to get 4K on mobile.
Thank you so much for sharing!
This smells like a bug on mobile…
I’ve seen the same strange behavior here…
On my SG5, all vertical resolutions < 265 require a 16:9 resolution
The "smallest" 16:9 resolution is 354×199
In comparison: The Nexus 7 did pass the test very well, not a single 2×2 video tag has been generated.
Our Nexus 9 and Nexus 5 passed every single test from 320 to 120 without any problems.
A Moto G2 has the smallest "4:3" resolution of 321×241 and a minimal 16:09 resolution of 322×181
I am still collecting more data …
Our WebRTC application is bandwidth sensitive and the resolution has still a deep impact on the total bandwidth needed.
With a variety of many different devices and results… how could we possibly set bandwidth friendly resolutions in a "standard" way without probing for an unknown amount of resolutions?
There is a Chromium bug discussion with some additional data here: https://bugs.chromium.org/p/webrtc/issues/detail?id=5597&sort=-id&colspec=ID%20Pri%20Mstone%20ReleaseBlock%20Component%20Status%20Owner%20Summary
Hi!
Do you know how to solve the “Orientation” problem???
I want it to give me landscape video when the person is using the phone in portrait….
I’ve tried the old and new constraints an none work… It always gives me portrait video….
Hello vuys;
in the article you are describing tests of front/back facing cameras in mobile headsets.
is there any way to push eg. Firefox mobile to support external USB camera connected to USB HOST port in mobile device?
I have tried
I have tried tried mobile versions of Firefox, Chrome but none of them lists external USB camera as available video ource for webrtc connection?
Thanks, man!
I noticed what I think are a couple of typos in the constraints code up above.
1) in your “old” code, you have an extra “{” in minWidth (line 6)
2) in your “new” code, you need to delete the extra “}” on line 7 since there is no “mandatory” component anymore that put it there in the old code.
Great writeup, thanks!
I’ve written a similar post about audio constraints @ https://addpipe.com/blog/audio-constraints-getusermedia/
I am looking for some pointers to keep the resolution at its best and drop the frames if required if WebRTC has low bandwidth. How can I achieve this?
Any help is appreciated.
Unless you have an unusual circumstance, It is generally best to use ‘ideal’ when setting up your constraints and then allow the WebRTC engine to optimize. The WebRTC engine includes a bandwidth optimizer that will alter the resolution, image quality, and framerate automatically based on the bandwidth it estimates.
Are you looking to drop frames before reducing the image quality/resolution when you experience bandwidth issues?