A playground for Simulcast without an SFU

Simulcast is one of the more interesting aspects of WebRTC for multiparty conferencing. In a nutshell, it means sending three different resolution (spatial scalability) and different frame rates (temporal scalability) at the same time. Oscar Divorra’s post contains the full details.

Usually, one needs a SFU to take advantage of simulcast. But there is a hack to make the effect visible between two browsers — or inside a single page. This is very helpful for single-page tests or fiddling with simulcast features, particular the ability to enable only certain spatial layers or to control the target bitrate of a particular stream.

The Playground

There are two variants of the playground, one for Chrome and another one for Firefox. The Chrome variant uses the same old SDP munging hack that we have first seen in Hangouts back in summer 2014. Firefox uses RTCRtpSender.setParameters  to enable simulcast. Neither is complying with the latest specification, but that has not stopped anyone from using it really.

Both variants show the video image, first the sender image and overall bitrate/frame rate graphs followed by the three different spatial streams, each with corresponding graphs for bitrate and framerate.

How bad is the hack required to accomplish this?

Since we don’t want to involve a server, we need to hack things. Fortunately there was a Chrome test which showed the idea. Both Chrome and Firefox use the RTP SSRC of a packet to route it to a certain mediastream. These SSRC can be found in the SDP offer:

Original Chrome SDP

As in 2014, the important thing here are the multiple a=ssrc:  lines as well as the a=ssrc-group:SIM.

Original Firefox SDP

In Firefox, the important simulcast bits are the a=rid  lines and the a=simulcast  line (4th from the bottom). Note: this is from an older version of the specification and subject to change.

webrtc hack

We need to convince our peer that it is actually receiving three different video streams  – a low, medium, and high bitrates – instead of just one. To accomplish that, we need to create our own SDP containing a different mapping from SSRC to track. This is a bit finicky but this site is called webrtchacks for a reason!
We end up transforming the Firefox offer into this:

This shows three different media sections (as specified in unified plan) which will trigger the ‘track’ event three times at the receiver, giving us three different MediaStream  objects to attach to video elements and will also allow making graphs for the individual bitrates using the getStats  API.

Refer to the source code for the SDP munging for how to create this.

Tuning the bitrate for individual layers

Chrome has long used a hardcoded table for the simulcast bitrates of the individual spatial layers.

The table shows the resolution (e.g. 1920×1080), the number of spatial layers (3) as well as the maximum, optimal and minimum bitrate.

Thanks to setParameters  we can now deviate from the bitrates defined in this table and get creative. If you run the example without any modifications, you can see that Simulcast sends at a bitrate of about 3.2mbps. This is split up into three different spatial streams of around 150kbps, 500kbps and 2500kbps which matches the settings in the table.

Let’s modify this by pasting this JavaScript into the console:

This sets the target bitrate of the mid-resolution (640×360) spatial layer to 300kbps and the target bitrate of the 720p stream to 400kbps. These bitrates are matched quite well by the encoder and their quality is surprisingly good, even at 400kbps for the 720p stream.

If we reduce the target bitrate for the 720p layer even more to 200kbps we can see a visual degradation as the framerate drops because only the base temporal layer is sent. 200kbps is about as low as you can go with a 720p stream…

The graphs in this picture (and below) show the bitrate on the left and the framerate on the right side.

This is not exactly news to anyone who successfully implemented simulcast in a SFU, but demonstrating the effect in a single page without any server in the middle is still amazing.

Bugs everywhere

One of the main motivations for building the page initially was to explore the simulcast support in Firefox. It did not perform as well as expected and the big question was whether this was because of Firefox or a bug in my SFU implementation. With the playground page I could demonstrate it wasn’t in the SFU but something is wrong in Firefox.

The bitrate for the mid-resolution layer is mere 300kbps instead of the 500kbps Chrome sends. And we only get 500kbps for the high-resolution layer, despite requesting more than 2000kbps. Part of it turned out to be a very subtle typo in the SDP munging which affected the settings for the low resolution layer. Easy to fix in the JavaScript.

Next, there was an issue with the high-resolution layer config getting modified — this should land in Firefox Nightly soon and will be uplifted to Beta and ESR. With the fix the bitrate sent is much higher:

The target bitrate for the mid-resolution layer also got changed to 500kbps per my request. Consider me a highly satisfied customer!

Jitsi’s Brian Baldino found another interesting issue. When disabling the high and medium spatial layers Chrome will keep sending at a bitrate of more than one megabit per second. This turns out to be padding data in order to keep the bitrate estimate high.

This actually turned out to be a very good reproduction of an older Chrome issue — hopefully the bad behaviour in a more concrete use-case and a reproduction inside a single page test make this easier to fix.

Last but not least you might have noticed a high frame rate for the local video in Chrome — close to 90 frames. It seems that Chrome adds up both the bitrate and framerate for all sending streams. Which is (probably) not quite correct so here goes another bug report .

Summary

Simulcast is quite important for building advanced WebRTC applications. Hopefully this playground makes it a bit more accessible to web developers. Special thanks to Florent Castelli for implementing setParameters  in Chrome, allowing more tinkering as well as Byron Campen for the quick turnaround on the Firefox issue.

{“author”: “Philipp Hancke“}