Update: Philipp continues to reverse engineer Hangouts using chrome://webrtc-internals. Please see the bottom section for new analysis he just put together in the past couple of days based on Chrome 38.

---

As initiators and major drivers of WebRTC, Google was often given a hard time for not supporting WebRTC in its core collaboration product. This recently changed when WebRTC support for Hangouts was added with Chrome 36.

So obviously we wanted to check out how this worked. We also were curious to see how a non-googler could make some practical use of chrome://webrtc-internals. Soon thereafter I came across a message from Philipp Hancke (aka Hornsby Cornflower) saying he had already starting looking at the new WebRTC hangouts with webrtc-internals. Fortunately I was able to convince him to share his findings and thorough analysis.

Philipp has been a real-time communications master for more than 10 years. He currently works for simpleWebRTC  and talky.io creator &yet. In addition, he is a major contributor to the Jitsi Meet project. As a long-time member of the XMPP Standards Foundation (XSF),  Philipp has contributed to a number of key XEPs (XMPP speak for an RFC). Incidentally, he is currently serving on the XSF council (which is XMPP speak for the circle of the initiated XMPP gurus that control the world).

You’ll see how the combo of his vast WebRTC knowledge and XMPP/Jingle background come together in the analysis below.

{“intro-by”: “chad“}

---

 

Google announced in late June that their Hangouts now supports WebRTC. At a technical level, this is a very good chance to analyze how Hangouts works using Chrome’s built-in WebRTC diagnostics tool – chrome://webrtc-internals.

The webrtc-internals page is an extremely useful tool for debugging WebRTC issues in Chrome. It shows all API calls of all PeerConnection objects along with additional statistics like bandwidth consumption in a very nice way. This allows us to observe what PeerConnection API calls are used by WebRTC without digging into the source code at all.

Findings

Non-standardized features

As my webrtc-internals analysis below reveals,  some of the non-standard features implemented in Chrome from when its first added WebRTC support are used to facilitate plug-in free Hangouts there. These features are:

• the use of security descriptions (SDES) for encryption
• RTP-based unreliable data channels
• per-candidate ICE parameters (and google-ice, an old variant of the ICE standard)

None of these are part of the official WebRTC spec. This is also explains why Firefox does not work with the WebRTC version of Hangouts.

Plan B multiplexing

Another reason for the lack of support is Firefox is because Hangouts sends multiple audio and video streams over a single PeerConnection. While Firefox does not support this at all, Chrome has used a variant called Plan B (which was not adopted by the IETFs RTCWEB working group) for a while. Basically Plan B calls setRemoteDescription/setLocalDescription repeatedly over the lifetime of the stream, adding source-specific media attribute lines (as described in RFC 5576) in the SDP. The setRemoteDescription calls trigger a onaddstream or onremovestream callback respectively.

Simulcast

This is not the first step Google makes to upgrade Hangouts to WebRTC. In August 2013, the video codec used changed from H.264 (presumably H.264 SVC, the scalable variant) to VP8.
As we will see later, the new WebRTC-hangouts make use of a technique called Simulcast which is similar to SVC. Possibly Google had already introduced this technique back then, making small, well defined steps in the upgrade process.

In simulcast, several video streams are sent, each with different resolutions and framerates. Conversely, SVC does this using a single video stream. When simulcast is used to send multiple streams over the same connection there is not much difference, even though SVC is slightly more efficient. One of the most important features of both SVC and simulcast is that a Selective Forwarding Unit (such as the Jitsi Videobridge or the VidyoRouter that is presumably still used by Hangouts) can forward low-bandwidth versions of the stream to certain clients without having to decode and re-encode the video in the process.

NaCL

Another issue that was quoted back in August 2013 as a reason that Hangouts did not switch to WebRTC yet was the hats feature. Now all people in the screenshots are wearing hats and Chrome Native Client extensions (NaCL) is required. This might mean that the hats feature (which requires facial recognition) needs more performance than currently available at the JavaScript layer to work well.

The goal of this step seems to have been rolling out compatibility with Chrome without making additional changes on the server-side infrastructure. Removing the non-standard elements like SDES and RTP-based data channels might happen in the future, but will not impact users in any way. Firefox compatibility will then mostly depend on how fast Mozilla is able to add multiple streams per PeerConnection.

Screen Sharing

Even though the use of Chrome Extensions for screensharing has been advocated by the WebRTC team, Hangouts does not require an extension to use the chrome.chooseDesktopMedia API.

Hangouts is mostly WebRTC?

As a summary, let us compare where what Chrome uses in this case is different from the WebRTC standard (either the W3C or the various IETF drafts):

Feature	WebRTC/RTCWeb Specifications	Chrome
SDES	MUST NOT offer SDES	uses SDES
data channels	DTLS/SCTP-based	unreliable RTP-based
ICE	RFC 5245	google-ice
Audio codec	Opus or G.711	ISAC
Multiple streams	undecided yet	Plan B
Simulcast	undecided yet	proprietary SDP extension

If you don’t believe me, or want to learn how to use webrtc-internals to extract this kind of information and more from your WebRTC session, then please review the detailed analysis section below.

Analysis

We are now going to take a quick look at a session we dumped. The full dump is available here, saved directly from webrtc-internals.

Beware, the following sections use quite a lot of SDP terminology, so reading the SDP anatomy blogpost is highly recommended.

RTCPeerConnection Constraints

The constraints used to create the RTCPeerConnection object:

 

Mandatory contraints

This shows two mandatory constraints:

DtlsSrtpKeyAgreement: false

Setting this to false disables DTLS-SRTP and re-enables the older SDES encryption. As Victor previously covered in this post, at IETF 87 in Berlin last July, it was decided that WebRTC is not going to use SDES anymore. Chrome recently disabled support for this by default. Technically this means that any claims that Hangouts now uses WebRTC are incorrect.

In terms of security not much is changed however, since Hangouts are not peer-to-peer. The bridge is acting a a man-in-the-middle, so DTLS-SRTP does not offer much advantage here. As mentioned earlier, this is one of the reasons that Hangouts does not work in Firefox, which does not support SDES at all.

RtpDataChannels: true

Before deciding to use SCTP over DTLS for WebRTC data channels, Chrome first implemented unreliable data channels over RTP. This is what the RTPDataChannels constraint does. It seems the video router Google uses for this does not support DTLS at all yet. This does not work in Firefox either.

Optional constraints

There are a couple of optional constraints used too:
googLog:…

The name implies that it enables some sort of logging.

googSkipEncodingUnusedStream:true

This is an undocumented flag, but seems quite interesting. Not encoding unused streams saves CPU obviously, but it’s not clear how “unused” is defined. Quite likely this relates to the use of simulcast which is described below.

googScreencastMinBitrate: 400

Another undocumented flag. It seems to set the minimum bitrate for screensharing.

googImprovedWifiBwe: true

This enables an improved bandwidth estimation algorithm which was mentioned by Justin and Serge in their WebRTC Update at the KrankyGeek event.

googHighBitrate:true, googVeryHighBitrate: true

Also mentioned in the update, this uses a higher initial bitrate. Without this, getting high-quality HD pictures would take quite long.

The RTCPeerConnection API calls

The dump shows a createLocalDataChannel call followed by a createOffer and two addStream calls, one for audio, one for video:

The order above is likely a bug – calling createOffer before adding the streams generates a receiveonly stream which we can see in the createOfferOnSuccess. The createOfferOnSuccess callbacks shows an SDP that is the result of some of the constraints mentioned above:

a=recvonly

This is caused by calling createOffer before the addStream calls.

a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:…

Since DtlsSrtpKeyAgreement is set to false SDES is used. There is no fingerprint attribute here either.

m=application 1 RTP/SAVPF 101

As mentioned above, RtpDataChannels are used.

setLocalDescription

Next, we see a setLocalDescription call:

Usually, one would not expect much difference between the SDP in the createOfferOnSuccess callback and the one used in setLocalDescription. Here, the changes are quite significant, so let’s take a closer look.

There is a a=ice-options:google-ice line at session level, moved there from the individual m-lines.

This means that google-ice, an old variant that predates RFC 5245 is used. This can be easily verified with tools like Wireshark. Compared to standard ICE the both username fragments that form the STUN username are not separated by a colon.

m=audio 1 RTP/SAVPF 103 111 0 8 106 105 13 126

This changes the order of the audio codecs described in the a=rtpmap lines. Basically this means that the ISAC codec (at 16khz) is used.

a=sendrecv

This makes the streams bidirectional, working around the wrong order of the createOffer / addStream calls.

a=ssrc:…

The number of a=ssrc lines is lower than what Chrome generates by default. But it turns out that actually the two additional lines that Chrome generates are redundant and will be removed at some point.

a=ssrc-group:SIM

This defines a group of synchronisation sources (as described in RFC 5576) with a “SIM” semantics. While this semantics is not registered with the IANA, the name SIM implies simulcast, i.e. sending different resolutions or framerates on the same connection. We can see three different SSRCs in this group, probably HD, 360p and a small resolution stream. It recently became possible to achieve similar results with getUserMedia, but this way to enable simulcast might be easier to optimize for the video encoder.

setRemoteDescription

The next thing we see is a setRemoteDescription call:

This shows the SDP coming from the video router. Again, a number of things are worth pointing out here:
a=ice-ufrag:1234567890123456
a=ice-pwd:123456789012345678901234

These are the router’s ICE ufrag and pwd attributes. At a first glance this looks like a really bad hack. However, it turns out later that these parameters are never used. They’re just here to make Chrome happy which enforces the presence of these attributes in a setRemoteDescription call.

a=x-google-flag:conference

It is not documented anywhere what this actually does. Probably it’s a hint for the encoder that there might be multiple audio streams to enable better echo cancellation.

Another setRemoteDescription?

This is immediately followed by another setRemoteDescription call, without waiting for the setRemoteDescription to even succeed:

The new setRemoteDescription call adds two a=ssrc lines in the audio section which triggers an addStream. This might be a mixed audio stream for the conference. Another setLocalDescription call (no changes) brings the signaling state back to stable. As a rule of thumb, you always want your signaling state to be stable.

ICE candidates

Next, the video router sends a couple of ICE candidates. Let’s look at the first one:

This line includes a username and password at the end which is an extension to the normal ice candidate syntax that is only supported by Chrome. It seems that hangouts might still use Jingle, the signaling protocol Google originally came up with for XMPP and Google Talk. Jingle allows individual ufrag and pwds for each transport-info message which are used to do trickle ice. More on that later.

Looking at the other candiates is interesting as well, in particular the third one:

The transport field of this candidate is ssltcp and the port is 443. This means it is running ICE-TCP prepended by something that looks like a TLS handshake on port 443. Port 443 is typically used for HTTPS traffic which will be open even in environments that block UDP traffic.

Further, it will work well with HTTPS proxy servers that support the CONNECT method. This method is similar to running TURN over TLS, with slightly less overhead for the server.

Plan B multi-streams

The next setRemoteDescription call adds a video stream:

Adding multiple audio/video streams this way is called “Plan B” and described here. It was actually decided at the IETF in Berlin last year that this is not the SDP way to do this, but we also use it alot in Jitsi Meet and it works pretty well.

At that point, the ICE connection is established and we see an unreliable data channel added with a onRemoteDataChannel callback. Note how the label of the channel correlates with the a=ssrc msid line in the SDP.

When another person joins, we see another setRemoteDescription call. This one adds the following lines in the video section:
a=fmtp:100 width=640
a=fmtp:100 height=360
a=fmtp:100 framerate=30

While Chrome does not parse this (and does not need to), it is very interesting because this is equivalent to a description-info action in Jingle as described here. As this is the only thing that changes here, I take it as another hint that hangouts still uses Jingle.

Next we see another two setRemoteDescription calls, adding an audio and video stream respectively. And a third one which adds an RTP datachannel.

While not shown in the dump, the same thing happens when a participant leaves. The a=ssrc lines are removed and onRemoveStream is called.

---

New Chrome 38 Analysis

In Chrome 38 (currently canary channel), the chrome://webrtc-internals page also shows any getUserMedia calls and the constraints used. Again, what has been tweaked is quite interesting:

Common constraints

sourceId

The sourceId constraint selects a specific device, identified by this id which can be stored in a cookie or localStorage setting.

The API for enumerating available devices is currently somewhat unstable but already allows changing the microphone and camera to be used in Chrome. It was originally MediaStream.getSources, but was renamed to Navigator.getMediaDevices and is soon going to be available as MediaDevices.enumerateDevices.

Audio constraints

There is quite a number of audio constraints here. The first set consisting of:

This probably gives you  the best echo cancellation available.

googAudioMirroring

seems to allow swapping the left and right channels for local audio (which is disabled usually), similar to how the local video is displayed mirrored by means of CSS.

googDucking: false

Ducking is described as using the default communications device on Windows in this chromium issue. This is particularly important for users with headsets.

It seems to be broken currently and is deactivated therefore.

chromeRenderToAssociatedSink: true

Quite likely, this sends audio output to the same device that is used for capturing audio. Again, this seems useful when using a headset.

Video constraints

minFrameRate: 30, minHeight: 720, minWidth: 1280, maxFrameRate: 30, maxWidth: 1280, maxHeight: 720

Acquires a 720p hd videostream at 30 frames per second.

googLeakyBucket: true

As explained in this bug report this changes the way the video encoding adapts to the available bandwidth, especially when sending large keyframes as it happens when sharing a full-HD screen.

googNoiseReduction: true

Likely removes the noise in the captured video stream at the expense of computational effort.

{“author”: “Philipp Hancke“}