Is Slack’s WebRTC Really Slacking? (Yoshimasa Iwase)

Earlier this month Fippo published a post analyzing Slack’s new WebRTC implementation. He did not have direct access or a team account to do a thorough deep dive – not to mention he is supposed to be taking some off this month. That left many with some open questions? Is there more to the TURN network? How does multi-party calling work? How exactly is Slack using the Janus gateway? Fortunately WebRTC has an awesomely active and capable community that quickly picked up the slack (pun intended).

Last week Yoshimasa Iwase published a great post giving slack a deeper dive.  Unfortunately Google translate did not give the Japanese translation justice, so I asked him if we could provide a translation with some added details here. Iwase-san has been deeply involved with WebRTC for several years at NTT Communications and as editor at HTML5Experts.jp (which is another great resource you should check out.) He has also helped organize WebRTC events in Japan and if you’ve seen him speak you won’t question he knows his stuff.

Check out his in depth analysis below.

{“editor”: “chad hart“}

Introduction

Inspired by the by Philipp Hancke’s (Fippo) Slack article, I started to analyze how Slack uses WebRTC more deeply. I really wanted to understand Slack’s WebRTC architecture to see if it could help inform other WebRTC engineers on how to build their own service or system. Slack has over 2 million active users at the time of writing, so seeing how they built their service should provide useful insight into how to build your own scalable WebRTC service.

Analysis method

As Philipp Hancke describes a in the Blackbox Exploration series of analysis, there are some useful tools to analyze WebRTC service. I chose the same tools:

• Chrome webrtc-internals to see information like SDP and candidates and dump logs – Slack only supports Chrome so I couldn’t check about:webrtc in Firefox
• JavaScript – their files are minified but we can check some functions like “RTCPeerConnection” anyway
• Wireshark capture

Below is the result of my analysis.

Slack doesn’t use P2P

It’s common to use P2P topology in 1 to 1 communication since this optimizes user experience and minimizes the number of servers you need to maintain. Despite this fact Slack forces all stream to use TURN server. In addition to TURN they use janus as an endpoint.

Here is the toplogy picture of their usage:

 

Even though you are going to Slack WebRTC chat with someone sitting next to you on the same LAN, the communication path will still follow the above topology. Their TURN server is deployed on AWS.  As far as I tested, there’s no TURN server in Tokyo region and my WebRTC client (Chrome) is connected to TURN server in Singapore region, which causes unnecessary latency. I’ll describe their TURN deployment in more detail next.

How do they force us to user TURN?

The answer is found in SDP. Let’s check  the SDP Offer at first and then the Answer.

Offer from Janus

From *1, Apparently Slack uses Janus as a server side WebRTC endpoint. Janus is WebRTC Gateway and can behaves as MCU, SFU, and other modes with its plugin architecture. (See Lorenzo Miniero’s post for more on WebRTC Gateways)

*2 shows “RTP/SAVPF” is still used. In fact Lorenzo  fixed this into “UDP/TLS/RTP/SAVPF” immediately after Fippo’s post. After Slack updates their Janus, we should see the SDP updated here.

From *3, Slack sends us some ICE candidates in vanilla ICE style, which is in common in MCU or SFU usage since candidates are obvious to MCU or SFU. The key point here is that the IP address are all private ones. The means we can’t connect each other in P2P topology.

How do we connect these Private IP

The answer is found in the ICE candidates received. If you check chrome://webrtc-internals you will find host/srflx/relay as usual. (srflx should be absent when your machine has global IP address)

“host” and “srflx” are meaningless because these address can’t connect *3’s IPs. “relay” – a.k.a. the TURN address – is the most important ICE candidate and here is the concrete result:

Obviously you can find the global IP address, “52.77.208.161”. This is the TURN server’s address Slack deployed in their system. Reverse looking up this global IP shows …

TURN server are deployed on AWS. Because AWS EC2 instance both global IP and private IP in VPC, TURN server should have private IP. This private IP can connect to *3 private IP.

Slack’s TURN server

Speaking of TURN server, Slack doesn’t use the active coturn project but the former RFC5766 TURN SERVER. This result found at Wireshark dump:

The TURN server’s version is 3.2.3.96 and actually this is old version. The latest version of coturn server is ”3.2.5.9” according to GitHub.

In addition to version here are some information about Slack’s TURN server:

• URN of the TURN server is   turn:slack-calls9.slack-core.com:22466  and it seems that there are about 23 TURN servers deployed around the world. You can check this by lookup from slack-calls1.slack-core.com  to slack-calls23.slack-core.com
• I was able to find TURN servers deployed at least in Singapore, Ireland, and Northern California
• TURN authentication information, user name and password, are dynamically obtained from JavaScript

Why do they force users to TURN server?

There might be some reasons:

1. To reduce call set up time like FaceTime and WhatsApp does
2. To make use of TURN function like authentication

That’s all I could find on their TURN setup, so go back to their SDP.

Answer from Janus

Janus’ SDP Answer was as follows:

Basically the structure is almost same as the offer. However, one of the points I’d like to describe is *4. (Actually in the offer you can see a=sendonly attribute but I just skipped it for simple explanation) This means there are two media streams between Client and Janus. One is sendonly and the other one is recvonly.

The explanation above is depicted as following image:

If Janus behaves as MCU, mixing the streams from each party in the stream, there should should only be a single stream down to a client (Browser or Desktop application). On the other hand, if Janus behaves as SFU the down stream will be the same number of the clients because SFU doesn’t combine the streams, it  just distributes them. I will investigate their multi-party calling model next.

How are they doing multiparty?

Slack provides multiparty voice chat to paying users. My team uses the paid service so we were able to do a 3-person voice chat experiment. The scenario of the experiment is pretty simple:

1. Member-A using Chrome starts to create a room
2. Member-B and Member-C using Desktop application join the room

After this experiment I got SDPs, dump data from chrome://webrtc-internals, and packet capture. Here is analysis result.

Slack’s Janus behaves as SFU

As I wrote before, Janus itself is a WebRTC Gateway and provides many functions with various plug-in. One of these plug-ins is for a videoconferencing SFU. Our analysis indicates Slack uses Janus as SFU. (I don’t know which SFU plugin is used: Janus officially provides or Slack originally developed)

Anyway, Here is the image of multiparty topology:

How did I find this topology?

The answer is also found in SDPs. In this experiment My browser Chrome got two recvonly offers from Janus.

1st offer

2nd offer

As you see, both SDPs have a=sendonly, which means 2 media streams going to my browser Chrome and means Slack’s Janus doesn’t behave as MCU.

These SDPs  show another interesting fact.

Multistream with Plan B or Unified Plan is not supported

There are a couple ways to send multiple streams: (This is well explained at Dr. Alex’s presentation slide)

• a. Multicast
  1. Multiple PeerConnection with single Mediastream
  2. Single PeerConnection with multiple Mediastream
• b. Simulcast
• c. SVC Encoding

[1.a] is available from the beginning of WebRTC, but consumes much resources such as ports. For [1.b] there are two major ways to realize: one is Plan B based and the other one is Unifid Plan based. (For your information, it seems Chrominium started to implement Unified Plan)

The MultiStream used in Slack is [1.a].

Signaling

Slack signaling is based on ScreenHero. You can find this from minified JavaScript code. Some snippet is like this:

Room based signaling is used which is common in multiparty conference scenarios.

Conclusion

In conclusion, here are some of the main highlights:

• No peer-to-peer – all calls are relayed by an AWS-based TURN network using an older TURN server project
• Using Janus as a SFU for providing multi-party calls
• Signaling looks to be room based from ScreenHero

As one of WebRTC engineers and one of Slack users, it’s kind of disappointing that Slack doesn’t fully utilize TURN feature, such as ‘TURN/TCP’ and ‘TURN/TLS’. Since Slack has over 2 million active users, with this TURN configuration many users under strict network environment are unable to use voice chat for now. I hope they’ll change their setting in the near future in order to give the Slack users better UX.

Slack already announced they’re going to add video chat feature so I’ll check their WebRTC usage when that is available and hopefully be back here to comment again.

{“author”: “Yoshimasa Iwase“}