The Open Source rfc5766-turn-server Project – Interview with Oleg Moskalenko

As Reid previously introduced in his An Intro to WebRTC’s NAT/Firewall Problem post, NAT traversal is often one the more mysterious areas of WebRTC for those without a VoIP background. When two endpoints/applications behind NAT wish to exchange media or data with each other, they use “hole punching” techniques in order to discover a direct communication path that goes from one peer to another through intervening NATs and routers but not traversing any relays. “Hole punching” techniques will fail if both hosts are behind certain types of NATs (e.g. symmetric NATs) or firewalls. In those cases, a direct communication path cannot be found and it’s necessary to use the services of an intermediate host that acts as a relay for the media or data packets, which typically sits on the public Internet. The TURN (Traversal Using Relays around Nat) protocol allows an endpoint (the TURN client) to request that a host (the TURN server) act as a relay. So far TURN, along with ICE and STUN, has seen little deployment. Now that it is a fundamental piece of WebRTC, it is gaining some momentum. In fact, at the IETF we’re now starting a new effort that will focus on enhancements to TURN/STUN that will be applicable to WebRTC deployments. This new effort is called TRAM (Turn Revised And Modernized), and we’re currently discussing its charter.

Today Chad and I had the chance to catch-up with Oleg Moskalenko, the guy behind the rfc5766-turn-server project: a widely used TURN and STUN server opensource implementation. Oleg is also an IETF contributor and in fact we have recently worked together on a proposal to provide restrictive firewall and HTTP-middlebox traversal for WebRTC (e.g. endpoints behind a corporate firewall or HTTP proxy that only allows HTTP traffic to pass through). Here is a summary of our conversation.

{“intro-by”, “victor”}

webrtcHacks: Why did you decide to make a TURN server?

Oleg: TURN server specs were originally designed to facilitate NAT traversal for RTP/RTCP media traffic, in “classic” VoIP solutions. In the recent years, TURN is expanding and becoming popular because it is a necessary part of the WebRTC infrastructure. TURN is changing from a relatively rare tool to an important part of a popular solution.

TURN is now used with different network protocols (UDP, TCP, TLS) and some applications are even using it as a solution for NAT traversal of general (non-media) traffic. But the main TURN application remains point-to-point connectivity for media in situations where making a direct connection between the peers is not possible. TURN serves as a media traffic re-translator.

webrtcHacks: Were the requirements to build a TURN server difficult? What were some of the main challenges you had to design for?

Oleg: We discovered several serious implementation challenges:

• Scalability – the amount of traffic is huge and a TURN server must be very efficient in forwarding lots of data. Each additional media stream adds significant load. Several hundreds or thousands of audio and video streams can consume enormous bandwidth. Since the TURN server is not merely a dumb packet forwarder, it requires very efficient implementation design and engineering.
• Cost –  since the amount of traffic the TURN server could potentially handle is huge, you do not want anyone without your permission to using it to consume valuable, limited, and expensive resources. You want your TURN server to be protected by efficient mechanisms to prevent overuse.
• Security- another reason for the security concern is that you do not want some spammers or hackers using your TURN server for hiding their identities and stealing capacity. If they can successfully allocate a relay endpoint on your TURN server, that would make your TURN server the source of their malicious traffic – and you may be liable for that.
• Quality – of course, the TURN server must be able not just to translate lots of traffic, it must also be able to do that with good network qualities  — latency, loss, jitter — so the media quality will not be suffering under load. That requires “soft real-time” reaction time.
• Hardware agnostic – if you are using the TURN server as an installable software tool, then, ideally, the TURN server must be able to utilize whatever hardware is the available– and whatever is the convenient for your systems.
• Load balancing – if your application is growing, no single server regardless of the underlying hardware will be able to withstand the load. This means that your TURN server must allow load-balancing solutions among multiple servers, and the scalability must be near-linear.
• Reliability- absolute stability is a must, of course.
• Flexibility – a TURN server can be used in different network conditions with different network topologies. A TURN server may be a public server on the Internet with one or several public network IP addresses. Alternatively it can be an “enterprise TURN server” that sits on the edge of a corporate network with several public IP addresses and several private network addresses. Being able to configure different topologies requires flexibility.

webrtcHacks: So how did you design rfc5766-turn-server to address these challenges?

Oleg: We designed our TURN server with all the above requirements in mind. There are several highlights-

The implementation has two distinct layers: the abstract TURN layer and the implementation layer. The implementation layer is an adaptation of the “abstract” layer to a typical modern POSIX system. This public project is mostly about supporting modern general-purpose POSIX systems, but it can be adapted to a proprietary specialized hardware environment, too. This may be required if there are requirements of extreme performance.

Let me focus on the “general-purpose” POSIX  implementation. We assumed that off-the-shelf hardware and software would be used, but we still wanted to have a high-performance system. Therefore the system is designed to be inherently multi-threaded thereby utilizing the modern system architectures that maximize available CPUs power.

It absolutely must be portable across all popular POSIX platforms, even the the less popular ones. We are trying every obtainable *NIX environment, from Cygwin to Solaris and we are making it work.

We made it to be easily compilable, installable and configurable.

We decided to optimized it primarily for large systems which must serve large number of users.

And of course we needed to provide full, comprehensive documentation.

webrtcHacks: Was building this in a multi-threaded design difficult?

Oleg: We went through several versions of  the threading model.  Each model was tested hard and evolved into the next version. Our current threading model uses the best possible available options of the system used by the TURN server. It automatically discovers the type of the used system and applies the best possible threading option that matches the available sockets implementation. Depending on the system, it uses either thread-per-UDP-network-endpoint model, or multiple symmetrical threads handling all UDP end points – like in 3.9 and newer Linux systems. For TCP endpoints, we are always using a symmetrical, multi-threading model. In this multi-threaded model, a new session is assigned to a single thread (chosen in a round-robin manner) and stays within that thread for its lifetime. That allows optimal CPU utilization without unnecessary context switches.

webrtcHacks: your project is called rfc5766-turn-server – is it safe to assume it is fully RFC5766 compliant?

Oleg: We decided, from the very beginning, that we are going to support all TURN related specs, in full, and we have meticulously followed the specifications. We are even adding some experimental things which will become parts of the TURN standard only in future like, DTLS protocol support and SHA256 hash function.

We did have to “tune” the TURN server specs so that it can be used even when the TURN server is behind a NAT itself – like in an Amazon AWS EC2 system

webrtcHacks: You mentioned this was designed to be reliable for a large number of users? How can you be sure of its reliability?

Oleg: The tests that the server has to endure are really hard. Our findings from testing and analysis contribute to the great stability of the TURN server.

webrtcHacks: Do you have any performance benchmarks on this you can share? I.e – a typical shared Amazon instance can achieve X number of VP8 calls?

Oleg: That kind of performance is hard to measure. Slight difference in the load or network conditions or hardware will result in a very different outcome. Different video or audio codec will also affect the result significantly. So I am not making formal benchmark tests because I see only problems in publishing the results. Somebody would be able to tweak slightly the test and get 10 times better performance, no matter what. But I am pretty sure that we provide the best real performance.

webrtcHacks: Do you expect to be able to make significant performance improvements to the design in the future?

webrtcHacks: So is the project basically done? What else do you have planned?

Oleg: Of course, nothing is perfect and we are planning some significant improvements in the future including:

• Configuration interface beyond the config files and the command-line parameters. We are thinking about GUI and/or CLI approaches;
• Possible user-space TCP/IP stack inclusion (or a kernel module utilization) for even higher performance.
• Statistics and monitoring facilities; possible integration with Ganglia monitoring system.

webrtcHacks: How do you expect the TURN standard to evolve?

Oleg: The TURN standard – RFC 5766 – and related RFCs have been around for years and the folks are getting some news ideas how to improve it. Among others, the most prominent items are:

• DTLS support
• Hash function mobility – SHA256 and possibly other hash functions. The existing standard supports only SHA1
• Security improvements – the current specs are proved to be rather inconvenient in WebRTC environment
• TURN mobility – when a TURN client is used on a mobile device, it may change its IP address and port while moving from one tower receiver to another. The app still wants to retain the same TURN session but the current specs do not allow this. Fortunately a new standard is in the works.
• Configurability – there is also some work in progress to make the TURN more configurable and IT-friendly in a corporate (enterprise) environment

webrtcHacks: Where can our readers find more information about your project?

Oleg: The best place to go is our project page – http://code.google.com/p/rfc5766-turn-server/. We frequently update a Wiki and have a very active discussion group.

{“interviewers”, [“chad“, “victor“]}

{“interviewee”, “Oleg Moskalenko“}