WebRTC has made getting and sending real time video streams (mostly) easy. The next step is doing something with them, and machine learning lets us have some fun with those streams. Last month I showed how to run Computer Vision (CV) locally in the browser. As I mentioned there, local is nice, but sometimes more performance is needed so you need to run your Machine Learning inference on a remote server. In this post I’ll review how to run OpenCV models server-side with hardware acceleration on Intel chipsets using Intel’s open source Open WebRTC Toolkit (OWT).
Walkthrough
All posts tagged Walkthrough
webrtcH4cKS: ~ First steps with QUIC DataChannels
QUIC-based DataChannels are being considered as an alternative to the current SCTP-based transport. The WebRTC folks at Google are experimenting with it:
Looking for feedback: QUIC based RTCQuicTransport and RTCIceTransport API's are available as origin trial in Chrome 73 for experimentation.https://t.co/KVVEVmggms
— WebRTC project (@webrtc) February 1, 2019
Let’s test this out. We’ll do a simple single-page example similar to the WebRTC datachannel sample that transfers text. It offers a complete working example without involving signaling servers and also allows comparing the approach to WebRTC DataChannels more easily.
webrtcH4cKS: ~ Let’s get better at fuzzing in 2019 – here’s how
Fuzzing overload. Image: Star Trek One Trek Mind #55: No Trouble With Tribbles
Fuzzing is a Quality Assurance and security testing technique that provides unexpected, often random data to a program input to try to break it. Natalie Silvanovich from Google’s Project Zero team has had quite some fun fuzzing various different RTP implementations recently.
She found vulnerabilities in:
- WebRTC — mostly issues in the RTP payload
- Facetime – a few out-of-bounds, stack corruption, and heap corruption issues
- Whatsapp and what didn’t work
In a nutshell, she found a bunch of vulnerabilities just by throwing unexpected input at parsers. The range of applications which were vulnerable to this shows that the WebRTC/VoIP community does not yet have a process for doing this work ourselves. Meanwhile, the WebRTC folks at Google will have to improve their processes as well.
webrtcH4cKS: ~ Computer Vision on the Web with WebRTC and TensorFlow
TensorFlow is one of the most popular Machine Learning frameworks out there – probably THE most popular one. One of the great things about TensorFlow is that many libraries are actively maintained and updated. One of my favorites is the TensorFlow Object Detection API. The Tensorflow Object Detection API classifies and provides the location of multiple objects in an image. It comes pre-trained on nearly 1000 object classes with a wide variety of pre-trained models that let you trade off speed vs. accuracy.
webrtcH4cKS: ~ Debugging VP8 is more fun than it used to be
Editor Note: Fippo uses a lot of advanced WebRTC terms below – if you are a regular reader of this blog then don’t let that scare you. Wireshark is a great tool for diagnosing media issues and inspecting signaling packets even if you’re not building a media server. {“editor”, “chad hart“}
Stuff breaks all the time and then you need to debug it. My favorite tool for this remains Wireshark as we have seen previously. Its fairly useful for debugging all the ICE and DTLS stuff but recently I’ve had to debug the media traffic itself.
webrtcH4cKS: ~ How to limit WebRTC bandwidth by modifying the SDP
WebRTC 1.0 uses SDP for negotiating capabilities between parties. While there are a growing number of objects coming to WebRTC to avoid this protocol from the 90’s , the reality is SDP will be with us for some time. If you want to do things like change codecs or adjust bandwidth limits, then you’re going to need to “munge” SDP for the time being.
At a recent WebRTC Boston, Nick Gauthier of MeetSpace described how he used SDP modification and other techniques to jam up to 10 video callers into a single conference without a media server. Not everyone has a good reason to do this, but there are certainly plenty of applications where having more precise control of your bandwidth consumption would be useful. You can see his video here or check out his technique and thorough explanation on how to munge SDP to adjust individual bandwidth usage below.
webrtcH4cKS: ~ Let’s Encrypt – how get to free SSL for WebRTC
Way back in 47 (version that is), Chrome started to mandate the use of HTTPS in conjunction with getUserMedia. To use HTTPS you need a SSL/TLS certificate. Xander Dumaine covered this a bit for us before, but I still see a lot of people out there struggle with it. As it so happens, the certificate for my own personal website is about to expire and I’m not too excited about paying $70/year to renew it. Fortunately, there is a new way to get certificates for free through Let’s Encrypt. Let’s Encrypt is a non-profit certificate authority that formed with the backing of many major industry players like Mozilla, Akamai, Cisco, and many others to simplify and automate the process of setting up encryption for your website. Oh, and its completely free.
Conference calling is a multi-billion dollar industry that is mostly powered by expensive, high-powered conferencing servers. Now you can replicate much of this functionality for free with a modern browser using the combination of WebRTC and WebAudio.
Like with video, multi-party audio can utilize a few architectures:
- Full mesh – each client sends their audio to every other client; the individual streams are then combined locally before they come out of your speaker
- Mixed with a conferencing server acting as a Multipoint Control Unit (MCU) – the MCU combines each stream and sends a single set to each client
- Routed with a conferencing server in a Selective Forwarding Unit (SFU) mode – each client sends a single stream to the server where it is replicated and sent to the others
This architecture represents a fourth type: client-mixed type where one of the clients acts like the server. This provides the server-less benefits of mesh conferencing without the excessive bandwidth usage and stream management challenges.
webrtcH4cKS: ~ Update: Anatomy of a WebRTC SDP (Antón Román)
Session Description Protocol (SDP) is a fundamental, but very unintuitive concept behind how WebRTC works today. Its no wonder that the Anatomy of a WebRTC SDP post and the interactive SDP guide by Quobis CTO, Antón Román has been so popular here on webrtcHacks. With all things WebRTC, things have changed and we were due for an update.
We also had some rendering issues on the interactive guide. After failing to figure out how to fix it, I decided to completely rewrite it. It is still has some issues, so please make your pull requests to fix and update it on our github repo here.
webrtcH4cKS: ~ Surviving Mandatory HTTPS in Chrome (Xander Dumaine)
“Only Secure Origins Are Allowed”
– Chrome 47
Chrome 47 now forces secure origins (mostly) with HTTPS. This can be a pain to deal with, but Xander Dumaine is here to help with some guidance. Xander is a Senior Software Engineer who deals with the good and bad of WebRTC for Interactive Intelligence in Raleigh, NC. He is helping maintain simpleWebRTC and organises the Triangle WebRTC Meetup group in that area.
{“editor”: “chad hart“}
Want to keep up on our latest posts? Please click here to subscribe to our mailing list if you have not already. We only email post updates. You can also follow us on twitter at @webrtcHacks for blog updates.
