Guide

webrtcH4cKS: ~ AIY Vision Kit Part 1: TensorFlow Computer Vision on a Raspberry Pi Zero

Posted by Chad Hart on March 6, 2018
Posted in: Guide.
Tagged: , , , , , .
19 comments

A couple years ago I did a TADHack  where I envisioned a cheap, low-powered camera that could run complex computer vision and stream remotely when needed. After considering what it would take to build something like this myself, I waited patiently for this tech to come. Today with Google’s new AIY Vision kit, we are pretty much there.

The AIY Vision Kit is a $45 add-on board that attaches to a Raspberry Pi Zero with a Pi 2 camera. The board includes a Vision Processing Unit (VPU) chip that runs Tensor Flow image processing graphs super efficiently. The kit comes with a bunch of examples out of the box, but to actually see what the camera see’s you need to plug the HDMI into a monitor. That’s not very useful when you want to put your battery powered kit in a remote location. And while it is nice that the rig does not require any Internet connectivity, that misses out on a lot of the fun applications. So, let’s add some WebRTC to the AIY Vision Kit to let it stream over the web. ...

Continue Reading

webrtcH4cKS: ~ Computer Vision on the Web with WebRTC and TensorFlow

Posted by Chad Hart on December 3, 2017
Posted in: Guide.
Tagged: , , .
22 comments

TensorFlow is one of the most popular Machine Learning frameworks out there – probably THE most popular one. One of the great things about TensorFlow is that many libraries are actively maintained and updated. One of my favorites is the TensorFlow Object Detection API.   The Tensorflow Object Detection API classifies and provides the location of multiple objects in an image. It comes pre-trained on nearly 1000 object classes with a wide variety of pre-trained models that let you trade off speed vs. accuracy. ...

Continue Reading

webrtcH4cKS: ~ How to capture & replay WebRTC video streams with video_replay (Stian Selnes)

Posted by Chad Hart on August 31, 2017
Posted in: Guide.
Tagged: , , , , .
10 comments

Decoding video when there is packet loss is not an easy task.  Recent Chrome versions have been plagued by video corruption issues related to a new video jitter buffer introduced in Chrome 58. These issues are hard to debug since they occur only when certain packets are lost. To combat these issues, webrtc.org has a pretty powerful tool to reproduce and analyze them called video_replay. When I saw another video corruption issue filed by Stian Selnes I told him about that tool. With an easy reproduction of the stream, the WebRTC video team at Google made short work of the bug. Unfortunately this process is not too well documented, so we asked Stian to walk us through the process of capturing the necessary data and using the video_replay tool. Stian, who works at Pexip, has been dealing with real-time communication for more than 10 years. He has  experience in large parts of the media stack with a special interest in video codecs and other types of signal processing, network protocols and error resilience. ...

Continue Reading

webrtcH4cKS: ~ WebRTC Externals – the cross-browser WebRTC debug extension

Posted by Philipp Hancke on July 28, 2017
Posted in: Guide.
Tagged: , , , .
4 comments
Garden Tools

I am a big fan of Chrome’s webrtc-internals tool. It is one of the most useful debugging tools for WebRTC and when it was added to Chrome back in 2012 it made my life a lot easier. I even wrote a lengthy series of blog post together with Tsahi Levent-Levi describing how to use it to debug issues recently.

Firefox has a similar about:webrtc page which shows the local and remote SDP for each page as well as a very useful grid of ICE candidates. But unlike Chrome it does not show the exact order of API calls or nice graphs obtained from the getStats API. I miss both features dearly. Edge and Safari don’t support similar debugging helpers currently either. ...

Continue Reading

webrtcH4cKS: ~ Reeling in Safari on WebRTC – A Closer Look at What’s Supported

Posted by Chad Hart on June 19, 2017
Posted in: Guide, Reverse-Engineering.
Tagged: , , , .
7 comments

Long have WebRTC developers waited for the day Apple would come around to WebRTC. It has not been simple for web developers and Apple due to their policy that requires web browsing functionality to use the WebKit engine along with Safari. This meant no WebRTC in Safari; no Firefox or Chrome WebRTC on iOS, no native WebView with WebRTC or iOS API’s (but plenty of 3rd party ones). Despite community efforts and active development inside the WebKit project, it was not entirely clear when there would be at launch. That changed earlier this month when Apple announced a WebRTC-enabled WebKit based on the Google-backed webrtc.org engine was coming to both High Sierra – the next version of OSX – and iOS 11. Even better, WebRTC is available today as part of the free Safari Technology Preview. ...

Continue Reading

webrtcH4cKS: ~ Am I behind a Symmetric NAT?

Posted by Philipp Hancke on May 15, 2017
Posted in: Guide.
Tagged: , , , , .
6 comments

NATs can be a nuisance for VoIP, particularly Symmetric NATs . Fortunately WebRTC includes tools for dealing with them. Image source: http://pinktentacle.com/

WebRTC establishes peer-to-peer connections between web browsers. To do that, it uses a set of techniques known as Interactive Connectivity Establishment or ICE. ICE allows clients behind certain types of routers that perform etwork Address Translation, or NAT, to establish direct connections. (See the WebRTC glossary entry for a good introduction.) One of the first problems is for a client to find what its public IP address is. To do so, the client asks a STUN server for its IP address.

NATs are boxes (physical or virtual) that connect our local private networks to the public internet. They do so by translating the internal IP addresses we use to public ones. They work differently from one another, which ends up requiring WebRTC to rely on both STUN and TURN in order to connect calls. For background on these, check out some of our past posts on this topic like this one and this one. ...

Continue Reading

webrtcH4cKS: ~ Debugging VP8 is more fun than it used to be

Posted by Philipp Hancke on March 28, 2017
Posted in: Guide.
Tagged: , , , , .
2 comments
insect

Editor Note: Fippo uses a lot of advanced WebRTC terms below – if you are a regular reader of this blog then don’t let that scare  you. Wireshark is a great tool for diagnosing media issues and inspecting signaling packets even if you’re not building a media server. {“editor”, “chad hart“}

Stuff breaks all the time and then you need to debug it. My favorite tool for this remains Wireshark as we have seen previously. Its fairly useful for debugging all the ICE and DTLS stuff but recently I’ve had to debug the media traffic itself. ...

Continue Reading

webrtcH4cKS: ~ How to limit WebRTC bandwidth by modifying the SDP

Posted by Nick Gauthier on October 26, 2016
Posted in: Guide.
Tagged: , , , .
12 comments

WebRTC 1.0 uses SDP for negotiating capabilities between parties.  While there are a growing number of objects coming to WebRTC to avoid this protocol from the 90’s , the reality is SDP will be with us for some time. If you want to do things like change codecs or adjust bandwidth limits, then you’re going to need to “munge” SDP for the time being.

At a recent WebRTC Boston, Nick Gauthier of MeetSpace described how he used SDP modification and other techniques to jam up to 10 video callers into a single conference without a media server. Not everyone has a good reason to do this, but there are certainly plenty of applications where having more precise control of your bandwidth consumption would be useful. You can see his video here or check out his technique and thorough explanation on how to munge SDP to adjust individual bandwidth usage below. ...

Continue Reading

webrtcH4cKS: ~ WebRTC media servers in the Cloud: lessons learned (Luis López Fernández)

Posted by Chad Hart on September 9, 2016
Posted in: Guide.
Tagged: , , , , , , , , , , .
3 comments

Media servers, server-side media handling devices, continue to be a popular topic of discussion in WebRTC. One reason for this because they are the most complex elements in a VoIP architecture and that lends itself to differing approaches and misunderstandings. Putting WebRTC media servers in the cloud and reliably scaling them is  even harder. Fortunately there are several community experts with deep expertise in this domain to help. One of those experts who has always been happy to share his learnings is past webrtcHacks guest author Luis López Fernández. ...

Continue Reading

webrtcH4cKS: ~ Let’s Encrypt – how get to free SSL for WebRTC

Posted by Chad Hart on August 1, 2016
Posted in: Guide.
Tagged: , , , , , , , , .
7 comments

Way back in 47 (version that is), Chrome started to mandate the use of HTTPS in conjunction with getUserMedia. To use HTTPS you need a SSL/TLS certificate.  Xander Dumaine covered this a bit for us before, but I still see a lot of people out there struggle with it. As it so happens, the certificate for my own personal website is about to expire and I’m not too excited about paying $70/year to renew it. Fortunately, there is a new way to get certificates for free through Let’s Encrypt. Let’s Encrypt is a non-profit certificate authority that formed with the backing of many major industry players like Mozilla, Akamai, Cisco, and many others to simplify and automate the process of setting up encryption for your website. Oh, and its completely free. ...

Continue Reading