I finally got around to finishing my first demo application. First, a word of caution – I don’t really know what I am doing. I am not a professional programmer and I don’t plan to be one. My JavaScript experience is limited to a node.js-based database API I wrote a few months ago for an email analytics project I was playing with and some w3schools.com tutorials. That being said, I have a long background in VoIP and this stuff is supposed to be easy right? Let’s find out.

Problem Statement

My daughter is almost 2 years old. Like most toddlers she likes to nap. Like most parents, nap time is a great time to get things done with minimal distractions. My wife and I typically just hook up a Skype feed on her and go about our business until we see or hear her wake up. The problem is she often doesn’t cry or make much sound at all when she gets up. If we’re not staring at the video feed then we might not notice her. Toddler wondering around with no one close by is not good.  🙁

Wouldn’t it be better if we could have a motion detector hooked up the video feed? This sounds like the type of problem you are supposed to be able to address with WebRTC and HTML5?

Here is my attempt.

Motion Detection

The motion detection part here is reliant on some video processing. My son plays some games like this one on pbskids.org that utilize motion detection from a web cam as a user interface, but with Flash. First I needed to see if anyone had done anything like this in JavaScript using getUserMedia from WebRTC instead. Fortunately the answer is yes, I came across this Magic Xylophone demo from soundstep that does exactly that. Even better, I found the js-motion-detection project on GitHub based on the original soundstep demo but abstracted a bit to make it easier to work with in other apps.

Check out the links above for an explanation of how this works – or go to Romuald Quantin’s the original explanation of these method’s on Abobe’s HTML5 Developer Network. Basically they taking the video stream form getUserMedia and drawing it to invisible html5 canvases. Within the canvas, a current frame is compared to a previous frame. If the difference exceeds a specified threshold a motion event is fired.

I wanted to keep this proof-of-concept very simple, so I ripped out a code and features I did not need for basic motion detection from this demo. Next step was to hook this into a remote feed instead of a local one.

WebRTC Attempt 1

As a studious observer of the WebRTC industry and editor of this blog, I knew building a WebRTC app from scratch would be more than a few minute exercise. To start out I wanted to focus on the front-end of things. I will save back-end implementation and setup for a future project/post.

Figuring out the front-end side of things still involves a decent amount of work. Basically at this point I had two options:

1. Do most everything myself – including:

• write all the createPeerConnection code myself
• Piggyback on some 3^rd-party signaling server,
• Interface all this with a UI for my app

2. I find a SDK that includes a hosted signaling service

Or is there a third way? I was fine with Skype, I just wanted some motion detection on top of it. Could I use something like the Chrome screen capture to do motion detection on an existing WebRTC app that has all the signaling and UI already taken care of? This sounds like a good hack, so I decided to investigate this option.

This ended up not being that easy. First it only works in Chrome and you need to enable a Chrome flag to do it:

Then you need it to run it on https which requires some more setup and a certificate.

There is a good illustration of this here: https://www.webrtc-experiment.com/Pluginfree-Screen-Sharing/

Basically I setup an iframe to point to apprtc.appspot.com, tawk.com, or your favorite web-based video stream of choice. Then I pointed this source at the motion detector. I wasn’t able to get this to work.

First, I ran into the cascading and blurry screen issues that are apparently open issues with the screen sharing implementation. To fix this I tried to hide some of the elements but the motion detector code did not seem to like that. Then I ran into permission issues too.

I couldn’t get this to work. If I fiddle around with this some more I might have some success. However, given the early state of the screenCapture, I decided to move on to a more traditional WebRTC implementation where I would actually initiate the calls myself rather than piggy-backing on someone else’s site.

WebRTC Attempt 2

As I stated earlier, I didn’t want to setup my own signaling server for this one so I needed a 3^rd party tool. To find one I went to the Tool Vendor Directory and selected “Front-end”, “Open-source”, and “Hosted”.

The WebRTC part

I settled on EasyRTC, largely because their back-end capabilities looked easier to implement for when I want to go down that route. (Note to other vendors – I am happy to try this on your platforms too.) I also needed to verify it would be easy to access the remote stream. Some libraries create this object dynamically which would make accessing it more difficult. In this case it was very simple – you simply specify the Element ID for a video object you created in the main HTML body:

The EasyRTC guys there were kind enough to get me an API key and build it into some demos for me. The code to get the WebRTC part setup was super simple – I just followed this “Video Conferencing – Trading Ease For Flexibility” section of their client API doc and added the API key they gave me to the my_init() function.

Make sure you can make calls

I use WebMatrix for my IDE (yeah, I am a n00b but I think it’s a slick product and has decent node.js support too). It lets you deploy locally using and picks a random port for you. When I first tried using this to connect to the EasyRTC server it did not work. Port 11771 is not exactly a common port. I could have manually assigned this port, tweaked my Windows Firewall, and then my router to get all the port settings right. WebMatrix lets you setup and deploy to the Azure cloud easy and free, so I just did that.

Motion detector

The next step was to get the motion detector setup and hook the remote media stream into it. The motion detector requires a few things on the html page:

• A <div> tag to identify the area where motion will be detected – the detector needs its sizing info
• A video tag for the stream to monitor – this is also needed for EasyRTC
• 3 canvases – one for the source, one for highlights (not used – see below), and one that contains the “blended” image where the real calculations are done
• A <div> tag to contain all hotSpots
• A <div> tag to identify an individual hotSpot.

Here is what the code looks like:

 

Note the main.css file is used to overlay these canvases in the same spot and keep them hidden. The CSS file also sets the “target” hotspot area to be the entire size of its containing div – the hotSpots div which is also set to take the entire area of its parent div. Really Good’s original js motion detector was setup to allow you to target motion within specific sub-areas of the video. For this demo I am looking at the entire video source, so this level of specificity was not needed. I could have tried to remove these functions from the motion.js file and underlying algorithm, but I figured it might come in handy for future versions. For example, if you have a feed that includes some object that does move, like an old clock or fan you might want to specify only part of the video to monitor.

Also note I really should have moved more of the sizing parameters into the CSS file – I’ll do that in a future version.

From here, there were only two variables I needed to set to the motion detector work:

Beeping

The Really Good’s code just displayed to the console when the ‘motion’ event was fired. I needed to add a beep to this, which was real simple. First I pulled a beep I liked from SoundJay. Then I used the Audio API to play it. Here is the full code for beep.js:

Testing and Tweaks

In my initial tests I noticed it was not very sensitive, especially when I used the 1.3 megapixel front facing camera on my old HTC Thunderbolt phone. To make it more sensitive, within the checkAreas function, I changed this value from 10 to 1:

I also did a minimal amount of formatting to the main page with the 480 x 800 pixel resulotion of my Thunderbolt in mind.

The sensitivity may be too high for some, but for my purposes I wanted to err on the side of over alerting. I tested it on my daughter this past weekend while she took her afternoon nap and it worked like a charm!

Here is a video demonstration:

Your Turn

Test it yourself here and let me know how it works. Just point 2 WebRTC-supporting browers to that url, click on the caller button when it pops up, and then do “Start Motion Detection” on either or both ends. When there is movement detected in the remote video your browser should beep.

All my code is available here on GitHub.

Thanks again to the Priologic EasyRTC team for letting everyone use their server for this hack.

{“author”, “chad“}