TensorFlow is one of the most popular Machine Learning frameworks out there – probably THE most popular one. One of the great things about TensorFlow is that many libraries are actively maintained and updated. One of my favorites is the TensorFlow Object Detection API.   The Tensorflow Object Detection API classifies and provides the location of multiple objects in an image. It comes pre-trained on nearly 1000 object classes with a wide variety of pre-trained models that let you trade off speed vs. accuracy.

The guides are great, but all of them rely on using images you need to add to a folder yourself.  I really wanted to hook this into a live WebRTC stream to do real-time computer vision with the web. I could not find any examples or guides on how to do this, so I am writing this. For RTC people, this will be a quick guide on how to use TensorFlow to process WebRTC streams. For TensorFlow people, this will be a quick intro on how to add WebRTC to your project. WebRTC people will need to get used to Python. TensorFlow people will need to get used to web interaction and some JavaScript.

This is not a reasonable getting started guide on either WebRTC or TensorFlow – for that you should see Getting Started with TensorFlow, Getting Started with WebRTC, or any of the innumerable guides on these topics out there.

Detecting Cats with Tensor Flow and WebRTC

Just show me how to do it

If you are just coming back for a quick reference or are too lazy to read text, here is a quick way to get started. You need to have Docker installed. Load a command prompt and then type:

Then point your browser to http://localhost:5000/local accept camera permissions and you should see something like this:

Architecture

Let’s start with a basic architecture that sends locally a local web camera stream from WebRTC’s getUserMedia to a Python server using the Flask web server and the TensorFlow Object Detection API. My setup looks like the graphic below.

Flask will serve the html and JavaScript files for the browser to render. getUserMedia.js will grab the local video stream. Then objDetect.js will use the HTTP POST method send images to the TensorFlow Object Detection API which will returns the objects it sees (what it terms classes) and their locations in the image. We will wrap up this detail in a JSON object and send it back to objDetect.js so we can show boxes and labels of what we see.

Setup

Setup and Prerequisites

Before we start we’ll need to setup Tensorflow and the Object Detection API.

Easy setup with Docker

I have done this a few times across OSX, Windows 10, and Raspbian (which is not easy).  There are a lot of version dependencies and getting it all right can be frustrating, especially when you just want to see something work first. I recommend using Docker to avoid the headaches. you will need to learn docker, but that is something you should probably know anyway and that time is way more productive than trying to build the right version of Protobuf. The TensorFlow project maintains some official Docker images, like tensorflow/tensorflow.

If you go the Docker route, then we can use the image I created for this post.  From your command line do the following:

Note the  $(pwd)  in the docker run  only works for Linux and the Windows Powershell. Use %cd%  on Windows 10 command line.

At this point you should be in the docker container. Now run:

This will use the latest TensorFlow docker image and attach port 5000 on the docker host machine to port 5000 , name the container tf-webrtchacks , map a local directory to a new /code  directory in the container, set that as the default directory where we will do our work,  and run a bash for command line interaction before we start.

If you are new to TensorFlow then you should probably start by following the instructions in tensorflow/tensorflow to run the intro Jupyter notebook and then come back to the command above.

The Hard Way

If you start from scratch you’ll need to install TensorFlow which has a lot of its own dependencies like Python. The TensorFlow project has guides for various platforms here: https://www.tensorflow.org/install/. The Object Detection API also has its own install instructions with a few additional dependencies. Once you get that far then do this:

That should install all the Python dependencies, copy over the appropriate Tensorflow Object API files, and install the Protobufs. If this does not work then I recommend inspecting the setup.py  and running the commands there manually to address any issues.

Code Walkthrough

Part 1 – Make sure Tensorflow works

To make sure the TensorFlow Object Detection API works, let’s start with a tweaked version of the official the Object Detection Demo Jupyter Notebook.  I saved this file as object_detection_tutorial.py.

If you cut and paste each section of the notebook, you should have this:

I am not going to review what the actual TensorFlow code does since that is covered in the Jupyter demo and other tutorials. Instead I’ll just focus on the modifications.

Things we don’t need

I commented out a few sections:

1. Changed some location references
2. Removed any references to the Python matplot for visualizing the output in GUI-based environments. This isn’t setup in my docker environment – keep those in there depending on how you are running things.

Object Detection API Outputs

As shown in line 111, the Object Detection API outputs 4 objects:

1. classes – an array of object names
2. scores – an array of confidence scores
3. boxes – locations of each object detected
4. num – the total number of objects detected

classes, scores, and boxes are all equal sized arrays that parallel to each other, so classes[n]  corresponds to scores[n]  and boxes[n] .

Since I took out the visualizations, we need some way of seeing the results, so let’s add this to the end of the file:

The first np.squeeze  part just reduces the multi-dimensional array output into a single dimension – just like in the original visualization code. I believe this is a byproduct of TensorFlow usually outputting a multidimensional array.

Then we set a threshold value for the scores it outputs. It appears TensorFlow defaults to 100 objects it returns. Many of these objects are nested within or overlap with higher confidence objects. I have not seen any best practices for picking a threshold value, but 50% seems to work well with the sample images.

Lastly we cycle through the arrays, just printing those that are over the threshold value.

If you run:

You should get the following output:

Part 2 – Making a Object API Web Service

In this section we’ll adapt the tutorial code to run as web service.  My Python experience is pretty limited (mostly Raspberry Pi projects), so please comment or submit a pull request on anything dumb I did so I can fix it.

2.1 Turn the Demo code into a service

Now that we got the TensorFlow Object API working, let’s wrap it into a function we can call. I copied the demo code into a new python file called object_detection_api.py. You’ll see I removed a bunch of lines that were unused or commented and the print details to console section (for now).

Since we will be outputting this to the web, it would be nice to wrap our output into a JSON object. To do that make sure to add an  import json to your imports and then add the following:

Next let’s make a get_objects  function reusing our code from before:

We have an image input and a threshold  value with a default of 0.5 . The rest is just restructured from the demo code.

Now lets add some more code to this function to take our values and output then into a JSON object:

This time we are using our Object class to create some initial metadata and add that to our output  list. Then we use our loop to add Object data to this list. Finally we convert this to JSON and return it.

After that, let’s make a test file (note to self- make tests first) to check it called object_detection_test.py:

That’s it. Now run

In addition to the console output from before you should see a JSON string:

2.2 Add a Web Server

We have our function – now we’ll make a web service out of it.

Start with a test route

We have a nice API we can easily add to a web service. I found Flask to be the easiest way to do this. Create a server.py and let’s do a quick test:

Now run the server:

Make sure it works

And then call the web service. In my case I just ran the following from my host machine (since my docker instance is now running the server in the foreground):

json.tool  will help format your output. You should see this:

Ok, next let’s make a real route by accepting a POST containing an image file and other parameters. Add a new /image  route under the /test  route function:

This takes an image from a form encoded POST with an optional threshold value and passes it to our object_detection_api.

Let’s test it:

You should see the same result as the /test  route above. Go ahead and put a path to any other local image you like.

Make it work for more than localhost

If you are going to be running your browser on localhost you probably don’t need to do anything, but that is not too realistic for a real service or even a lot of tests. Running a web service across networks and with other sources means you’ll need to deal with CORS. Fortunately that’s easy to fix by adding the following before your routes:

Make it secure origin friendly

Its best practice to use HTTPS with WebRTC since browsers like Chrome and Safari will only work out of the box secure origins (though Chrome is fine with localhost and you can enable Safari to allow capture on insecure sites – jump to the Debug tools section here ). To do this you’ll need to get some SSL certificates or generate some self-hosted ones. I put mine in the ssl/  directory and then changed the last app.run line to:

If you’re using a self-signed cert, you’ll probably need to add the --insecure  option when testing with CURL:

Since it is not strictly required and a bit more work to make your certificates, I kept the SSL version commented out at the bottom of server.py .

For a production application you would likely use a proxy like nginx to send HTTPS to the outside while keeping HTTP internally (in addition to making a lot of other improvements).

Add some routes to serve our web pages

Before we move on to the browser side of things, let’s stub out some routes that we’ll need later. Put this after the index()  route:

That’s about it for Python. Now we’ll dig into JavaScript with a bit of HTML.

Part 3 – Browser Side

Before we start, make a static directory at the root of your project. We’ll serve our HTML and JavaScript from here.

Now let’s start by using WebRTC’s getUserMedia to grab a local camera feed. From there we’ll send snapshots of that to the Object Detection Web API we just made, get the results, and then display them in real time over the video using the canvas.

HTML

Let’s make our local.html file first:

This is what this webpage does:

• uses the WebRTC adapter.js polyfill
• Sets some styles to
  • put the elements on top of each other
  • put the video on the bottom so we can draw on top of it with the canvas
• Create a video element for our getUserMedia stream
• Link to a JavaScript file that calls getUserMedia
• Link to a JavaScript file that will interact with our Object Detection API and draw boxes over our video

Get the camera stream

Now create a local.js file in the static directory and add this to it:

You’ll see here we first set some constraints. In my case I asked for an 1280×720 video but require something between 640×480 and 1920×1080. Then we make our getUserMedia with those constraints and assign the resulting stream to the video object we created in our HTML.

Client side version of the Object Detection API

The TensorFlow Object Detection API tutorial includes code that takes an existing image, sends it to the actual API for “inference” (object detection) and then displays boxes and class names for what it sees. To mimic that functionality in the browser we need to:

1. Grab images – we’ll create a canvas to do this
2. Send those images to the API – we will pass the file as part of a form-body in a XMLHttpRequest for this
3. Draw the results over our live stream using another canvas

To do all this create a objDetect.js file in the static folder.

Initialization and setup

We need to start by defining some parameters:

You’ll notice I included some of these as data-  elements in my HTML code. I ended up using this code for a few different project and wanted to reuse the same codebase and this made it easy. I will explain these as they are used.

Setup our video and canvas elements

We need a variable for our video element, some starting events, and we need to create the 2 canvas’ mentioned above.

The drawCanvas  is for displaying our boxes and labels. The imageCanvas  is for uploading to our Object Detection API. We add the drawCanvas to the visible HTML so we can see it when we draw our object boxes. Next we’ll go to the bottom of ObjDetect.js and work our way up function by function.

Kicking off the program

Trigger off video events

Let’s get our program started. First, let’s trigger off some video events:

We start with looking for both the  onplay  event and loadedmetadata  events for our video – there is no image processing to be done without video. We need the metadata to set our draw canvas size to match the video size in the next section.

Start our main object detection subroutine

While the drawCanvas  has to be the same size as the video element, the imageCanvas  is never displayed and only sent to our API. This size can be reduced with the uploadWidth  parameter at the beginning of our file to help reduce the amount of bandwidth needed and lower the processing requirements on the server.  Just note, reducing the image might impact the recognition accuracy, especially if you go too small.

While we’re here we will also set some styles for our drawCanvas. I chose cyan  but pick any color you want. Just make sure it has a lot of contrast with your video feed for good visibility.

toBlob conversion

After we have set our canvas sizes we need to figure out how to send the image. I was doing this a more complex way and then I saw Fippo’s grab() function at the last Kranky Geek WebRTC event, so I switched to the simple  toBlob method. Once the image is converted to a blob we’ll send it to the next function we will create – postFile .

One note – Edge does not seem to support the HTMLCanvasElement.toBlob  method. It looks like you can use the polyfill recommended here or the msToBlob  instead, but I have not had a chance to try either.

Sending the image to the Object Detection API

Our  postFile takes the image blob as an argument. To send this data we’ll POST it as form data using XHR. Remember, our Object Detection API also takes an optional threshold value, so we can include that here too. To make this easy to adjust without touching this library, this is one of the parameters you can include in a data-  tag we set up in the beginning.

Once we have our form set, we  use XHR to send it and wait for a response. Once we get the returned objects we can draw them (see the next function). And that’s it. Since we want to do this continually we’ll just keep grabbing a new image and sending it again right after we get a response from the previous API all.

Drawing Boxes and Class Labels

Next we need a function to draw the Object API outputs so we can actually check what is being detected:

Since we want to have a clean drawing board for our rectangles every time, we start by clearing the canvas with clearRect . Then we just filter our items with a class_name and perform our drawing operation on each.

The coordinates passed in the objects  object are percentage units of the image size. To use them with the canvas we need to convert them to pixel dimensions. We also check if our mirror parameter is enabled. If it is is then we’ll flip the x-axis to match the flipped mirror view of the video stream. Finally we write the object  class_name  and draw our rectangles.

Try it!

Now go to your favorite WebRTC browser and put your URL in. If you’re running on the same machine that will be http://localhost:5000/local (or https://localhost:5000/local  if you setup your certificates).

Optimizations

The setup above will run as many frames as possible through the server. Unless setup Tensorflow with GPU optimizations, this will chew up a lot of CPU (like a whole core for me), even if nothing has changed. It would be more efficient to limit how often the API is called and to only invoke the API when there is new activity in the video stream. To do this I made some modifications to the objDetect.js in a new file objDetectOnMotion.js.

This is mostly the same, except I added 2 new functions. First, instead of grabbing the image every time, we’ll use a new function sendImageFromCanvas()  that only send the image if it has changed within a given framerate – a new updateInterval  parameter which the maximum the API can be called. We’ll use a new canvas and context for this

That code is pretty simple:

imageChangeThreshold  is a percentage representing the percentage of pixels that changed. We’ll take this and pass it to an imageChange  function that returns a true or false if the threshold has been exceeded. Here is that function:

The above is a much improved version of how I did motion detection long ago in the Motion Detecting Baby Monitor hack. It starts by measuring the RGB color values of each pixel. If those values exceed an absolute difference of 10 across the aggregate color values for that pixel then the pixel is deemed to have changed. The 10 is a little arbitrary, but seemed to be a good value in my tests. If the number of change pixels crosses the threshold  than the function returns true.

After researching this a bit more I saw other algorithms typically convert to greyscale since color is not a good indicator of motion. Applying a gaussian blur can also smooth out encoding variances. Fippo had a great suggestion to look into Structural Similarity algorithms as used by the test.webrtc.org to detect video activity (see here). More to come here.

Works on any video element

This code will actually work on any <video>  element, including a remote peer’s video in a WebRTC peerConnection. I did not want to make this post/code any longer and complex, but I did include a video.html file in the static folder as an illustration:

You should see this (running on the video I took from my How to Train a Dog with JavaScript project):

Try it with your own video. Just be aware of CORS issues if you are using a video hosted on another server.

This was a long one

This took quite some time to get going, but now I hope to get on to the fun part of trying different models and training my own classifiers. The published Object Detection API is designed for static images. A model tuned for video and object tracking would be great to try.

In addition, there is a long list of optimizations to make here. I was not running this with a GPU, which would make a huge difference in performance. It takes about 1 core to support 1 client at a handful of frames, and I was using the fastest/least accurate model. There is a lot of room to go here to improve performance. It would also be interesting to see how well this performs in a GPU cloud network.

There is no shortage of ideas for additional posts in this area – any volunteers?

{“author”: “chad hart“}