Let’s get better at fuzzing in 2019 – here’s how

Fuzzing is a Quality Assurance and security testing technique that provides unexpected, often random data to a program input to try to break it. Natalie Silvanovich from Google’s Project Zero team has had quite some fun fuzzing various different RTP implementations recently.

She found vulnerabilities in:

• WebRTC — mostly issues in the RTP payload
• Facetime – a few out-of-bounds, stack corruption, and heap corruption issues
• Whatsapp and what didn’t work

In a nutshell, she found a bunch of vulnerabilities just by throwing unexpected input at parsers. The range of applications which were vulnerable to this shows that the WebRTC/VoIP community does not yet have a process for doing this work ourselves. Meanwhile, the WebRTC folks at Google will have to improve their processes as well.

Let’s make it a new years resolution to get better at this stuff in 2019, ok?

Natalie’s Project Zero WebRTC fuzzing was done using the video_replay tool in a mostly end-to-end way. This is necessary but we will start a bit lower-level, using libfuzzer and walk you through an example of actual code taken from the Janus server, the janus_rtcp_get_remb  function. This function is quite self-contained and turned out to contain a number of nasty issues which makes it a great example.

Fuzzing REMB RTCP messages

Receiver Estimated Max Bitrate (REMB) is a certain kind of RTCP packet that is commonly used by WebRTC for coordinating packing send rates. The packet format is as follows, taken from the REMB draft:

RTCP is an interesting target for fuzzing because it does contain so much indirect control information.  I was actually quite surprised the WebRTC code did not contain any issues with RTCP parsing. The RTCP parser is being fuzzed though.

Fuzzing Walkthrough

Setup

First, in order to fuzz a target in isolation we need to copy the function and some of its header dependencies into a separate file.

Then you need to setup the fuzzing tools and have clang installed on your machine. You can grab a copy of the fuzzit repository here and checkout out the individual versions later.

Next, we need to write a fuzz target as described in the libfuzzer tutorial. It is a simple function that looks like this:

This function will be called with an array of bytes of a specified length many times. It is going to call our actual target, janus_rtcp_get_remb .

Check out the initial version and compile this with address sanitizer and fuzzer enabled:

Test the fuzzer

You should have a binary called jrtcp  now. Run it with a maximum length of 1500 which is the maximum practical length of a UDP packet.  For the sake of reproducibility we are providing a random seed 123456 — if this is not chosen the whole process is a bit more randomized and you might get different crashes.

If you need help on the usage, add -help=1 .

Running the fuzzer will output something like this:

Examining issues

libfuzzer will provide you with an example of the input data as a file:

If you do a hexdump of that file you will see the actual packet:

So a single-byte input crashes the function. We can run the fuzz target with the crash sample as input to reproduce:

Adjusting for the real RTCP header sizes

The problem is the cast to an RTCP header which has a size of 4 bytes. And the input is only a single byte long. Arguably that is a bit of an artificial bug since in production the input will have to be at least two bytes long in order to run through the DTLS/STUN/RTP demultiplexing process. The right place to enforce this kind of “external” behavior is to return early in the fuzz target.

Anyway… we can fix that by rejecting any input that is less than the sizeof janus_rtcp_header. Check out the next version or fix it yourself.
Run the fuzzer again with the same input to verify the crash is fixed.

Compile again, run it again with no arguments. It still crashes…

More fixes

As you can see, the fuzzer is smart enough to have figured out that we reject all inputs that don’t pass the “RTCP version must be 2” check so the first byte passes that.

This is a compound RTCP packet. Kind of… its length is given as 40 32bit-words or 320 bytes. However, the input is only 165 bytes long, leading to an out-of-bounds error when trying to cast the next janus_rtcp_header . Fixing this is pretty easy, or so it seems:

Fixing this and repeating the above jrtcp  call, the fuzzer runs through and everything and seems fine.

Now, compile and run the fuzzer again without any argument. It still crashes… this time during the second iteration of the RTCP loop.
It turns out we need to be a bit more careful and actually take into account that our starting position is not zero.
Let’s keep track of this using an offset… see the github repository for the next version.

Compile, run again with the crash file to verify this is fixed.
Run again without providing a crash. It still crashes. Boo…
At least its a short crash this time:

Almost there..

We are getting closer, this is crashing inside our REMB parsing part at least. The packet is too short for the cast to janus_rtcp_fb  and janus_rtcp_fb_remb . We need at least 24 bytes starting at the current offset to parse an REMB packet. Changing the check around this to

fixes that.

Compile the new version, run it. It doesn’t crash anymore. Woho!

Let it run for 20 million iterations. That looks reasonably robust now. Janus was lucky it did not attempt interpreting the “Num SSRCs” field in the REMB packet. This could easily have created another out of bounds read, trying to read 20 SSRCs when the outer RTCP length was way shorter than that.

Looking at the fuzzer corpus

So now we know how to do some basic fuzzing. One of the cool features of libfuzzer is that it is guided by code coverage. So it will inspect the code that is run and use that information to generate new input in order to hit every line and branch. Using these techniques, it has reconstructed image formats even.

With the latest version, create an empty directory named “corpus” and run

for another 20 million iterations. The directory will now contain a number of samples that libfuzzer considers to be “covering” the code we provided it with. You can use these examples for writing unit tests.
Using the corpus again for future fuzzer runs will speed up the process. You can also use a corpus such as the one from the WebRTC project to seed the fuzzing process.

Inspecting the corpus actually showed an additional problem. The RTCP version check should be done for any compound packet, not just the first one.
Again, easy to fix.

Use a fuzzer!

Using a fuzzer as part of your development will make your code better. But don’t forget that libfuzzer should not be the only layer of fuzzing you use. WebRTC has been using fuzzers to test at a function level for a while but it did not fuzz the complete call. This omission resulted in seven vulnerabilities. End-to-end fuzzing of a live system is still required!

Finding these tiny bugs in the Janus RTCP parsing code did not take long, less than an hour including writing a nice email to the folks at MeetEcho. The code isn’t bad, it has been running in production for quite a while without issues, yet these bugs went unnoticed for years. Not for much longer, stay tuned for their response 🙂

Until then… go forth and fuzz things.

{“author”: “Philipp Hancke“}