Debugging WebRTC media issues, especially video, often requires access to the unencrypted RTP payloads. We talked about this back in 2017 already and had a great blog post on using the libWebRTC “video_replay” tool. While that post has aged remarkably well, video_replay has improved significantly, in particular since it is now possible to create the pcap directly from Chrome. video_replay has become my go-to tool for debugging video-related problems such as corruption or freezes and is frequently used in bug reports.
Getting access to unencrypted RTP from a stock Chrome build
As the 2017 blog post mentions, Chrome has a command line flag to disable the WebRTC/SRTP encryption that works in Chrome Beta and Canary. However, this changes the SDP, which makes it less useful for debugging issues arising in production.
Get a log using the RtpDump Flag
Chrome has been supporting dumping the RTP packets before and after encryption into the log files since December 2020 Chrome (four years already, time flies!). This can be enabled by starting Chrome with the following flags:
1 2 |
chrome –enable-logging --v=1 --force-fieldtrials=WebRTC-Debugging-RtpDump/Enabled/ |
Adding the --use-fake-device-for-media-stream
flag is also advisable when attaching the pcap file to a public bug report.
Note that starting Chrome with those flags requires a new session. It will not work when the command just starts a new window in an existing session.
This dumps packets at the lowest level possible, either before encryption for outgoing packets or after decryption as part of libWebRTC’s srtp_session.
The result is a relatively large log file since each packet is dumped as a hexadecimal string in text2pcap format:
1 2 3 4 5 6 7 8 9 10 11 |
O 16:26:15.717 000000 b0 61 3a 3f 00 00 00 00 97 e3 51 20 be de 00 03 90 30 41 00 01 22 9e de c6 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ff # RTP_DUMP |
This shows the direction of the packet (outgoing or incoming), the timestamp, a line number (which does not really matter), and the packet content followed by the hexadecimal packet representation. In this example, an RTX probing packet with all zeroes. (See Kaustav Kosh’s post on probing for details on the bandwidth probing topic).
The location of these saved logs varies depending on the platform.
Extract the RTP_DUMP for Wireshark
The next step is to extract all lines that end with RTP_DUMP
(or SCTP_DUMP
for data channel traffic). I find this easiest to do with the classic unix grep
command:
1 |
grep RTP_DUMP chrome_debug.log |
And then converting the text file (which can still be easily edited) to a pcap or pcapng in recent versions:
1 |
text2pcap -D -u 1000,2000 -t %H:%M:%S.%f in.txt out.pcap |
Alternatively, this can be combined with the grep
1 |
grep RTP_DUMP path/to/chrome_debug.log | text2pcap -D -u 1000,2000 -t %H:%M:%S.%f - out.pcap |
I made an example pcap generated this way using video_replay commands so you can quickly try it yourself – webrtchacks.pcap.
The resulting PCAP file can be opened with Wireshark which remains one of my favorite tools.
The “Low-level WebRTC Protocols” course I have done with Tsahi Levent-Levi has a lot of sessions on using it if you want to learn how to understand the actual packet contents.
Using Wireshark dissectors
One of the biggest advantages of Wireshark is that it can parse many network protocols. It includes excellent support for most packets you encounter in WebRTC.
You can use this by right-clicking on the packet and selecting “Decode as” and then “RTP”.
Having a static dissector entry for “decode port 2000 as RTP” (or another port which you can change in the text2pcap command above) will save you a lot of time:
Remember to click “save” and sometimes clean up the mapping list.
You can also enable the dissectors for VP8 (discussed previously here), H264, Opus, and RED in the same window by creating a mapping between the payload type and the dissector.
Wireshark can filter using these dissectors, e.g. by using rtp.timestamp eq 1688970056
or by the rtp.ssrc
. This helps one quickly find the packet in question and can often show where something goes wrong:
Using video_replay
video_replay comes as part of the built-in libWebRTC tools. As such you will need to check out and build it yourself. The basic instructions can be found in the WebRTC git repository.
In a nutshell, install the prerequisite software and then fetch the webrtc
repository. We assume you are going to use out/Default
as the output path:
1 2 3 4 5 6 |
mkdir webrtc-checkout cd webrtc-checkout/ fetch --nohooks webrtc gclient sync cd src gn gen out/Default |
Note that when you need H.264 support you need to configure your build with:
1 2 |
rtc_use_h264 = true ffmpeg_branding = "Chrome" |
I typically do this by editing out/Default/args.gn
and then regenerate the build files using the gn gen out/Default
command. Next use ninja
to build video_replay:
1 |
ninja -C out/Default/ video_replay |
This will create a binary out/Default/video_replay
.
video_replay flags
Running out/Default/video_replay –help
shows the currently supported command line arguments:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 |
--codec (Video codec); default: "VP8"; --config_file (config file); default: ""; --decoder_bitstream_filename (Decoder bitstream output file); default: ""; --decoder_ivf_filename (Decoder ivf output file); default: ""; --disable_decoding (Disable video decoding.); default: false; --disable_preview (Disable decoded video preview.); default: false; --ext_map (RTP extension to ID map in the format of EXT1:ID,EXT2:ID,EXT3:ID Known extensions are: TOFF - kTimestampOffsetUri ABSSEND - kAbsSendTimeUri ABSCAPT - kAbsoluteCaptureTimeUri ROT - kVideoRotationUri CONT - kVideoContentTypeUri DD - kDependencyDescriptorUri LALOC - kVideoLayersAllocationUri TWCC - kTransportSequenceNumberUri TWCC2 - kTransportSequenceNumberV2Uri DELAY - kPlayoutDelayUri COLOR - kColorSpaceUri ); default: ; --extend_run_time_duration (Extends the run time of the receiving client after the last RTP packet has been delivered. Typically useful to let the last few frames be decoded and rendered. Duration given in seconds.); default: 0; --flexfec_payload_type (FLEXFEC payload type); default: 120; --force_fieldtrials (Field trials control experimental feature code which can be forced. E.g. running with --force_fieldtrials=WebRTC-FooFeature/Enabled/ will assign the group Enable to field trial WebRTC-FooFeature. Multiple trials are separated by "/"); default: ""; --input_file (input file); default: ""; --media_payload_type (Media payload type); default: 123; --media_payload_type_rtx (Media over RTX payload type); default: 98; --out_base (Basename (excluding .jpg) for raw output); default: ""; --red_payload_type (RED payload type); default: 118; --red_payload_type_rtx (RED over RTX payload type); default: 99;g --render_height (Height of render window); default: 480; --render_width (Width of render window); default: 640; --simulated_time (Run in simulated time); default: false; --ssrc (Incoming SSRC); default: 12648429; --ssrc_flexfec (Incoming FLEXFEC SSRC); default: 195935983; --ssrc_rtx (Incoming RTX SSRC); default: 195939069; --start_timestamp (RTP start timestamp, packets with smaller timestamp will be ignored (no wraparound)); default: 0; --stop_timestamp (RTP stop timestamp, packets with larger timestamp will be ignored (no wraparound)); default: 4294967295; --ulpfec_payload_type (ULPFEC payload type); default: 119; |
Here is a brief explanation for each of them:
Flag | Description | Default value |
input_file | the input pcap | must be set |
codec | the codec to interpret the primary payload type as | VP8 |
config_file | it is possible to use a config file instead of flags but the format is undocumented and arcane | not used |
decoder_bitstream_filename, decoder_ivf_filename, out_base | described below | not used |
disable_decoding | disable decoding, just reassembling the frames from the pcap | false |
disable_preview | disables the preview window | false |
ext_map | used to configure RTP header extensions | not used |
extend_run_time_duration | necessary to play until the actual end of the stream sometimes | 0 |
force_fieldtrials | used to configure field trials | not used |
media_payload_type, media_payload_type_rtx | the RTP payload type for the media and retransmissions | need to be configured |
red_payload_type, red_payload_type_rtx, ulpfec_payload_type, flexfec_payload_type | the RTP payload type for video redundancy, retransmissions thereof ulpfec and FlexFEC | need to be configured |
render_width / render_height | the size of the preview window | 640 and 480 |
simulated_time | whether to run in simulated time, useful when writing jpegs or the bitstream | false |
ssrc, ssrc_rtx, ssrc_flexfec | the RTP ssrcs for the media, retransmissions and FlexFEC | need to be configured |
start_timestamp / stop_timestamp | RTP timestamps to start with and stop at. The start must be a key frame. Can be used to only look at a section of the PCAP | not used |
Which flags you need to configure depends on what you want to achieve.
At minimum you need:
- codec (VP8, H264, AV1)
- input_file (the pcap you want to analyze)
- media_payload_type (the RTP payload type of the media packets)
- media_payload_type_rtx (the RTP payload type of the retransmission packets)
- ssrc and ssrc_rtx (the RTP ssrc of the media and retransmission packets respectively)
In our example, we would run:
1 2 |
video_replay --codec=H264 --input_file webrtchacks.pcap --media_payload_type=106 --media_payload_type_rtx=107 --ssrc=199935102 --ssrc_rtx=3053316777 |
That will show a window with the decoded media stream. This is the first step of any investigation, check if the issue gets reproduced with video_replay.
Note that for H264, video_replay does not support hardware decoders so will use ffmpeg for decoding instead. This might yield different results than Chrome. Exporting the bitstream as described below is the recommended alternative in these cases.
Advanced usages
frames to jpeg
What comes next depends on your goal. For corruption issues typically, you can output each frame encoded as a JPEG file using --out_base=/tmp/frame-
:
1 2 3 |
out/Default/video_replay --codec=H264 --input_file webrtchacks.pcap --media_payload_type=106 --media_payload_type_rtx=107 --ssrc=199935102 --ssrc_rtx=3053316777 --out_base=/tmp/file- |
That will create a lot of JPEG files named “frame-” in the /tmp directory. Each frame has the RTP timestamp as part of the filename. This allows narrowing down the range of frames using the –start_timestamp
and –stop_timestamp
arguments (but note that you must always start at a keyframe)
which in turns allows you to minimize the pcap file attached to a bug report.
Export to IVF
Exporting to the IVF format using --decoder_ivf_filename
works like this:
1 2 3 |
out/Default/video_replay --codec=H264 --input_file webrtchacks.pcap --media_payload_type=106 --media_payload_type_rtx=107 --ssrc=199935102 --ssrc_rtx=3053316777 --decoder_ivf_filename=webrtchacks.ivf |
That format allows the reassembled bitstream to be played with video players such as mplayer. Or feed the stream to ffmpeg for conversion into an MP4 (as described below) for sharing with people who do not want to build video_replay themselves.
Using the simulated time argument is quite useful for these conversions which do not need to happen in real-time.
Raw bitstream
Exporting the reassembled bitstream “raw” using --decoder_bitstream_filename
is particularly useful for H264 where you can take the reassembled bitstream and use it in stream analyzers:
1 2 3 |
out/Default/video_replay --codec=H264 --input_file webrtchacks.pcap --media_payload_type=106 --media_payload_type_rtx=107 --ssrc=199935102 --ssrc_rtx=3053316777 --decoder_bitstream_filename=webrtchacks.h264 |
I often use this web-based analyzer to take a quick look at the NAL units in the browser.
And it can be quickly converted to an MP4 using ffmpeg:
1 |
ffmpeg -i webrtchacks.h264 -vcodec copy webrtchacks.mp4 |
which finally shows you the content of our example pcap:
Playing back audio
The PCAP file with the unencrypted audio can also be used for audio playback. This can be done with either WebRTCs neteq_rtpplay binary – similar to video_replay. Or use Wireshark’s built-in RTP player, which is sufficient for most cases. Wireshark’s player supports all the important pieces like Opus decoding and audio redundancy. Ensure your dissectors are configured to dissect and the RTP payload types used in the SDP along these lines:
Resolve WebRTC issues faster with video_replay
video_replay gets frequent mentions in the WebRTC issue tracker (albeit half of them are probably from me). The reason? It saves the developers looking at the issue from “just log in to my service and spend hours reproducing my bug”.
Saying “here is how you reproduce using video_replay in less than a minute” provides a very low bar for having someone examine your issue.
See here for a great report with a quick fix for a regression that failed to parse rather arcane H.264 NAL units. video_replay made identifying and testing the fix a matter of “just a few minutes”.
{
“author”: “Philipp Hancke“,
“editor”: “Chad Hart“,
“special-thanks”: “Brian Baldino for putting me in touch with video_replay user Kacper Waśniowski, who reviewed the post for technical accuracy and provided feedback ”
}
video_replay has been really useful for us to debug a frame corruption issue we have been seeing with our SFU, we recently got a patch merged for it (https://bugs.chromium.org/p/webrtc/issues/detail?id=390329776)
Once we had a reproducible packet dump, took <1 hr to troubleshoot the issue and make a patch in the webrtc source 🙂
Saw it, it was a nice fix!