Comparing WebRTC video codecs with a proper dataset

How this was measured

The dataset

The Video Conferencing Dataset is Microsoft’s open corpus for exactly this kind of work: 160 clips of a person on camera, 1080p at 30 fps, 10 seconds each, spanning a deliberate range of camera qualities and spatial and temporal complexity. It is split into four scenarios of 40 clips, and that split is what makes it useful here, because conferencing codecs meet very different content depending on what the user has switched on.

th

Talking head. Plain webcam capture, speaker gesturing or reading. The closest thing to an unmodified call.

th-ob

Opaque background. The background is replaced, as Teams and Meet do by default in many deployments. Synthetic, flat and highly compressible.

th-bb

Background blur. Heavy blur applied behind the speaker, which removes most high-frequency detail from the majority of the frame.

th-m

Mobile. Recorded on phones, and portrait at 1080×1920. Excluded here, since the landscape bitrate ladder does not apply to it.

That leaves 120 landscape clips, 40 per scenario. Clips are served from a public Azure blob under the Community Data License Agreement (Permissive 2.0), in both raw YUV and MP4. The MP4s were used and decoded to raw I420 per clip. Because the decoded MP4 is the common reference for every codec, source compression does not bias one codec against another. It does, however, mean the experiment measures performance on this distribution rather than on pristine camera output.

The libWebRTC tooling

No standalone encoder CLI was involved. Everything ran through WebRTC’s own codec test harness, video_codec_perf_tests, defined in modules/video_coding/BUILD.gn. Its VideoCodecTest.DISABLED_EncodeDecode case is a flag-driven single-run entry point, disabled by default so it does not run in CI, which encodes a clip, decodes it back and reports per-frame statistics:

This matters because the harness instantiates the real CreateBuiltinVideoEncoderFactory and CreateBuiltinVideoDecoderFactory, so every run exercises the same encoder wrappers, the same rate-control glue and the same libyuv scaling path that a live PeerConnection would use. Downscaling from the 1080p source to each rung happens inside the harness rather than in FFmpeg, and PSNR is computed against that scaled source, which is what keeps the four resolutions self-consistent. The codec pairs are the software stack WebRTC ships: libvpx for VP8 and VP9, OpenH264 encoding with the FFmpeg decoder for H.264, and libaom encoding with dav1d decoding for AV1.

Each run prints psnr_y_db, psnr_u_db, psnr_v_db, qp, encoded_bitrate_kbps, bitrate_mismatch_pct, encode_time_us and decode_time_us, aggregated over the 300 frames, and those lines are parsed straight from stdout. Adding --write_csv also drops a per-frame CSV next to the run, one row per encoded frame carrying timestamp_rtp, per-frame PSNR, QP, frame size and the keyframe flag.

Adding VMAF

PSNR is computed inside the harness, but VMAF is not, so the second metric needs the pixels. --dump_encoder_input and --dump_decoder_output write the frames the encoder was handed and the frames that came back out of the decoder, as y4m, the uncompressed frame-sequence container FFmpeg and libvmaf both read. That is the same pair CalcPsnr consumes, so both metrics score byte-identical data and no separate encode is involved:

Two details are load-bearing. Encoders drop frames under rate control, so the decoder output can be shorter than the encoder input, and comparing the two sequentially would misalign everything after the first drop. The harness itself is immune because it pairs by RTP timestamp, but the y4m dumps carry no timestamps, so the decoded-frame index list is recovered from the per-frame CSV and the reference is subset to match. And libvmaf silently returns 0.0 when a .y4m and a raw .yuv are mixed in one call rather than reporting an error, which is why the upscaled comparison feeds raw on both sides.

The operating points

Bitrates come from WebRTC itself rather than from round numbers. The target_bitrate column of kSimulcastFormatsVP8 in video/config/simulcast.cc is what a sender would actually allocate to a layer at each resolution, so those are the points measured: 150, 500, 2500 and 4000 kbps for 180p, 360p, 720p and 1080p. All four codecs were held to the VP8 ladder so the comparison is like-for-like, even though AV1 and VP9 would in practice draw from kSimulcastFormatsVP9, which the same file defines just below it. Those are meaningfully lower: 142, 420, 1524 and 3367 kbps for the same four rungs, so WebRTC already assumes VP9 needs 16% less bitrate at 1080p and 39% less at 720p to do the same job. Holding VP9 and AV1 to the VP8 numbers therefore hands them more bitrate than a real sender would, and the quality gaps below should be read as the ceiling of what the newer codecs buy, not as what a deployed ladder would deliver. One point per resolution means this is a fixed-operating-point comparison, not a rate-distortion sweep, and so it yields no BD-rate: the Bjøntegaard delta, the single number that says how much bitrate one codec saves over another at matched quality, is computed from a curve through several rates per codec, and there is one rate per codec here.

Every run is L1T1 — one spatial layer, one temporal layer, so no simulcast and no SVC, just a plain single stream — with no additional temporal layers, a single keyframe at frame 0, frame dropping left on, and two cores, over all 300 frames of each clip. That comes to 1,920 encode-decode runs for the four-codec comparison, plus a further 1,920 for the single-pass timing measurement that the encode-cost chart draws on.

Why a subset. On the easiest clips, the ones with a replaced or blurred background, AV1 and OpenH264 sometimes reach their best-quality setting and simply cannot use all the bitrate they were given, delivering as little as 60% of it. Averaging those in would compare codecs at different rates. It affected 116 of the 1,920 runs, all of them AV1 or OpenH264; VP8 and VP9 always spent their budget. The full table reports both views, with a per-resolution count in its “saturated” column.

Reading the ranking

The ordering starts with VP8 < H.264 at every rung, while VP9 and AV1 take the top two spots and trade the lead. The intervals are uneven. Moving off VP8 to VP9 or AV1 is worth 2.2–3.6 dB, and to OpenH264 only 0.4–1.8 dB; choosing AV1 over VP9 is worth between −0.20 and +0.82 dB depending on resolution. At 180p VP9 is actually ahead of AV1 by 0.20 dB, and AV1’s lead only becomes clear at 360p and above, peaking at +0.82 dB at 360p before narrowing again to +0.17 dB at 1080p.

H.264 here is OpenH264, which is what WebRTC ships rather than what the format can do. It is the weakest of the three modern options on quality but by far the cheapest to run.

The ranking depends on what is behind the speaker

This is the part of the dataset that earns its keep. Splitting the same 120 clips by scenario turns a single ordering into three different ones, and the effect is not small: for OpenH264 the spread between the easiest and hardest scenario at 1080p is 5.58 dB, larger than the entire 3.62 dB gap between VP8 and AV1 in the pooled result. Which background mode your users have switched on matters more, for that encoder, than which codec you picked. A corpus of plain webcam footage would have shown none of this.

All three panels below are PSNR-Y in dB, one per scenario, on the same axis so they can be read against each other. The table beneath them is at 1080p. What VMAF makes of the same split is covered further down.

Three things fall out of this. OpenH264 is content-dependent to a degree the others are not: on a replaced background it gains 0.19 dB over VP8, which is nothing, but under background blur it gains 3.66 dB and lands within half a dB of VP9. The AV1-versus-VP9 answer flips: AV1 leads on plain talking head and on blur, but on opaque replaced backgrounds VP9 is ahead by 0.40 dB. And the synthetic backgrounds are not simply easier, they are differently hard: blur is easier for every codec, while a flat replaced background is the scenario that separates the codecs most, suggesting that the modern codecs exploit its large uniform regions more effectively than OpenH264.

PSNR and VMAF, side by side

Each run in the VMAF pass was scored with both VMAF, Netflix’s perceptual metric, and PSNR on exactly the same pixels: the frames the harness fed the encoder against the frames its decoder returned. Nothing was re-encoded between the two scores, so the panels below are two readings of the same 1,920 VMAF-pass runs rather than separate experiments. Both use all 120 clips, which is why the left panel sits a little above the rate-matched chart further up.

Why carry both. VMAF is intended to track perceived picture quality better than PSNR, but the v0.6.1 model used here was trained for entertainment viewing rather than video conferencing. PSNR is the one you can actually get here: it is cheap enough to compute per frame inside the encoder loop, it is what video_codec_perf_tests reports, and it is now in getStats() in the browser, so it is the only one of the two you can read off a live call. In this workflow, VMAF needs both sequences and a second pass. Running them together checks whether the cheap number tells the same story: where they agree, PSNR is a useful proxy, and where they disagree, the metric choice is part of the result.

The two axes are deliberately not shared: decibels and VMAF points are different quantities and putting them on one scale would invent a relationship that does not exist. What is comparable is the shape, and it mostly agrees. The broad separation survives: VP8 trails H.264, while VP9 and AV1 remain well ahead. The top two trade places depending on resolution and metric.

Two rankings actually flip, and the table shows them differently. At 180p the sign changes: PSNR puts OpenH264 ahead of VP8 and VMAF puts it behind, marked in amber. At 720p and 1080p the order changes without any sign changing, so read the two bold cells in each block instead: PSNR’s leader is VP9 and VMAF’s is AV1. Both flips involve differences small enough that the metric choice determines the ranking.

But only for the close calls. VP9 and AV1 beat VP8 on 120 of 120 clips under both metrics at every rung: perfect unanimity, nothing to argue about. OpenH264 against VP8 is the one contested comparison, and there the two metrics pick the same per-clip winner only 75–88% of the time. That is not VMAF being erratic. It is two metrics resolving a genuinely close race differently, which is the useful warning: a sub-decibel PSNR difference is not a result.

Where the OpenH264 disagreement comes from

Splitting that contested comparison by scenario resolves it. The pooled 180p flip is not a property of OpenH264, it is the average of two scenarios where VMAF punishes it and one where VMAF rewards it more than PSNR does.

^* The fourth column scores the same encodes the way a viewer meets them: decoded, then scaled back up to a 1920×1080 window with a bicubic filter and compared against the original 1080p source, which is Netflix’s convention for comparing encodes made at different resolutions. At the 1080p rung it is the same computation as the second column.

On blurred background the two metrics simply agree, and VMAF is the more generous of the two. OpenH264 beats VP8 on every rung under both, and VMAF makes the gap larger at three of the four: +1.79 against +1.23 dB at 180p, +2.48 against +2.05 at 360p, +3.78 against +3.09 at 720p, converging only at 1080p where PSNR says +3.80 and VMAF +3.74. Per-clip agreement runs 80% to 98%, the highest anywhere in this dataset. Scored the way a viewer receives it, the advantage is bigger again, +2.74 at 180p against a PSNR reading of +1.23. Whatever OpenH264 is doing with a heavily blurred background, it is not an artifact of PSNR’s preference for smooth error.

The reverse holds on the other two scenarios. On plain talking head and on an opaque replaced background, VMAF is consistently harsher on OpenH264 than PSNR is: at 180p it goes from −0.05 dB in PSNR to −1.12 VMAF points on talking head and from −0.46 dB to −1.39 VMAF points on opaque, and at 1080p on opaque PSNR calls it a tie at +0.05 dB while VMAF gives it +0.47 points on only 28 of 40 clips. That is where the pooled flip is manufactured. One more thing falls out: on blurred background at 720p, VMAF puts OpenH264 ahead of VP9, 95.07 against 94.66, while PSNR has VP9 ahead 48.23 to 47.88. The cheapest encoder in the set wins a rung outright on the perceptual metric.

The vmaf_v0.6.1neg model, which removes the enhancement-gain term that rewards sharpening, was recorded alongside. It widens every gap rather than closing it: pooled over all runs VP9 goes from +3.50 to +3.89 and AV1 from +3.91 to +4.40 against VP8. Whatever the modern codecs are winning on here, it is not a filtering artifact that VMAF over-credits.

QP does not travel between codecs

Quantizer is the number people reach for when they want a cheap quality signal, because it is in every encoder’s output and needs no reference frame. Across codecs it is meaningless, and this dataset shows why. Each codec reports QP on its own scale, and the scales are not merely differently sized, they are differently shaped: a quantizer index is an index into that codec’s own step-size table, not a quantity of distortion.

Which QP is this? The value recorded here is EncodedImage::qp_, which is the same field that accumulates into qpSum in getStats(), so these are the numbers an application actually sees. Each encoder fills it from a different place:

VP8

VP8E_GET_LAST_QUANTIZER, 0–127 (libvpx_vp8_encoder.cc:1328, where the local is candidly named qp_128). The 0–63 variant VP8E_GET_LAST_QUANTIZER_64 is also called, but only to feed the temporal-layer controller, not qp_.

VP9

VP8E_GET_LAST_QUANTIZER again, reusing the VP8 control id, but on VP9 that returns 0–255 (libvpx_vp9_encoder.cc:1816).

AV1

AOME_GET_LAST_QUANTIZER, 0–255 (libaom_av1_encoder.cc:1151).

H.264

Not an encoder control at all: the slice QP is parsed back out of the bitstream by H264BitstreamParser::GetLastSliceQp(), 0–51 (h264_encoder_impl.cc:632).

So the same struct field carries four different quantities, one of them recovered by re-parsing the encoder’s own output. This is not merely a reporting quirk, it is webrtc-stats issue 799: WebCodecs specifies 0–63 for AV1, while Chromium reports the libaom qindex, so getStats() returns AV1 QP values above 63 and hardware encoders may well pick a different convention again. The tree is not even internally consistent on this point. The newer encoder behind LibaomAv1EncoderFactory uses AOME_GET_LAST_QUANTIZER_64 and reports 0–63 (libaom_av1_encoder_v2.cc:788), and the speed controller in the old encoder divides by 4 with the comment “Use [0, 63] range instead of [0, 255]” (libaom_av1_encoder.cc:128). Two AV1 QP conventions are live in the same checkout.

Ranks are 1 = best, so a metric that carried across codecs would put the same codec first in both columns. Nothing does. The thresholds in the third column are WebRTC’s own: the QualityScaler decides when to drop resolution from QP, and it needs a completely different pair of numbers per codec to do it, which is the clearest in-tree statement that these values do not share a scale.

Normalising does not rescue it. At 1080p, VP8 sits at 20% of its scale, AV1 at 31%, VP9 at 36% and H.264 at 43%. That ordering is still not the quality ordering, which runs VP8, H.264, VP9, AV1 from worst to best. The honest use of QP is within one codec, as a relative signal over time, which is exactly what the QualityScaler uses it for. Comparing a VP9 QP against an H.264 QP, or reading a single QP as an absolute quality level, produces conclusions the reference measurement contradicts.

Full results

Bold marks the best codec in each column within a resolution block. “Saturated” counts clips undershooting target by more than 20%. PSNR and bitrate come from the 1,920-run quality pass; the two timing columns come from the separate single-pass timing run, so they match the encode-cost chart above rather than being spliced from a different set of machine conditions.

On the “vs target” column: setting the saturated runs aside, VP9 lands essentially exactly on target (mean −0.01%, sd 0.61) and VP8 carries a consistent 1% offset with an equally tight spread. AV1 is unbiased but noisier (sd 2.30), and OpenH264 is the one outlier, both low and wide (mean −2.06%, sd 2.52). It is a small effect against the several-dB differences above, and it is OpenH264 specifically, not H.264.

The saturated runs set aside above are a hard QP clamp, not a controller decision. OpenH264 forces iMinQp to at least GOM_MIN_QP_MODE = 12 and iMaxQp to MAX_LOW_BR_QP = 42 whenever the caller leaves them unset (rc.h:70, applied at encoder_ext.cpp:383), and WebRTC’s h264_encoder_impl.cc never sets either. 44 of the 480 OpenH264 runs come back with a mean QP below 14, spread evenly across all four rungs, and they miss target by a mean of 31.8% and a worst case of 65.6%. Re-running the worst of them with --write_csv, 275 of its 300 frames sit exactly at QP 12 and the median frame is 120 bytes against a 625-byte budget: the encoder has run out of quality to buy and simply declines the rest of the bitrate. That is a clamp, and it accounts for 37 of the 45 runs more than 20% short.

Clear of the clamp the controller is still the loosest of the four. Taking only runs with a mean QP of 20 or more, OpenH264 lands at −1.88% with sd 2.74 and 13.8% of runs more than 5% short, worst case −15.3%. VP8 is −1.13% with sd 0.61 and VP9 ±0.11% with sd 0.55, and neither misses by more than 5% on a single run out of 480. So the answer is both: a clamp on easy content, and a genuinely wider controller everywhere else.

Method and limits

• Source. The VCD MP4 distribution, roughly 11.5 Mbps H.264 at 1080p30, decoded to raw I420 per clip. The decoded MP4 is the common reference, so source compression does not favor one codec; however, the results may not generalize to pristine camera output.
• Scope. The 120 landscape clips only: 40 each of talking head, opaque background and blurred background. The 40 portrait mobile clips were excluded because the landscape bitrate ladder does not apply to them.
• Configuration. L1T1, with no additional temporal layers, one keyframe at frame 0, frame dropping enabled, two cores per run. 300 frames per clip.
• One operating point per resolution. No rate sweep, so no BD-rate. These are fixed-bitrate comparisons at the points WebRTC’s simulcast ladder would actually pick.
• AV1 does not reproduce exactly. The identical AV1 configuration has now been re-run twice, and neither replication was bit-exact: the timing pass differed on 148 of 480 runs (median 0.06 dB, worst 0.98 dB) and the VMAF pass on 176 of 480 (median 0.26 dB, worst 1.62 dB), both concentrated at 720p and 1080p. That is libaom’s multi-threaded encode at two cores, not measurement error. It does not move the aggregates, AV1’s 1080p mean shifted by 0.006 dB between the first two passes, but per-clip AV1 figures should be read as approximate. VP8, H.264 and VP9 reproduced exactly in both replications, 0 of 480 runs differing each time, which is what isolates the effect to libaom.
• VMAF is out of domain. The vmaf_v0.6.1 model is trained for 1080p HDTV viewing on Netflix catalog content, not video conferencing. It is a standard perceptual metric and is useful here for relative comparisons, but its absolute values on this corpus should not be treated as calibrated opinion scores.
• Neither metric charges for a dropped frame. Rate control skips frames, and a skipped frame is simply absent from both the PSNR average and the VMAF average. It is very unevenly distributed: OpenH264 dropped at least one frame in 120 of its 480 runs (25.0%), against 18 for VP8, 13 for VP9 and 22 for AV1, concentrated on the two low-bitrate rungs. The drops are the opposite failure from the undershoot above: they cluster at high QP, 40–44% of runs with a mean QP of 24 or more against 9% of runs at the QP 12 clamp. In the per-frame trace of the worst run every one of its 18 drops directly follows a burst frame of 3–13× the per-frame budget, against a median frame of 0.15×, so the encoder underspends, is caught out by a scene change, overshoots hard and then skips a frame to pay the leaky bucket back. Some part of what OpenH264 wins on quality it buys by not sending frames, and no number on this page subtracts that. The frames_decoded column in the VMAF dataset records it per run.
• The upscaled VMAF depends on the scaler. Bicubic is the convention, but a browser compositor may use something else, so the absolute “on a 1080p screen” values move with that choice. The ordering across rungs does not.

Reproducing this

Everything on this page comes from three CSV files, and every one of them is regenerable from the commands below. No step depends on state held anywhere else.

Environment

Machine

A single x86-64 Linux desktop, unchanged across every pass. Every run is pinned to --num_cores=2, so the thread count of the host only affects how many runs proceed at once, not what any one run measures. Absolute timings, and even the ratios between bars, may change on another machine because the codecs use different optimizations and scale differently with the hardware.

libWebRTC

Built from the tree at c9baebb35d. The binary is video_codec_perf_tests; the quality, VMAF and timing passes all use the same one.

args.gn

is_debug = false, rtc_use_h264 = true, ffmpeg_branding = "Chrome", proprietary_codecs = true, symbol_level = 1.

External

libvmaf f5f3664 with its built-in models, so no model file is needed on disk. FFmpeg 6.1.1 is used only to decode the source MP4s and to perform the bicubic upscale.

autoninja and ninja.py both fail against a Siso-generated state file in this checkout; calling siso ninja directly is the way through.

The dataset

The clip list ships with the dataset repository, and the clips themselves come from a public Azure blob. Only the three landscape scenarios are used, giving the 120 clips; each is decoded to raw I420 once and every run for that clip reads the same file.

The three passes

Each script is unattended, resumable, and streams results as they land, so an interrupted run loses at most the work in flight. All three live in the repository root.

vcd_run.py

The quality pass: 4 codecs × 4 resolutions × 120 clips. Produces vcd-psnr-results.csv, 1,920 rows.

vcd_vmaf_full.py

The VMAF pass, same grid, adding the y4m dumps and both scoring conventions. Produces vcd-vmaf-full.csv, 1,920 rows, and re-derives PSNR as a cross-check.

vcd_timing.py

Timing for all four codecs in one pass so the bars are mutually comparable. Produces vcd-timing.csv, 1,920 rows.

How the aggregates are defined

Three rules turn those rows into the numbers above, and all three are recomputable from the CSVs alone.

• Rate-matched subset. A clip enters the headline PSNR chart at a given resolution only if all four codecs delivered within ±10% of target there, so the comparison is at equal rate. That is abs(bitrate_mismatch_pct) ≤ 10 for all four, giving n = 93, 96, 71 and 93 clips. The side-by-side PSNR and VMAF panels deliberately skip this and use all 120, because their job is to compare metrics rather than to rank codecs.
• Saturated. A run undershooting its target by more than 20% is counted in the “saturated” column: the encoder hit its quality ceiling and could not spend the budget.
• Frame alignment. Decoded frame indices come from the per-frame CSV as (timestamp_rtp - 90000) / 3000 over rows carrying a PSNR value. When that list is shorter than 300, the reference is subset to it before VMAF runs.