In the WebRTC standardisation post I mentioned that one of the controversial discussions in the IETF context was the mandatory to implement (MTI) video codec battle for WebRTC. While there are some technical arguments on the topic (i.e this VP8 vs H.264 – subjective evaluation and this performance comparisons discussion), there is no dispute both are high quality and efficient video codecs. The issue here is all about IPR and licensing as described in this interesting and ongoing discussion: “VP8 vs H.264 – the core issue“.
Why a mandatory WebRTC video codec matters
Some of you might be wondering why the IETF is having such difficulties reaching consensus on this topic. To answer this, let’s start by describing the importance/relevance of selecting a mandatory to implement (MTI) video codec to begin with. The short answer is Interoperability. As described in the specifications, the goal of having a mandatory to implement codec is to prevent negotiation failure, not to preempt or prevent negotiation. As discussed in previous posts, codecs in WebRTC are negotiated via de SDP Offer/Answer mechanism, and the lack of a common codec will result in negotiation and session setup failure. Having a MTI codec does not mean this is the only codec that can be negotiated, but it does mean that as long as the two parties conform to the specification the MTI can be used always as last-resort if no other options are available. As one can imagine, the absence of a MTI video codec might easily lead to fragmented deployments with some services using one codec and some browsers using others and no way for them to communicate. So it seems that guaranteeing video interoperability is something good. However, should we force implementors to pay licensing fees to accomplish that? That is the main question here.
As this debate has raged on, the lack of decision in this area has been a blocking issue in field trials and proof of concepts (POC). I’m involved in a number of projects where customers are willing to extend (and interwork) ‘legacy’ videoconferencing services over the WebRTC domain. Most of those ‘legacy’ platforms are based on H.264 codec (just few are ‘upgrading’ to VP8) while Chrome and Firefox only implement VP8 (do you remember this – Google Dropping H.264 Codec from Chrome Browser?). Few of the WebRTC-to-SIP interworking gateways in the market include some video transcoding capabilities. Those that do are mostly based on software with varying degrees of scalability. In most cases video transcoding is just a ‘roadmap item’ as it seems some vendors are waiting for this to settle down before deciding whether to implement it or not. That means for the time being we need to use VP8-capable softphones on the SIP/IMS side if we want to test/demonstrate interworked video sessions. A couple of weeks ago I participated at the Iberian Telecom Summit in Madrid, and one of the main topics of discussion was whether customers were willing to pay for video transcoding in a WebRTC context. The answer was not clear at all.
During the last IETF in Berlin this topic was not on the agenda – if you remember, main discussion there was DTLS-SRTP vs SDES. When I started preparing for the next IETF meeting several weeks ago (which is next week in Vancouver), I wondered whether the IETF would finally address this topic or would keep postponing it. So I decided to send this email to the IETF RTCWEB WG and the answer I got was the following plan, which basically means we’ll try to pick a MTI video codec while in Vancouver. Of course there were voices against this plan with some arguing video codec should not be an IETF issue at all but should be covered by the W3C instead. So far those arguments have led to nowhere. However, I wouldn’t be surprised to hear the same type of arguments at the beginning of the upcoming session. The agenda for the meeting can be found here and, as it can be seen, the video codec discussion will take place on Thursday. If you’re willing to follow the discussion remotely you can use any of the tools listed here.
The combatants
As indicated in the plan, the chairs asked for proposals by Oct 6 2013, there are two proposals on the table:
- one advocating for H.264 as MTI codec for WebRTC and
- another advocating for VP8 as MTI codec for WebRTC.
The VP8 proposal is only backed by Google, the H.264 proposal is backed by Ericsson, Nokia (or shall I say Microsoft?), BlackBerry, Qualcomm, Orange, Cisco, Microsoft and Apple — note that codecs is the only WebRTC-related subject where Apple has participated and expressed their opinion. Note also that Mozilla is not co-authoring any of those drafts. In the past they did comment on the topic via the mailing list but recently it seems to me they’ve been silent on the topic – if they didn’t, then my apologies as I missed it. Overall, while it seems to me they would be very happy to see VP8 adopted as MTI Video Codec for WebRTC, it is not totally clear to me how unhappy they would be if H.264 was selected instead.
VP8 proposal backers | H.264 proposal backers |
|
|
When it comes to implementations, as mentioned, both Chrome and Firefox only implement VP8 for WebRTC video. There has been some confusion on this as Firefox has some support for H.264 but it’s clarified in the following email from Mozilla:
1 2 |
What that article says is that Mozilla is adding select platform-supplied codecs to Firefox for non-WebRTC uses Our implementation of audio codecs for WebRTC continues to support PCMU, PCMA, Opus, and nothing else; <strong>our implementation use VP8 exclusively for video</strong>. |
This basically means that H.264 support is only for decoding (relying on underlying OS support) and is not available for WebRTC. In any case, I am wondering whether at some point Mozilla would consider taking advantage of this for WebRTC too. This topic is especially interesting for Firefox OS based devices, where Mozilla is collaborating with Qualcomm, one of the H.264 proponents.
The only ‘browser’ I know of that supports H.264 for pseudo-WebRTC is Ericsson’s Bowser (I did not forget the ‘r’). I am not sure if this is being actively maintained any longer as I’ve seen several emails on the mailing list complaining about it:
1 |
Looks like it was more of a push to try to get H.264 supported. I don't think any work has been done since last year. |
In any case, none of the customers I’ve seen playing with WebRTC considered Bowser more than a nice experiment. For mobile environments they are currently testing mobile versions for Chrome and Firefox on top of Android. Here’s an example from earlier this year.
Last week Microsoft Open Tech published a first implementation of the W3C ORTC, the competing API we presented in this article. Quoting Microsoft’s own words,
1 |
The <a href="http://html5labs.interoperabilitybridges.com/prototypes/object-rtc/object-rtc/info">prototype</a> demonstrates how to support <strong>H.264/AVC</strong> video without SDP, by building the appropriate JavaScript code and without introducing any changes in the relevant parts of the ORTC specification. |
What the laywers have to say
Going back to the core issue of the matter, the above mentioned IETF drafts include considerations on VP8 and H.264 IPR/licensing.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
The IETF has a long tradition of preferring non-encumbered IPR whenever possible, and especially to avoid IPR where using the technology requires making agreements with and payments to third parties as part of the cost of doing business. Among the reasons for this tradition is that the requirement for IPR agreements severely distorts the competitive landscape, and especially that it seriously hampers people attempting to implement standards in open source, or other business models where counting the number of installations or users is difficult, expensive or simply impossible. As of this moment (October 4, 2013), the following IPR disclosures are filed in the IETF IPR database: o <a href="https://datatracker.ietf.org/ipr/1571/">https://datatracker.ietf.org/ipr/1571/</a> - by Google, declaring that the technology is royalty-free. o <a href="https://datatracker.ietf.org/ipr/2035/">https://datatracker.ietf.org/ipr/2035/</a> - <strong>by Nokia, which does not declare a royalty-free license</strong>. The licensing terms for Google's IPR are available at <a href="http://www.webmproject.org/license/additional/">http://www.webmproject.org/license/additional/</a>. The Nokia IPR mentioned above includes IPR that has been asserted in ongoing litigation in Germany (Nokia v. HTC, District Court in Mannheim, Germany. 7 O 201/12); on one of the patents, <strong>the court has ruled that the phones in question (which support VP8) are not infringing</strong>. As mentioned in <a href="http://blog.webmproject.org/2013/08/good-news-from-germany">http://blog.webmproject.org/2013/08/good-news-from-germany</a>.html?m=0; the case is still ongoing. The following companies have asserted that <strong>any IPR relevant to VP8 they might have is available for licensing by Google under a royalty free license</strong>; the licensing terms are available at <a href="http://www.webm-ccl.org/vp8/agreement/">http://www.webm-ccl.org/vp8/agreement/</a>, as well as details on the licensors: o CIF Licensing LLC o France Telecom o Fraunhofer-Gesellschaft zur Foerderung der angewandten Forschung e.V. o Fujitsu Limited o Koninklijke Philips Electronics N.V. o LG Electronics Inc. o Mitsubishi Electric Corporation o MPEG LA, LLC o NTT DOCOMO, INC (*) o Panasonic Corporation o Samsung Electronics Co., Ltd. o Siemens Corporation (*) The license can be executed on-line from the link given above. |
Licensing considerations for H.264
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
Royalty Free for Innovation, Low-volume Shipments MPEG-LA released their AVC Patent Portfolio License already in 2004 and in 2010 they announced that H.264 encoded Internet video is free to <strong>end users will never be charged royalties</strong> [<a title=""MPEG LAs AVC License Will Not Charge Royalties for Internet Video that is Free to End Users through Life of License"" href="http://tools.ietf.org/html/draft-burman-rtcweb-h264-proposal-02#ref-MPEGLA">MPEGLA</a>]. Real-time generated content, the content most applicable to WebRTC, was free already from the establishment of the MPEG-LA license [<a href="http://tools.ietf.org/html/draft-burman-rtcweb-h264-proposal-02#ref-MPEGLA-License">MPEGLA-License</a>]. <strong>License fees for products that decode and encode H.264 video remain though</strong>. Those fees [<a href="http://tools.ietf.org/html/draft-burman-rtcweb-h264-proposal-02#ref-MPEGLA-Terms">MPEGLA-Terms</a>] are, and will very likely continue to be for the lifetime of MPEG-LA pool, $0.20 per codec or less. To paraphrase, the MPEG LA license does allow up to 100K units per year, per legal entity/company (type "a" sublicensees in MPEG LA's definition), to be shipped <strong>for zero ($0) royalty cost. This should be adequate for many WebRTC innovators or start-ups to try out new implementations on a large set of users before incurring any patent royalty costs</strong>, a benefit to selecting a H.264/AVC profile as the mandatory codec. Higher H.264/AVC Profile Tools Bundled It should be noted that when one licenses the MPEG LA H.264/AVC pool, patents for higher profile tools - such as CABAC, 8x8 - are bundled in with those required for the Constrained Baseline Profile. Thus, these could optionally be used by WebRTC implementers to achieve even greater performance or efficiencies than using H.264 Constrained Baseline Profile alone. It can also be noted that for MPEG-LA, since one license covers both an encoder and decoder, there is no additional cost of using an encoder to an implementation that supports decoding of H.264. Licensing Stability H.264 is a mature codec with a mature and well-known licensing model. It is a well-established fact that not all H.264 right holders are MPEG-LA pool members. H.264 is however an ITU/ISO/IEC international standard, developed under their respective patent policies, and all contributors must license their patents under Reasonable And Non-Discriminatory (RAND) terms. In the field of video coding, most major research groups interested in patents do contribute to the ITU/ISO/IEC standards process and are therefore bound by those terms. <strong>VP8 is a much younger codec than H.264 and it is fair to say that the licensing situation is less clear than for H.264</strong>. Google has provided their patent rights on VP8, including patents owned by 11 patent holders [<a href="http://tools.ietf.org/html/draft-burman-rtcweb-h264-proposal-02#ref-MpegLaVp8">MpegLaVp8</a>], under a open source friendly license with very restrictive reciprocity conditions. Recently, VP8 was adopted as Working Draft for Video Coding for Browsers in MPEG, which is the first step in becoming an MPEG standard. As such, it will have to follow the ISO/IEC/ITU common patent policy [<a href="http://tools.ietf.org/html/draft-burman-rtcweb-h264-proposal-02#ref-IsoIecItuPolicy">IsoIecItuPolicy</a>], but IPR statements cannot be expected there for still some time. There is no guarantee that IPR statements in MPEG will be royalty free (option 1), but may just as well be "Fair, Reasonable And Non-Discriminatory" (FRAND, option 2). This indicates that <strong>the licensing situation for VP8 has still not been settled</strong>. |
Each of us can draw its own conclusions here. Note the following indicated by a Skype (Microsoft) employee:
1 2 3 4 5 |
On the IPR issue, Google reached agreement with 11 patent holders. There are at least 31 companies in the MPEG-LA H.264 pool. There is considerable technical overlap between VP8 and H.264. My employer is one of those in the H.264 pool, and wasn’t one of the 11 companies Google reached an agreement with. Draw your own conclusions and take your own IPR risks. Personally <strong>I’d rather the IPR devil I know vs. the IPR devil I don’t know</strong>. Google could fix this for most potential users (through indemnification, similar to what Oracle offers its Linux licensees) but has chosen not to. You can draw your own conclusion there, too. |
How will this end?
What will be the outcome of the discussion next week in Vancouver? Who knows, but I’m sure we’ll see some room flooding with folks that never attended before an RTCWEB WG (or even IETF) meeting in order to try to influence the decision. The following are potential options:
- H.264 is selected as MTI Video Codec for WebRTC — but, would Chrome and Firefox support H.264 for WebRTC if the IETF mandates that codec? Remember we’ve recently seen Google switching Hangouts from H.264 to VP8.
- VP8 is selected as MTI Video Codec for WebRTC — according to Google it seems pretty much every SoC vendor is now shipping HW VP8 acceleration but, would Apple and Microsoft (incl. IE, Lync, Skype) implement VP8 just because the IETF mandates that codec? Would that be yet another reason for them to not join the standard WebRTC bandwagon?
- Both H.264 and VP8 are selected as MTI Video Codecs for WebRTC — not sure this is realistic from the implementation perspective. While I could speculate H.264 proponents might potentially live with this decision, I don’t think the same would be true for the VP8 side. Ideally this would allow browser-to-browser and browser-to-something-else communication without requiring video transcoding.
- An older codec whose IPR has expired is selected as MTI Video Codec for WebRTC — so far no one has formally proposed this option in a draft but it would follow a “better-than-nothing” approach. This would basically mean having a last-resort that then let clients upgrade to VP8 or H264 as they see fit. This codec would likely not be as technically as good as VP8 or H.264. It is also interesting to note that codecs could potentially be implemented as a plug-in.
- No MTI Video Codec for WebRTC is selected — again, no one has formally proposed this option in a draft, but it would basically mean that Video communication would not be a guaranteed feature in WebRTC — in other words, and as someone put it on the mailing list, in the event that a common Video Codec cannot be negotiated during video session setup, the session should fall-back to voice-only negotiation where there is a MTI (G.711 and Opus). To be frank, I wouldn’t be very happy with this but I tend to think this is the direction we’re heading towards.
I’ll post an update next week after the meeting. Have you already subscribed to our mailing-list, if not please click here as it just takes few seconds to join it. You can also send me an email to [email protected] or follow me on Twitter at @victorpascual.
In the meantime… what’s your opinion?
{“author”, “victor“}
{“editor”, “chad“}
Lorenzo Miniero says
How the 100k units per year should “be adequate for many WebRTC innovators or start-ups to try out new implementations on a large set of users before incurring any patent royalty costs” is a mystery to me. What happens after you try it out? You cannot afford it and you cast it aside. I’m part of a startup and I’m well aware of the fact we wouldn’t absolutely be able to afford the licensing costs of H.264 for the applications we’re building on top of WebRTC (conferencing, streaming, etc.).
The choice of H.264 would only result in a full stop to all of the innovativations and fresh ideas people have brought to WebRTC so far, because the licensing burden would not only apply to us, but to tons of other implementors that are doing great stuff right now.
Most of the H.264 proponents have contributed zero or close to zero when it comes to WebRTC: they may be doing something on the ML, but I’ve seen nothing relevant when it comes to implementations, nice scenarios and so on (and no, I don’t consider Bowser relevant, and I don’t even consider ORTC at all: as if we needed another Silverlight vs. Flash or ASP vs PHP clone). The only stuff I’ve seen done was great browser implementations by Google and Mozilla, and excellent applications by WebRTC enthusiasts that at times pushed the concept beyond what I thought possible. When you don’t write a single line of code, just stay on the side watching, and then eventually try to impose your codec, I can only see one thing: you’re scared you won’t be able to compete based on merits, and so you’re bullying to impose your coded, to make money out of other people work or stop those who can’t afford to pay you, no matter how great their efforts were. This is defending the status quo, something that unfortunayely is not new in the IETF where we all know how strong large companies are: and something I will never accept or tolerate, especially when a technology that could affect, in a positive way, the future of multimedia technology is involved.
If it can’t be VP8, let it be something else (H.263 or heck, even H.261 would work at this point) but I will never accept H.264. Legacy applications are called like that for a reason: WebRTC is moving towards the future, and a rock tied to its foot to keep it from evolving is the last thing it needs.
Peter Dunkley says
The quote from the Skype (Microsoft) employee is interesting, but so was the response to that message on the mailing list which pointed out that if Skype (Microsoft) wants that position to be considered they really should submit an IPR declaration to the IETF.
I have a concern about the table showing Google as the only VP8 backer. It is correct that Google is the only big organisation backing VP8. However, I am sure there are many, many, small start-ups (who don’t have the time or money to engage with the IETF) that support VP8 simply because there is no royalty-free option for H.264 at this time (I agree with the sentiments of the first commenter here – the 100k unit thing is just a fig-leaf). I believe that the most innovate use of WebRTC will be from these smaller companies and I do worry that the need to use a royalty-bearing codec will stifle that.
To be blunt, the big organisations that are in favour of using H.264 can afford to switch to VP8 more than smaller organisations can afford to worry about H.264 licensing. Even the administration required to count your H.264 usage (required even if the first 100k units are free) may be too onerous for one/two man organisations with downloadable software products (what do you count, downloads, license keys, API keys – how do you manage all this without cost?).
VP8 might not be royalty free in the future, but that still makes it a far better choice than H.264 which is not royalty free now.
Gustavo says
1) I don’t know why any non-browser vendor opinion matters (aka. Cisco, Ericsson, Orange….). So even if that table shows a lot of companies in the right side, Google and Mozilla should be 90% of the decision and Opera, Safari and MS rest 10%.
2) The only future proof approach in my opinion is VP8 as it ensures a simple transition to VP9.
Gustavo says
“Most of the H.264 proponents have contributed zero or close to zero when it comes to WebRTC”
+1
“If it can’t be VP8, let it be something else (H.263 or heck, even H.261 would work at this point) but I will never accept H.264. ”
++++1
Lawrence Byrd says
I think you are right – it is about to get really ugly! Thank you, Victor, for your comprehensive overview and good fortunes in Vancouver. I had not quite appreciated for imbalance of those at the table for H.264 until I stared at your table, which will make it a tough fight for a royalty-free VP8/VP9 in WebRTC. Lorenzo Miniero above is right that all the small start-ups and innovators who want licensing-unencumbered video codecs are not represented except through Google. You also point to the considerable FUD floating about – “who knows is VP8/VP9 is unencumbered, we may still be able to get you!”. Google has now spent over $300M trying to “free” all the codecs and code and it is disingenuous for companies to keep turning up at IETF saying “well Google hasn’t negotiated/paid/fought with us yet” within a standards body supposedly committed to “a long tradition of preferring non-encumbered IPR whenever possible, and especially to avoid IPR where using the technology requires making agreements with and payments to third parties as part of the cost of doing business”. Fare forward.
Victor Pascual says
UPDATE: Cisco to open source H.264 implementation, distribute binary module and absorb MPEG-LA license fees http://blogs.cisco.com/collaboration/open-source-h-264-removes-barriers-webrtc/
See below the announcement from Cisco
———- Forwarded message ———-
From: Jonathan Rosenberg (jdrosen)
Date: Wed, Oct 30, 2013 at 1:28 PM
Subject: [rtcweb] Cisco to open source its H.264 implementation and absorb MPEG-LA licensing fees
To: “[email protected]”
I’d like to make an announcement material to the conversations around MTI video codecs in rtcweb.
Cisco is announcing today that we will take our H.264 implementation, and open source it under BSD license terms. Development and maintenance will be overseen by a board from industry and the open source community. Furthermore, we will provide a binary form suitable for inclusion in applications across a number of different operating systems (Windows, MacOS, Linux x86, Linux ARM and Android ARM), and make this binary module available for download from the Internet. We will not pass on our MPEG-LA licensing costs for this module, and based on the current licensing environment, this will effectively make H.264 free for use on supported platforms.
We believe that this contribution to the community can help address the concerns many have raised around selection of H.264 as MTI. I firmly believe that with H.264 we can achieve maximal interoperability and now, do it with open source and for free (well, at least for others – its not free for Cisco J)
More information on the open source project can be found at http://www.openh264.org, which is sparse now but more coming soon.
Thx,
Jonathan R.
Victor Pascual says
Mozilla to bring real-time H.264 support to their browser
https://blog.mozilla.org/blog/2013/10/30/video-interoperability-on-the-web-gets-a-boost-from-ciscos-h-264-codec/
Cullen Jennings says
All the people working on getting the openh264.org and announcements out over the past few weeks were reading this article but sort of afraid to post anything on given what was coming. Great article – In my mind the big thing that has changed today with respect to this article is that Firefox is going to have both VP8 and H.264.
We don’t think choosing an older codec really works – if you look at the video, it is so awful, no one wants it. Thus we think we have to break the log jam at IETF. H.264 as MTI does that and we hope that browsers support VP8, H264, VP9, H265, Daala and any good codec. If the handful of major browsers have wide codec support, and a common fallback of H.264, many applications can be build that use the codec they want and things will work well.
Give users the choice, but make sure there is a common fallback. And my view is microsoft and apple were never going to do VP8 as the MTI.
Lorenzo Miniero says
And my view is microsoft and apple were never going to do VP8 as the MTI.
“were never going”? Cullen, you’re talking as if the call for consensus had already been taken 😉
Lawrence Byrd says
I think Cullen makes a key point here – if Microsoft and Apple are ever going to eventually arrive on board the WebRTC train the path of them having to adopt Google-driven VP8/VP9 as the MTI in the standard is probably not the most plausible path. So this is a very interesting response by Cisco to the video codec dilemma. Plus, in some sense, Cisco is putting a challenge on the table here to get Apple to do the same thing – make H.264 easily available in iOS to other apps without further fees.
Lorenzo Miniero says
As Shakespeare would have said, from my point of view it’s “Much Ado About Nothing”.
I don’t see any real value in this Cisco opening. First of all, as per the FAQ you cannot ship the module when you’re installing your application that depends on it: you have to download it on the fly, possibly getting the free license somehow in the process. Far from ideal, especially in case the servers are not reachable for any reason. Besides, as far as I’ve understood the “free” only applies to Cisco patents: the FAQ again says that this does not cover other potent holders tha may assert claims, which is EXACTLY the same situation VP8 is allegedly in now.
Make the codec REALLY free and then we’ll get somewhere.
Lou says
I find it unbelievably rude to credit Wikipedia for the painting illustrating your article, where in fact the work of Ilya Efimovich Efimovich Repin, courtesy of the Pushkin Museum. As if I were to credit Saint Gobain for the Chateau Margaux that I drank last night :o).
Other than that, great article !
Chad Hart says
That was my laziness. I apologize for the poor and incorrect image citation. It has been updated. -chad
Victor Pascual says
Ericsson Research has decided to remove Bowser from the Apple App Store and Google Play https://groups.google.com/forum/#!topic/ericsson-labs-web-rtc/XwHDJTpKPkU