Is it possible to include h264 hardware decoding without including h264 software decoding?
https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/vaapi_h264.c

H264 patents: https://www.mpegla.com/wp-content/uploads/avc-att1.pdf

• Graphics cards that offer H264 hw decoding have paid the license fee.
• Cisco pays the capped license fee for slow openh264 software decoding.

We can't ship h264 - the problem is that the license fee has to pay who ships is. For instance Fedora provides only dnf repo and openh264 is downloaded from cisco directly by dnf update.

(In reply to Martin Stránský [:stransky] (ni? me) from comment #2)

We can't ship h264 - the problem is that the license fee has to pay who ships is.

hardware decoding: The graphics card ships a licensed hardware decoder. Everyone has paid the fee as part of their graphics card.
That's why Firefox can already use H264 hardware decoding on Windows and MacOS without problem.

software decoding: Microsoft ships a licensed software decoder. Linux distributions don't, but Cisco provides openh264 for download.

For instance Fedora provides only dnf repo and

VAAPI driver/API - already required for VP8+VP9:
Fedora doesn't want to install https://packages.debian.org/stable/intel-media-va-driver-non-free because it includes non-free parts for hardware encoding on older Intel chips,
but Fedora could package https://packages.debian.org/stable/intel-media-va-driver to simply talk to the hardware without doing fancy stuff in software:

https://github.com/intel/media-driver/

Media driver supports two build types as below

• Full Feature Build is default driver build, which supports all feature by hardware accelerator and close source shaders(media kernel binaries). Ubuntu intel-media-va-driver-non-free package is generated from this build type.
• Free Kernel Build, enables fully open source shaders(media kernels) and hardware features but the features would be limited. Ubuntu intel-media-va-driver package is generated from this build type.

(In reply to Darkspirit from comment #3)

because it includes non-free parts for hardware encoding on older Intel chips

Well, you do lose compliant film grain synthesis from AV1 decoding.

The video processing pipeline is also stripped of some features on newer chips (de-noise, some color grading, HDR tone mapping) and completely absent on chips older than Ice Lake.

On the encoding side there's lack of MPEG-2 and some HEVC formats. The only encoding missing from some "older" chips (Ice Lake, Kaby Lake, Coffee Lake, Whiskey Lake, Comet Lake and Amber Lake) is VP8, but that was phased out from newer chips for decoding as well.

(In reply to Jan Alexander Steffens [:heftig] from comment #4)
I filed this bug because removal of slow openh264 (I remember it as barely usable) was considered in bug 1737113.

Only relevant for Linux distributions:
The MIT-licensed Intel media driver provides the VAAPI interface to talk to the licensed H264/VP8/VP9/AV1 Intel hardware decoder.
The full variant has additional non-free shaders.
The free variant provides basic/sufficient decoding features. It would be nice if Fedora could install it by default if it doesn't already.

• Irrelevant: The free variant can't do any encoding on my Macbook Pro (Intel Iris Graphics 6100 (BDW GT3)), but the full (partly-non-free) variant could.
• Only relevant for bug 1652958: "AV1 film grain kernel is not open source yet which doesn't necessarily match the libaom algorithm or film grain synthesis." Users who need this could install the full (partly-non-free) variant.
• Irrelevant: MPEG2
• Irrelevant: VP8 is not used by YouTube anymore, Firefox' internal VP8 software decoder is sufficient on such newer hardware.
• non-free 10-bit VP9/HEVC tone mapping seems to be an optional feature that must be manually called.
• Firefox doesn't seem to use the optional VAAPI denoise filter.

I filed https://github.com/intel/media-driver/issues/1276.

I understand you. The question is if Mozilla can ship the vaapi_h264.c file and if that file doesn't violate any h264 patent. For instanace does radeon ship h264 decoder interface in open source drivers?

(let's skip the fact that shipping HW only h264 decoder may be confusing)

• Vendors need to have a license to sell/provide a h264 decoder to users. If an entity provides/sells a h264 decoder to more than 100,000 users per year, it must pay license fees.
  https://www.mpegla.com/wp-content/uploads/avcweb.pdf#page=10 (for comparison: https://www.mpegla.com/wp-content/uploads/m4vweb.pdf#page=7)

  Includes right to manufacture and sell AVC encoders and decoders with the right of End Users to use them for personal and consumer (including internal business) purposes without remuneration but not for other use

• On Raspberry Pi, the Video4Linux2 kernel interface is able to talk to the unlocked/licensed h264 hardware decoder.
  Users can additionally buy an MPEG2 or VC1 license key to unlock those hardware decoders as well.
  You give their decoder a stream of h264 and get images via dmabuf back (ffmpeg h264_v4l2m2m). The hardware decoder vendor, Raspberry Pi trading Ltd., has not negotiated with patent holders separately, but uses the MPEG-LA service to get their license. Otherwise they would not be allowed to sell an unlocked h264 hardware decoder to users.

(In reply to Martin Stránský [:stransky] (ni? me) from comment #7)

I understand you. The question is if Mozilla can ship the vaapi_h264.c file and if that file doesn't violate any h264 patent.

• If a h264 software or hardware decoder is found, Firefox already talks to it:
  • On Windows and Mac, Firefox already puts H264 data into an API that talks to the unlocked/licensed hardware decoder.
    Firefox, Windows and MacOS are not decoding the h264 video, but the unlocked hardware decoder is.
    Microsoft Windows offers fallback to an internal licensed h264 software decoder they provide. One can additionally buy a HEVC software decoder as Windows plugin.
  • If the openh264 software decoder of a Cisco customer is present on Linux, Firefox already uses it.
    Firefox already contains the code to be able to talk to openh264.
  • If FFmpeg (an API bundled with a possibly unlicensed H264 software decoder) is available on Linux, Firefox already uses it (media.ffmpeg.enabled == true).
  • Linux distributions usually can't install an FFmpeg h264 software decoder package by default, Firefox thus can't rely on it.
  • vaapi_h264.c is not a decoder, but talks to the VAAPI API. Firefox contains code that talks to similar APIs on other platforms.

For instanace does radeon ship h264 decoder interface in open source drivers?

• Yes, H.264/libva-mesa-driver: https://wiki.archlinux.org/title/Hardware_video_acceleration#VA-API_drivers
  https://packages.debian.org/stable/mesa-va-drivers is the VAAPI interface to talk to the unlocked h264 hardware decoder sold as part of AMD Radeon graphics cards.
  You fear that Fedora might become responsible for letting its users talk to their hardware by installing VAAPI drivers?
  Fedora already installs the Radeon VAAPI driver (debian: mesa-va-drivers) by default, but apparently not the free variant of the Intel VAAPI driver (debian: intel-media-va-driver):
  https://mastransky.wordpress.com/2020/06/03/firefox-on-fedora-finally-gets-va-api-on-wayland/

  AMD open source drivers decode video with radeonsi_drv_video.so library which is provided by mesa-dri-drivers package and it comes with Fedora by default.

  Internal hardware and software technologies of graphics cards that provide OpenGL, HDMI, etc. are most likely patent-protected.
  Fedora already provides Mesa to let its users talk to OpenGL/HDMI graphics hardware with internal proprietary firmware and patent-protected hardware.
  Even if Fedora would choose to not install VAAPI drivers (mesa-va-drivers/intel-media-va-driver) by default, users would only need to install the VAAPI driver then.
  At the moment, Firefox users additionally need to install FFmpeg - which contains unneeded parts like a probably unlicensed h264 software decoder - to be able to talk to the VAAPI driver (mesa-va-drivers/intel-media-va-driver) to talk to the h264 hardware decoder inside the graphics card.

• The interesting question is if Intel and AMD are licensed to sell an h264 hardware decoder to users or if they might violate patent rights. If they are not licensed, they must have some kind of non-disclosure agreement of h264 hardware decoder technology with their distributors, so that those are obliged to be licensed before publishing/selling an unlocked h264 hardware decoder to users. If they have such a legal construct, then it might even be that each online or retail shop that sells this hardware decoder to users might be the one who needs to pay the license fee because it publishes the unlocked/unrestricted patent-protected technology to a user. Then it would already affect Linux laptop makers with their own online shop.

(In reply to Darkspirit from comment #9)

...

I think the best would be to talk to some Mozilla legal team (assuming there's one). We'd need their ack anyway (or from some other official Mozilla devs).

(In reply to Martin Stránský [:stransky] (ni? me) from comment #7)

<...> The question is if Mozilla can ship the vaapi_h264.c file and if that file doesn't violate any h264 patent. <...>

Any software that uses VA-API for H.264 decoding, have to manage Decoded Picture Buffer, as VA-API expects already sorted lists of reference frames. There is at least one active patent, US7206347, which mentions DPB.

Here is a cross-chart from MPEGLA regarding H.264 patent pool: https://www.mpegla.com/wp-content/uploads/avcCrossRefChart.pdf which you may find interesting.

DPB is used by fill_vaapi_ReferenceFrames() in https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/vaapi_h264.c

I took all "Decoded picture buffer" patents from https://www.mpegla.com/wp-content/uploads/avcCrossRefChart.pdf,
sorted by them patent holder (https://www.mpegla.com/wp-content/uploads/avc-att1.pdf),
grouped EP patents and added expiration dates where possible (https://patents.google.com).
I haven't grouped patent families (that's when patents have the same content).

US+EU patents seem to have a maximum lifetime of 20 years.

Thanks Jan, so looks like we have to wait to 2028-02-20, right?

(In reply to Darkspirit from comment #13)

I took all "Decoded picture buffer" patents <...>

If there is a legal team at Mozilla, please go ask them instead. Discussing patent applicability details in public may be harmful.

Summary of the bugreport is highly misleading. In its current form it strongly implies that DPB is the only affected aspect of H.264, which may not be the case.