Steps to reproduce

1. Enable gfx.webrender.software. I'm using Windows 10 so gfx.webrender.software.d3d11 is also enabled.
2. Restart Firefox.
3. Load https://bugzilla.mozilla.org/show_bug.cgi?id=1482920#c12
4. Look for random red dots in the video thumbnail in bug 1482920 comment 12.

Expected result

No red dots.

Actual result

Random red dots near the center of the video thumbnail. See the attached screenshot. Zooming the page in and out with Ctrl+= and Ctrl+- will change the location of the red dots.

If I disable gfx.webrender.software and restart Firefox, the red dots go away.

I can reproduce this bug in Firefox 87, Beta 88, and Nightly 89.

The video in bug 1511215 comment 0 has even more red dots than the video in comment 0. I see now that the red dots don't just affect the video thumbnail; they linger in video playback.

Seems to occur for me on Linux too

FWIW, I just tested on macOS (11.2.3) and I can't reproduce the red dots.

gfx.webrender.debug.picture-caching shows the bug happening at picture cache tile boundaries and at primitive boundaries, which would imply that some YUV value is probably generating an out-of-bounds RGB value that overflows something in the shader slow paths, but not when in the fast-paths inside the interior of the tile.

A failure to repro on macOS is possibly because the video hits different compositing paths and/or uses different formats than Windows and Linux do.

Looking in the profiler shows that we are using the brush_yuv_image shader, and this hits a specific fast-path on Windows and Linux because the video is supplied in a planar format there. On macOS, we use a packed YUV 422 format, which would fall off that fast-path and seems to avoid that problem.

This expands on an earlier fix from bug 1698009. It turns out we can occasionally find
YUV values which can still produce negative RGB values if only Y is clamped. The final
solution to this is just to clamp the output RGB values rather than input YUV values.

Since this is only used when we fall off the SWGL fast-paths (which properly handle
this clamping already), the performance impact of the extra clamping should be negligible.

https://hg.mozilla.org/mozilla-central/rev/577145ad8319

Comment on attachment 9213968 [details]
Bug 1703402 - Clamp RGB components after YUV conversion for SWGL blending. r?jrmuizel

Beta/Release Uplift Approval Request

• User impact if declined: Visual artifacts during playback on some videos.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Just adds some clamping to out of range colors values on video, but otherwise doesn't really change much else. Only affects SW-WR rollout.
• String changes made/needed:

Comment on attachment 9213968 [details]
Bug 1703402 - Clamp RGB components after YUV conversion for SWGL blending. r?jrmuizel

Fixes an SW-WR rendering bug. Approved for 88.0b9.

https://hg.mozilla.org/releases/mozilla-beta/rev/057ecbffdf23