On bug 1587421 comment 12 :callahad reported the following issue and I'm copy/pasting it here because bug 1587421 is a 3-year-old bug related to the release of macOS 10.15

I experienced this last week on a fresh install of macOS Monteray (12.3.1).

As expected, the dylib did have the com.apple.quarantine xattr set.

"libgmpopenh264.dylib" can't be opened because Apple cannot check it for malicious software.

This software needs to be updated.
Contact the developer for more information.

Firefox downloaded this file on May 3, 2022.

This appears to be a regression introduced by bug 1746675 late in 2021 due to a renaming copy/paste error. Bug 1746675 changed GMPExtractorWorker.js to use IOUtils.removeMacXAttr() when it should have used IOUtils.delMacXAttr() added just before in bug 1736331. Using the wrong function name doesn't result in an error because GMPExtractorWorker.js gracefully handles the function being missing because it's coded to be platform-generic and does a runtime check for the existence of removeMacXAttr expecting it to only be present when running on macOS.

The fix will be to change GMPExtractorWorker.js code to call IOUtils.delMacXAttr() and it would be safer to use #if defined(XP_MACOSX) instead of checking for the existence of the function first. We should quickly land the rename fix and uplift it.

An additional complication is that we should add code to handle cases where the plugins have already been downloaded without clearing the attribute and now will fail to load. This could be a follow up fix.

And we should have tests for this!

For users hitting this problem now, one workaround would be to clear the quarantine attribute on the dylib file by running xattr -d com.apple.quarantine <filename>. Or using the following find and exec commands to remove the attribute on the two supported plugins. This has yet to be verified.

$ find ~/Library/Application\ Support/Firefox/Profiles/ -name libgmpopenh264.dylib -exec xattr -d com.apple.quarantine {} \;
$ find ~/Library/Application\ Support/Firefox/Profiles/ -name libwidevine.dylib -exec xattr -d com.apple.quarantine {} \;

Set release status flags based on info from the regressing bug 1746675

Use the correct function name (delMacXAttr vs removeMacXAttr) to remove the com.apple.quarantine extended attribute from downloaded plugin files.

Add test code to ensure extracted plugin files do not have the quarantine attribute.

I manually verified that the posted patches successfully clear com.apple.quarantine when installing the plugin.

And without this fix, the new test code causes a failure on try TEST-UNEXPECTED-FAIL | toolkit/modules/tests/xpcshell/test_GMPInstallManager.js | test_GMPExtractor_paths - [test_GMPExtractor_paths : 1] Missing expected exception The 'com.apple.quarantine' attribute should not be present.

This fix should resolve the problem for new profiles and existing profiles that install an updated plugin.

In addition to this fix, we also need to deal with profiles that have already downloaded a plugin dylib without clearing the attribute causing the dylib to fail to load even after this fix lands because the fix re-enables the code to remove the attribute at install time. I will file a new bug to cover this.

https://hg.mozilla.org/mozilla-central/rev/e9122f049d1b

Comment on attachment 9280389 [details]
Bug 1773207 - "libgmpopenh264.dylib" can't be opened because Apple cannot check it for malicious software r?#media-playback-reviewers!

Beta/Release Uplift Approval Request

• User impact if declined: Some sites such as WebRTC sites and possibly video playback sites using Widevine (such as Netflix and Amazon Prime Video) may not work and trigger an error dialog (see description) explaining that Apple has not verified the .dylib file.

Bug 1773600 is also needed to fully address this problem, but the fix is separate and not dependent. This fix addresses the problem for newly installed plugin files (when the plugin is downloaded). Bug 1773600 is needed to fix plugin files already downloaded.

• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Install Firefox on a fresh macOS install and verify that Netflix or Amazon Prime video is functional and that a site that uses OpenH264 works. For OpenH264, the problem was reported on site https://meetings.dialpadbeta.com on https://bugzilla.mozilla.org/show_bug.cgi?id=1587421#c15
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): The change is small and it only re-enables code that has been present in Firefox for a few years, but was accidentally disabled in Firefox 97.

The code that is being re-enabled clears the com.apple.quarantine attributed from downloaded plugin files.

• String changes made/needed:
• Is Android affected?: No

It worth noting that this problem will show a lot more in 101 because of bug 1766311 which will cause us to try to use the openh264 instead of the hardware encoders.

Comment on attachment 9280389 [details]
Bug 1773207 - "libgmpopenh264.dylib" can't be opened because Apple cannot check it for malicious software r?#media-playback-reviewers!

Beta/Release Uplift Approval Request

• User impact if declined: Some sites such as WebRTC sites and possibly video playback sites using Widevine (such as Netflix and Amazon Prime Video) may not work and trigger an error dialog (see description) explaining that Apple has not verified the .dylib file.

Bug 1773600 is also needed to fully address this problem, but the fix is separate and not dependent. This fix addresses the problem for newly installed plugin files (when the plugin is downloaded). Bug 1773600 is needed to fix plugin files already downloaded.

We are not clear on how widespread this problem is. Bug 1766311 changed Firefox to use libgmpopenh264.dylib in more cases which might lead this bug being hit in more cases.

• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Install Firefox on a fresh macOS install and verify that Netflix or Amazon Prime video is functional and that a site that uses OpenH264 works. For OpenH264, the problem was reported on site https://meetings.dialpadbeta.com on https://bugzilla.mozilla.org/show_bug.cgi?id=1587421#c15
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): The change is small and it only re-enables code that has been present in Firefox for a few years, but was accidentally disabled in Firefox 97.

The code that is being re-enabled clears the com.apple.quarantine attributed from downloaded plugin files.

• String changes made/needed:
• Is Android affected?: No

Comment on attachment 9280389 [details]
Bug 1773207 - "libgmpopenh264.dylib" can't be opened because Apple cannot check it for malicious software r?#media-playback-reviewers!

Approved for 102 beta 8, thanks.

https://hg.mozilla.org/releases/mozilla-beta/rev/ff5b6fafa490

Reproduced the issue on Firefox 103.0a1.pkg (2022-06-07) under macOS 12.4 by following the info from Comment 10.

The issue is no longer reproducible on latest Firefox 103.0a1.pkg (2022-06-14) under the same system.

Verified on 102.0b8.pkg (treeherder build) as well on the same system and the issue is fixed.

Catalin, when you reproduced was widevine (Netflix or https://bitmovin.com/demos/drm) also broken?

Nope, Netflix was working. I reproduced it on https://meetings.dialpadbeta.com after entering in a call session.

Catalin, I want to confirm that our telemetry matches what users see. Can you reproduce the issue and paste the contents of the WEBRTC_SOFTWARE_H264_ENABLED histogram from about:telemetry?

Sure thing Jeff, here's the histogram content:

WEBRTC_SOFTWARE_H264_ENABLED
22 samples, average = 1, sum = 22

0 | 0 0%
1 |######################### 22 100%
2 | 0 0%

I'll attach a screenshot with the error as well.

Ok, yeah, this confirms that our telemetry doesn't properly detect this scenario. We think we have openh264 but it doesn't actually work.

It seems like it should also be possible to fix this by shipping a signed openh264 dylib. I've filed bug 1774221 about doing that.

(In reply to Jeff Muizelaar [:jrmuizel] from comment #20)

It seems like it should also be possible to fix this by shipping a signed openh264 dylib. I've filed bug 1774221 about doing that.

Agreed and once the fix for bug 1774221 is shipping, this problem should not be reproducible. With bug 1774221, the plugin will be signed and notarized. In testing it loads without removing the quarantine attribute.

Given that, I'm going to cancel the request to uplift this fix to Release. And in the future, we can remove the code that clears the quarantine attribute on plugin files. I'll file a bug to cover that.