When the Mac signing format changed in bug 1046306 we started to sign each partner repack after adding the customizations (bug 1058119). Previously it was possible for the app signature to exclude the distribution directory where partner changes are stored. There was no need to sign because the original app sig wasn't invalidated by repacking. Signing a partner build with the newer format results in differences from the regular build beyond the distribution directory: * the signature at Firefox.app/Contents/_CodeSignature/CodeResources includes hashes of distribution.ini and other partner-specific changes * the binaries get re-signed, eg XUL. (This might be a Mozilla implementation choice) When it comes time to update, Balrog will offer the same partial & complete updates that the regular build is given, but the partial updates generated from the regular builds won't apply. The updater does size and CRC checks on the existing file before applying a patch, so files differing from the regular release will cause an error. Firefox then fails over to the complete update, which succeeds. Partials updates to later releases succeed. The partner customizations are still present and appear to work, but no longer appear in the signature (see next comment for details). The open questions here are * does this mean Mac partner repacks are 'broken' ? In what situations ? * do we have more update orphans for this subset of the user population ?

Test scenario: 60.0.1 --> 61.0 --> 61.0.1 for Mac EME-free en-US. * fresh install of 60.0.1, run once and exit * modify channel-prefs.js to point to channel 'nthomas' * update to 61.0, partial fails: > EXECUTE PATCH Contents/_CodeSignature/CodeResources > LoadSourceFile: destination file size 23979 does not match expected size 23773 * failover to complete works OK * difference to regular 61.0 release: > $ diff -rq Firefox\ 61.0.app/ Firefox\ test.app/ > Files Firefox 61.0.app/Contents/Resources/defaults/pref/channel-prefs.js and Firefox test.app/Contents/Resources/defaults/pref/channel-prefs.js differ > Only in Firefox test.app/Contents/Resources: distribution i.e. regular build plus partner customizations (not in sig though!) and testing channel * update to 61.0, partial works fine * difference to regular 61.0.1 release: > $ diff -rq Firefox\ 61.0.1.app/ Firefox\ test.app/ > Files Firefox 61.0.1.app/Contents/Resources/defaults/pref/channel-prefs.js and Firefox test.app/Contents/Resources/defaults/pref/channel-prefs.js differ > Only in Firefox test.app/Contents/Resources: distribution i.e. same difference as after updating to 61.0 * difference to EME-free 61.0.1 release: > $ diff -rq Firefox\ 61.0.1\ EME-free.app/ Firefox\ test.app/ > Files Firefox 61.0.1 EME-free.app/Contents/Library/LaunchServices/org.mozilla.updater and Firefox test.app/Contents/Library/LaunchServices/org.mozilla.updater differ > Files Firefox 61.0.1 EME-free.app/Contents/MacOS/XUL and Firefox test.app/Contents/MacOS/XUL differ > Files Firefox 61.0.1 EME-free.app/Contents/MacOS/crashreporter.app/Contents/MacOS/crashreporter and Firefox test.app/Contents/MacOS/crashreporter.app/Contents/MacOS/crashreporter differ > Files Firefox 61.0.1 EME-free.app/Contents/MacOS/crashreporter.app/Contents/MacOS/minidump-analyzer and Firefox test.app/Contents/MacOS/crashreporter.app/Contents/MacOS/minidump-analyzer differ > Files Firefox 61.0.1 EME-free.app/Contents/MacOS/firefox and Firefox test.app/Contents/MacOS/firefox differ > Files Firefox 61.0.1 EME-free.app/Contents/MacOS/firefox-bin and Firefox test.app/Contents/MacOS/firefox-bin differ > Files Firefox 61.0.1 EME-free.app/Contents/MacOS/libfreebl3.dylib and Firefox test.app/Contents/MacOS/libfreebl3.dylib differ > Files Firefox 61.0.1 EME-free.app/Contents/MacOS/liblgpllibs.dylib and Firefox test.app/Contents/MacOS/liblgpllibs.dylib differ > Files Firefox 61.0.1 EME-free.app/Contents/MacOS/libmozavcodec.dylib and Firefox test.app/Contents/MacOS/libmozavcodec.dylib differ > Files Firefox 61.0.1 EME-free.app/Contents/MacOS/libmozavutil.dylib and Firefox test.app/Contents/MacOS/libmozavutil.dylib differ > Files Firefox 61.0.1 EME-free.app/Contents/MacOS/libmozglue.dylib and Firefox test.app/Contents/MacOS/libmozglue.dylib differ > Files Firefox 61.0.1 EME-free.app/Contents/MacOS/libnss3.dylib and Firefox test.app/Contents/MacOS/libnss3.dylib differ > Files Firefox 61.0.1 EME-free.app/Contents/MacOS/libnssckbi.dylib and Firefox test.app/Contents/MacOS/libnssckbi.dylib differ > Files Firefox 61.0.1 EME-free.app/Contents/MacOS/libnssdbm3.dylib and Firefox test.app/Contents/MacOS/libnssdbm3.dylib differ > Files Firefox 61.0.1 EME-free.app/Contents/MacOS/libplugin_child_interpose.dylib and Firefox test.app/Contents/MacOS/libplugin_child_interpose.dylib differ > Files Firefox 61.0.1 EME-free.app/Contents/MacOS/libsoftokn3.dylib and Firefox test.app/Contents/MacOS/libsoftokn3.dylib differ > Files Firefox 61.0.1 EME-free.app/Contents/MacOS/pingsender and Firefox test.app/Contents/MacOS/pingsender differ > Files Firefox 61.0.1 EME-free.app/Contents/MacOS/plugin-container.app/Contents/MacOS/plugin-container and Firefox test.app/Contents/MacOS/plugin-container.app/Contents/MacOS/plugin-container differ > Files Firefox 61.0.1 EME-free.app/Contents/MacOS/updater.app/Contents/MacOS/org.mozilla.updater and Firefox test.app/Contents/MacOS/updater.app/Contents/MacOS/org.mozilla.updater differ > Files Firefox 61.0.1 EME-free.app/Contents/Resources/defaults/pref/channel-prefs.js and Firefox test.app/Contents/Resources/defaults/pref/channel-prefs.js differ > Files Firefox 61.0.1 EME-free.app/Contents/_CodeSignature/CodeResources and Firefox test.app/Contents/_CodeSignature/CodeResources differ i.e. code signature differs (last line), and binaries were resigned. Repack customizations same in both.

I just ran a quick check on this repack: http://archive.mozilla.org/pub/firefox/candidates/60.0.2-candidates/build1/partner-repacks/ironsource/ironsource-google/v1/mac/ And while there definitely was some weirdness on the first update (The first update didn't appear succeed, but did the second time), I did get successfully updated to 61.0.2 after the second update and all the customizations stuck. So I think I saw what you did, but everythign still worked. Side note, we don't have a lot of partners doing Mac builds.

(In reply to Mike Kaply [:mkaply] from comment #2) > Side note, we don't have a lot of partners doing Mac builds. I was surprised to find that if you enumerate all the active combinations of partner, distro, platform, and locale then Mac is about 24% of the total. EME-free on all locales is quite a big driver of that. Actual active users depends on a lot of downstream factors though so I'm digging into metrics data to try to get some actual numbers. If you have any experience there, or know of good starting points or gotchas then I'd love to hear.

> If you have any experience there, or know of good starting points or gotchas then I'd love to hear. My main experience is the non EME stuff. Partners just don't typically ask for Mac.

https://sql.telemetry.mozilla.org/queries/56756#148404 is an attempt to use telemetry data to look at the version distribution of Mac pings, from both regular and partner installs, for the second half of June. The dataset is limited to update_enabled = true, which sounds like the pref and lack of r/w access might still block updates. It's all the pings, rather than our usual 'active user' definition of loading a few urls. There aren't any huge differences there IMO. Slightly more partner builds on 60.0, 59.0.2, and 58.0.2, but the long tail looks similar, rather than elevated for really old installs. 59.0.3 was also a Windows only release. See the chart for a visual representation of that, but beware the log scale.

On the basis of the previous comment, and from what I find searching around that Gatekeeper only checks the signature the first time an app is opened, I'm going to mark this as WONTFIX.