With bug 1546299 we got signed builds of geckodriver, and given its dependency on bug 1471004 the notarization service should have been used. But the geckodriver binary cannot be started on MacOS 10.15 as of today:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=geckodriver&selectedJob=270854573

Aki and Johann, can you please have a look what's broken here? Right now it means users on Catalina cannot run automated tests via Selenium. So I hope that we can find a solution hopefully soon. Thanks!

Hey Henrik! Thanks for looping us in. Do you have an error or a screenshot? I haven't migrated my machine to Catalina yet, so I can't reproduce.

Andreas migrated to Catalina, so he has.

Also have a look at https://github.com/mozilla/geckodriver/issues/1629 where I got the information initially.

Oh no, disaster! I haven't upgraded to 10.15 and won't for a jolly long while so I can't help investigate here. Hopefully things are clear. Haik is very knowledgable about these things, I think, and may be able to assist in a pinch.

Ah, we never added notarization for the iscript sign_geckodriver behavior. We can add it, but we will only be able to notarize geckodriver with the nightly- or release-signing certs. Likely that will mean we should stick with geckodriver signing in nightly graphs.

wip PR here.

Aki, would you mind explaining why we have this hard requirement on nightly or release? We thought that we can build (already done) and sign (bug 1577110) geckodriver based on toolchain builds, which are way faster and better to handle. Now it looks like that due to notarization we were working towards a dead-end.

Apple will only notarize a build that has been signed with a valid [shipping] key, like the nightly or release keys. Also, it's a security hole to just allow people to sign any arbitrary binary with the nightly or release keys, so we have checks in place to make sure the requests come from a valid nightly or release graph.

So I think what I miss is the difference to the chain-of-trust requirement. Given that toolchain jobs ensure to keep that, and as given by Johan we should be able to sign also non-nightly and non-release builds on mozilla-central. But what makes notarization different to that?

Ahh, sorry. You're right; I'm wrong. If the task runs on an on-push or action graph on mozilla-central with CoT enabled, we should be able to nightly-sign and notarize geckodriver. Apologies for the confusion!

Puh! This is lovely to hear! Is there anything else I can help with? What do you think how long it might take to get the notarization working for the geckodriver signing? We just released the 0.26.0 release, but might be able to just replace the binary with the new one on the github releases page.

Hm, getting

Downloaded ticket has been stored at file:///var/folders/z4/f2_cz9d95jdctsrtnr3016zm00001_/T/6c927b89-3903-421a-815e-3952f32b6a89.ticket.
Could not remove existing ticket from geckodriver/Contents/CodeResources -- file:///builds/scriptworker/aki/work/0/ because an error occurred. Error Domain=NSCocoaErrorDomain Code=512 "“CodeResources” couldn’t be removed." UserInfo={NSFilePath=/builds/scriptworker/aki/work/0/geckodriver/Contents/CodeResources, NSUserStringVariant=(
    Remove
), NSUnderlyingError=0x7f9a85d85c10 {Error Domain=NSPOSIXErrorDomain Code=20 "Not a directory"}}
The staple and validate action failed! Error 73.

geckodriver is a file, not a directory. I wonder if we have to create a dmg or pkg first, and notarize that?

Aki, might the info from https://developer.apple.com/documentation/security/notarizing_your_app_before_distribution/resolving_common_notarization_issues help here?

In this document they talk about binary or app bundle. So I assume notarizing binaries should be possible, and maybe the code which is currently in use by us to do that, doesn't take just binaries into account.

That document points out:

If you see error -73 while using Xcode 10, you can resolve this issue by upgrading to Xcode 10.1 or later. Also, in this case, check to ensure that the disk image or flat installer package you’re notarizing is writable so you can attach the ticket to the package with the stapler utility.

(emphasis mine).

We're already using Xcode 10.2.x. However, we're not notarizing a disk image or flat installer package; we're trying to notarize a single executable binary. We likely either need to notarize properly, which probably means creating a dmg package containing the executable, or we need to bypass the security restrictions on machines that run geckodriver.

What about a zip file as we offer it for download? While stapling to a binary is currently not possible (see the below link), notarizing the zip archive might be somewhat doable? But I do not really fully understand the paragraph given that all items have to be stapled:

https://developer.apple.com/documentation/xcode/notarizing_your_app_before_distribution/customizing_the_notarization_workflow#3087720

I assume that both the unsigned and signed versions of geckodriver are affected by that?

Aiui, the link says you need to staple everything inside the zipfile. We can't staple a bare geckodriver, so putting it in a zipfile doesn't help our stapling situation. Is there someone who can create a geckodriver.app for mac? We can notarize an app.

As a workaround, users may be able to run xattr to remove the quarantine attribute from geckodriver:
https://derflounder.wordpress.com/2012/11/20/clearing-the-quarantine-extended-attribute-from-downloaded-applications/

Good information. Andreas, would you mind to test if that works for you on 10.15?

On macOS 10.15 (Catalina) with geckodriver 0.26.0:

% xattr geckodriver
com.apple.macl
com.apple.quarantine
% ./geckodriver
Killed: 9
% xattr -r -d com.apple.quarantine geckodriver
% xattr geckodriver
com.apple.macl
% ./geckodriver
^C

So it seems users can use xattr(1) to remove the quarantine check manually.

(In reply to Aki Sasaki [:aki] (he/him) (UTC-7) from comment #15)

Aiui, the link says you need to staple everything inside the
zipfile. We can't staple a bare geckodriver, so putting it in a
zipfile doesn't help our stapling situation. Is there someone who
can create a geckodriver.app for mac? We can notarize an app.

Thank you, Aki, for investigating this!

What I have trouble understanding is how Apple is expecting
non-interactive, “terminal-based” programs to be distributed under
the new notarisation regime. geckodriver is not a program it is
suitable to distribute as an application bundle (.app) because it
is not an interactive program like Firefox.

I have a few perhaps stupid questions:

1. If we could distribute a .dmg instead of a .zip, would we be
   able to notarise the .dmg?

2. If we can’t notarise .dmg’s, it is more suitable for us to
   distribute geckodriver in a .pkg installer. Do you have any experience
   crafting one? Presumably we can do this in automation too.

(In reply to Andreas Tolfsen 「:ato」 from comment #19)

(In reply to Aki Sasaki [:aki] (he/him) (UTC-7) from comment #15)

Aiui, the link says you need to staple everything inside the
zipfile. We can't staple a bare geckodriver, so putting it in a
zipfile doesn't help our stapling situation. Is there someone who
can create a geckodriver.app for mac? We can notarize an app.

Thank you, Aki, for investigating this!

What I have trouble understanding is how Apple is expecting
non-interactive, “terminal-based” programs to be distributed under
the new notarisation regime. geckodriver is not a program it is
suitable to distribute as an application bundle (.app) because it
is not an interactive program like Firefox.

I'm not sure. But we're able to launch executables under an .app via commandline, e.g. mkdir /tmp/foo && /Applications/Firefox.app/Contents/MacOS/firefox --profile /tmp/foo

I have a few perhaps stupid questions:

1. If we could distribute a .dmg instead of a .zip, would we be
   able to notarise the .dmg?

We notarize Firefox.app that's in a .dmg currently, but it's an .app , and we extract it from the dmg before signing and notarizing. I think a bare geckodriver in a .dmg would hit the same problems.

2. If we can’t notarise .dmg’s, it is more suitable for us to
   distribute geckodriver in a .pkg installer. Do you have any experience
   crafting one? Presumably we can do this in automation too.

We do create a .pkg using pkgbuild, but it's from the existing .app structure. We may be able to create a .pkg with just geckodriver and the appropriate directory structure, but I'm not 100% sure if that will notarize by itself or not. If we're able to create that pkg, we'd need to install it, and then run geckodriver from the installed location.

It looks like our options are:

• use xattr and run geckodriver from commandline
• create an .app and notarize it; install the .app and run geckodriver from commandline
• create a .pkg and notarize it; install the .pkg and run geckodriver from commandline
  • this may or may not require creating an .app first

I found this in the Apple documentation:

Alternatively, you can put apps, kernel extensions, and other
software in a container, like a disk image, and notarize the
container. The notary service accepts disk images (UDIF format),
signed flat installer packages, and ZIP archives. It processes
nested containers as well, like packages inside a disk image.

For geckodriver on macOS we currently produce a gunzipped tarball
(.tar.gz), but according to this it should be possible to notarise
.dmg and .zip archives.

As I’m completely unaware of where any of the packaging code lives,
could someone point me to the right resources to try out this change?

We have, e.g., https://searchfox.org/mozilla-central/source/build/package/mac_osx/make-diskimage , https://searchfox.org/mozilla-central/source/python/mozbuild/mozpack/dmg.py . Found via https://searchfox.org/mozilla-central/search?q=DMG_&path= .

Please also note that the notarization issue for users only occurs when the binary gets downloaded interactively by the user via a web browser. Then the quarantined bit gets set. If the download happens with eg. curl or wget instead, the binary can directly be used.

Aha, that sounds familiar. Since we have both the curl/wget option, and the xattr option, are we good here?

I documented the problem and the workarounds in https://firefox-source-docs.mozilla.org/testing/geckodriver/Notarization.html, then updated the changelog.

If we can somewhat easily change the geckodriver packaging to .dmg and .dmgs indeed can be notarised, I would suggest we make that change before the 0.27.0 release. I haven’t followed up on your suggestions from comment #22 as I’m struggling to figure out where the packaging is actually made. I should add I’m not familiar with TaskCluster at all.

(In reply to Andreas Tolfsen 「:ato」 from comment #25)

I documented the problem and the workarounds in https://firefox-source-docs.mozilla.org/testing/geckodriver/Notarization.html, then updated the changelog.

If we can somewhat easily change the geckodriver packaging to .dmg and .dmgs indeed can be notarised, I would suggest we make that change before the 0.27.0 release. I haven’t followed up on your suggestions from comment #22 as I’m struggling to figure out where the packaging is actually made. I should add I’m not familiar with TaskCluster at all.

The packaging is all done here: https://searchfox.org/mozilla-central/rev/55aa17110091deef24b913d033ccaf58f9c6d337/taskcluster/scripts/misc/build-geckodriver.sh#56-59.

(In reply to Aki Sasaki [:aki] (he/him) (UTC-7) from comment #22)

We have, e.g., https://searchfox.org/mozilla-central/source/build/package/mac_osx/make-diskimage , https://searchfox.org/mozilla-central/source/python/mozbuild/mozpack/dmg.py . Found via https://searchfox.org/mozilla-central/search?q=DMG_&path= .

make-diskimage appears unused. Looking at dmg.py, I expect the trickiest part will be arranging for all of the macOS-specific tools to be available on the Linux builders (everything is cross compiled in the toolchain tasks). It might be that the existing builder toolchains for cross compiling Just Work, though :)

Actually so far we haven't heard any complains that the binary cannot be started. Maybe it's because it is a command line application.

Are there any updates on this? I can use Selenium with the Chrome handler on MacOS, but Geckodriver won't start. Should I use the workaround?

Sorry, but we don't have an update yet. And yes, please use the suggested workaround to get geckodriver working, or download the archive via curl/wget.

We carry this issue from one geckodriver release to the next. Lets discuss what we can or even want to do here.

Aki, maybe you could give some brief information what would change if we enable notarization? Would we have to ship a DMG instead of just a binary for MacOS? What else would be required these days to get notarization enabled? Thanks.

That appears to be the answer in comment 13, yes. I think the .app or .pkg is more of a requirement than .dmg though.

Henrik: Reminder to file another bug to update the documentation

We discussed this last week and we do not intent to actually work on this in the foreseen future. Instead we want to improve our documentation to make it clear that this is expected and what needs to be done to get geckodriver running if downloaded via the browser. This will also allow us to get rid of the known issues entry in the release notes, which we carry along for quite some time now.

I'm going to file a new documentation bug soon and as such leaving needinfo set.

Closing as inactive for now given that we do not have the time to work on this.

I think I can move this forward, assuming geckodriver notarization is still desired?

Similar to bug 1774669 (openh264), we can notarize the plain geckodriver executable -- we just can't staple the notarization. For openh264 we've verified that means the quarantine attribute is not needed. I think the .dmg solution is still better: then we could staple, and presumably there would then be no further reliance on network connectivity to the notarization server. For now I can add notarization of the plain executable.

With https://github.com/mozilla-platform-ops/ronin_puppet/pull/405, mac_notarize_geckodriver behavior is supported, but currently unused.

(In reply to Geoff Brown [:gbrown] from comment #35)

I think I can move this forward, assuming geckodriver notarization is still desired?

Yes, that would be great Geoff! Given that we cannot easily change the distribution of the package we have to continue shipping the executable. If there is a way to notarize it we would kinda appreciate a working solution. Thanks a lot.

Similar to https://phabricator.services.mozilla.com/D153115, but for geckodriver.

I'll go ahead and switch that over: We'll start notarizing geckodriver. As I noted earlier, we still can't staple the notarization to geckodriver, so the existing workaround will still be needed for offline use. I'll leave-open for changing the packaging, which I think is the best long-term solution.

Thanks Geoff! I would be fine to just get the bug marked as fixed once the patch landed on central and to file a new one for the DMG case if that is really needed.

https://hg.mozilla.org/mozilla-central/rev/fc1b2face187

Release Note Request (optional, but appreciated)
[Why is this notable]:
[Affects Firefox for Android]:
[Suggested wording]:
[Links (documentation, blog post, etc)]:

geckodriver has nothing to do with Firefox so none of these flags are needed. Please do not set these flags randomly. Thanks.