With bug 1427849 and bug 1546299 we got signed builds for geckodriver based on the binary as produced in the Firefox build jobs, and located in the common test archive.

Once bug 1534533 landed, we will be able to change the signing jobs to no longer depend/require a full build, but just the geckodriver toolchain task. This will give us signed builds way faster, and solves all known issues for releasing geckodriver from Taskcluster.

This work has to be done before we can stop producing geckodriver builds via the build job.

Johann, just to get a confirmation from you in regards of signing toolchain build jobs, mind replying to comment 0 if that would be possible? Mainly that we keep the chain of trust, and are allowed to sign those builds. Thanks.

Yes, it looks feasible to me! Toolchain tasks (like this one[1]) outputs both public/chain-of-trust.json and public/chain-of-trust.json.sig, so Chain of Trust is able to trace this task back to the tree. Therefore, it will be happy.

Please let me know if need more information.

[1] https://tools.taskcluster.net/groups/MFD7X6NMTpmWY3gKrYaSDA/tasks/NjKrxvOBS5avXZoCt8LQmg/runs/0/artifacts

Great. So I had a look into this issue and tried to change that signing job to depend on the toolchain build job but failed. Mainly because of the way how it has been implemented with transforms. It's way different compared to normal jobs.

Would you mind giving me a hint in how to change the code so it no longer depends on the geckodriver-repack job but uses the builds as generated by the geckodriver toolchain job?

https://searchfox.org/mozilla-central/source/taskcluster/ci/geckodriver-signing/kind.yml
https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/transforms/geckodriver_signing.py

https://searchfox.org/mozilla-central/source/taskcluster/ci/toolchain/geckodriver.yml

The geckodriver-repack job wouldn't be needed anymore, and could also be removed once that switch has been done.

I'm testing locally by comparing before and after the patch with

./mach taskgraph full --json --target-kind geckodriver-signing --tasks '.*geckodriver-signing.*' > f.json

It's pretty clear that this can be made to work, but I'm not sure how
we want to handle the nightly choices here. Toolchains don't really
respect nightly in the way that builds do.

jlorenzo: take a look at my commit and see what needs to change? Current diff for one task looks like:

-  "geckodriver-signing-linux-nightly/opt": {
+  "geckodriver-signing-linux32/opt": {
     "attributes": {
       "always_target": false,
-      "build_platform": "linux-nightly",
+      "build_platform": "linux32",
       "build_type": "opt",
       "kind": "geckodriver-signing",
-      "nightly": true,
       "repackage_type": "repackage-signing",
       "run_on_projects": [
-        "all"
+        "trunk"
       ],
       "shipping_phase": null,
       "shipping_product": null
     },
     "dependencies": {},
     "kind": "geckodriver-signing",
-    "label": "geckodriver-signing-linux-nightly/opt",
+    "label": "geckodriver-signing-linux32/opt",
     "optimization": null,
     "release_artifacts": [
       "public/geckodriver.tar.gz",
@@ -34,7 +33,7 @@
       },
       "extra": {
         "index": {
-          "rank": 0
+          "rank": 1569621433
         },
         "parent": "",
         "treeherder": {
@@ -48,15 +47,15 @@
             "platform": "linux32"
           },
           "symbol": "s",
-          "tier": 2
+          "tier": 1
         },
         "treeherder-platform": "linux32/opt"
       },
       "metadata": {
-        "description": "Signing Geckodriver for build 'linux-nightly/opt' ([Treeherder push](https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=1e0685b06c61f350da1973d419223883ba64d3e2))",
-        "name": "geckodriver-signing-linux-nightly/opt",
-        "owner": "btara@mozilla.com",
-        "source": "https://hg.mozilla.org/mozilla-central/file/1e0685b06c61f350da1973d419223883ba64d3e2/taskcluster/ci/geckodriver-signing"
+        "description": "Signing Geckodriver for build 'linux32/opt' ([Treeherder push](https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=d23a0a0ffa93fe273c571592bdbff6aa68f6ca2b))",
+        "name": "geckodriver-signing-linux32/opt",
+        "owner": "dluca@mozilla.com",
+        "source": "https://hg.mozilla.org/mozilla-central/file/d23a0a0ffa93fe273c571592bdbff6aa68f6ca2b/taskcluster/ci/geckodriver-signing"
       },
       "payload": {
         "maxRunTime": 600,
@@ -69,7 +68,7 @@
               "public/geckodriver.tar.gz"
             ],
             "taskId": {
-              "task-reference": "<geckodriver-repack>"
+              "task-reference": "<toolchain>"
             },
             "taskType": "repackage"
           }
@@ -78,37 +77,36 @@
       "priority": "medium",
       "provisionerId": "scriptworker-prov-v1",
       "routes": [
-        "tc-treeherder.v2.mozilla-central.1e0685b06c61f350da1973d419223883ba64d3e2.36626"
+        "tc-treeherder.v2.mozilla-central.d23a0a0ffa93fe273c571592bdbff6aa68f6ca2b.36627"
       ],
       "scopes": [
-        "project:releng:signing:cert:nightly-signing"
+        "project:releng:signing:cert:dep-signing"
       ],
       "tags": {
-        "createdForUser": "btara@mozilla.com",
+        "createdForUser": "dluca@mozilla.com",
         "kind": "geckodriver-signing",
-        "label": "geckodriver-signing-linux-nightly/opt",
+        "label": "geckodriver-signing-linux32/opt",
         "retrigger": "false"
       },
-      "workerType": "signing-linux-v1"
+      "workerType": "depsigning"
     }
   },

We used the dependency on nightly jobs before to lower the amount of geckodriver builds to be signed because geckodriver gets built for each and every Firefox build. Given that we have a toolchain job now, which only gets run when source code changes happen (this is right or?), it shouldn't matter anymore. The amount of jobs is kinda low.

Chatted with :whimboo and :nalexander. Nick will update the ticket with a summary.

This flattens geckodriver-{repack,signing} into one task that takes
a toolchain-geckodriver-* task as input. It's very close to a
general "sign toolchain artifacts" task.

Open questions:

• I've called it geckodriver-signing2 for now, and not removed the
  other tasks. Happy to rename whatever we like.
• Can we sign stuff in public/build on macOS? I can link the error,
  which I think is a missing mkdir -p .../public/build. (It's just
  failing to tar a result.)
• We can bikeshed where to put these on Treeherder at some point. The
  tasks right now are with the build platforms, so I put them there.
  I think a "Signed toolchain" row (platform) makes more sense, but I
  don't really care.

A try build before I squashed is at:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a65c83402515975c381998b615dcfd7bf46bafec

Apparently I can't ask for feedback, so I asked for review. Questions welcome, folks!

As discussed with Andreas today we do not want to hold back the 0.26.0 release. We can live with the MSVCRT dependency until the 0.26.1 or 0.27.0 release.

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #10)

As discussed with Andreas today we do not want to hold back the 0.26.0 release. We can live with the MSVCRT dependency until the 0.26.1 or 0.27.0 release.

OK, that's grand for me. Hopefully I'll get more significant feedback from Johan and Aki sometime this week.

As it looks like due to bug 1588081 we won't be able to switch the signing jobs to built on the toolchain jobs. The notarization service only works for Nightly and Release keys.

If that is the case we might have to get rid of the toolchain jobs, and update the normal builds for geckodriver for bug 1442253, and bug 1584530. :/

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #12)

As it looks like due to bug 1588081 we won't be able to switch the signing jobs to built on the toolchain jobs. The notarization service only works for Nightly and Release keys.

That was actually a wrong information. So lets forget about this comment.

Aki, were you able to make some steps forward in having a solution here? When it comes to the next release, which we actually want to have soon, it would be great to have this working. Thanks!

See the latest comments in bug 1588081. We have workarounds; if we want a fully notarized geckodriver someone needs to create a notarizable package format.

Could we for now just switch the job dependency to the toolchain task, and then implement the specific package format? Having this bug open causes a lot of trouble for us when releasing geckodriver. Or is this causing a lot of extra work?

I believe it'll work, give it a try.

Note that Chris started working on that by end of last week. Here a recent try build which already looks kinda promising:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8a8fd0953aefdbbdd9228753d9d172478dc4fe12

Unfortunately, I started working on this without having looked at nalexander's patches. It would have saved me a lot of time!

Ultimately, this is pretty close to D47518. I worked around the mac signing issue by changing the output directory of the toolchain builds to remove the 'build/' intermediate directory. I had to add an extra comment in the build script to allow force-rebuilding of the toolchains, since the toolchain optimization hashing doesn't take artifact paths into account.

I think this should allow geckodriver builds to happen on autoland and central. Signing will happen on central per push I think.

I did need to change the archive format for windows and mac, to .zip and .tar.bz2 respectively.

This does not solve the macOS notarization issue.

(In reply to Chris AtLee [:catlee] from comment #19)

Ultimately, this is pretty close to D47518. I worked around the mac signing issue by changing the output directory of the toolchain builds to remove the 'build/' intermediate directory. I had to add an extra comment in the build script to allow force-rebuilding of the toolchains, since the toolchain optimization hashing doesn't take artifact paths into account.

This is great to hear Chris! Thanks a lot for your time and helping us with that part!

I think this should allow geckodriver builds to happen on autoland and central. Signing will happen on central per push I think.

I did need to change the archive format for windows and mac, to .zip and .tar.bz2 respectively.

Oh, this is exactly what we would need for our releases: https://github.com/mozilla/geckodriver/releases. So that's perfect.

This does not solve the macOS notarization issue.

Ok, that can be delayed and will now block the general release from TC meta bug.

If a task has explicitly specified artifact paths, don't additionally specify
the default paths. If the task has private artifacts, having a directory
that uploads public artifacts seems like an attractive nuissance.

https://hg.mozilla.org/mozilla-central/rev/b76e3e988fdd

We probably need to switch to the dep-signing pool, or switch the scope to nightly-signing / release-signing for the m-c bustage.

Backed out for breaking geckodriver signing toolchains

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&selectedTaskRun=NPDuX4pFRDe8KS9hgIf_vA-0&resultStatus=testfailed%2Cbusted%2Cexception&revision=ead8f0367372fe5767d33a0aa0a95b07ee76ea75

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=301362783&repo=mozilla-central&lineNumber=24

Backout: https://hg.mozilla.org/mozilla-central/rev/50244579252a229131ae04a1d2c60682611e5702

Ah, it's a combination of linux-signing pool in the transform, along with no shippable or nightly attribute in the task, I'm guessing.

Yeah, the intent was to use nightly signing for the toolchains on m-c. I'll try and figure out how to make that happen.

https://hg.mozilla.org/mozilla-central/rev/09443cbbdfc9

Backed out for toolchain bustages.

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedTaskRun=SBa3YaftRZq-TDRjQqJNAw-0&resultStatus=testfailed%2Cbusted%2Cexception&revision=33277064c34351810b7ca34691733c41157b3b79

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=302402130&repo=autoland

Backout: https://hg.mozilla.org/integration/autoland/rev/2a9cf23c30710f2384a9aee5bad39fc2d80236e5

Looks like we still have one thing to land here...

https://hg.mozilla.org/mozilla-central/rev/72f6e2d07667

To fix the regression for external tools, which expect the location of the geckodriver binary artifact at public/build/, we agreed on backing out the following changeset that was optional for this bug anyway:

https://hg.mozilla.org/mozilla-central/rev/72f6e2d07667

Once done please do NOT reopen this bug. I will file a follow-up bug for the additional fix. Thanks.

Actually a follow-up patch was provided on bug 1639955 that should have fixed the regression now. So there is no need to backout this specified changeset anymore.