Now that we have signed packages on Linux and Windows (bug 1427849) we can get started to work on improvements for releasing geckodriver from Taskcluster. For MacOS (bug 1546299) we will pick the unsigned binaries for the time being.

Important for a release would be the changeset on mozilla-central, the release is based off. Depending on that commit a script could query the Taskcluster index for geckodriver artifacts of the signing jobs, or in case of MacOS the repack job.

For automatically creating a release PyGithub could be used. With it a draft release can be created, and assets be uploaded.

I still wonder if we would have to keep the code up-to-date on the releases branch on Github given that creating a release needs a target_commitish as reference. But maybe this will even work with an invalid setting. Lets see.

Note, that I don't think that it makes sense to have a separate job running in Taskcluster which collects the geckodriver binary artifacts. It's bad use of the workers.

For better referencing geckodriver binaries in Taskcluster we would need a route to have it available as index.

It could look like:
index.{trust-domain}.v2.{branch}.geckodriver.nightly.year.month.day.revision (for repack job)
index.{trust-domain}.v2.{branch}.signed-geckodriver.nightly.year.month.day.revision. (for signing job)

For example:
index.gecko.v2.mozilla-central.geckodriver.nightly.2019.06.11.0679bf09303e
index.gecko.v2.mozilla-central.signed-geckodriver.nightly.2019.06.11.0679bf09303e

A final release script as run on a specific changeset could then grab the binaries for each of the supported platforms, and provides them under a new index like:

index.{trust-domain}.v2.{branch}.signed-geckodriver.release.{version}.{platform}

which would be:

index.gecko.v2.mozilla-central.signed-geckodriver.release.0_24_0.win64

Another option could be to ignore the platform and encode it as part of the file name as we currently do for Github releases:

route: index.gecko.v2.mozilla-central.signed-geckodriver.release.0_24_0
artifacts: geckodriver-v0.24.0-linux32.tar.gz

Not sure how expensive extra routes are, and if it would be ok to have those for all the geckodriver builds even we create those for Nightlies only at the moment (which might change in the future).

John, maybe you can give some feedback?

Just as note: an example for geckoview can be found here:
https://hg.mozilla.org/mozilla-central/rev/9463e541aa94

If we may want to release via TC so that the geckodriver releases are queryable via Taskcluster index, we would need an Action with a release scope due to the security sensitive operation. Dustin pointed me to https://hg.mozilla.org/ci/ci-configuration/file/tip/grants.yml for some examples. The job could be triggered via Treeherder via Add new jobs only by a specific group of people.

Sorry for the delay.

As far as I know, indexes are pretty cheap to set up. The main issue I know is to keep people updated that a change is about to happen. We've had this in geckoview, for instance. It took us a few tries before having the index that suited all our needs.

index.{trust-domain}.v2.{branch}.signed-geckodriver.nightly.year.month.day.revision. (for signing job)
This scheme looks good to me. You may want to expose by revision (without knowing the date), or just get the latest. If so, I'd recommend:

index.{trust-domain}.v2.{branch}.signed-geckodriver.nightly.revision.{revision}
index.{trust-domain}.v2.{branch}.signed-geckodriver.nightly.latest
# And similarly
index.{trust-domain}.v2.{branch}.signed-geckodriver.nightly.pushdate.{year}.{month}.{day}.{revision}
index.{trust-domain}.v2.{branch}.signed-geckodriver.nightly.pushdate.{year}.{month}.{day}.latest
index.{trust-domain}.v2.{branch}.signed-geckodriver.nightly.pushdate.{year}.{month}.{day}.{build_id}

A final release script as run on a specific changeset could then grab the binaries for each of the supported platforms, and provides them under a new index like:

I remember some tasks doing something similar. I don't know if it was re-exposing the artifacts or tinkering with the index. I don't see any tasks like this anymore. Maybe they went away with shippable builds. Callek, do something like this ring a bell to you?

(In reply to Johan Lorenzo [:jlorenzo] from comment #4)

A final release script as run on a specific changeset could then grab the binaries for each of the supported platforms, and provides them under a new index like:

I remember some tasks doing something similar. I don't know if it was re-exposing the artifacts or tinkering with the index. I don't see any tasks like this anymore. Maybe they went away with shippable builds. Callek, do something like this ring a bell to you?

So let's back up slightly, the intent is to create a single task findable by one or more index's and has links to artifacts created across multiple other tasks?

We do not have this functionality in tree atm, but Taskcluster supports it. Releng last used it with direct API calls using buildbot bridge (before we were able to kill buildbot) https://github.com/mozilla-releng/buildbot-bridge/blob/0d0c169e6f5d6e6d014a007cd2070939234c8dec/bbb/tcutils.py#L48

Documentation is at https://docs.taskcluster.net/docs/reference/platform/queue/api#createArtifact (see "Redirect Artifact Request")

There is code in tree that does let us pop up an index on a previously completed task (kicks off https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/morph.py#205) though the intent of this is mostly keeping the queue's list of indexes shorter (so we don't hit AMQP hard limits) and it is implemented as a node.js thing.. defined with the in-tree docker image: https://searchfox.org/mozilla-central/source/taskcluster/docker/index-task/Dockerfile

I could see the actual request being useful but no support for it right now, I would imagine we would want to do something with in-tree and *-worker support directly, so we could e.g. define a given directory or file as "redirect" and then we read any files there as if they are text files and parse the url as redirect-target, and validate we are not breaking tooling assumptions by doing so...

Does that help? Would a zoom discussion be more useful?

Currently I'm not working on this bug.

With the signed build switched over to the toolchain builds we now have artifacts for the latest signed builds of geckodriver:

https://firefox-ci-tc.services.mozilla.com/tasks/index/gecko.v2.mozilla-central.latest.geckodriver

Sadly that won't help us to get a release out. Therefore we would need similarly to Firefox Nightly a static entry in the index to reference a specific build. Maybe something similar to Nightly here?

https://firefox-ci-tc.services.mozilla.com/tasks/index/gecko.v2.mozilla-central.signed.nightly

Maybe we could even get the latest.geckodriver under signed like:

https://firefox-ci-tc.services.mozilla.com/tasks/index/gecko.v2.mozilla-central.signed.geckodriver.latest
https://firefox-ci-tc.services.mozilla.com/tasks/index/gecko.v2.mozilla-central.signed.geckodriver.revision.[xyz]

Chris, would that be doable?

There are several routes already specified for geckodriver. e.g.:

gecko.v2.mozilla-central.latest.geckodriver.win64
gecko.v2.mozilla-central.pushdate.2020.05.20.20200520152823.geckodriver.win64
gecko.v2.mozilla-central.pushdate.2020.05.20.latest.geckodriver.win64
gecko.v2.mozilla-central.pushlog-id.37435.geckodriver.win64
gecko.v2.mozilla-central.revision.5415da14ec9a2f4749e8f405d6111a6f40e8138f.geckodriver.win64
gecko.v2.trunk.revision.5415da14ec9a2f4749e8f405d6111a6f40e8138f.geckodriver.win64

Do any of these work for you?

Oh cool! Yes the gecko.v2.mozilla-central.revision.*.geckodriver.* path looks great. That's what we definitely can make use of here. Thanks!

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #9)

Oh cool! Yes the gecko.v2.mozilla-central.revision.*.geckodriver.* path looks great. That's what we definitely can make use of here. Thanks!

This is in-place now. See:

https://firefox-ci-tc.services.mozilla.com/tasks/index/gecko.v2.mozilla-central.revision.f0e47b29f8c4840d9f9fbe971991f86179b55314.geckodriver

I wonder what's the best way to keep the builds around forever. As suggested archive.mozilla.org might be the best place. Callek and Tom might be able to give more details.