With bug 1493948 we will repack the geckodriver binary into its own artifact by extracting it from the common test package.

Ideally this should be a toolchain build task like cbindgen but for all supported platforms by geckodriver.

When moving to the toolchain task we want to have binaries of geckodriver present for each commit. Right now the geckodriver-repack job only runs for Nightly builds of Firefox. I assume that we could make this job dependent on changes for the appropriate folder in hg, so that we don't actually have to built it for each and every release, but only when changes happen.

Hi Mike, I have a couple of questions about the implementation here and I hope that you can help.

1. When we have a toolchain task to build geckodriver where in the taskgraph would it need to be injected. Given that it's not needed for a Firefox build, I'm not sure where the reference needs to be added. We would also need release and debug builds of it (compiled with cargo).

2. I assume we can run a build only when the source directories of geckodriver have been changed? It might be similar to what we use with the docker jobs.

3. How does it look like with builds on linux32, and win32? Are those not possible with a toolchain task? I cannot find any references under taskcluster/.

Thanks a lot!

Tom probably can help you better than I would.

minidump_stackwalk would probably be a good place to start, as an example of building a tool from source in mozilla-central, though it doesn't handle dependencies properly. This is another example that does handle that, but hasn't landed yet.

That is probably enough to get you started, but feel free to ask more questions, either here, or in #ci on IRC.

s/probably/hopefully/

This is getting in the way of Bug 1525126, and from there in the way of Bug 1561939 (and its immediate follow-up to work on Android). Touching geckodriver takes ages 'cuz it's doing a Nightly build...

(In reply to Nick Alexander :nalexander [he/him] from comment #6)

This is getting in the way of Bug 1525126, and from there in the way of Bug 1561939 (and its immediate follow-up to work on Android). Touching geckodriver takes ages 'cuz it's doing a Nightly build...

Maybe this is easy? Try build percolating at https://treeherder.mozilla.org/#/jobs?repo=try&revision=12f8c4b028d95868571bd3e8dce2d616b28b12dc.

So this is interesting. https://treeherder.mozilla.org/#/jobs?repo=try&revision=a4a8c297c41f429ef54b3edbf46d2847122637bb seems to fail with linker errors that we've seen before... but sccache isn't involved in these builds. See https://bugzilla.mozilla.org/show_bug.cgi?id=1525760#c11.

glandium: any thoughts? The Linux build seems okay, so perhaps the cross builds aren't setting an appropriate linker? (I culted build-minidump-stackwalker for this, BTW.)

You're missing the binutils toolchain for mac, and the Windows build is using the windows-gnu target, not windows-msvc, so lld is not the right linker for that. But it seems you should use the windows-msvc target. minidump-stackwalker is a bad example for that because it doesn't build with MSVC/clang-cl, so we have to use the windows-gnu target there.
You may want to look at the sccache build script instead. Or rust-size. Or cbindgen.

There was quite a bit of discussion of this in #build, and the
consensus was that geckodriver should be built as a stand-alone Rust
crate and not as part of Firefox/Gecko (say, as a new
--enable-project target). This follows that approach, and the
expression, modeled off of cbindgen, is pretty straight-forward.

The zip dependency doesn't appear to require bzip2, and bzip2 is
tricky to cross-compile, so the feature set is restricted to ease the
build task.

Eventually testing/geckodriver can be removed from the top-level
Rust workspace, the geckodriver-signing tasks migrated to these
toolchain tasks, consumers be migrated to the signing tasks, and
geckodriver removed from the "common" test archive.

Try build percolating at https://treeherder.mozilla.org/#/jobs?repo=try&revision=a5085c72353a0fc590ca3384950cba0f3e084cd9.

glandium: so, Win32 turns out to be exactly as irritating as you'd expect :( Can you look at https://treeherder.mozilla.org/#/jobs?repo=try&author=nalexander%40mozilla.com&selectedJob=261848754 and make a recommendation? Is it better to set a linker for host and target (duplicating what you just did for Gecko, essentially) or is it better to build as --enable-project and try to take advantage of what's been done? I will say that --enable-project was not straight-forward; we configure a ton of cross-compiling stuff for macOS in mozconfigs, all of which have to be ported over and subtly updated (i.e., toolchain download paths are different).

Maybe we can land this and follow-up with Win32?

Can I entertain you with cross compilation?
https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=261955851&revision=797b287baa019e3b559d002aa4aca8d580ab48f9

There was quite a bit of discussion of this in
firefox-build-system-reviewers, and the consensus was that geckodriver
should be built as a stand-alone Rust crate and not as part of
Firefox/Gecko (say, as a new --enable-project target). This follows
that approach, and the expression, modeled off of cbindgen but updated
to cross compile from a Linux host to all targets, is pretty
straight-forward.

A sparse profile would be nice, but the way that the Gecko Cargo
workspace works means that the profile must accumulate Rust code from
many locations.

If we want to, eventually testing/geckodriver can be removed from the
top-level Rust workspace, the geckodriver-signing tasks migrated to
these toolchain tasks, consumers migrated to the signing tasks, and
geckodriver removed from the "common" test archive.

Depends on D43645

https://treeherder.mozilla.org/#/jobs?repo=try&revision=62835b50cbf35d5de185778d3de1d70a090b967b looks happy. Windows outputs are .tar.bz2 (just following cbindgen, I think) and contains just geckodriver.exe.

https://hg.mozilla.org/mozilla-central/rev/ded4116115e8
https://hg.mozilla.org/mozilla-central/rev/d2bdeab3b280

Here a triggered task:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=geckodriver&revision=7004b8779a36084c898634e5e31d8f406815d68a

I have some remaining questions:

1. Right now I cannot see that this toolchain build task has been run at all since it was merged to mozilla-central. So I assume it is only build for any kind of change in the mozbase-rs, webdriver, or geckodriver crates? Also once we have other jobs which depend on the toolchain job, I would assume that it automatically figured out if a build is necessary or now?

2. Why do we have the limitation to trunk and try, and not for release branches in the TaskCluster config? You didn't answer my question in the Phabricator review from yesterday. Do we now need a follow-up to also enable it for release branches?

Otherwise thanks a lot for helping us getting this done! It looks fantastic.

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #19)

Here a triggered task:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=geckodriver&revision=7004b8779a36084c898634e5e31d8f406815d68a

I have some remaining questions:

1. Right now I cannot see that this toolchain build task has been run at all since it was merged to mozilla-central. So I assume it is only build for any kind of change in the mozbase-rs, webdriver, or geckodriver crates? Also once we have other jobs which depend on the toolchain job, I would assume that it automatically figured out if a build is necessary or now?

toolchain kind tasks are only triggered on demand, so either via another task depending on them or via a try build. I expect to land browsertime-related tasks very soon (Bug 1566174, IIRC) which should start producing these archives regularly.

2. Why do we have the limitation to trunk and try, and not for release branches in the TaskCluster config? You didn't answer my question in the Phabricator review from yesterday. Do we now need a follow-up to also enable it for release branches?

Ah, I see that I wrote responses, went to update the patch, and forgot to submit the responses. I've done that now. Just to have the response here as well: I think I culted from cbindgen, which also makes little sense to me: it's also macOS-specific. I dropped the limitation, so whatever the toolchain default is will prevail. I think that's probably correct.

Otherwise thanks a lot for helping us getting this done! It looks fantastic.

Happy to help! I looked at the signing tasks and they looked a little more complicated than I could bite off. If you could move them over that would be awesome-possum.

(In reply to Nick Alexander :nalexander [he/him] from comment #20)

Happy to help! I looked at the signing tasks and they looked a little more complicated than I could bite off. If you could move them over that would be awesome-possum.

Yes, we will have a look at it, but your bug 1525126 has more priority right now.