I would like this to be the last release of linux32. I thought it had been done in the past after discussions

0.25 download stats

    geckodriver-v0.25.0-linux32.tar.gz (1.37 MiB) - downloaded 2,414 times. Last updated on 2019-09-10
    geckodriver-v0.25.0-linux64.tar.gz (1.39 MiB) - downloaded 2,239,404 times. Last updated on 2019-09-10
    geckodriver-v0.25.0-macos.tar.gz (1.76 MiB) - downloaded 127,321 times. Last updated on 2019-09-10
    geckodriver-v0.25.0-win32.zip (1.28 MiB) - downloaded 29,655 times. Last updated on 2019-09-10
    geckodriver-v0.25.0-win64.zip (1.34 MiB) - downloaded 923,359 times. Last updated on 2019-09-10

0.24 download stats

    geckodriver-v0.24.0-linux32.tar.gz (2.78 MiB) - downloaded 45,093 times. Last updated on 2019-01-28
    geckodriver-v0.24.0-linux64.tar.gz (2.76 MiB) - downloaded 26,755,159 times. Last updated on 2019-01-28
    geckodriver-v0.24.0-macos.tar.gz (2.00 MiB) - downloaded 2,199,968 times. Last updated on 2019-01-28
    geckodriver-v0.24.0-win32.zip (3.73 MiB) - downloaded 414,485 times. Last updated on 2019-01-28
    geckodriver-v0.24.0-win64.zip (4.46 MiB) - downloaded 9,417,329 times. Last updated on 2019-01-28

The download numbers are not worth the support.

(In reply to David Burns :automatedtester from comment #1)

I would like this to be the last release of linux32. I thought it
had been done in the past after discussions

We have in the past only discussed deprecation of ARMv7 HF builds.
The intent was sent in September 2018 and we finally stopped
shipping them in 0.24.0.

The main reason to deprecate the ARM build was the extra work it
would cause us to stand up the builds on TaskCluster, but we certainly
also confirmed with the download stats that we wouldn’t be breaking
too many users.

The situation with Linux 32-bit builds is different because they
are already provided on TaskCluster. What is the justification for
turning off something that already works and which has not been a
maintenance burden?

It turns out that the 0.26.0 release should have to be released soon. As such we will only wait for the fix which removes the MSVC dependency on Windows, and the final Android patch to be landed. I will move all the other dependencies to bug 1584911, which tracks the 0.27.0 release.

I will also leave bug 1574446 as dependency for now given that we should test Andreas' patch, and get it landed.

Note that we have an unfortunate situation with the 0.26.0 release. Even with all the patches landed soon, we might not be able to release that version. Reason for that is the signing part. We announced that binaries are signed on Windows and Mac for the 0.25.0 release, but signing only works for the geckodriver binaries as built as part of the Firefox build job, but not for the toolchain job. Given that the MSVC fix only went into the latter, we would have to get bug 1577110 fixed first, or stop the signing for now and loose this new feature temporarily.

Given that not that many people have noticed the MSVC problem I would tend to delay the 0.26.0 release a bit further, and wait for bug 1577110. As I got informed from Nick on IRC he already had a discussion with Johann, and it might not take that long anymore to have signed builds available for the toolchain job.

As such I'm adding bug 1577110 as dependency. If you disagree feel free to remove it.

ok. There seemed to be a non-trivial amount of work going on around this which is what prompted my comment. If it changes then we should really evaluate it as the amount of usage is incredibly low and linux dists are removing 32 version.

(In reply to David Burns :automatedtester from comment #5)

ok. There seemed to be a non-trivial amount of work going on around this which is what prompted my comment. If it changes then we should really evaluate it as the amount of usage is incredibly low and linux dists are removing 32 version.

No, the amount of work to keep Linux 32bit was actually pretty low. Thanks to the now existing toolchain tasks it basically took me about 5min + the computation time for the try builds, which are 3min.

Nick, any crazy idea in how we could get around that problem from comment 4? If we could get the default geckodriver builds as produced by the Firefox build job to be statically linked, it would also solve the issue and we could get a release out. But not sure where the flag would have to be added.

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #7)

Nick, any crazy idea in how we could get around that problem from comment 4? If we could get the default geckodriver builds as produced by the Firefox build job to be statically linked, it would also solve the issue and we could get a release out. But not sure where the flag would have to be added.

First, let me make sure I understand the situation. Is it true that the static linking issue predates the toolchain-geckodriver tasks, or is that a regression from the toolchain-ification? If it's not a regression from that work, is it just that this is the first release that will have Mozilla-signed binaries, and therefore it's the first time we've seen this regression matter?

Second, I did some poking and it looks like it would be possible to work around the existing issue in the binaries produced during build-* tasks. We need to include -Ctarget-feature=+crt-static for geckodriver targeting Windows around here. There's an information architecture mismatch because we want this to be program-specific but the current setup doesn't allow per-program anything. But given the limited use of RUST_PROGRAMS (only geckodriver, I think!) it feels possible. Bug 1561749 is the template, essentially.

It feels like a little more investigation into signing the toolchain builds is warranted before we build a not-so-perfect thing for this use case. Do we have the option:

• Of shipping unsigned builds? If this is the first signed release, can we push that out a release?
• Of shipping signed but requiring the MSVCRT? It sounds like this is not critical, in that there's definitely a solid work-around, and the impacted community is quite small.

(In reply to Nick Alexander :nalexander [he/him] from comment #8)

First, let me make sure I understand the situation. Is it true that the static linking issue predates the toolchain-geckodriver tasks, or is that a regression from the toolchain-ification? If it's not a regression from that work, is it just that this is the first release that will have Mozilla-signed binaries, and therefore it's the first time we've seen this regression matter?

Yes, that problem exists since we started to compile geckodriver in-tree in combination with having the MSVC dependency on Rust binaries. So it's clearly not related to any kind of your work for the toolchain build jobs.

Note that we already released geckodriver 0.25.0 with the signing feature on Windows and Mac announced to the public. Releasing 0.26.0 without it would be bad (see my comment 4).

It feels like a little more investigation into signing the toolchain builds is warranted before we build a not-so-perfect thing for this use case. Do we have the option:

• Of shipping unsigned builds? If this is the first signed release, can we push that out a release?

See above. I would only do that if there is no other chance to get Windows builds decoupled from the MSVC dependency.

• Of shipping signed but requiring the MSVCRT? It sounds like this is not critical, in that there's definitely a solid work-around, and the impacted community is quite small.

Yes, so far not that many people complained about it. So it should be fine. But it's not only me who can decide that. Andreas and David, what is your take on that?

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #9)

• Of shipping signed but requiring the MSVCRT? It sounds like this is not critical, in that there's definitely a solid work-around, and the impacted community is quite small.

Yes, so far not that many people complained about it. So it should be fine. But it's not only me who can decide that. Andreas and David, what is your take on that?

If there is a small group having this problem then let's leave this as a low priority issue and if it become more prevalent then let's schedule time to fix it.

We duplicate the documentation for moz:firefoxOptions between
firefox-source-docs.m.o and MDN. This redirects users to look at MDN.

Backed out 5 changesets (Bug 1573798) for toolchain bustage.

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=270692063&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=de299b36996e7e751e2fb005744d97b0e36a718a

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=270691438&repo=autoland

Backout link: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=1ba43d20a0b2fe9076fc6e8966abe53a0e3acc09

[task 2019-10-10T15:04:46.549Z]   # Bug 1584530: don't require the Microsoft MSVC runtime to be installed.
[task 2019-10-10T15:04:46.549Z]   export RUSTFLAGS="-Ctarget-feature=+crt-static"
[task 2019-10-10T15:04:46.549Z]   ;;
[task 2019-10-10T15:04:46.549Z] # OSX cross builds are a bit harder
[task 2019-10-10T15:04:46.549Z] x86_64-apple-darwin)
[task 2019-10-10T15:04:46.549Z]   export PATH="$MOZ_FETCHES_DIR/llvm-dsymutil/bin:$PATH"
[task 2019-10-10T15:04:46.549Z]   export PATH="$MOZ_FETCHES_DIR/cctools/bin:$PATH"
[task 2019-10-10T15:04:46.549Z]   export RUSTFLAGS="-C linker=$GECKO_PATH/taskcluster/scripts/misc/osx-cross-linker"
[task 2019-10-10T15:04:46.549Z]   ;;
[task 2019-10-10T15:04:46.549Z] esac
[task 2019-10-10T15:04:46.549Z] + case "$TARGET" in
[task 2019-10-10T15:04:46.549Z] + export PATH=/builds/worker/fetches/llvm-dsymutil/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[task 2019-10-10T15:04:46.549Z] + PATH=/builds/worker/fetches/llvm-dsymutil/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[task 2019-10-10T15:04:46.549Z] + export PATH=/builds/worker/fetches/cctools/bin:/builds/worker/fetches/llvm-dsymutil/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[task 2019-10-10T15:04:46.549Z] + PATH=/builds/worker/fetches/cctools/bin:/builds/worker/fetches/llvm-dsymutil/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[task 2019-10-10T15:04:46.549Z] + export 'RUSTFLAGS=-C linker=/builds/worker/workspace/build/src/taskcluster/scripts/misc/osx-cross-linker'
[task 2019-10-10T15:04:46.549Z] + RUSTFLAGS='-C linker=/builds/worker/workspace/build/src/taskcluster/scripts/misc/osx-cross-linker'
[task 2019-10-10T15:04:46.549Z] 
[task 2019-10-10T15:04:46.549Z] export PATH="$(cd $MOZ_FETCHES_DIR && pwd)/rustc/bin:$PATH"
[task 2019-10-10T15:04:46.549Z] cd $MOZ_FETCHES_DIR && pwd)/rustc/bin:$PATH"
[task 2019-10-10T15:04:46.549Z] cd $MOZ_FETCHES_DIR && pwd)/rustc/bin:$PATH
[task 2019-10-10T15:04:46.549Z] cd $MOZ_FETCHES_DIR && pwd
[task 2019-10-10T15:04:46.549Z] ++ cd /builds/worker/fetches
[task 2019-10-10T15:04:46.549Z] ++ pwd
[task 2019-10-10T15:04:46.549Z] + export PATH=/builds/worker/fetches/rustc/bin:/builds/worker/fetches/cctools/bin:/builds/worker/fetches/llvm-dsymutil/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[task 2019-10-10T15:04:46.549Z] + PATH=/builds/worker/fetches/rustc/bin:/builds/worker/fetches/cctools/bin:/builds/worker/fetches/llvm-dsymutil/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[task 2019-10-10T15:04:46.549Z] 
[task 2019-10-10T15:04:46.549Z] cd $GECKO_PATH/testing/geckodriver
[task 2019-10-10T15:04:46.549Z] + cd /builds/worker/workspace/build/src/testing/geckodriver
[task 2019-10-10T15:04:46.549Z] 
[task 2019-10-10T15:04:46.549Z] cp $GECKO_PATH/.cargo/config.in $GECKO_PATH/.cargo/config
[task 2019-10-10T15:04:46.549Z] + cp /builds/worker/workspace/build/src/.cargo/config.in /builds/worker/workspace/build/src/.cargo/config
[task 2019-10-10T15:04:46.550Z] 
[task 2019-10-10T15:04:46.550Z] cargo build --frozen --verbose --release --target "$TARGET"
[task 2019-10-10T15:04:46.550Z] + cargo build --frozen --verbose --release --target x86_64-apple-darwin
[task 2019-10-10T15:04:46.991Z] error: the lock file /builds/worker/workspace/build/src/Cargo.lock needs to be updated but --frozen was passed to prevent this
[fetches 2019-10-10T15:04:46.994Z] removing /builds/worker/fetches
[fetches 2019-10-10T15:04:47.850Z] finished
[taskcluster 2019-10-10 15:04:50.828Z] === Task Finished ===
[taskcluster 2019-10-10 15:04:51.502Z] Unsuccessful task run with exit code: 101 completed in 127.966 seconds

Cargo.lock file was not updated.

https://hg.mozilla.org/mozilla-central/rev/ac0d7d1920f8
https://hg.mozilla.org/mozilla-central/rev/7e45efb434c4
https://hg.mozilla.org/mozilla-central/rev/f749dac93775
https://hg.mozilla.org/mozilla-central/rev/2561e326e39b
https://hg.mozilla.org/mozilla-central/rev/030c7f016214

It's not done yet. We still have to do the release.

Andreas, before we release 0.26.0 please make sure to have a look at https://github.com/mozilla/geckodriver/issues/1629. As it looks like on Catalina users have problems with allowing MacOS to check for malicious software. Could this be related to our signed binaries?

If we are notorized and signed we should be fine. This is going to be the same problem that Firefox will have. Has someone from releng checked that it is notorized?

We should release 0.26.0 despite the binary on macOS not being
notarised, then follow up with a new release once we have notarisation
sorted.

The situation is the same if macOS try to download earlier releases
of geckodriver.

I thought it was done in https://bugzilla.mozilla.org/show_bug.cgi?id=1546299 which has a dep on OSX notarization.

Exactly that's why I'm a bit scared what's going on. The signed builds should be notarized. The only thing I could see is that maybe we haven't uploaded the signed macOS binary to the Github releases page? Andreas, it would be great if you could have at least a look at it. If the above assumption might be the reason, please take care and upload the signed builds and not the unsigned.

The notarization problems are now covered by bug 1588081. As it looks like it's a problem on the releng side. Maybe if they get the code changes which are part of a different repository pushed to production, we only would have to rebuilt the geckodriver signing job via Treeherder afterward. As such I would propose to just wait a bit with the release, or we can follow-up and replace the signed Mac build on the Github releases page?

I verified that even the signed builds are affected by the notarisation
issue. The 0.26.0 signed builds still fail to launch on macOS
Catalina. We should follow up in bug 1588081.

As we can’t make any binary modifications to the 0.26.0 release, I
would still suggest that we proceed with publishing the release
binaries and follow up with a point release containing a Catalina
fix.

I published all the signed release packages of geckodriver 0.26.0 from e9783a644016 but I could only find a Windows 2012 opt job with the signature Windows 2012 opt Geckodriver geckodriver-signing-win32-nightly/opt Gd(s). I uploaded the Windows 32-bit .zip package, but where do I find the 64-bit package?

The download stats of 0.25.0 (previous version) were:

    geckodriver-v0.25.0-linux32.tar.gz (1.37 MiB) - downloaded 5,058 times. Last updated on 2019-09-10
    geckodriver-v0.25.0-linux64.tar.gz (1.39 MiB) - downloaded 4,853,425 times. Last updated on 2019-09-10
    geckodriver-v0.25.0-macos.tar.gz (1.76 MiB) - downloaded 262,610 times. Last updated on 2019-09-10
    geckodriver-v0.25.0-win32.zip (1.28 MiB) - downloaded 59,295 times. Last updated on 2019-09-10
    geckodriver-v0.25.0-win64.zip (1.34 MiB) - downloaded 1,956,103 times. Last updated on 2019-09-10

For the reason that win64 is a prominent platform, I’ve marked 0.26.0 as a ‘pre-release’ on GitHub until we’ve resolved this. I don’t know if this makes an actual difference.

(In reply to Andreas Tolfsen 「:ato」 from comment #28)

As we can’t make any binary modifications to the 0.26.0 release, I
would still suggest that we proceed with publishing the release
binaries and follow up with a point release containing a Catalina
fix.

As I noted above we wouldn't have to. All the bits and pieces which needed are in a different repository. Once that other patch lands, and get published, we would only have to re-run the signing job for MacOS. We could than take the newly generated binary.

Maybe we can just replace the binary of the 0.26.0 release on the Github release page once the issue has been fixed? Given that there are no code changes, we would still have the exact same bits in it. But not sure if Github allows that.

(In reply to Andreas Tolfsen 「:ato」 from comment #29)

I published all the signed release packages of geckodriver 0.26.0 from e9783a644016 but I could only find a Windows 2012 opt job with the signature Windows 2012 opt Geckodriver geckodriver-signing-win32-nightly/opt Gd(s). I uploaded the Windows 32-bit .zip package, but where do I find the 64-bit package?

Not sure why those weren't running. If that happens again we should report a bug. For now I triggered the win64 repack and signing job but just using the "Add new jobs" feature of Treeherder. The signed builds can be found here:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=geckodriver&revision=e9783a644016aa9b317887076618425586730d73&selectedJob=270989977

I'm going to add the win64 binary now, and also will add the note about the MSCVRT dependency again which will be necessary for windows users.

Also I have found bug 1442253 which is about the size of the binaries on Linux. As we agreed on already for the 0.25.0 release we don't want to ship signed builds on Linux yet. Instead the binaries from the toolchain jobs have to be used. I have triggered the necessary jobs respectively now, and will replace the binaries once I'm back online.

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #31)

As I noted above we wouldn't have to. All the bits and pieces
which needed are in a different repository. Once that other patch
lands, and get published, we would only have to re-run the signing
job for MacOS. We could than take the newly generated binary.

Maybe we can just replace the binary of the 0.26.0 release on the
Github release page once the issue has been fixed? Given that
there are no code changes, we would still have the exact same bits
in it. But not sure if Github allows that.

OK, good!

That will inevitably change the binary, but since we don’t yet
publish signature files for macOS I suppose that would be acceptable.

GitHub does allow us replace the attached assets.

(In reply to Andreas Tolfsen 「:ato」 from comment #29)

I published all the signed release packages of geckodriver
0.26.0 from e9783a644016 but I could only find a Windows 2012 opt job with the signature Windows 2012 opt Geckodriver geckodriver-signing-win32-nightly/opt Gd(s).
I uploaded the Windows 32-bit .zip package, but where
do I find the 64-bit package?

Not sure why those weren't running. If that happens again we
should report a bug. For now I triggered the win64 repack
and signing job but just using the "Add new jobs" feature of
Treeherder.

Thanks! 🙌🏻

Let’s keep an eye on Gd(s) in the future.

https://github.com/mozilla/geckodriver/releases/tag/v0.26.0

This is still not fixed yet. The binaries for linux 32/64 still have to be replaced with those from the toolchain jobs!

I would really suggest that we update the releasing docs to not having such a faulty release process.

(In reply to Andreas Tolfsen 「:ato」 from comment #35)

https://github.com/mozilla/geckodriver/releases/tag/v0.26.0

I wouldn't have expected that you already release it! Will you do the binary replacements for Linux? Turns out I do not have that much time here at the airport.

I finally pushed the correct Linux binaries for both 32 bit and 64 bit, removed the hash files because those are for different binaries, and marked it as release again. I hope that we didn't confuse users and CI systems that much.

We will later decide what to do with MacOS.

Maybe in the future Andreas can update the Changes.md file, and I will do all the releases bits given that I'm more familiar with all the latest changes and workarounds?

Actually a notification has to be sent to the tooling mailing list. Andreas, will you do that?

Thanks for the reminder, but I had this prepared in my drafts folder
ready for despatching today 😉

https://lists.mozilla.org/pipermail/tools-marionette/2019-October/000116.html