..including the special-casing of https://taskcluster.net.

There are only 78 occurrences of `taskcluster.net` in the Gecko tree. This might be a walk in the park! I'll break fixing those down into microcommits.

Pete, many of these are cases where the code is making an externally-usable URL, while there are many cases (not showing up in this gist) that use the taskcluster proxy. With respect to bug 1460015, the process sort of needs both rootUrls then: `http://taskcluster` (for the proxy) and `https://firefox-taskcluster.prod.mozaws.net` (for the externally-valid URLs). What do you think about workers providing *two* environment variables to convey this information? Maybe TASKCLUSTER_ROOT_URL (http://taskcluster) and TASKCLUSTER_PUBLIC_ROOT_URL (https://firefox-taskcluster.prod.mozaws.net)? The other option is to only provide TASKCLUSTER_ROOT_URL=https://firefox-taskcluster.prod.mozaws.net and require tasks that use the proxy to override that value when using the proxy (in which case, do we provide the appropriate URL in another env var, or does everyone just hard-code http://taskcluster?) I'm not sure which is best - what do you think?

Would the ROOT_URL be something that it'd be reasonable for the worker to provide? And, if that is the case, are there any cases where a task reasonably wants to use both the public and private URL? If not, the worker could provide one or the other, depending on whether the task has the proxy enabled. Even if so, it seems reasonable to provide both to the task. That way, the in-tree code can be used without change on multiple clusters (production and staging?).

> Would the ROOT_URL be something that it'd be reasonable for the worker to provide? I think so. > And, if that is the case, are there any cases where a task reasonably wants to use both the public and private URL? Sure, the decision task both wants to create tasks (using the proxy) and include public URLs in descriptions, etc. > If not, the worker could provide one or the other, depending on whether the task has the proxy enabled. Even if so, it seems reasonable to provide both to the task. That way, the in-tree code can be used without change on multiple clusters (production and staging?). That's the ultimate goal, is that a task graph will easily run in either cluster.

(In reply to Dustin J. Mitchell [:dustin] pronoun: he from comment #5) > > And, if that is the case, are there any cases where a task reasonably wants to use both the public and private URL? > > Sure, the decision task both wants to create tasks (using the proxy) and > include public URLs in descriptions, etc. My first reaction to this, is that we'd probably want to have the created tasks use the environment variables as much as possible. However, on reflection, there are probably lots of places where this isn't feasible, or sensible.

An example of a place that we can't use an env var is in the `mounts` property of generic-worker tasks -- those must be fully qualified URLs when the task is created. Is that what you mean?

Oops, that's a terrible example, since generic-worker takes a taskId. We could also change things to be able to take taskIds and paths, instead of installer URLs. Per discussion with Pete, we're going to go with TASKCLUSTER_ROOT_URL and TASKCLUSTER_PROXY_URL. The latter is only set when running in a task with a proxy, and must be used intentionally (and without credentials). Pete is thinking about whether it makes sense to add support for this variable to clients, using the proxy by default if set. But that doesn't block this particular project.

Mike, is the workaround landed in bug 1433033 to un-urlescape Location headers still required? It's a bit trickier in a redeployability scenario since it's not clear what the redirect would be (we are probably not redeploying cloud-mirror, so it's even more than just a rootUrl problem). If it is still an issue, is it practical to consider just re-escaping a portion of the path (regardless of the URL)? Or, could we lean on upstream to fix the bug? Thanks!

and while I have you Mike, any idea what this error means? https://taskcluster-artifacts.net/TBSZsIM1Tju_2F8mCA6m_A/0/public/logs/live_backing.log it looks like it couldn't reach snapshots over IPv6, which makes sense since EC2 doesn't support v6. Any idea how my changeset would have led to that?

(In reply to Dustin J. Mitchell [:dustin] pronoun: he from comment #13) > and while I have you Mike, any idea what this error means? > > https://taskcluster-artifacts.net/TBSZsIM1Tju_2F8mCA6m_A/0/public/logs/ > live_backing.log > it looks like it couldn't reach snapshots over IPv6, which makes sense since > EC2 doesn't support v6. Any idea how my changeset would have led to that? That happens sometimes. If EC2 doesn't support ipv6, we should disable ipv6 at the kernel level, then. (In reply to Dustin J. Mitchell [:dustin] pronoun: he from comment #11) > Mike, is the workaround landed in bug 1433033 to un-urlescape Location > headers still required? It's a bit trickier in a redeployability scenario > since it's not clear what the redirect would be (we are probably not > redeploying cloud-mirror, so it's even more than just a rootUrl problem). > > If it is still an issue, is it practical to consider just re-escaping a > portion of the path (regardless of the URL)? Or, could we lean on upstream > to fix the bug? Thanks! I don't know. Try reverting the non-debian_package.py parts of 3bebaac29a1587e940136857dbf9fe6dae6ecffe and see what happens? That said, the workaround is a no-op if there's no redirection to cloud-mirror, so...

(In reply to Dustin J. Mitchell [:dustin] pronoun: he from comment #13) > and while I have you Mike, any idea what this error means? > > https://taskcluster-artifacts.net/TBSZsIM1Tju_2F8mCA6m_A/0/public/logs/ > live_backing.log > it looks like it couldn't reach snapshots over IPv6, which makes sense since > EC2 doesn't support v6. Any idea how my changeset would have led to that? https://phabricator.services.mozilla.com/D7365

Notes to self: this is ready for experimentation, but still failing with https://taskcluster-artifacts.net/ALFKJKt7Tou3lkgGO6vQdQ/0/public/logs/live_backing.log I'm guessing because the existing image_builder image doesn't set TASKCLUSTER_ROOT_URL.

^^ rebased, linted, etc. Fingers crossed!

In deb7-base: [task 2018-12-10T22:37:49.158Z] executing ['sh', '-x', '-c', ' /builds/worker/checkouts/gecko/mach taskcluster-build-image -t "debian7-base:90ba1640122af940143803aa3aac2f342617e3b720266d87da89d2907fc46dcf" "debian7-base"'] [task 2018-12-10T22:37:49.160Z] + /builds/worker/checkouts/gecko/mach taskcluster-build-image -t debian7-base:90ba1640122af940143803aa3aac2f342617e3b720266d87da89d2907fc46dcf debian7-base [task 2018-12-10T22:37:49.623Z] Step 0 : FROM debian:$DIST-$BASE_TAG [task 2018-12-10T22:37:49.938Z] Pulling repository debian [task 2018-12-10T22:38:15.785Z] Tag $DIST-$BASE_TAG not found in repository debian also, the docker image hashes seem to be different in CI than when run manually..

`./mach vendor python taskcluster-urls==11.0.0`.

Eventually, workers will provide these variables directly (https://bugzilla.mozilla.org/show_bug.cgi?id=1460015). But for now, this ensures that TASKCLUSTER_ROOT_URL is set everywhere, and TASKCLUSTER_PROXY_URL is set wherever the proxy is active. The setup for the mach commands defaults to https://taskcluster.net for user convenience. When the production instance's URL changes, we can simply change that default. This changes the docker build process to propagate TASKCLUSTER_ROOT_URL into the docker images, and for good measure includes some code to use that value to generate debian repo paths.

This provides an easy way to encode an artifact URL in static data such as taskcluster/ci/nightly-l10n/kind.yml. This could be used in mozharness_test.py, for example, as well -- but other code (such as to support backfilling) expects `task-reference` there. To avoid breaking such subtle bits, those can continue using `task-reference` with URLs generated based on TASKCLUSTER_ROOT_URL.

This involves changing the calling signature (dropping some unused features) and updating all three call sites to match. The function implementation is also modified to use get_artifact_url.

Pushed by dmitchell@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/85d7f8b330eb vendor taskcluster-urls r=firefox-build-system-reviewers,gps https://hg.mozilla.org/integration/autoland/rev/6dc9522ee0bf set TASKCLUSTER_ROOT_URL and TASKCLUSTER_PROXY_URL; r=tomprince,glandium https://hg.mozilla.org/integration/autoland/rev/b93a0fcc86f3 add {artifact-reference: ..} r=tomprince https://hg.mozilla.org/integration/autoland/rev/0c4101230d4d use {artifact-reference: ..} in diffoscope; r=glandium https://hg.mozilla.org/integration/autoland/rev/fb7cfca6ebc3 use {artifact-reference: ..} for symbol uploads; r=ted https://hg.mozilla.org/integration/autoland/rev/5c92bb5ac895 Replace uses of `get_taskcluster_artifact_url` with `artifact-reference`; r=dustin https://hg.mozilla.org/integration/autoland/rev/d3dfbd848236 use taskcluster-urls to create taskcluster URLs r=tomprince https://hg.mozilla.org/integration/autoland/rev/c8b7e620eabf use taskcluster_urls to generate interactive URL r=bstack https://hg.mozilla.org/integration/autoland/rev/3017e5890739 move list_task_group to taskgraph.taskcluster.util r=bstack https://hg.mozilla.org/integration/autoland/rev/e1ebad5d89c5 update libdmg-hfsplus to use TASKCLUSTER_ROOT_URL; r=glandium https://hg.mozilla.org/integration/autoland/rev/5eb2c77d70a9 update fetch-content to use TASKCLUSTER_ROOT_URL; r=tomprince https://hg.mozilla.org/integration/autoland/rev/0b229b00818a update diffoscope tasks to use TASKCLUSTER_ROOT_URL; r=glandium https://hg.mozilla.org/integration/autoland/rev/90f3faa36137 update pipfile-updates to use TASKCLUSTER_ROOT_URL; r=tomprince https://hg.mozilla.org/integration/autoland/rev/6d17291b8b92 update funsize scripts to use TASKCLUSTER_ROOT_URL; r=sfraser https://hg.mozilla.org/integration/autoland/rev/f4945658d45f update periodic-updates to use TASKCLUSTER_ROOT_URL; r=sfraser https://hg.mozilla.org/integration/autoland/rev/31e500489665 generate portable URLs for Android mach commands; r=nalexander

https://hg.mozilla.org/mozilla-central/rev/85d7f8b330eb https://hg.mozilla.org/mozilla-central/rev/6dc9522ee0bf https://hg.mozilla.org/mozilla-central/rev/b93a0fcc86f3 https://hg.mozilla.org/mozilla-central/rev/0c4101230d4d https://hg.mozilla.org/mozilla-central/rev/fb7cfca6ebc3 https://hg.mozilla.org/mozilla-central/rev/5c92bb5ac895 https://hg.mozilla.org/mozilla-central/rev/d3dfbd848236 https://hg.mozilla.org/mozilla-central/rev/c8b7e620eabf https://hg.mozilla.org/mozilla-central/rev/3017e5890739 https://hg.mozilla.org/mozilla-central/rev/e1ebad5d89c5 https://hg.mozilla.org/mozilla-central/rev/5eb2c77d70a9 https://hg.mozilla.org/mozilla-central/rev/0b229b00818a https://hg.mozilla.org/mozilla-central/rev/90f3faa36137 https://hg.mozilla.org/mozilla-central/rev/6d17291b8b92 https://hg.mozilla.org/mozilla-central/rev/f4945658d45f https://hg.mozilla.org/mozilla-central/rev/31e500489665

Pushed by mozilla@jorgk.com: https://hg.mozilla.org/comm-central/rev/3409b3f717ea Port bug 1492664 - set TASKCLUSTER_ROOT_URL and TASKCLUSTER_PROXY_URL. rs=bustage-fix

This is breaking developer artifact builds - they are failing to find the revisions because the root url isn't set: ====== 0:20.32 Attempting to find a pushhead containing 204ab379fb829827a82efd9ef72bb18acdcf42f6 on mozilla-central. 0:21.10 Attempting to find a pushhead containing 204ab379fb829827a82efd9ef72bb18acdcf42f6 on integration/mozilla-inbound. 0:21.96 Attempting to find a pushhead containing 204ab379fb829827a82efd9ef72bb18acdcf42f6 on releases/mozilla-beta. 0:23.16 Retrieving the last 50 pushheads starting with id 110633 on integration/mozilla-inbound 0:24.02 Retrieving the last 50 pushheads starting with id 35230 on mozilla-central 0:24.77 Tried 70 pushheads, no built artifacts found. 0:24.79 make[3]: *** [recurse_artifact] Error 1 ====== If I do: TASKCLUSTER_ROOT_URL=https://taskcluster.net ./mach build then it works correctly.

Backed out 16 changesets (Bug 1492664) for breaking developer artifact builds, requested by standard8 a=backout Backout push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception%2Cusercancel%2Crunnable&revision=28f92797e661694da8f10e3a9a8fbcfff49c919c

I suspect that backout is going to break things even worse :)

There was an issue with the Ns jobs on the backout, but I don't see any more discussion in irc so I guess that's not a showstopper. So there's no great rush to reland thus.

https://hg.mozilla.org/integration/mozilla-inbound/rev/0699d3873e440453ce7d5cc7c398a1d076b92033 Bug 1492664 - vendor taskcluster-urls; r=gps https://hg.mozilla.org/integration/mozilla-inbound/rev/5ea6f03f845e49d503f5d0283557f54561c41654 Bug 1492664 - set TASKCLUSTER_ROOT_URL and TASKCLUSTER_PROXY_URL; r=tomprince,glandium https://hg.mozilla.org/integration/mozilla-inbound/rev/a972d6b4434edcbce1f93fb1629a6b81ca7bb585 Bug 1492664 - set TASKCLUSTER_ROOT_URL for mach build, too; r=froydnj https://hg.mozilla.org/integration/mozilla-inbound/rev/9c35dd209c6b407bc3a45ce7b4c27272ef1bb486 Bug 1492664 - add {artifact-reference: ..}; r=tomprince https://hg.mozilla.org/integration/mozilla-inbound/rev/c77ae59aba41a65f0e9d4896f9dfe3bf8a0e8353 Bug 1492664 - use {artifact-reference: ..} for symbol uploads; r=ted https://hg.mozilla.org/integration/mozilla-inbound/rev/c566f1c8dcdfe6fdf73b0c0d8da66a14a7bfec08 Bug 1492664 - use {artifact-reference: ..} for l10n; r=aki https://hg.mozilla.org/integration/mozilla-inbound/rev/335a92b1f42444dbcc4057f14bafab7f73f59e66 Bug 1492664 - use {artifact-reference: ..} in diffoscope; r=glandium https://hg.mozilla.org/integration/mozilla-inbound/rev/e7f30540870849cf6ba49957d37190fbd89b5f0d Bug 1492664: Replace uses of `get_taskcluster_artifact_url` with `artifact-reference`; r=dustin https://hg.mozilla.org/integration/mozilla-inbound/rev/29f33f22fd8b6998276e10c10aac2e36e64af4bb Bug 1492664 - use taskcluster-urls to create taskcluster URLs; r=tomprince https://hg.mozilla.org/integration/mozilla-inbound/rev/77699b51336b804336222192734922f76ea7ac10 Bug 1492664 - use taskcluster_urls to generate interactive URL; r=bstack https://hg.mozilla.org/integration/mozilla-inbound/rev/870c8cec99e5f81814a8064a785c4c96f5f1aeca Bug 1492664 - move list_task_group to taskgraph.taskcluster.util; r=bstack https://hg.mozilla.org/integration/mozilla-inbound/rev/911b4b0fb6835fc7d0caaea5d64a590563ca3104 Bug 1492664 - update libdmg-hfsplus to use TASKCLUSTER_ROOT_URL; r=glandium https://hg.mozilla.org/integration/mozilla-inbound/rev/742b038bb1dd39c029afce73eb3f5b683cb590f2 Bug 1492664 - update fetch-content to use TASKCLUSTER_ROOT_URL; r=tomprince https://hg.mozilla.org/integration/mozilla-inbound/rev/4c63a04fdd47c7cb8dc7350ae7fd6d861b525a2d Bug 1492664 - update diffoscope tasks to use TASKCLUSTER_ROOT_URL; r=glandium https://hg.mozilla.org/integration/mozilla-inbound/rev/c1ea4a10cc8d3631d61c1172624373e3901debd7 Bug 1492664 - update pipfile-updates to use TASKCLUSTER_ROOT_URL; r=tomprince https://hg.mozilla.org/integration/mozilla-inbound/rev/06c466ab432372eb645f07ca1ff215052492b358 Bug 1492664 - update funsize scripts to use TASKCLUSTER_ROOT_URL; r=sfraser https://hg.mozilla.org/integration/mozilla-inbound/rev/d9a7f2ce49c3634baab1535315f9ba9f8f3b14ea Bug 1492664 - update periodic-updates to use TASKCLUSTER_ROOT_URL; r=sfraser https://hg.mozilla.org/integration/mozilla-inbound/rev/bf6d089640eb8c79b183129186cc936ac854a0bf Bug 1492664 - generate portable URLs for Android mach commands; r=nalexander https://hg.mozilla.org/integration/mozilla-inbound/rev/c82285d253de952fe6388e2ec5060057e2e8e80b Bug 1492664 - update references in docs; r=froydnj https://hg.mozilla.org/integration/mozilla-inbound/rev/2d876c4ece8b472b4c51d2240626974e96929a6e Bug 1492664 - fix list_task_group to use GET; r=Callek

https://hg.mozilla.org/integration/mozilla-inbound/rev/a7d50dbb2c8e7297b586c366c5a57d2a22e24132 Bug 1492664 - use correct path in l10n artifact-references; a=bustage CLOSED TREE

https://hg.mozilla.org/mozilla-central/rev/0699d3873e44 https://hg.mozilla.org/mozilla-central/rev/5ea6f03f845e https://hg.mozilla.org/mozilla-central/rev/a972d6b4434e https://hg.mozilla.org/mozilla-central/rev/9c35dd209c6b https://hg.mozilla.org/mozilla-central/rev/c77ae59aba41 https://hg.mozilla.org/mozilla-central/rev/c566f1c8dcdf https://hg.mozilla.org/mozilla-central/rev/335a92b1f424 https://hg.mozilla.org/mozilla-central/rev/e7f305408708 https://hg.mozilla.org/mozilla-central/rev/29f33f22fd8b https://hg.mozilla.org/mozilla-central/rev/77699b51336b https://hg.mozilla.org/mozilla-central/rev/870c8cec99e5 https://hg.mozilla.org/mozilla-central/rev/911b4b0fb683 https://hg.mozilla.org/mozilla-central/rev/742b038bb1dd https://hg.mozilla.org/mozilla-central/rev/4c63a04fdd47 https://hg.mozilla.org/mozilla-central/rev/c1ea4a10cc8d https://hg.mozilla.org/mozilla-central/rev/06c466ab4323 https://hg.mozilla.org/mozilla-central/rev/d9a7f2ce49c3 https://hg.mozilla.org/mozilla-central/rev/bf6d089640eb https://hg.mozilla.org/mozilla-central/rev/c82285d253de https://hg.mozilla.org/mozilla-central/rev/2d876c4ece8b https://hg.mozilla.org/mozilla-central/rev/a7d50dbb2c8e

[Tracking Requested - why for this release]:

Backout by archaeopteryx@coole-files.de: https://hg.mozilla.org/mozilla-central/rev/d0e13414d651 Backed out 21 changesets for breaking cron task for nightlies. a=backout

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=79cf24341024b84654bb33561e929322afd80aae&selectedJob=218088766 Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=218088766&repo=mozilla-central&lineNumber=2598 Backout: https://hg.mozilla.org/mozilla-central/rev/d0e13414d6512c9fe84911a0dd730e4fb4a28c27

https://hg.mozilla.org/integration/mozilla-inbound/rev/cc63b775369504c9299a88d9649e37f3f1d4c792 Bug 1492664 - vendor taskcluster-urls; r=gps https://hg.mozilla.org/integration/mozilla-inbound/rev/bdaa57b4a2fdbc596f61ec77a42c4322ef9b48c1 Bug 1492664 - set TASKCLUSTER_ROOT_URL and TASKCLUSTER_PROXY_URL; r=tomprince,glandium https://hg.mozilla.org/integration/mozilla-inbound/rev/5a7bea3fb23b1c208725be620aa881645af40f8a Bug 1492664 - add {artifact-reference: ..}; r=tomprince https://hg.mozilla.org/integration/mozilla-inbound/rev/edc2a5ed02cc62e0e6c5411e7160a4fd51415cbe Bug 1492664 - use {artifact-reference: ..} for symbol uploads; r=ted https://hg.mozilla.org/integration/mozilla-inbound/rev/aca20c673dd548d73399df4778faac77b722c44a Bug 1492664 - use {artifact-reference: ..} for l10n; r=aki https://hg.mozilla.org/integration/mozilla-inbound/rev/fdbc5b9746a2c417fbb3a27d96475e92dce7d2db Bug 1492664 - use {artifact-reference: ..} in diffoscope; r=glandium https://hg.mozilla.org/integration/mozilla-inbound/rev/943f04bc1c706c5b1422bb746ef9acba7ecf285e Bug 1492664: Replace uses of `get_taskcluster_artifact_url` with `artifact-reference`; r=dustin https://hg.mozilla.org/integration/mozilla-inbound/rev/d3efbfd8e16250141fd58a9383678bbcf0ae6615 Bug 1492664 - use taskcluster-urls to create taskcluster URLs; r=tomprince https://hg.mozilla.org/integration/mozilla-inbound/rev/1786aeb1e5c5227104c40dde4049f247071050d1 Bug 1492664 - use taskcluster_urls to generate interactive URL; r=bstack https://hg.mozilla.org/integration/mozilla-inbound/rev/7433ce13608181f6e2738127c6fd0ab572109526 Bug 1492664 - move list_task_group to taskgraph.taskcluster.util; r=bstack https://hg.mozilla.org/integration/mozilla-inbound/rev/aa78a567e81e3405b2eaf24ad2132f837c0f14fc Bug 1492664 - update libdmg-hfsplus to use TASKCLUSTER_ROOT_URL; r=glandium https://hg.mozilla.org/integration/mozilla-inbound/rev/fbc8bc72b14ef0ca4b051b767f766535271edc44 Bug 1492664 - update fetch-content to use TASKCLUSTER_ROOT_URL; r=tomprince https://hg.mozilla.org/integration/mozilla-inbound/rev/5f370da981050137d554e1d7d356a47606d158d0 Bug 1492664 - update diffoscope tasks to use TASKCLUSTER_ROOT_URL; r=glandium https://hg.mozilla.org/integration/mozilla-inbound/rev/9bedcff5f5f4157e7f8c4e3cb97fc9d395bbaa0a Bug 1492664 - update pipfile-updates to use TASKCLUSTER_ROOT_URL; r=tomprince https://hg.mozilla.org/integration/mozilla-inbound/rev/f61ad7db75d493a04804eb9d797016a36d6e91a4 Bug 1492664 - update funsize scripts to use TASKCLUSTER_ROOT_URL; r=sfraser https://hg.mozilla.org/integration/mozilla-inbound/rev/1f7b787349b838850fc4b73e29191efbf564d483 Bug 1492664 - update periodic-updates to use TASKCLUSTER_ROOT_URL; r=sfraser https://hg.mozilla.org/integration/mozilla-inbound/rev/3808fda372b11df2e3db56c763b9364215f5fefe Bug 1492664 - generate portable URLs for Android mach commands; r=nalexander https://hg.mozilla.org/integration/mozilla-inbound/rev/9dfd14d495fecce69c0b500787a21c16d25322e9 Bug 1492664 - update references in docs; r=froydnj

(landed to inbound since autoland couldn't rebase but gave no useful output, but there were no conflicts on inbound)

https://hg.mozilla.org/mozilla-central/rev/cc63b7753695 https://hg.mozilla.org/mozilla-central/rev/bdaa57b4a2fd https://hg.mozilla.org/mozilla-central/rev/5a7bea3fb23b https://hg.mozilla.org/mozilla-central/rev/edc2a5ed02cc https://hg.mozilla.org/mozilla-central/rev/aca20c673dd5 https://hg.mozilla.org/mozilla-central/rev/fdbc5b9746a2 https://hg.mozilla.org/mozilla-central/rev/943f04bc1c70 https://hg.mozilla.org/mozilla-central/rev/d3efbfd8e162 https://hg.mozilla.org/mozilla-central/rev/1786aeb1e5c5 https://hg.mozilla.org/mozilla-central/rev/7433ce136081 https://hg.mozilla.org/mozilla-central/rev/aa78a567e81e https://hg.mozilla.org/mozilla-central/rev/fbc8bc72b14e https://hg.mozilla.org/mozilla-central/rev/5f370da98105 https://hg.mozilla.org/mozilla-central/rev/9bedcff5f5f4 https://hg.mozilla.org/mozilla-central/rev/f61ad7db75d4 https://hg.mozilla.org/mozilla-central/rev/1f7b787349b8 https://hg.mozilla.org/mozilla-central/rev/3808fda372b1 https://hg.mozilla.org/mozilla-central/rev/9dfd14d495fe