Filed by: dvarga [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=249560102&repo=mozilla-central
Full log: https://queue.taskcluster.net/v1/task/KWfd0nF6ScK8bCRgfRTqYw/runs/2/artifacts/public/logs/live_backing.log

[task 2019-06-01T12:55:51.065Z] 12:55:51     INFO -  Allowed: Path "classes.jar!/org/mozilla/gecko/util/HardwareUtils.class" has architecture-specific versions:
[task 2019-06-01T12:55:51.065Z] 12:55:51     INFO -    arm64-v8a, x86_64 -> 08fa78d2df2892d19e41093b6a62b5c4f26bd331
[task 2019-06-01T12:55:51.065Z] 12:55:51     INFO -    armeabi-v7a, x86 -> e736af50bcd8e6911a03efd21c8d64f71dd49f7c
[task 2019-06-01T12:55:51.065Z] 12:55:51     INFO -  Allowed: Path "classes.jar!/org/mozilla/geckoview/BuildConfig.class" has architecture-specific versions:
[task 2019-06-01T12:55:51.066Z] 12:55:51     INFO -    arm64-v8a, x86_64 -> 7a2c07b51d70508f565168967aae7e9839a65c01
[task 2019-06-01T12:55:51.066Z] 12:55:51     INFO -    armeabi-v7a, x86 -> f7903879967f89626e18609b3ca045f261ab5415
[task 2019-06-01T12:55:51.066Z] 12:55:51     INFO -  Disallowed: Path "localization/dsb/toolkit/about/aboutSupport.ftl" has architecture-specific versions:
[task 2019-06-01T12:55:51.066Z] 12:55:51     INFO -    arm64-v8a, armeabi-v7a -> 29571e7a0dc499980118d72ada3c06f052383aeb
[task 2019-06-01T12:55:51.066Z] 12:55:51     INFO -    x86, x86_64 -> 636213355b8bef8d6679b96028069b81585b510f
[task 2019-06-01T12:55:51.066Z] 12:55:51     INFO -  Makefile:178: recipe for target 'recurse_android-fat-aar-artifact' failed
[task 2019-06-01T12:55:51.066Z] 12:55:51    ERROR -  make[3]: *** [recurse_android-fat-aar-artifact] Error 1
[task 2019-06-01T12:55:51.066Z] 12:55:51     INFO -  make[3]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2019-06-01T12:55:51.066Z] 12:55:51     INFO -  /builds/worker/workspace/build/src/config/recurse.mk:32: recipe for target 'android-fat-aar-artifact' failed
[task 2019-06-01T12:55:51.066Z] 12:55:51    ERROR -  make[2]: *** [android-fat-aar-artifact] Error 2
[task 2019-06-01T12:55:51.066Z] 12:55:51     INFO -  /builds/worker/workspace/build/src/config/rules.mk:400: recipe for target 'default' failed
[task 2019-06-01T12:55:51.066Z] 12:55:51    ERROR -  make[1]: *** [default] Error 2
[task 2019-06-01T12:55:51.067Z] 12:55:51     INFO -  client.mk:125: recipe for target 'build' failed
[task 2019-06-01T12:55:51.067Z] 12:55:51     INFO -  make: *** [build] Error 2
[task 2019-06-01T12:55:51.076Z] 12:55:51     INFO -  0 compiler warnings present.
[task 2019-06-01T12:55:51.124Z] 12:55:51    ERROR - Return code: 2
[task 2019-06-01T12:55:51.124Z] 12:55:51  WARNING - setting return code to 2
[task 2019-06-01T12:55:51.124Z] 12:55:51    FATAL - 'mach build -v' did not run successfully. Please check log for errors.
[task 2019-06-01T12:55:51.124Z] 12:55:51    FATAL - Running post_fatal callback...
[task 2019-06-01T12:55:51.124Z] 12:55:51    FATAL - Exiting -1
[task 2019-06-01T12:55:51.124Z] 12:55:51     INFO - [mozharness: 2019-06-01 12:55:51.124751Z] Finished build step (failed)
[task 2019-06-01T12:55:51.124Z] 12:55:51     INFO - Running post-run listener: _parse_build_tests_ccov
[task 2019-06-01T12:55:51.124Z] 12:55:51     INFO - Running post-run listener: _shutdown_sccache
[task 2019-06-01T12:55:51.124Z] 12:55:51     INFO - Running post-run listener: _summarize
[task 2019-06-01T12:55:51.125Z] 12:55:51    ERROR - # TBPL FAILURE #
[task 2019-06-01T12:55:51.125Z] 12:55:51     INFO - [mozharness: 2019-06-01 12:55:51.125020Z] FxDesktopBuild summary:
[task 2019-06-01T12:55:51.125Z] 12:55:51    ERROR - # TBPL FAILURE #
[taskcluster 2019-06-01 12:55:51.603Z] === Task Finished ===
[taskcluster 2019-06-01 12:55:51.725Z] Artifact "public/build/geckoview_example.apk" not found at "/builds/worker/workspace/build/src/obj-firefox/gradle/build/mobile/android/geckoview_example/outputs/apk/withGeckoBinaries/debug/geckoview_example-withGeckoBinaries-debug.apk"
[taskcluster 2019-06-01 12:55:51.929Z] Artifact "public/build/target.maven.zip" not found at "/builds/worker/workspace/build/src/obj-firefox/gradle/build/mobile/android/geckoview/target.maven.zip"
[taskcluster 2019-06-01 12:55:52.041Z] Artifact "public/build/geckoview-androidTest.apk" not found at "/builds/worker/workspace/build/src/obj-firefox/gradle/build/mobile/android/geckoview/outputs/apk/androidTest/withGeckoBinaries/debug/geckoview-withGeckoBinaries-debug-androidTest.apk"
[taskcluster 2019-06-01 12:55:52.141Z] Artifact "public/android/maven" not found at "/builds/worker/workspace/build/src/obj-firefox/gradle/build/mobile/android/geckoview/maven/"
[taskcluster 2019-06-01 12:55:52.691Z] Unsuccessful task run with exit code: 255 completed in 654.15 seconds

:cpeterson, can you please assign someone to take a look at this? It's perma failing on central.

:nalexander is there any chance you could take a look at this? Seems to be related to Bug 1508976.
Thank you.

(In reply to Natalia Csoregi [:nataliaCs] from comment #2)

:nalexander is there any chance you could take a look at this? Seems to be related to Bug 1508976.
Thank you.

This is definitely fallout from Bug 1508976, but my guess is this file was updated between my try pushes (green Nightly builds a day or so ago) and this Nightly build. I will address it, probably by allowing that file to vary based on the architecture (which seems surprising, but who knows) or by making that file not vary.

(In reply to Nick Alexander :nalexander [he/him] from comment #3)

(In reply to Natalia Csoregi [:nataliaCs] from comment #2)

:nalexander is there any chance you could take a look at this? Seems to be related to Bug 1508976.
Thank you.

This is definitely fallout from Bug 1508976, but my guess is this file was updated between my try pushes (green Nightly builds a day or so ago) and this Nightly build. I will address it, probably by allowing that file to vary based on the architecture (which seems surprising, but who knows) or by making that file not vary.

I have dug into this. What is happening is this:

• the Ngv task is deterministic
• Ngv depends on N for arm, aarch64, x86, x86_64
• each N task is not deterministic: the exact l10n changeset pulled depends on the tip of the l10n HG repository

So as soon as two N tasks pull different l10n repositories, there can be different localized files in the omnijar. And this is what happened in this case:

From https://taskcluster-artifacts.net/flL6RBHpTM2C0EYtYUMlCA/0/public/logs/live_backing.log

[task 2019-06-01T11:47:28.244Z] 11:47:28     INFO -  11:47:28     INFO -  ensuring https://hg.mozilla.org/l10n-central/dsb@default is available at dsb
[task 2019-06-01T11:47:29.048Z] 11:47:29     INFO -  11:47:29     INFO -  (sharing from new pooled repository 02b3824f870839d34fc9e23d3d339c4a1e0a5c75)
[task 2019-06-01T11:47:29.598Z] 11:47:29     INFO -  11:47:29     INFO -  requesting all changes
[task 2019-06-01T11:47:29.991Z] 11:47:29     INFO -  11:47:29     INFO -  adding changesets
[task 2019-06-01T11:47:30.159Z] 11:47:30     INFO -  11:47:30     INFO -  adding manifests
[task 2019-06-01T11:47:30.274Z] 11:47:30     INFO -  11:47:30     INFO -  adding file changes
[task 2019-06-01T11:47:30.949Z] 11:47:30     INFO -  11:47:30     INFO -  added 1125 changesets with 5473 changes to 962 files (+23 heads)
[task 2019-06-01T11:47:31.022Z] 11:47:31     INFO -  11:47:31     INFO -  new changesets 02b3824f8708:d0a643f4596a
[task 2019-06-01T11:47:31.244Z] 11:47:31     INFO -  11:47:31     INFO -  searching for changes
[task 2019-06-01T11:47:31.244Z] 11:47:31     INFO -  11:47:31     INFO -  no changes found
[task 2019-06-01T11:47:31.869Z] 11:47:31     INFO -  11:47:31     INFO -  (pulling to obtain default)
[task 2019-06-01T11:47:32.387Z] 11:47:32     INFO -  11:47:32     INFO -  (remote resolved default to d0a643f4596a9793f0739db55ff575534f2402c2; result is not deterministic)
[task 2019-06-01T11:47:32.388Z] 11:47:32     INFO -  11:47:32     INFO -  (revision already present locally; not pulling)
[task 2019-06-01T11:47:32.454Z] 11:47:32     INFO -  11:47:32     INFO -
[task 2019-06-01T11:47:32.454Z] 11:47:32     INFO -  11:47:32     INFO - updating [==>                                                         ]  41/656
[task 2019-06-01T11:47:32.454Z] 11:47:32     INFO -  11:47:32     INFO -
[task 2019-06-01T11:47:32.454Z] 11:47:32     INFO -  11:47:32     INFO - 656 files updated, 0 files merged, 0 files removed, 0 files unresolved
[task 2019-06-01T11:47:32.454Z] 11:47:32     INFO -  11:47:32     INFO -  updated to d0a643f4596a9793f0739db55ff575534f2402c2

From https://taskcluster-artifacts.net/f7dJ3wCtT7i4PZU2BGVjgQ/0/public/logs/live_backing.log

[task 2019-06-01T11:07:44.965Z] 11:07:44     INFO -  11:07:44     INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', '--config', 'extensions.robustcheckout=/builds/worker/workspace/build/src/testing/mozharness/external_tools/robustcheckout.py', 'robustcheckout', u'https://hg.mozilla.org/l10n-central/dsb', u'dsb', '--sharebase', u'/builds/hg-shared', '--branch', u'default']
[task 2019-06-01T11:07:44.965Z] 11:07:44     INFO -  11:07:44     INFO - Copy/paste: hg --config ui.merge=internal:merge --config extensions.robustcheckout=/builds/worker/workspace/build/src/testing/mozharness/external_tools/robustcheckout.py robustcheckout https://hg.mozilla.org/l10n-central/dsb dsb --sharebase /builds/hg-shared --branch default
[task 2019-06-01T11:07:45.016Z] 11:07:45     INFO -  11:07:45     INFO -  (using Mercurial 4.8.1)
[task 2019-06-01T11:07:45.016Z] 11:07:45     INFO -  11:07:45     INFO -  ensuring https://hg.mozilla.org/l10n-central/dsb@default is available at dsb
[task 2019-06-01T11:07:45.385Z] 11:07:45     INFO -  11:07:45     INFO -  (sharing from new pooled repository 02b3824f870839d34fc9e23d3d339c4a1e0a5c75)
[task 2019-06-01T11:07:45.634Z] 11:07:45     INFO -  11:07:45     INFO -  requesting all changes
[task 2019-06-01T11:07:45.839Z] 11:07:45     INFO -  11:07:45     INFO -  adding changesets
[task 2019-06-01T11:07:45.952Z] 11:07:45     INFO -  11:07:45     INFO -  adding manifests
[task 2019-06-01T11:07:46.057Z] 11:07:46     INFO -  11:07:46     INFO -  adding file changes
[task 2019-06-01T11:07:46.809Z] 11:07:46     INFO -  11:07:46     INFO -  added 1123 changesets with 5466 changes to 961 files (+23 heads)
[task 2019-06-01T11:07:46.876Z] 11:07:46     INFO -  11:07:46     INFO -  new changesets 02b3824f8708:50c5ef41adfd
[task 2019-06-01T11:07:46.991Z] 11:07:46     INFO -  11:07:46     INFO -  searching for changes
[task 2019-06-01T11:07:46.991Z] 11:07:46     INFO -  11:07:46     INFO -  no changes found
[task 2019-06-01T11:07:47.116Z] 11:07:47     INFO -  11:07:47     INFO -  (pulling to obtain default)
[task 2019-06-01T11:07:47.366Z] 11:07:47     INFO -  11:07:47     INFO -  (remote resolved default to 50c5ef41adfd89777d01dd8e3facf12834a11061; result is not deterministic)
[task 2019-06-01T11:07:47.366Z] 11:07:47     INFO -  11:07:47     INFO -  (revision already present locally; not pulling)
[task 2019-06-01T11:07:47.431Z] 11:07:47     INFO -  11:07:47     INFO -  655 files updated, 0 files merged, 0 files removed, 0 files unresolved
[task 2019-06-01T11:07:47.431Z] 11:07:47     INFO -  11:07:47     INFO -  updated to 50c5ef41adfd89777d01dd8e3facf12834a11061

Now, what to do? Ideally all the jobs would share an l10n head (per locale). My guess is that they do so for mozilla-beta and mozilla-release, but I will confirm. It's not great to ship a Nightly GeckoView where the locale translations are different based on ... something (although that's what we're doing right now, acccidentally: two arch-specific Nightly GeckoView AARs can have different translations).

It's easy enough to work around this in the fat AAR code -- we can just ignore differences in DTD and FTL files -- but it feels wrong. Nonetheless I will prepare such a patch right now; we can make this more sensible in follow-up.

This works around the following situation: the Ngv/fat AAR task is
deterministic, but it depends on N tasks for multiple architectures,
adn the exact l10n changeset pulled by each N task depends on the tip
of the l10n HG repository at the time the N task executes.

For now, just ignore the obvious localized resources: *.ftl, *.dtd,
*.properties.

nataliaCs: I asked for review from the build peers, but that won't happen before Monday. There's no real reason to back this out: it can sit broken and can get starred to this ticket for a few days without consequence.

(In reply to Nick Alexander :nalexander [he/him] from comment #6)

nataliaCs: I asked for review from the build peers, but that won't happen before Monday. There's no real reason to back this out: it can sit broken and can get starred to this ticket for a few days without consequence.

That's alright, thank you for looking into this. Will leave a memo for the other sheriffs.

https://hg.mozilla.org/mozilla-central/rev/7bf7f0a78f3c