Something has affected the Android l10n repacks for Aurora 34. The list of apks is empty. Unfortunately, I cannot provide any more detail than that.

More detail, and moving over to releng for investigation: central stopped having dates dirs after http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2014-08-29-08-07-46-mozilla-central-android-l10n/, and aurora disappeared on merge day.

That'd be because bug 1056128 didn't realize that bug 1043023 forgot about the existence of http://mxr.mozilla.org/build/source/mozharness/configs/single_locale/mozilla-central_android.py#111

Thanks for the diagnosis, philor. Can we get a releng owner for updating the repack machines?

Jordan: do changes from mozharness get merged into ash-mozharness at some interval, or should I land this change there as well?

And this time, the most recent patch that handle m-c and aurora.

(In reply to Chris Cooper [:coop] from comment #4) > Created attachment 8484349 [details] [diff] [review] > [mozharness] Update JDK to 1.7.0 in mock config > > Jordan: do changes from mozharness get merged into ash-mozharness at some > interval, or should I land this change there as well? this is a process that may need to be improved. I think up till now we have relied on ash-mozharness hackers to merge in mainline mozharness and TBH I bet aki would have been one of the main guys to do that. I will bring this up at our next mozharness meeting as it seems ash-mozharness is not getting upstream changes much recently. Until we figure out what we want ash to look like on a weekly basis, let's land these changes there too.

Comment on attachment 8484353 [details] [diff] [review] [mozharness] Update JDK to 1.7.0 in mock config, v2 Review of attachment 8484353 [details] [diff] [review]: ----------------------------------------------------------------- actually I'm assuming this is something that will only be ran on on the mozilla-aurora correct? If so, I think it is fine for ash to wait for the next mainline merge. this patch lgtm. I'm just noticing from the context lines in the diff that http://mxr.mozilla.org/build/source/mozharness/configs/single_locale/mozilla-aurora_android.py#95 and http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/config.py#1480 differ more than just the recently changed jdk: e.g. zlib-devel vs zlib-devel-1.2.3-27.el6.i686. mozilla-aurora_android.py appears to have many more mock packages. Is that expected behavior or are these meant to be in sync?

This at least affects central, too. See my comment 1.

(In reply to Jordan Lund (:jlund) from comment #7) > Comment on attachment 8484353 [details] [diff] [review] > [mozharness] Update JDK to 1.7.0 in mock config, v2 > > Review of attachment 8484353 [details] [diff] [review]: > ----------------------------------------------------------------- > > actually I'm assuming this is something that will only be ran on on the > mozilla-aurora correct? If so, I think it is fine for ash to wait for the > next mainline merge. > > this patch lgtm. I'm just noticing from the context lines in the diff that > http://mxr.mozilla.org/build/source/mozharness/configs/single_locale/mozilla- > aurora_android.py#95 and > http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/config.py#1480 > differ more than just the recently changed jdk: e.g. zlib-devel vs > zlib-devel-1.2.3-27.el6.i686. mozilla-aurora_android.py appears to have many > more mock packages. > > Is that expected behavior or are these meant to be in sync? I think this is fodder for a separate bug. I had a quick look, and there are other non-trivial discrepancies between the mock_packages defined in buildbot-configs vs mozharness.

Comment on attachment 8484353 [details] [diff] [review] [mozharness] Update JDK to 1.7.0 in mock config, v2 Review of attachment 8484353 [details] [diff] [review]: ----------------------------------------------------------------- https://hg.mozilla.org/build/mozharness/rev/1f4fb0a896d7

(In reply to Chris Cooper [:coop] from comment #9) > I think this is fodder for a separate bug. I had a quick look, and there are > other non-trivial discrepancies between the mock_packages defined in > buildbot-configs vs mozharness. On second thought, I'm not sure how much this matters *right now*. The package differences are between an actual Android build in the buildbot-configs and a single local repack in the mozharness configs, so it's entirely possible that those two different processes would require different tools. When we move the mainline Android builds to mozharness, we'll need to be sure the configs match, obviously. There is, however, definite room for improvement in each package list. There are multiple gcc packages being installed and I'm pretty sure only one compiler is actually being used in the build, likely the most recent. The mozharness list seems to be more explicit in terms of dependent packages that are installed. There are also some flat-out duplicates in the mozharness list: ccache and yasm are both repeated.

Merged to production, and deployed.

Well, that failed interestingly. It's in the mock_packages list, it's in the mock_mozilla commandline, and then it gets totally ignored and not installed.

I think the mock_target needs to change as well, as per http://hg.mozilla.org/build/buildbot-configs/rev/c2ea3f8a9ae5

(In reply to Chris Cooper [:coop] from comment #14) > I think the mock_target needs to change as well, as per > http://hg.mozilla.org/build/buildbot-configs/rev/c2ea3f8a9ae5 https://hg.mozilla.org/build/mozharness/rev/5e8520714013

We're installing the new JDK now, but we're hitting the following error: 04:17:12 INFO - Running command: ['mock_mozilla', '-r', 'mozilla-centos6-x86_64-android', '-q', '--cwd', '/builds/slave/m-cen-and-l10n_1-0000000000000/build', '--unpriv', '--shell', '/usr/bin/env MOZ_AUTOMATION=1 "LESSOPEN=|/usr/bin/lesspipe.sh %s" TMOUT=86400 CVS_RSH=ssh LOGNAME=cltbld USER=cltbld MOZ_OBJDIR=obj-l10n PATH=/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/ SYMBOL_SERVER_USER=ffxbld DISPLAY=:2 CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_SIGNING_SERVERS=gpg:signcode:mar:jar:b2gmar:signing4.srv.releng.scl3.mozilla.com:9100,gpg:signcode:mar:jar:b2gmar:signing5.srv.releng.scl3.mozilla.com:9100,gpg:signcode:mar:jar:b2gmar:signing6.srv.releng.scl3.mozilla.com:9100,gpg:dmg:mar:mac-signing2.srv.releng.scl3.mozilla.com:9100,gpg:dmg:mar:mac-signing3.srv.releng.scl3.mozilla.com:9100,gpg:dmg:mar:mac-signing4.srv.releng.scl3.mozilla.com:9100,dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9100,dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9100,dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9100 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_mob/ HG_SHARE_BASE_DIR=/builds/hg-shared MOZ_CRASHREPORTER_NO_REPORT=1 SYMBOL_SERVER_HOST=symbolpush.mozilla.org EN_US_BINARY_URL=http://stage.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android/en-US SHIP_LICENSED_FONTS=1 TINDERBOX_OUTPUT=1 "MOZ_SIGN_CMD=python /builds/slave/m-cen-and-l10n_1-0000000000000/build/tools/release/signing/signtool.py --cachedir /builds/slave/m-cen-and-l10n_1-0000000000000/build/signing_cache -t /builds/slave/m-cen-and-l10n_1-0000000000000/token -n /builds/slave/m-cen-and-l10n_1-0000000000000/nonce -c /builds/slave/m-cen-and-l10n_1-0000000000000/build/tools/release/signing/host.cert -f jar -H gpg:signcode:mar:jar:b2gmar:signing4.srv.releng.scl3.mozilla.com:9100 -H gpg:signcode:mar:jar:b2gmar:signing5.srv.releng.scl3.mozilla.com:9100 -H gpg:signcode:mar:jar:b2gmar:signing6.srv.releng.scl3.mozilla.com:9100 -H gpg:dmg:mar:mac-signing2.srv.releng.scl3.mozilla.com:9100 -H gpg:dmg:mar:mac-signing3.srv.releng.scl3.mozilla.com:9100 -H gpg:dmg:mar:mac-signing4.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9100" LC_ALL=C TOOLTOOL_HOME=/builds _=/tools/buildbot/bin/python LD_LIBRARY_PATH=/lib:/tools/gcc-4.7.2-0moz1/lib:/tools/gcc-4.7.2-0moz1/lib64 LOCALE_MERGEDIR=/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/merged/ MAIL=/var/spool/mail/cltbld MOZ_UPDATE_CHANNEL=nightly HOSTNAME=b-linux64-ix-0006.build.releng.scl3.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/ffxbld_dsa HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/m-cen-and-l10n_1-0000000000000 PROPERTIES_FILE=/builds/slave/m-cen-and-l10n_1-0000000000000/buildprops.json CCACHE_DIR=/builds/ccache CCACHE_COMPRESS=1 TOOLTOOL_CACHE=/builds/tooltool_cache tools/release/signing/verify-android-signature.sh --tools-dir=tools/ --nightly --apk=/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/fennec-35.0a1.ar.android-arm.apk'] in /builds/slave/m-cen-and-l10n_1-0000000000000/build 04:17:12 INFO - Copy/paste: mock_mozilla -r mozilla-centos6-x86_64-android -q --cwd /builds/slave/m-cen-and-l10n_1-0000000000000/build --unpriv --shell "/usr/bin/env MOZ_AUTOMATION=1 \"LESSOPEN=|/usr/bin/lesspipe.sh %s\" TMOUT=86400 CVS_RSH=ssh LOGNAME=cltbld USER=cltbld MOZ_OBJDIR=obj-l10n PATH=/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/ SYMBOL_SERVER_USER=ffxbld DISPLAY=:2 CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_SIGNING_SERVERS=gpg:signcode:mar:jar:b2gmar:signing4.srv.releng.scl3.mozilla.com:9100,gpg:signcode:mar:jar:b2gmar:signing5.srv.releng.scl3.mozilla.com:9100,gpg:signcode:mar:jar:b2gmar:signing6.srv.releng.scl3.mozilla.com:9100,gpg:dmg:mar:mac-signing2.srv.releng.scl3.mozilla.com:9100,gpg:dmg:mar:mac-signing3.srv.releng.scl3.mozilla.com:9100,gpg:dmg:mar:mac-signing4.srv.releng.scl3.mozilla.com:9100,dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9100,dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9100,dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9100 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_mob/ HG_SHARE_BASE_DIR=/builds/hg-shared MOZ_CRASHREPORTER_NO_REPORT=1 SYMBOL_SERVER_HOST=symbolpush.mozilla.org EN_US_BINARY_URL=http://stage.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android/en-US SHIP_LICENSED_FONTS=1 TINDERBOX_OUTPUT=1 \"MOZ_SIGN_CMD=python /builds/slave/m-cen-and-l10n_1-0000000000000/build/tools/release/signing/signtool.py --cachedir /builds/slave/m-cen-and-l10n_1-0000000000000/build/signing_cache -t /builds/slave/m-cen-and-l10n_1-0000000000000/token -n /builds/slave/m-cen-and-l10n_1-0000000000000/nonce -c /builds/slave/m-cen-and-l10n_1-0000000000000/build/tools/release/signing/host.cert -f jar -H gpg:signcode:mar:jar:b2gmar:signing4.srv.releng.scl3.mozilla.com:9100 -H gpg:signcode:mar:jar:b2gmar:signing5.srv.releng.scl3.mozilla.com:9100 -H gpg:signcode:mar:jar:b2gmar:signing6.srv.releng.scl3.mozilla.com:9100 -H gpg:dmg:mar:mac-signing2.srv.releng.scl3.mozilla.com:9100 -H gpg:dmg:mar:mac-signing3.srv.releng.scl3.mozilla.com:9100 -H gpg:dmg:mar:mac-signing4.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9100\" LC_ALL=C TOOLTOOL_HOME=/builds _=/tools/buildbot/bin/python LD_LIBRARY_PATH=/lib:/tools/gcc-4.7.2-0moz1/lib:/tools/gcc-4.7.2-0moz1/lib64 LOCALE_MERGEDIR=/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/merged/ MAIL=/var/spool/mail/cltbld MOZ_UPDATE_CHANNEL=nightly HOSTNAME=b-linux64-ix-0006.build.releng.scl3.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/ffxbld_dsa HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/m-cen-and-l10n_1-0000000000000 PROPERTIES_FILE=/builds/slave/m-cen-and-l10n_1-0000000000000/buildprops.json CCACHE_DIR=/builds/ccache CCACHE_COMPRESS=1 TOOLTOOL_CACHE=/builds/tooltool_cache tools/release/signing/verify-android-signature.sh --tools-dir=tools/ --nightly --apk=/builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/fennec-35.0a1.ar.android-arm.apk" 04:17:12 INFO - TOOLS DIR: tools/ 04:17:12 INFO - Archive: /builds/slave/m-cen-and-l10n_1-0000000000000/build/mozilla-central/obj-l10n/dist/fennec-35.0a1.ar.android-arm.apk 04:17:12 INFO - inflating: META-INF/NIGHTLY.DSA 04:17:12 FATAL - Invalid 04:17:12 FATAL - Signature is invalid! 04:17:12 FATAL - Running post_fatal callback... 04:17:12 FATAL - Exiting -1 Taken from: http://buildbot-master85.srv.releng.scl3.mozilla.com:8001/builders/Android%202.3%20mozilla-central%20l10n%20nightly-1/builds/2/steps/run_script/logs/stdio

I've been able to replicate this in staging, but that doesn't put me too far ahead since I still don't understand how the signature works here. I did look at the timing of this failure. The mozilla-central nightly repack succeeded on Aug 29, and then failed for the first time on Aug 30. The failure then migrated to aurora with the uplift on Sep 03. Something landed on Aug 29 to cause this failure. m-c Aug 29 nightly rev: d728b2f31c56 m-c Aug 30 nightly rev: eed9fe35a00d

Extra complication could be that we might have busted the builds by the jdk change, and then did something else hiding the second regression range.

(In reply to Axel Hecht [:Pike] from comment #19) > Extra complication could be that we might have busted the builds by the jdk > change, and then did something else hiding the second regression range. https://hg.mozilla.org/mozilla-central/rev/ce756025b879 ?

Thank you all for looking into this. Have there been any new updates?

Sorry, this shouldn't have sat in the releng queue for this long. This signature problems seems to have been introduced with the switch to the new JDK (1.7.0). Someone from the mobile team who knows about Android signing will need to dig into this.

In the interests of keeping this moving a little, even though this is far from my area of expertise: tl;dr: the Android build-and-sign toolchain in-tree appears to be completely fine, and the bug is elsewhere. Here's where signing happens. http://mxr.mozilla.org/mozilla-central/source/config/android-common.mk#17 As you can see, it calls out to MOZ_SIGN_COMMAND if defined, and otherwise calls out to RELEASE_JARSIGNER. That this works locally (using DEBUG_JARSIGNER), and that the builds on FTP install on devices (presumably using RELEASE_JARSIGNER or MOZ_SIGN_COMMAND), implies that this process is working. Otherwise we'd be screaming blue murder 'cos our own Nightly instances on personal phones aren't updating. Indeed, that updates are working means that the output is being signed with the same keys as before, and the signature is still valid. (Android doesn't allow upgrades across keys.) That is: Android says our signing stuff still works. To me that points the finger at signtool.py, which is the thing that's barfing, and isn't something I'm familiar with at all. Alternatively, whatever the buildbots are using as MOZ_SIGN_COMMAND (which is perhaps also signtool.py?) is producing a valid signature for Android, but not something that signtool.py understands. In either case: who owns signtool.py, what is it, and can we assign this bug to them?

signtool.py is releng, I'll take a look at this

It turns out that the new toolkit also dumps SHA256 of the certificate, while we don't have in the file we compare the output against. Ignoring that line solves the issue. We could probably also change the verification script to use different signatures per branch.

Comment on attachment 8497777 [details] [diff] [review] no_sha256.diff https://hg.mozilla.org/build/tools/rev/c708687699ca

I retriggered one of the chunks on m-c and it worked. I see a bunch of files in http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-l10n/ now. Leaving this open to verify tomorrow.