After landing bug 1018222, we're should have single-locale Aurora builds for both the hsb and dsb locales in http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-aurora-android-l10n/ but none have kicked off since landing the bug. Each locale has mobile resources in their repositories.

Looks like our automation tracks maemo-locales, not all-locales: https://github.com/mozilla/build-mozharness/blob/master/configs/multi_locale/mozilla-aurora_android.json#L5 https://mxr.mozilla.org/mozilla-aurora/source/mobile/android/locales/maemo-locales

We have single locale builds for fy-NL, which is not in that file https://mxr.mozilla.org/mozilla-aurora/source/mobile/android/locales/maemo-locales But in this https://mxr.mozilla.org/mozilla-aurora/source/mobile/android/locales/all-locales#26

The automation for including a locale in the multilocale build should be tracking maemo-locales, but single-locale builds should be tracking all-locales.

Whoops, good point...looking into this further.

Found this in a job that tried to build dsb: 02:34:02 ERROR - Can't determine the upload url for dsb! 02:34:02 ERROR - You either need to run --upload-repacks before --create-nightly-snippets, or specify the 'snippet_base_url' in self.config! Same thing for hsb. I'm out of my league now...Aki?

http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-l10n/mozilla-aurora-android-l10n_1-unknown-bm71-build1-build1.txt.gz bld-linux64-spot-126 02:29:37 INFO - ( echo 'export ANT_HOME=/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/apache-ant'; echo 'export PATH=/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/gcc/bin:/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/apache-ant/bin:/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/'; ) > obj-l10n/.mozconfig.mk 02:29:37 ERROR - client.mk:316: *** Could not find autoconf 2.13. Stop. 02:29:37 ERROR - Return code: 2 02:29:37 FATAL - Configure failed! 02:29:37 FATAL - Running post_fatal callback... 02:29:37 FATAL - Exiting -1 http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-l10n/mozilla-aurora-android-l10n_2-unknown-bm82-build1-build6.txt.gz b-linux64-hp-0033 02:20:44 ERROR - caught OS error 2: No such file or directory while running ['make', '-f', 'client.mk', 'configure'] 02:20:44 FATAL - Configure failed! 02:20:44 FATAL - Running post_fatal callback... 02:20:44 FATAL - Exiting -1 http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-l10n/mozilla-aurora-android-l10n_3-unknown-bm71-build1-build1.txt.gz bld-linux64-spot-1010 02:25:43 INFO - ( echo 'export ANT_HOME=/builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/apache-ant'; echo 'export PATH=/builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/gcc/bin:/builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/apache-ant/bin:/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/'; ) > obj-l10n/.mozconfig.mk 02:25:45 ERROR - client.mk:316: *** Could not find autoconf 2.13. Stop. 02:25:45 ERROR - Return code: 2 02:25:45 FATAL - Configure failed! 02:25:45 FATAL - Running post_fatal callback... 02:25:45 FATAL - Exiting -1 http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-l10n/mozilla-aurora-android-l10n_4-unknown-bm71-build1-build5.txt.gz bld-linux64-spot-127 02:29:15 INFO - ( echo 'export ANT_HOME=/builds/slave/m-aurora-and-l10n_4-0000000000/build/mozilla-aurora/apache-ant'; echo 'export PATH=/builds/slave/m-aurora-and-l10n_4-0000000000/build/mozilla-aurora/gcc/bin:/builds/slave/m-aurora-and-l10n_4-0000000000/build/mozilla-aurora/apache-ant/bin:/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/'; ) > obj-l10n/.mozconfig.mk 02:29:15 ERROR - client.mk:316: *** Could not find autoconf 2.13. Stop. 02:29:15 ERROR - Return code: 2 02:29:15 FATAL - Configure failed! 02:29:15 FATAL - Running post_fatal callback... 02:29:15 FATAL - Exiting -1 http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-l10n/mozilla-aurora-android-l10n_5-unknown-bm73-build1-build3.txt.gz bld-linux64-spot-443 02:29:54 INFO - ( echo 'export ANT_HOME=/builds/slave/m-aurora-and-l10n_5-0000000000/build/mozilla-aurora/apache-ant'; echo 'export PATH=/builds/slave/m-aurora-and-l10n_5-0000000000/build/mozilla-aurora/gcc/bin:/builds/slave/m-aurora-and-l10n_5-0000000000/build/mozilla-aurora/apache-ant/bin:/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/'; ) > obj-l10n/.mozconfig.mk 02:29:54 ERROR - client.mk:316: *** Could not find autoconf 2.13. Stop. 02:29:54 ERROR - Return code: 2 02:29:54 FATAL - Configure failed! 02:29:54 FATAL - Running post_fatal callback... 02:29:54 FATAL - Exiting -1

It sounds like there's at least two different problems here. The logs that Aki points at show complete failure of repacks altogether, starting with this morning's builds. The error I found was from http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-l10n/mozilla-aurora-android-l10n_1-unknown-bm85-build1-build2.txt.gz, which didn't hit the autoconf error.

dsb_failure: "dsb failed in make installers-dsb! Can't create a snippet for dsb without an upload url." The installer failed, therefore there was no upload, therefore there was no upload url. Search for | ERROR -|. dsb core dumped: 02:30:00 INFO - /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/android-sdk-linux/build-tools/19.0.2/aapt: /lib/libz.so.1: no version information available (required by /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/android-sdk-linux/build-tools/19.0.2/aapt) 02:30:01 INFO - make[2]: *** [.aapt.nodeps] Segmentation fault (core dumped) 02:30:01 INFO - make[2]: *** Deleting file `.aapt.nodeps' 02:30:01 INFO - make[2]: Leaving directory `/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/obj-l10n/mobile/android/base' 02:30:01 ERROR - make[1]: *** [repackage-zip] Error 2 02:30:01 INFO - make[1]: Leaving directory `/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/obj-l10n/mobile/android/locales' 02:30:01 INFO - make: *** [repackage-zip-dsb] Error 2 02:30:02 ERROR - Return code: 2 af: 02:27:00 INFO - /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/obj-l10n/_virtualenv/bin/python -m mozbuild.action.preprocessor -I /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/af/mobile/profile/bookmarks.inc -DAB_CD=af /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/mobile/locales/generic/profile/bookmarks.json.in -o .deps/generated_af/bookmarks.json 02:27:01 ERROR - Traceback (most recent call last): 02:27:01 INFO - File "/tools/python27/lib/python2.7/runpy.py", line 162, in _run_module_as_main 02:27:01 INFO - "__main__", fname, loader, pkg_name) 02:27:01 INFO - File "/tools/python27/lib/python2.7/runpy.py", line 72, in _run_code 02:27:01 INFO - exec code in run_globals 02:27:01 INFO - File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozbuild/action/preprocessor.py", line 16, in <module> 02:27:01 INFO - main(sys.argv[1:]) 02:27:01 INFO - File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozbuild/action/preprocessor.py", line 12, in main 02:27:01 INFO - pp.handleCommandLine(args, True) 02:27:01 INFO - File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozbuild/preprocessor.py", line 472, in handleCommandLine 02:27:01 INFO - self.processFile(input=input, output=out) 02:27:01 INFO - File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozbuild/preprocessor.py", line 380, in processFile 02:27:01 INFO - self.do_include(input, False) 02:27:01 INFO - File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozbuild/preprocessor.py", line 774, in do_include 02:27:01 INFO - self.handleLine(l) 02:27:01 INFO - File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozbuild/preprocessor.py", line 552, in handleLine 02:27:01 INFO - self.write(aLine) 02:27:01 INFO - File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozbuild/preprocessor.py", line 421, in write 02:27:01 INFO - filteredLine = self.applyFilters(aLine) 02:27:01 INFO - File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozbuild/preprocessor.py", line 403, in applyFilters 02:27:01 INFO - aLine = f[1](aLine) 02:27:01 INFO - File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozbuild/preprocessor.py", line 731, in filter_substitution 02:27:01 INFO - return self.varsubst.sub(repl, aLine) 02:27:01 INFO - File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozbuild/preprocessor.py", line 729, in repl 02:27:01 INFO - raise Preprocessor.Error(self, 'UNDEFINED_VAR', varname) 02:27:01 INFO - mozbuild.preprocessor.Error: ('/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/mobile/locales/generic/profile/bookmarks.json.in', 6, 'UNDEFINED_VAR', 'bookmarks_aboutBrowser') 02:27:01 ERROR - make[3]: *** [.deps/generated_af/bookmarks.json] Error 255 02:27:01 INFO - make[3]: *** Deleting file `.deps/generated_af/bookmarks.json' 02:27:01 INFO - make[3]: Leaving directory `/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/obj-l10n/mobile/locales' 02:27:01 ERROR - make[2]: *** [libs-af] Error 2 02:27:01 INFO - make[2]: Leaving directory `/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/obj-l10n/mobile/locales' 02:27:01 ERROR - make[1]: *** [libs-af] Error 2 02:27:01 INFO - make[1]: Leaving directory `/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/obj-l10n/mobile/android/locales' 02:27:01 INFO - make: *** [repackage-zip-af] Error 2

So, it sounds like we're missing a package for autoconf-2.13, and also missing a packaging that something in the l10n logic depends on? Not sure exactly, and I don't see anything that's changed around the installed packages recently...

Possibly of note: the log I linked to shows that the repack was done on a machine that was re-imaged recently. Aki's logs are a mix of recently reimaged machines, and spot instances (which re-image at least once a day).

This might be the root error in Aki's logs: 02:25:43 INFO - client.mk:213: obj-l10n/.mozconfig.mk: No such file or directory

I can't find any relevant changes to anything in buildbot-configs, buildbotcustom, mozharness, or even mozilla-aurora...

Looking further it appears that Aki's log (http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-l10n/mozilla-aurora-android-l10n_3-unknown-bm71-build1-build1.txt.gz) never installs mock packages, even though they're in the config. My log log (http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-l10n/mozilla-aurora-android-l10n_1-unknown-bm85-build1-build2.txt.gz) does. Aki's: 02:24:48 INFO - ##### 02:24:48 INFO - ##### Running setup step. 02:24:48 INFO - ##### 02:24:48 INFO - Running main action method: setup 02:24:48 INFO - Copying /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/mobile/android/config/mozconfigs/android/l10n-nightly to /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/.mozconfig 02:24:48 INFO - Running command: ['cat', '/builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/.mozconfig'] 02:24:48 INFO - Copy/paste: cat /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/.mozconfig 02:24:48 INFO - . "$topsrcdir/mobile/android/config/mozconfigs/common" 02:24:48 INFO - # L10n Mine: 02:16:27 INFO - ##### 02:16:27 INFO - ##### Running setup step. 02:16:27 INFO - ##### 02:16:27 INFO - Running main action method: setup 02:16:27 INFO - Copying /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/mobile/android/config/mozconfigs/android/l10n-nightly to /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/.mozconfig 02:16:27 INFO - Getting output from command: ['mock_mozilla', '-r', 'mozilla-centos6-x86_64', '--print-root-path'] 02:16:27 INFO - Copy/paste: mock_mozilla -r mozilla-centos6-x86_64 --print-root-path 02:16:28 INFO - Reading from file tmpfile_stdout 02:16:28 INFO - Output received: 02:16:28 INFO - /builds/mock_mozilla/mozilla-centos6-x86_64/root/ 02:16:28 INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stderr',), kwargs: {}, attempt #1 02:16:28 INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stdout',), kwargs: {}, attempt #1 02:16:28 INFO - no previous package list found 02:16:28 INFO - Running command: ['mock_mozilla', '-r', 'mozilla-centos6-x86_64', '--init'] 02:16:28 INFO - Copy/paste: mock_mozilla -r mozilla-centos6-x86_64 --init I think the overall failure of repacks is more fallout from bug 1025842. So, that needs to be fixed before we can confirm any sort of fix for hsb and dsb being missing. However, I think the issues with dsb and hsb are with the locales themselves. For hsb I found this: 02:25:58 ERROR - /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/obj-l10n/mobile/android/base/res/values/strings.xml:976: error: Error parsing XML: undefined entity 02:25:58 ERROR - /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout/bookmark_edit.xml:12: error: Error: No resource found that matches the given name (at 'hint' with value '@string/bookmark_edit_name'). 02:25:58 ERROR - /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout/bookmark_edit.xml:19: error: Error: No resource found that matches the given name (at 'hint' with value '@string/bookmark_edit_location'). 02:25:58 ERROR - /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout/bookmark_edit.xml:27: error: Error: No resource found that matches the given name (at 'hint' with value '@string/bookmark_edit_keyword'). 02:25:58 ERROR - /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout/browser_toolbar.xml:56: error: Error: No resource found that matches the given name (at 'contentDescription' with value '@string/menu'). 02:25:58 ERROR - /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout/browser_toolbar.xml:100: error: Error: No resource found that matches the given name (at 'contentDescription' with value '@string/edit_mode_cancel'). 02:25:58 ERROR - /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout-large-v11/browser_toolbar.xml:53: error: Error: No resource found that matches the given name (at 'contentDescription' with value '@string/back'). 02:25:58 ERROR - /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout-large-v11/browser_toolbar.xml:87: error: Error: No resource found that matches the given name (at 'contentDescription' with value '@string/menu'). 02:25:58 ERROR - /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout-large-v11/browser_toolbar.xml:112: error: Error: No resource found that matches the given name (at 'contentDescription' with value '@string/edit_mode_cancel'). (and much more - look at http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-l10n/mozilla-aurora-android-l10n_3-unknown-bm86-build1-build2.txt.gz for the full list) And for dsb I found similar: 02:31:22 ERROR - /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/obj-l10n/mobile/android/base/res/values/strings.xml:976: error: Error parsing XML: undefined entity 02:31:22 ERROR - /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout/bookmark_edit.xml:12: error: Error: No resource found that matches the given name (at 'hint' with value '@string/bookmark_edit_name'). 02:31:22 ERROR - /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout/bookmark_edit.xml:19: error: Error: No resource found that matches the given name (at 'hint' with value '@string/bookmark_edit_location'). 02:31:22 ERROR - /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout/bookmark_edit.xml:27: error: Error: No resource found that matches the given name (at 'hint' with value '@string/bookmark_edit_keyword'). 02:31:22 ERROR - /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout/browser_toolbar.xml:56: error: Error: No resource found that matches the given name (at 'contentDescription' with value '@string/menu'). (full log at http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-l10n/mozilla-aurora-android-l10n_1-unknown-bm84-build1-build5.txt.gz) Jeff, any idea about the two errors above?

Rail tells me that bug 1025842 doesn't impact in-house machines like the ones these builds run on, so it's probably unrelated. I'm filing a new bug for the general issue with these repacks.

Oh, that above reads dramatically. What can I do as translator? According to the sign-off pages for hsb and dsb there are no errors - two files are still missing but they are not on Pootle yet.

@Michael Unfortunately nothing you can do, we just need to figure out the issue. Dashboard reports only missing productization files for hsb/dsb (Jeff, do we have bugs for those?), no errors. The strings reported as missing are definitely there, and they look OK, not sure if the build system gets mad at the missing files.

I tried a dsb repack locally, and I get an apk (can't test it, I don't have an Android device anymore) It fails with cp: /src/central/releases/mozilla-aurora/obj-repack-android/mobile/android/base/fennec_ids.txt: No such file or directory after successfully verifying the apk. No idea what that means. IOW, I don't see anything on the localization side being wrong right now. And then I looked at ftp (should have done that before), and today dsb and hsb did in fact get apks generated, http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-aurora-android-l10n/ has http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-aurora-android-l10n/fennec-32.0a2.hsb.android-arm.apk and http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-aurora-android-l10n/fennec-32.0a2.dsb.android-arm.apk

I'm resolving this FIXED, as the infra backout seems to have helped, it'd be good to still keep watching the builds if these are just random outages. At least I don't know if we understand what broke the builds when.

@Comment 17: The file fennec_ids.txt does not exist on Pootle, just as the files chrome/region.properties and searchplugins/list.txt. Unfortunately I do not have an Android device, either. I tried to test the Upper Sorbian apk in the emulator Jar of Beans, installation succeeded but when starting Aurora I get the message "Aurora has been stopped". I have a very old computer, 1.024 MB RAM only. Maybe that's the reason for the stop.

I don't understand that this bug has been marked as "resolved" - there are no builds for dsb and hsb now. Still some days ago there were Aurora builds at least. I don't understand, either that now beta builds are created for dsb and hsb. Please don't say it is because of the three character language code. I saw that Maithili has a beta build - it has a three character language code as well - mai.

Sorry, it must be "**no** beta builds" instead of "**now** beta builds".

The aurora builds are busted in general, that's bug 1062880. doesn't have anything to do with what was going on in this bug.