Similar to the fx_desktop_build TC uploads, we'll need to upload the locale repacks to taskcluster from desktop_l10n.py

I tried to re-use what I could, but buildbase.py and desktop_l10n.py don't have access to all the same things (eg: things like query_revision() and how the upload files are retrieved are different). Still, there could probably be more things shared between them. Suggestions are welcome!

I missed this in my review queue today. I will get to it by tomorrow. Apologies for the delay mshal.

Comment on attachment 8621817 [details] [diff] [review] l10n-upload Review of attachment 8621817 [details] [diff] [review]: ----------------------------------------------------------------- looks good. a lot of query_pushdate and taskcluster_upload are copies from buildbase.py. I'm okay leaving that as is for now but if we require the need to do this for a new build variant again with a separate script entry point, we should be making some common util functions or class that can be shared. Long term wise it would be nice for desktop_l10n.py and fx_desktop_build.py to both inherit from BuildScript but that's wee bit out of scope :) ::: mozharness/mozilla/building/buildbase.py @@ +1386,5 @@ > self.log_obj, > ) > > + index = self.config.get('taskcluster_index', 'index.garbage.staging') > + # TODO: Bug 1165980 - these should be in tree fear not, it will all be in the tree soon ;) ::: scripts/desktop_l10n.py @@ +1390,5 @@ > + > + task = tc.create_task(routes) > + tc.claim_task(task) > + > + for upload_file in self.upload_files: I wonder if it be worth log dumping upload_files so we can clearly see what get_upload_files ended up getting from each repack after it ignored the duplicates. I guess we can look at that by scrolling through each submission in the logs too.. @@ +1405,5 @@ > + > + mozilla_dir = self.config['mozilla_dir'] > + repo = None > + for repository in self.config['repos']: > + if repository.get('dest') == mozilla_dir: seems fragile but I like your creativity! :)

Updated to log the list of upload files, and also removed the duplicate create_virtualenv() per our IRC discussion. r+ carried forward

Comment on attachment 8624727 [details] [diff] [review] l10n-upload https://hg.mozilla.org/build/mozharness/rev/d25f811a8515

I think this broke l10n :/ 05:10:10 INFO - ##### 05:10:10 INFO - ##### Running taskcluster-upload step. 05:10:10 INFO - ##### 05:10:10 INFO - Running main action method: taskcluster_upload 05:10:10 FATAL - Uncaught exception: Traceback (most recent call last): 05:10:10 FATAL - File "/builds/slave/m-aurora-lx-l10n-ntly-1-000000/scripts/mozharness/base/script.py", line 1664, in run 05:10:10 FATAL - self.run_action(action) 05:10:10 FATAL - File "/builds/slave/m-aurora-lx-l10n-ntly-1-000000/scripts/mozharness/base/script.py", line 1606, in run_action 05:10:10 FATAL - self._possibly_run_method(method_name, error_if_missing=True) 05:10:10 FATAL - File "/builds/slave/m-aurora-lx-l10n-ntly-1-000000/scripts/mozharness/base/script.py", line 1547, in _possibly_run_method 05:10:10 FATAL - return getattr(self, method_name)() 05:10:10 FATAL - File "scripts/scripts/desktop_l10n.py", line 1367, in taskcluster_upload 05:10:10 FATAL - self.activate_virtualenv() 05:10:10 FATAL - File "/builds/slave/m-aurora-lx-l10n-ntly-1-000000/scripts/mozharness/base/python.py", line 423, in activate_virtualenv 05:10:10 FATAL - execfile(activate, dict(__file__=activate)) 05:10:10 FATAL - IOError: [Errno 2] No such file or directory: '/builds/slave/m-aurora-lx-l10n-ntly-1-000000/build/venv/bin/activate_this.py' 05:10:10 FATAL - Running post_fatal callback... 05:10:10 FATAL - Exiting -1

Looks like we clobber the "build" directory because PurgeMixin.clobber() thinks this is a nightly build and we need to force clobber. And pip install happens before this step. 03:24:20 INFO - ##### 03:24:20 INFO - ##### Running clobber step. 03:24:20 INFO - ##### 03:24:20 INFO - Running main action method: clobber 03:24:20 INFO - Clobbering because we're a nightly build 03:24:20 INFO - rmtree: /builds/slave/m-aurora-lx-l10n-ntly-1-000000/build

Unfortunately my staging builds were working because I disabled clobbers so that I could iterate more than 4 times a day :(. This moves the create_virtualenv() to before the activate, but we have to disable mock while doing so or we can't find the virtualenv executable. I tested on staging with win/mac/linux and clobbers enabled, but it's possible another one of my local patches makes that unreliable.

Comment on attachment 8626639 [details] [diff] [review] l10n-create-virtualenv lgtm

Comment on attachment 8626639 [details] [diff] [review] l10n-create-virtualenv https://hg.mozilla.org/build/mozharness/rev/2076d7939ccf

Seems that N2 on the mac got unlucky in some way, N3 is green though, https://treeherder.mozilla.org/logviewer.html#?job_id=1709144&repo=mozilla-central

Central seems to be fixed, but aurora is busted, same bug or existing?

Aurora seems busted. :/ 04:20:46 FATAL - Uncaught exception: Traceback (most recent call last): 04:20:46 FATAL - File "/builds/slave/m-aurora-l64-l10n-ntly-3-00000/scripts/mozharness/base/script.py", line 1664, in run 04:20:46 FATAL - self.run_action(action) 04:20:46 FATAL - File "/builds/slave/m-aurora-l64-l10n-ntly-3-00000/scripts/mozharness/base/script.py", line 1606, in run_action 04:20:46 FATAL - self._possibly_run_method(method_name, error_if_missing=True) 04:20:46 FATAL - File "/builds/slave/m-aurora-l64-l10n-ntly-3-00000/scripts/mozharness/base/script.py", line 1547, in _possibly_run_method 04:20:46 FATAL - return getattr(self, method_name)() 04:20:46 FATAL - File "scripts/scripts/desktop_l10n.py", line 1401, in taskcluster_upload 04:20:46 FATAL - tc.create_artifact(task, upload_file) 04:20:46 FATAL - File "/builds/slave/m-aurora-l64-l10n-ntly-3-00000/scripts/mozharness/mozilla/taskcluster_helper.py", line 86, in create_artifact 04:20:46 FATAL - content_length = os.path.getsize(filename) 04:20:46 FATAL - File "/tools/buildbot/lib/python2.7/genericpath.py", line 49, in getsize 04:20:46 FATAL - return os.stat(filename).st_size 04:20:46 FATAL - OSError: [Errno 2] No such file or directory: '/builds/slave/m-aurora-l64-l10n-ntly-3-00000/build/mozilla-aurora/obj-l10n/dist/update/firefox-40.0a2.sq.linux-x86_64.partial.20150629004008-20150629055645.mar' 04:20:46 FATAL - Running post_fatal callback... 04:20:46 FATAL - Exiting -1 It sounds like there is some clobbering happening for each locale (not sure though). You may need to refactor the code so it uploads artifact after each "make upload".

Ahh, this was addressed on central in bug 1173998 - looks like we just need to uplift http://hg.mozilla.org/mozilla-central/rev/763336d4cbb2. I've requested approval in that bug. The removal was changed to just remove the current locale's partial mar rather than all locale's. Unfortunately doing a taskcluster upload after each locale doesn't work too well since we don't have a good mechanism in mozharness to periodically reclaim the task yet.

Macs are still busted, now it seems that the bundle name on aurora doesn't match its expectations. Windows and Linux seem fixed.

The mac issue is bug 1178785.

These have been uploading for a while now, and will be indexed better in bug 1133074.