There is no updates for builds in http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-aurora-android-l10n/ I would be wonderful to have updates in Fennec instead of downloading them manually.

Yes, this is known. I know how to tackle this, but I have a couple items with a hard deadline of Jan 31 so I may have to work on this afterwards.

Aki, can you shuffle things around here? Jan 31 means that we're basically not having completed infrastructure for l10n for Firefox 11 on Aurora, and possibly into Beta. CCing folks as I expect this bug to show up on https://wiki.mozilla.org/Release_Management/Road_to_Mobile_Beta_11#L10N once I actually edit that section.

Considering the work items I would be shuffling around would be required for a beta release of Firefox 11, nightlies have to take a lower priority.

I'm going to tackle this by porting *mobile?repacks* in http://hg.mozilla.org/build/tools/file/00bc79985c7d/scripts/l10n to mozharness, with nightly snippet creation.

any idea on the date when this will be ready?

I hope to be code complete on the script late today, and with tying it into our infrastructure by early next week.

(In reply to Aki Sasaki [:aki] from comment #6) > I hope to be code complete on the script late today, and with tying it into > our infrastructure by early next week. I have a working script creating updates for single locale repacks from the command line (staging). I need to tie it into buildbot to get that live, plus I need review for both portions, but things are looking good.

This patch refactors a bit of mozharness, moves some files around, and ports Android nightly- and release- single-locale repacks to mozharness, with snippet creation. * Created mozilla-{central,aurora} nightly configs for single locale android * Created mozilla-{beta,release} release configs for single locale android * Fixed --list-actions when c['default_actions'] is specified * Moved JarsignerErrorList from sign_android.py to mozharness.base.errors * Added mozharness.base.parallel with ChunkingMixin * Allowed for multiple self.query_env() variables * Moved generic add_failure() and query_failure() methods to BaseScript as you suggested * Added BaseScript.summarize_success_count() since I found myself repeating the same code over and over. Naming suggestions welcome :) * Added mozharness.base.signing with query_filesize() and query_sha512sum() from sign_android.py * Renamed c['share_base'] to c['vcs_share_base'] for more explicit variable recognition * Moved mozharness.buildbot to mozharness.mozilla.buildbot * Moved mozharness.l10n.* to mozharness.mozilla.l10n.* * Changed LocalesMixin.query_locales() to use ChunkingMixin.query_chunked_list() if 'total_locale_chunks' and 'this_locale_chunk' are set * Added a list_locales() method and action * Added a mozharness.mozilla.release with query_release_config() from sign_android.py. I'm not sure we need a mozharness.mozilla.release and mozharness.mozilla.signing, but that's what I have right now. * Added a mozharness.mozilla.signing with create_complete_snippet() and verify_android_signature() from sign_android.py * Added scripts/mobile_l10n.py that replaces everything mobile in http://hg.mozilla.org/build/tools/file/00bc79985c7d/scripts/l10n , with snippet creation and upload Tested mozilla-{central,aurora} nightlies and mozilla-beta release repacks. The release repacks don't create snippets; sign_android.py will handle that. In a followup bug, I'd like to add per-locale logging and reporting. However, I'm short on time and I have per-locale summaries at the end of the log. Rail: I know this is a huge huge patch. Let me know if you need me to explain this or split this up or what, and I'll be around to buy you beer. Next I need to tie this into buildbot.

This is basically a |git diff master| from the tip of the android-l10n branch ( https://github.com/escapewindow/mozharness/tree/android-l10n ), revision 04a43533344df7e2a49b09773eb82d4dd0097963 .

I see no danger in this as I see zero chance of a chemspill 10.0.x needing mobile desktop repacks.

This still needs testing. I'll land my mozharness patch in a user repo and test mobile repacks in buildbot.

Still needs more testing, but the command line for the nightly looks good now.

Nightly: great success. http://dev-master01.build.mozilla.org:8052/builders/Android%20mozilla-central%20l10n%20nightly%201%2F5/builds/3/steps/run_script/logs/stdio Need to test release; also need to sleep.

Release: great success. http://dev-master01.build.mozilla.org:8052/builders/release-mozilla-beta-android_repack_1%2F2/builds/2/steps/run_script/logs/stdio This turned red because the gd locale is broken.

Comment on attachment 594406 [details] [diff] [review] mozharness mobile_l10n.py Review of attachment 594406 [details] [diff] [review]: ----------------------------------------------------------------- Overall comments. * There are a lot of unused imports in the libs/files, but I think I need to file a separate file for this. * I understand that we need these changes "yesterday", so I tried to avoid being a style/grammar nazi. :) However there are 2 changes that should be fixed before we land this: aus_server name from branchConfig and opening files in binary mode. ::: configs/single_locale/mozilla-aurora_android.py @@ +10,5 @@ > +STAGE_SSH_KEY = "~/.ssh/ffxbld_dsa" > +#AUS_SERVER = "dev-stage01.build.sjc1.mozilla.com" > +AUS_SERVER = "aus2-staging.mozilla.org" > +AUS_USER = "ffxbld" > +AUS_SSH_KEY = "~/.ssh/ffxbld_dsa" Could you check if we use ffxbld for aus uploading here? At least for Firefox releases we use cltbld user. @@ +51,5 @@ > + "JARSIGNER": "%(abs_work_dir)s/" + JARSIGNER, > + "LOCALE_MERGEDIR": "%(abs_merge_dir)s/", > + }, > + # TODO ideally we could get this info from a central location. > + # However, the agility of these individual config files might trump that. yeah, could use releaseConfig here, maybe from the script/libs. ::: configs/single_locale/mozilla-central_android.py @@ +10,5 @@ > +STAGE_SSH_KEY = "~/.ssh/ffxbld_dsa" > +#AUS_SERVER = "dev-stage01.build.sjc1.mozilla.com" > +AUS_SERVER = "aus2-staging.mozilla.org" > +AUS_USER = "ffxbld" > +AUS_SSH_KEY = "~/.ssh/ffxbld_dsa" the same here ::: configs/single_locale/release_mozilla-beta_android.py @@ +7,5 @@ > +EN_US_BINARY_URL = "http://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/%(version)s-candidates/build%(buildnum)d/unsigned/android/en-US" > +#STAGE_SERVER = "dev-stage01.build.sjc1.mozilla.com" > +STAGE_SERVER = "stage.mozilla.org" > +STAGE_USER = "ffxbld" > +STAGE_SSH_KEY = "~/.ssh/ffxbld_dsa" the smae @@ +31,5 @@ > + "revision": "default", > + "dest": "tools" > + },{ > + "repo": "http://hg.mozilla.org/l10n/compare-locales", > + "revision": "RELEASE_0_9_4" Shouldn't we use a repo from hg.m.o/build/ ? and RELEASE_AUTOMATION tag? ::: configs/single_locale/release_mozilla-release_android.py @@ +7,5 @@ > +EN_US_BINARY_URL = "http://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/%(version)s-candidates/build%(buildnum)d/unsigned/android/en-US" > +#STAGE_SERVER = "dev-stage01.build.sjc1.mozilla.com" > +STAGE_SERVER = "stage.mozilla.org" > +STAGE_USER = "ffxbld" > +STAGE_SSH_KEY = "~/.ssh/ffxbld_dsa" ... ::: mozharness/base/signing.py @@ +57,5 @@ > + # TODO this should be parallelized with the to-be-written BaseHelper! > + def query_sha512sum(self, file_path): > + self.info("Determining sha512sum for %s" % file_path) > + m = hashlib.sha512() > + fh = open(file_path) could you open files using binary mode here? fh = open(file_path, 'rb') It's very important for hashing if you use it on windows (I know, that we're are not going to run it in windows now, but since it's in library, it should work properly). ::: mozharness/mozilla/release.py @@ +73,5 @@ > + self.release_config['old_buildnum'] = rc['oldBuildNumber'] > + self.release_config['ftp_server'] = rc['stagingServer'] > + self.release_config['ftp_user'] = c.get('ftp_user', rc['hgUsername']) > + self.release_config['ftp_ssh_key'] = c.get('ftp_ssh_key', rc['hgSshKey']) > + self.release_config['aus_server'] = rc['stagingServer'] I think this won't work for production. you should use branchConfig['aus2_host'] instead

Comment on attachment 595123 [details] [diff] [review] run pyflakes Can you update the header to add 'easy_install pyflakes' ? And this is cool, thanks!

Comment on attachment 595123 [details] [diff] [review] run pyflakes http://hg.mozilla.org/build/mozharness/rev/607f89614e8c

Please don't add tags to the repo in hg.m.o/l10n/compare-locales, but just to /build/compare-locales. The tags are the primary reason why I moved to a different repo to begin with.

(In reply to Axel Hecht [:Pike] from comment #20) > Please don't add tags to the repo in hg.m.o/l10n/compare-locales, but just > to /build/compare-locales. The tags are the primary reason why I moved to a > different repo to begin with. This script doesn't add any tags.

(In reply to Rail Aliiev [:rail] from comment #16) Ok, all but one thing addressed in https://github.com/escapewindow/mozharness/commits/android-l10n . > * There are a lot of unused imports in the libs/files, but I think I need to > file a separate file for this. I fixed this once I knew to use pyflakes. I still kind of prefer to have a number of these imported for easy usage later, but no biggie. > * I understand that we need these changes "yesterday", so I tried to avoid > being a style/grammar nazi. :) Thank you :) Feel free to file followup bugs? > However there are 2 changes that should be > fixed before we land this: aus_server name from branchConfig and opening > files in binary mode. Ok. > > > +AUS_SERVER = "aus2-staging.mozilla.org" > > +AUS_USER = "ffxbld" > > +AUS_SSH_KEY = "~/.ssh/ffxbld_dsa" > > Could you check if we use ffxbld for aus uploading here? At least for > Firefox releases we use cltbld user. This is for nightlies only; those are uploaded as ffxbld. > @@ +51,5 @@ > > + "JARSIGNER": "%(abs_work_dir)s/" + JARSIGNER, > > + "LOCALE_MERGEDIR": "%(abs_merge_dir)s/", > > + }, > > + # TODO ideally we could get this info from a central location. > > + # However, the agility of these individual config files might trump that. > > yeah, could use releaseConfig here, maybe from the script/libs. This is largely for nightlies; importing from config.py is ugly enough that I'm avoiding it. > @@ +31,5 @@ > > + "revision": "default", > > + "dest": "tools" > > + },{ > > + "repo": "http://hg.mozilla.org/l10n/compare-locales", > > + "revision": "RELEASE_0_9_4" > > Shouldn't we use a repo from hg.m.o/build/ ? and RELEASE_AUTOMATION tag? Yes, though l10n/compare-locales is more up to date. Looks like these fail whether we use l10n/compare-locales or build/compare-locales, so I'll use the tried-and-tested one. > ::: mozharness/base/signing.py > @@ +57,5 @@ > > + # TODO this should be parallelized with the to-be-written BaseHelper! > > + def query_sha512sum(self, file_path): > > + self.info("Determining sha512sum for %s" % file_path) > > + m = hashlib.sha512() > > + fh = open(file_path) > > could you open files using binary mode here? > fh = open(file_path, 'rb') > It's very important for hashing if you use it on windows (I know, that we're > are not going to run it in windows now, but since it's in library, it should > work properly). Done! > ::: mozharness/mozilla/release.py > @@ +73,5 @@ > > + self.release_config['old_buildnum'] = rc['oldBuildNumber'] > > + self.release_config['ftp_server'] = rc['stagingServer'] > > + self.release_config['ftp_user'] = c.get('ftp_user', rc['hgUsername']) > > + self.release_config['ftp_ssh_key'] = c.get('ftp_ssh_key', rc['hgSshKey']) > > + self.release_config['aus_server'] = rc['stagingServer'] > > I think this won't work for production. you should use > branchConfig['aus2_host'] instead As discussed in person, I *really* don't want to touch config.py. I'm going to add an aus2_host in releaseConfig first, then use it.

This was generated by git diff 04a43533344df7e2a49b09773eb82d4dd0097963 from the android-l10n branch, revision fd05c0b9ccb33d48fe672837707ddb2615f2546c ( https://github.com/escapewindow/mozharness/commits/android-l10n )

Comment on attachment 595180 [details] [diff] [review] (mozharness) changes on top of the previous mozharness patch yay!

Comment on attachment 595176 [details] [diff] [review] (configs) same, with releaseConfig['ausHost'] http://hg.mozilla.org/build/buildbot-configs/rev/5fbd5ea58118

Comment on attachment 595177 [details] [diff] [review] (custom) same, with releaseConfig['ausHost'] http://hg.mozilla.org/build/buildbotcustom/rev/80ea7d6565a2

Comment on attachment 595180 [details] [diff] [review] (mozharness) changes on top of the previous mozharness patch http://hg.mozilla.org/build/mozharness/rev/cdeccf7f7fa8

This is live now.

I think this broke in 10.0.1 :( All of the linux-mobile repacks failed with (and presumably the other platforms will too): scripts/scripts/mobile_l10n.py --cfg single_locale/release_mozilla-release_linux-mobile.py --total-chunks 6 --this-chunk 2 in dir /builds/slave/rel-m-rel-lnx-mb-rpk-2/. (timeout 1200 secs) watching logfiles {} argv: ['scripts/scripts/mobile_l10n.py', '--cfg', 'single_locale/release_mozilla-release_linux-mobile.py', '--total-chunks', '6', '--this-chunk', '2'] Traceback (most recent call last): File "scripts/scripts/mobile_l10n.py", line 490, in <module> single_locale = MobileSingleLocale() File "scripts/scripts/mobile_l10n.py", line 136, in __init__ require_config_file=require_config_file File "/builds/slave/rel-m-rel-lnx-mb-rpk-2/scripts/mozharness/base/vcs/vcsbase.py", line 107, in __init__ super(VCSScript, self).__init__(**kwargs) File "/builds/slave/rel-m-rel-lnx-mb-rpk-2/scripts/mozharness/base/script.py", line 560, in __init__ **kwargs) File "/builds/slave/rel-m-rel-lnx-mb-rpk-2/scripts/mozharness/base/config.py", line 211, in __init__ self.parse_args() File "/builds/slave/rel-m-rel-lnx-mb-rpk-2/scripts/mozharness/base/config.py", line 361, in parse_args self.set_config(parse_config_file(options.config_file)) File "/builds/slave/rel-m-rel-lnx-mb-rpk-2/scripts/mozharness/base/config.py", line 155, in parse_config_file raise IOError, "Can't find %s in %s!" % (file_name, search_path) IOError: Can't find single_locale/release_mozilla-release_linux-mobile.py in ['.', '/builds/slave/rel-m-rel-lnx-mb-rpk-2/scripts/scripts/../configs', '/builds/slave/rel-m-rel-lnx-mb-rpk-2/scripts/scripts/../../configs']! program finished with exit code 1 elapsedTime=0.139206

(In reply to Ben Hearsum [:bhearsum] from comment #31) > I think this broke in 10.0.1 :( > "/builds/slave/rel-m-rel-lnx-mb-rpk-2/scripts/mozharness/base/config.py", > line 155, in parse_config_file > raise IOError, "Can't find %s in %s!" % (file_name, search_path) > IOError: Can't find single_locale/release_mozilla-release_linux-mobile.py in > ['.', '/builds/slave/rel-m-rel-lnx-mb-rpk-2/scripts/scripts/../configs', > '/builds/slave/rel-m-rel-lnx-mb-rpk-2/scripts/scripts/../../configs']! > program finished with exit code 1 > elapsedTime=0.139206 + [...] 'single_locale/release_%s_%s.py' % (releaseConfig['sourceRepositories']['mobile']['name'], platform), so, looks like these vars are not what was thought here. (e.x. /configs/single_locale/release_mozilla-release_android.py exists) which means ...['name'] is obviously mozilla-release but platform here is 'linux-mobile' not 'android' http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/release-fennec-mozilla-release.py#81 is what we are using here, while misc.py uses just the keys from 'platforms' Hope that helps

Sorry Ben; I thought Lukas was running 10.0.1. I intentionally landed this even though it would break chemspill mobile repacks, as we have never really shipped mobile desktop builds and so a chemspill of an unshipped platform seemed lower priority than unblocking 11.0. For 10.0.1 we should ignore mobile desktop repacks; for 10.0.2+ we should stop running them.