Closed Bug 718385 Opened 13 years ago Closed 13 years ago

Create update channels for single locale aurora builds for Firefox Mobile

Categories

(Release Engineering :: General, defect, P1)

ARM
Android
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: willyaranda, Assigned: mozilla)

References

Details

Attachments

(5 files, 4 obsolete files)

There is no updates for builds in http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-aurora-android-l10n/ I would be wonderful to have updates in Fennec instead of downloading them manually.
Yes, this is known. I know how to tackle this, but I have a couple items with a hard deadline of Jan 31 so I may have to work on this afterwards.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Aki, can you shuffle things around here? Jan 31 means that we're basically not having completed infrastructure for l10n for Firefox 11 on Aurora, and possibly into Beta. CCing folks as I expect this bug to show up on https://wiki.mozilla.org/Release_Management/Road_to_Mobile_Beta_11#L10N once I actually edit that section.
Considering the work items I would be shuffling around would be required for a beta release of Firefox 11, nightlies have to take a lower priority.
Blocks: 697384
Priority: -- → P3
Assignee: nobody → aki
OS: Mac OS X → Android
Priority: P3 → P1
Hardware: x86 → ARM
I'm going to tackle this by porting *mobile?repacks* in http://hg.mozilla.org/build/tools/file/00bc79985c7d/scripts/l10n to mozharness, with nightly snippet creation.
any idea on the date when this will be ready?
I hope to be code complete on the script late today, and with tying it into our infrastructure by early next week.
(In reply to Aki Sasaki [:aki] from comment #6) > I hope to be code complete on the script late today, and with tying it into > our infrastructure by early next week. I have a working script creating updates for single locale repacks from the command line (staging). I need to tie it into buildbot to get that live, plus I need review for both portions, but things are looking good.
Attached patch mozharness mobile_l10n.py (deleted) — Splinter Review
This patch refactors a bit of mozharness, moves some files around, and ports Android nightly- and release- single-locale repacks to mozharness, with snippet creation. * Created mozilla-{central,aurora} nightly configs for single locale android * Created mozilla-{beta,release} release configs for single locale android * Fixed --list-actions when c['default_actions'] is specified * Moved JarsignerErrorList from sign_android.py to mozharness.base.errors * Added mozharness.base.parallel with ChunkingMixin * Allowed for multiple self.query_env() variables * Moved generic add_failure() and query_failure() methods to BaseScript as you suggested * Added BaseScript.summarize_success_count() since I found myself repeating the same code over and over. Naming suggestions welcome :) * Added mozharness.base.signing with query_filesize() and query_sha512sum() from sign_android.py * Renamed c['share_base'] to c['vcs_share_base'] for more explicit variable recognition * Moved mozharness.buildbot to mozharness.mozilla.buildbot * Moved mozharness.l10n.* to mozharness.mozilla.l10n.* * Changed LocalesMixin.query_locales() to use ChunkingMixin.query_chunked_list() if 'total_locale_chunks' and 'this_locale_chunk' are set * Added a list_locales() method and action * Added a mozharness.mozilla.release with query_release_config() from sign_android.py. I'm not sure we need a mozharness.mozilla.release and mozharness.mozilla.signing, but that's what I have right now. * Added a mozharness.mozilla.signing with create_complete_snippet() and verify_android_signature() from sign_android.py * Added scripts/mobile_l10n.py that replaces everything mobile in http://hg.mozilla.org/build/tools/file/00bc79985c7d/scripts/l10n , with snippet creation and upload Tested mozilla-{central,aurora} nightlies and mozilla-beta release repacks. The release repacks don't create snippets; sign_android.py will handle that. In a followup bug, I'd like to add per-locale logging and reporting. However, I'm short on time and I have per-locale summaries at the end of the log. Rail: I know this is a huge huge patch. Let me know if you need me to explain this or split this up or what, and I'll be around to buy you beer. Next I need to tie this into buildbot.
Attachment #594406 - Flags: review?(rail)
This is basically a |git diff master| from the tip of the android-l10n branch ( https://github.com/escapewindow/mozharness/tree/android-l10n ), revision 04a43533344df7e2a49b09773eb82d4dd0097963 .
I see no danger in this as I see zero chance of a chemspill 10.0.x needing mobile desktop repacks.
This still needs testing. I'll land my mozharness patch in a user repo and test mobile repacks in buildbot.
Still needs more testing, but the command line for the nightly looks good now.
Attachment #594409 - Attachment is obsolete: true
Attached patch (buildbotcustom) fix release config path (obsolete) (deleted) — Splinter Review
Attachment #594410 - Attachment is obsolete: true
Attachment #594408 - Flags: review?(rail)
Attachment #594451 - Flags: review?(rail)
Attachment #594408 - Flags: review?(rail) → review+
Attachment #594451 - Flags: review?(rail) → review+
Comment on attachment 594406 [details] [diff] [review] mozharness mobile_l10n.py Review of attachment 594406 [details] [diff] [review]: ----------------------------------------------------------------- Overall comments. * There are a lot of unused imports in the libs/files, but I think I need to file a separate file for this. * I understand that we need these changes "yesterday", so I tried to avoid being a style/grammar nazi. :) However there are 2 changes that should be fixed before we land this: aus_server name from branchConfig and opening files in binary mode. ::: configs/single_locale/mozilla-aurora_android.py @@ +10,5 @@ > +STAGE_SSH_KEY = "~/.ssh/ffxbld_dsa" > +#AUS_SERVER = "dev-stage01.build.sjc1.mozilla.com" > +AUS_SERVER = "aus2-staging.mozilla.org" > +AUS_USER = "ffxbld" > +AUS_SSH_KEY = "~/.ssh/ffxbld_dsa" Could you check if we use ffxbld for aus uploading here? At least for Firefox releases we use cltbld user. @@ +51,5 @@ > + "JARSIGNER": "%(abs_work_dir)s/" + JARSIGNER, > + "LOCALE_MERGEDIR": "%(abs_merge_dir)s/", > + }, > + # TODO ideally we could get this info from a central location. > + # However, the agility of these individual config files might trump that. yeah, could use releaseConfig here, maybe from the script/libs. ::: configs/single_locale/mozilla-central_android.py @@ +10,5 @@ > +STAGE_SSH_KEY = "~/.ssh/ffxbld_dsa" > +#AUS_SERVER = "dev-stage01.build.sjc1.mozilla.com" > +AUS_SERVER = "aus2-staging.mozilla.org" > +AUS_USER = "ffxbld" > +AUS_SSH_KEY = "~/.ssh/ffxbld_dsa" the same here ::: configs/single_locale/release_mozilla-beta_android.py @@ +7,5 @@ > +EN_US_BINARY_URL = "http://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/%(version)s-candidates/build%(buildnum)d/unsigned/android/en-US" > +#STAGE_SERVER = "dev-stage01.build.sjc1.mozilla.com" > +STAGE_SERVER = "stage.mozilla.org" > +STAGE_USER = "ffxbld" > +STAGE_SSH_KEY = "~/.ssh/ffxbld_dsa" the smae @@ +31,5 @@ > + "revision": "default", > + "dest": "tools" > + },{ > + "repo": "http://hg.mozilla.org/l10n/compare-locales", > + "revision": "RELEASE_0_9_4" Shouldn't we use a repo from hg.m.o/build/ ? and RELEASE_AUTOMATION tag? ::: configs/single_locale/release_mozilla-release_android.py @@ +7,5 @@ > +EN_US_BINARY_URL = "http://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/%(version)s-candidates/build%(buildnum)d/unsigned/android/en-US" > +#STAGE_SERVER = "dev-stage01.build.sjc1.mozilla.com" > +STAGE_SERVER = "stage.mozilla.org" > +STAGE_USER = "ffxbld" > +STAGE_SSH_KEY = "~/.ssh/ffxbld_dsa" ... ::: mozharness/base/signing.py @@ +57,5 @@ > + # TODO this should be parallelized with the to-be-written BaseHelper! > + def query_sha512sum(self, file_path): > + self.info("Determining sha512sum for %s" % file_path) > + m = hashlib.sha512() > + fh = open(file_path) could you open files using binary mode here? fh = open(file_path, 'rb') It's very important for hashing if you use it on windows (I know, that we're are not going to run it in windows now, but since it's in library, it should work properly). ::: mozharness/mozilla/release.py @@ +73,5 @@ > + self.release_config['old_buildnum'] = rc['oldBuildNumber'] > + self.release_config['ftp_server'] = rc['stagingServer'] > + self.release_config['ftp_user'] = c.get('ftp_user', rc['hgUsername']) > + self.release_config['ftp_ssh_key'] = c.get('ftp_ssh_key', rc['hgSshKey']) > + self.release_config['aus_server'] = rc['stagingServer'] I think this won't work for production. you should use branchConfig['aus2_host'] instead
Attachment #594406 - Flags: review?(rail) → review-
Attached patch run pyflakes (deleted) — Splinter Review
Attachment #595123 - Flags: review?(aki)
Comment on attachment 595123 [details] [diff] [review] run pyflakes Can you update the header to add 'easy_install pyflakes' ? And this is cool, thanks!
Attachment #595123 - Flags: review?(aki) → review+
Please don't add tags to the repo in hg.m.o/l10n/compare-locales, but just to /build/compare-locales. The tags are the primary reason why I moved to a different repo to begin with.
(In reply to Axel Hecht [:Pike] from comment #20) > Please don't add tags to the repo in hg.m.o/l10n/compare-locales, but just > to /build/compare-locales. The tags are the primary reason why I moved to a > different repo to begin with. This script doesn't add any tags.
(In reply to Rail Aliiev [:rail] from comment #16) Ok, all but one thing addressed in https://github.com/escapewindow/mozharness/commits/android-l10n . > * There are a lot of unused imports in the libs/files, but I think I need to > file a separate file for this. I fixed this once I knew to use pyflakes. I still kind of prefer to have a number of these imported for easy usage later, but no biggie. > * I understand that we need these changes "yesterday", so I tried to avoid > being a style/grammar nazi. :) Thank you :) Feel free to file followup bugs? > However there are 2 changes that should be > fixed before we land this: aus_server name from branchConfig and opening > files in binary mode. Ok. > > > +AUS_SERVER = "aus2-staging.mozilla.org" > > +AUS_USER = "ffxbld" > > +AUS_SSH_KEY = "~/.ssh/ffxbld_dsa" > > Could you check if we use ffxbld for aus uploading here? At least for > Firefox releases we use cltbld user. This is for nightlies only; those are uploaded as ffxbld. > @@ +51,5 @@ > > + "JARSIGNER": "%(abs_work_dir)s/" + JARSIGNER, > > + "LOCALE_MERGEDIR": "%(abs_merge_dir)s/", > > + }, > > + # TODO ideally we could get this info from a central location. > > + # However, the agility of these individual config files might trump that. > > yeah, could use releaseConfig here, maybe from the script/libs. This is largely for nightlies; importing from config.py is ugly enough that I'm avoiding it. > @@ +31,5 @@ > > + "revision": "default", > > + "dest": "tools" > > + },{ > > + "repo": "http://hg.mozilla.org/l10n/compare-locales", > > + "revision": "RELEASE_0_9_4" > > Shouldn't we use a repo from hg.m.o/build/ ? and RELEASE_AUTOMATION tag? Yes, though l10n/compare-locales is more up to date. Looks like these fail whether we use l10n/compare-locales or build/compare-locales, so I'll use the tried-and-tested one. > ::: mozharness/base/signing.py > @@ +57,5 @@ > > + # TODO this should be parallelized with the to-be-written BaseHelper! > > + def query_sha512sum(self, file_path): > > + self.info("Determining sha512sum for %s" % file_path) > > + m = hashlib.sha512() > > + fh = open(file_path) > > could you open files using binary mode here? > fh = open(file_path, 'rb') > It's very important for hashing if you use it on windows (I know, that we're > are not going to run it in windows now, but since it's in library, it should > work properly). Done! > ::: mozharness/mozilla/release.py > @@ +73,5 @@ > > + self.release_config['old_buildnum'] = rc['oldBuildNumber'] > > + self.release_config['ftp_server'] = rc['stagingServer'] > > + self.release_config['ftp_user'] = c.get('ftp_user', rc['hgUsername']) > > + self.release_config['ftp_ssh_key'] = c.get('ftp_ssh_key', rc['hgSshKey']) > > + self.release_config['aus_server'] = rc['stagingServer'] > > I think this won't work for production. you should use > branchConfig['aus2_host'] instead As discussed in person, I *really* don't want to touch config.py. I'm going to add an aus2_host in releaseConfig first, then use it.
Attachment #594408 - Attachment is obsolete: true
Attachment #595176 - Flags: review?(rail)
Attachment #594451 - Attachment is obsolete: true
Attachment #595177 - Flags: review?(rail)
Attachment #595176 - Flags: review?(rail) → review+
This was generated by git diff 04a43533344df7e2a49b09773eb82d4dd0097963 from the android-l10n branch, revision fd05c0b9ccb33d48fe672837707ddb2615f2546c ( https://github.com/escapewindow/mozharness/commits/android-l10n )
Attachment #595180 - Flags: review?(rail)
Attachment #595177 - Flags: review?(rail) → review+
Comment on attachment 595180 [details] [diff] [review] (mozharness) changes on top of the previous mozharness patch yay!
Attachment #595180 - Flags: review?(rail) → review+
Attachment #595176 - Flags: checked-in+
Attachment #595177 - Flags: checked-in+
Comment on attachment 595180 [details] [diff] [review] (mozharness) changes on top of the previous mozharness patch http://hg.mozilla.org/build/mozharness/rev/cdeccf7f7fa8
Attachment #595180 - Flags: checked-in+
This is live now.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
I think this broke in 10.0.1 :( All of the linux-mobile repacks failed with (and presumably the other platforms will too): scripts/scripts/mobile_l10n.py --cfg single_locale/release_mozilla-release_linux-mobile.py --total-chunks 6 --this-chunk 2 in dir /builds/slave/rel-m-rel-lnx-mb-rpk-2/. (timeout 1200 secs) watching logfiles {} argv: ['scripts/scripts/mobile_l10n.py', '--cfg', 'single_locale/release_mozilla-release_linux-mobile.py', '--total-chunks', '6', '--this-chunk', '2'] Traceback (most recent call last): File "scripts/scripts/mobile_l10n.py", line 490, in <module> single_locale = MobileSingleLocale() File "scripts/scripts/mobile_l10n.py", line 136, in __init__ require_config_file=require_config_file File "/builds/slave/rel-m-rel-lnx-mb-rpk-2/scripts/mozharness/base/vcs/vcsbase.py", line 107, in __init__ super(VCSScript, self).__init__(**kwargs) File "/builds/slave/rel-m-rel-lnx-mb-rpk-2/scripts/mozharness/base/script.py", line 560, in __init__ **kwargs) File "/builds/slave/rel-m-rel-lnx-mb-rpk-2/scripts/mozharness/base/config.py", line 211, in __init__ self.parse_args() File "/builds/slave/rel-m-rel-lnx-mb-rpk-2/scripts/mozharness/base/config.py", line 361, in parse_args self.set_config(parse_config_file(options.config_file)) File "/builds/slave/rel-m-rel-lnx-mb-rpk-2/scripts/mozharness/base/config.py", line 155, in parse_config_file raise IOError, "Can't find %s in %s!" % (file_name, search_path) IOError: Can't find single_locale/release_mozilla-release_linux-mobile.py in ['.', '/builds/slave/rel-m-rel-lnx-mb-rpk-2/scripts/scripts/../configs', '/builds/slave/rel-m-rel-lnx-mb-rpk-2/scripts/scripts/../../configs']! program finished with exit code 1 elapsedTime=0.139206
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to Ben Hearsum [:bhearsum] from comment #31) > I think this broke in 10.0.1 :( > "/builds/slave/rel-m-rel-lnx-mb-rpk-2/scripts/mozharness/base/config.py", > line 155, in parse_config_file > raise IOError, "Can't find %s in %s!" % (file_name, search_path) > IOError: Can't find single_locale/release_mozilla-release_linux-mobile.py in > ['.', '/builds/slave/rel-m-rel-lnx-mb-rpk-2/scripts/scripts/../configs', > '/builds/slave/rel-m-rel-lnx-mb-rpk-2/scripts/scripts/../../configs']! > program finished with exit code 1 > elapsedTime=0.139206 + [...] 'single_locale/release_%s_%s.py' % (releaseConfig['sourceRepositories']['mobile']['name'], platform), so, looks like these vars are not what was thought here. (e.x. /configs/single_locale/release_mozilla-release_android.py exists) which means ...['name'] is obviously mozilla-release but platform here is 'linux-mobile' not 'android' http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/release-fennec-mozilla-release.py#81 is what we are using here, while misc.py uses just the keys from 'platforms' Hope that helps
Sorry Ben; I thought Lukas was running 10.0.1. I intentionally landed this even though it would break chemspill mobile repacks, as we have never really shipped mobile desktop builds and so a chemspill of an unshipped platform seemed lower priority than unblocking 11.0. For 10.0.1 we should ignore mobile desktop repacks; for 10.0.2+ we should stop running them.
Status: REOPENED → RESOLVED
Closed: 13 years ago13 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: