This could be a simple mozconfig change, but I'm going to guess we have a lot of hardcoded BRANCH/mobile/ paths in buildbotcustom and mozharness. The mobile world will break if bug 701833 is ready to land and does so before we have this in place.

If the locales dir changes, repacks will break unless we change mobileDirName. http://hg.mozilla.org/build/tools/file/de826f781b54/scripts/l10n/nightly-mobile-repacks.py#l33 http://hg.mozilla.org/build/tools/file/de826f781b54/scripts/l10n/release-mobile-repacks.py#l37 However, I'd rather not hardcode "mobile/xul"; passing this in as an argument will allow us to use the same script for mobile/xul and mobile/android if needed. Plus, it'll be mobile/xul on mozilla-central, but mozilla-aurora, mozilla-beta, and mozilla-release will still have mobile/ until that patch merges.

locales_file and locales_dir will need to change in the mozharness multilocale configs. In addition, I think we'll need another set of configs to differentiate native and xul android builds. http://hg.mozilla.org/build/mozharness/file/cbcc9a5d5549/configs/multi_locale/mozilla-central_linux-android.json#l5

This will fix the mozconfig location for mobile/xul builds. However, a) I need to pass in the mobile/xul or mobile/android or mobile/ directory to the single locale repack scripts, which will also take a buildbotcustom patch; b) this patch will change the mozconfig location for *all* android and mobile desktop builds, including aurora, beta, and release, even before the mobile/xul patch has merged into those repositories; and c) this patch doesn't necessarily allow for a native android platform. I think (a) should be easy enough to add once we have the appropriate single locale script patches. For (c), I'm thinking about biting the bullet and renaming 'linux-android' to 'android-xul' and adding 'android' (native). This exacerbates (b), however, since the platform rename would affect aurora, beta, and release. To resolve that, we can c1) bite the bullet and rename across all repos, with appropriate mozconfig merges etc. where needed, or c2) leave the platform named 'linux-android' on aurora, beta, and release, while building the platforms named 'android-xul' and 'android' on mozilla-central and m-c-based project branches. This makes try ugly and may be confusing.

Alternately, for (c), we could have 'linux-android' refer to android (xul), and add an 'android' platform that is native. This is confusing for humans but would be nicer for automation. Whatever solution we choose here, I'll need to add and/or rename the appropriate mozharness multilocale config files.

Sorry for not making this transparent, I'm not yet sure what exactly to do in bug 701833 wrt l10n. There's still more data tickling in on that, even.

(In reply to Axel Hecht [:Pike] from comment #5) > Sorry for not making this transparent, I'm not yet sure what exactly to do > in bug 701833 wrt l10n. There's still more data tickling in on that, even. That's ok. The patches themselves don't look as difficult as figuring out logistics. I'm going to semi-randomly choose one of these solutions to write patches for, and adjust if/when we make decisions or find out more information.

Assuming that mobile/config and mobile/locales stay where they are, we can leave linux-android alone to build android (xul). This patch adds android (native). Currently it points at mobile/ and the same mozconfigs still, since Doug's patch and the subsequent birch merge haven't landed. However, running this on my staging master with a mozharness patch that copies the m-c linux-android multilocale config to the appropriately-named android-native config, I was able to get a green mozilla-central multilocale Android native nightly, although it borked on upload again due to Coop's post_upload.py patch on dev-stage01.

Comment on attachment 574129 [details] [diff] [review] mobile single locale repack mobile/xul support 8.0.1 is non-android, and this patch should work as-is for the existing mobile/ layout. Landing to have one fewer loose end to deal with while staging patches. http://hg.mozilla.org/build/tools/rev/20ea6a354051

Per email, the mobile directories will be mobile/xul/ config locales mobile/android/ config locales Right now I'm going with 'android' (native) and 'android-xul' (xul) as the platform names.

This passes checkconfig and I see Android XUL builders. TODO * mozconfig changes * mozilla-tests changes * trychooser * project-branch changes (birch will run 'android' only, no 'android-xul') * verify my paths match against dougt's user repo (pending user repo) * test test test

android/native/ will not be landing at the same time, but after discussion with mobile + releng I'll turn this on and hide the perma-red builds until it lands.

Adding the configs ahead of time doesn't hurt, and will make merge time a lot simpler.

These are the mozconfigs in buildbot-configs that our automation will fall back to if they can't find the appropriate mozconfigs in mobile/xul/config/ or mobile/android/config/. These update the aurora, beta, and release mozconfigs but won't break existing builds on those platforms, because those branches are currently using the 'linux-android' platform. Doug: I think these paths are correct. Could you review and/or reassign the r? to someone who can?

Comment on attachment 574703 [details] [diff] [review] mozconfigs for 'android', 'android-xul', 'android-debug' platforms I do not see the hg copies. Maybe those are hidden by your hgrc. Spoke to aki about this a bit. patch looks fine.

Comment on attachment 574703 [details] [diff] [review] mozconfigs for 'android', 'android-xul', 'android-debug' platforms http://hg.mozilla.org/build/buildbot-configs/rev/d92391c7908f

This patch creates the needed builders; I've tested a mozilla-aurora nightly. I can't test the android and android-xul builds until I get a user repo or patch afaict. However, when I attach a tegra to my test master, it doesn't launch any aurora tests. When I reconfig my test master to clean configs, it launches aurora tests. I need to figure this out pronto.

This works! HOWEVER, as is, it will kill all tegra tests on a single production or staging master; this passes the magical 1024 MAX_BROKER_REFS threshold. I was able to work around this with the various methods: a) Reduce the platform list: 'android' and 'android-xul' can't be on the same test master. (linux-android has few enough builders that it doesn't matter.) We could spin up another master with the other platform. This would split our tegra pool since we don't have slavealloc for tegras. b) Reduce the ACTIVE_BRANCHES. If we split branches across masters, we'll split our pool just like splitting tegra platforms. If, however, we turn off a number of these project branches that aren't actually running any android tests, we reduce the builder count without affecting anything in the real world. I'm not currently sure if we can cover all needed branches and stay under 1024 builders. Alternately, if we can update the mozilla-tests configs to only turn on the test platforms and suites we want per branch, we may get under 1024 (this would be even fewer builders than turning off all unneeded branches). This will require some precise config. c) Split unit test and talos tegras on the same master. This will split our pool. d) Or we can bump MAX_BROKER_REFS: import twisted.spread.pb twisted.spread.pb.MAX_BROKER_REFS = 2048 Afaict this needs to go into the tegra's buildbot.tac as well as the test master's buildbot.tac. After I added this to tegra-018 and tegra-028 and updated my dev-master01 buildbot.tac, I was able to run tests with all the builders enabled. I'd like review on this; I'll also keep hammering on it in staging all day Wednesday to try to prepare for Thursday's downtime.

Comment on attachment 574838 [details] [diff] [review] configs - tegra_android-o Feedback only, as this patch may change slightly before it's ready to land. However, it's tested and I only think there will be small changes in the final patch.

Comment on attachment 574697 [details] [diff] [review] mozharness - android and android-xul multilocale configs for all branches http://hg.mozilla.org/build/mozharness/rev/ba962b2f5475

Also adds jsreftest-3 and crashtest-3, and removes trailing whitespace.

As the patch currently stands, birch will be building out of mobile/android/ as of Thursday. I pinged Doug to ask what the timing is; I can build out of mobile/ instead until they're ready (linux-android instead of android).

still investigating, but it looks like some tegra builders are showing up on non-tegra masters, looking into why

Comment on attachment 574838 [details] [diff] [review] configs - tegra_android-o Review of attachment 574838 [details] [diff] [review]: ----------------------------------------------------------------- Looks good! Without actually testing it myself on dev-master01, I don't see any issues other than mentioned below. ::: mozilla-tests/BuildSlaves.py.template @@ +14,5 @@ > "w764": "pass", > "vista": "pass", > "tegra_android": "pass", > + "tegra_android-xul": "pass", > + "tegra_android-o": "pass", We'll need to deploy the changes to the real BuildSlaves.py using puppet, similar to http://hg.mozilla.org/build/puppet-manifests/rev/93793d168bdd ::: mozilla-tests/config.py @@ +963,5 @@ > + if 'linux-android' in BRANCHES[branch]['platforms']: > + del BRANCHES[branch]['platforms']['linux-android'] > +#------------------------------------------------------------------------- > +# End 11.0 hacks. > +#------------------------------------------------------------------------- I don't know for sure that this is deleting everything properly for a generic test master. Given that limit_platforms is set in a way that excludes android on all non-tm01 masters, this is probably not a concern. ::: mozilla/BuildSlaves.py.template @@ +9,5 @@ > 'macosx64': 'pass', > 'try-macosx': 'pass', > 'linux-android': 'pass', > + 'android': 'pass', > + 'android-xul': 'pass', this also needs to be deployed with puppet ::: mozilla/config.py @@ +655,5 @@ > + 'SYMBOL_SERVER_PATH': SYMBOL_SERVER_MOBILE_PATH, > + 'SYMBOL_SERVER_SSH_KEY': "/home/cltbld/.ssh/ffxbld_dsa", > + 'POST_SYMBOL_UPLOAD_CMD': SYMBOL_SERVER_POST_UPLOAD_CMD, > + 'TINDERBOX_OUTPUT': '1', > + 'AKI_DELETEME': '1', this is set in a couple places, looks like debugging code or something. Obviously, should be removed from the final landing @@ +1318,5 @@ > +# Delete the below lines when 11.0 merges into release > +#------------------------------------------------------------------------- > +del BRANCHES['mozilla-release']['platforms']['android'] > +del BRANCHES['mozilla-release']['platforms']['android-debug'] > +del BRANCHES['mozilla-release']['platforms']['android-xul'] In general, I prefer the approach of never having added these platforms instead of deleting them, but I understand the reason and think this is fine. Having really clear comments about how to delete these things is great!

Comment on attachment 574838 [details] [diff] [review] configs - tegra_android-o http://hg.mozilla.org/build/buildbot-configs/rev/767226f46ea5

Comment on attachment 574944 [details] [diff] [review] android trychooser buildbotcustom http://hg.mozilla.org/build/buildbotcustom/rev/017c767bcfef

Comment on attachment 575063 [details] [diff] [review] puppet diff http://hg.mozilla.org/build/puppet-manifests/rev/7be2a11d14ee

Comment on attachment 575075 [details] [diff] [review] buildbot.tac update for MAX_BROKER_REFS http://hg.mozilla.org/build/tools/rev/3301fdc6c361

Due to https://bugzilla.mozilla.org/show_bug.cgi?id=701833#c35 .

Comment on attachment 574943 [details] [diff] [review] trychooser: linux-android -> android, android-xul http://hg.mozilla.org/build/tools/rev/bfd03d6b52a6

Comment on attachment 575160 [details] [diff] [review] disable multilocale on m-c and birch http://hg.mozilla.org/build/buildbot-configs/rev/38b1c8f36d9d

Not sure if this will work, but I'm guessing ? We need to be able to turn off 'android' while keeping 'android-xul' jobs unhidden.

Comment on attachment 575183 [details] [diff] [review] disable multilocale on m-c and birch, buildbotcustom edition http://hg.mozilla.org/build/buildbotcustom/rev/f9d1cc967b95

Comment on attachment 575169 [details] [diff] [review] differentiate android and android-xul http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/rev/a43187ed8661 Not entirely sure if this will fix things, but Philor r+ed it. Landed. Also not sure how to get this live.

Tbpl and trychooser need updating; everything else is done. Mobile is filing bugs for followup for elf-hack, m-c + birch localization, and merging mobile/android into m-c.

Aki, can you describe the build process you're implementing, in particular the release process, on some 10 steps? I could use that for digesting the changes on the l10n front.

(In reply to Axel Hecht [:Pike] from comment #40) > Aki, can you describe the build process you're implementing, in particular > the release process, on some 10 steps? I could use that for digesting the > changes on the l10n front. I would be happy to describe or discuss anything to help you with mobile localization; I'm not entirely sure what you're asking for right now, specifically. I'll try figuring out what would be helpful and write something up; if you have specific questions or if you want a real time discussion that works too.

Lukas updated trychooser; Philor verified tbpl-dev is good with my patch, which doesn't need to land in tbpl production until mobile/android/ lands in m-c. Added a TODO item to my list to help Axel with comment 40. I'm going to call this done.

(In reply to Aki Sasaki [:aki] from comment #42) > Added a TODO item to my list to help Axel with comment 40. I'm going to continue this discussion in bug 702302 (or bug 697384 ?).

Precise question is: Doug has asked us to localize two different applications for xul and native UI, not sure if that's messaged to you guys. In particular that that creates two independent l10n-changesets.json, with different locales and revisions for each. Is that the same thing you implemented? He did mention he had it figured out with you, just checking the technical details.

(In reply to Axel Hecht [:Pike] from comment #44) > Precise question is: Doug has asked us to localize two different > applications for xul and native UI, not sure if that's messaged to you guys. > In particular that that creates two independent l10n-changesets.json, with > different locales and revisions for each. Is that the same thing you > implemented? He did mention he had it figured out with you, just checking > the technical details. Is that needed? If, say, XUL UI locale revision is X native UI locale revision is X + 5 Is there a negative impact to specifying X + 5 as the revision for both products? Also, I'm not absolutely sure about this, but I'm pretty sure we can solve this with platforms in l10n-changesets.json, e.g. 'android' and 'android-xul'. Even if we *don't* solve it there, we'll be turning off multilocale and turning on single locale. If we repack more locales than we plan on shipping, we can not ship certain locales even if we repack them. Does that help?

This is fall-out from not having a single top-level locales/en-US directory for the versions, I can't work around that in another way that shipping two different applications, which doug asked me to do, very explicitly. And that will lead to two different sets of l10n changesets, double ship-it etc. We'll try to get a better setup for the discussion of how to build and release a localized fennec, given travel and thanksgiving next week, timing will be tough-to-unfortunate. PS: didn't realize the l10n-changesets-vs-release aspect among all the various bits of fallout during that chat with Doug.

I honestly don't see a need for that. We won't be getting 2 go's; we'll be getting a single go that tells us to build XUL or native or both. We're controlling where we run localization by platform-specific variables. android builds out of mobile/android (so mobile/android/locales); android-xul builds out of mobile/xul. I might be overlooking something, but if you unhardcode 'mobile/' and replace it with 'mobile/android/' for the android platform and 'mobile/xul/' for android-xul I think everything will be fine.

I can't do that, it's just not how the system works, let alone the complete business logic in the l10n dashboard on top of that.

Mobile believes that string changes in mobile/xul will be a rare exception, so if it helps to point the dashboard to mobile/android and not track mobile/xul, that will probably be easiest.

Let's stop the back and forth here, and wait for the real discussion offline. Believe and assumption has got us where we are today, so it's clearly not enough. (And, no, the XUL ui needs a complete redo now l10n wise, even for just a gecko stability update)

(In reply to Axel Hecht [:Pike] from comment #50) > Let's stop the back and forth here, and wait for the real discussion > offline. Believe and assumption has got us where we are today, so it's > clearly not enough. > > (And, no, the XUL ui needs a complete redo now l10n wise, even for just a > gecko stability update) Sure. I think the mobile team has gotten that working locally, so that's good news. I'm on PTO next week, but if you need to meet I can try to make myself available for that.