Essentially, we should be able to do something like the following (from a Gaia git clone): make profile LOCALES="en-US es pt-BR" LOCALE_BASEDIR="../gaia-l10n" in order to produce a multilocale version of Gaia.

Per e-mail with Stas, he's working on this. Moving to a more correct component, too.

I'd like to first implement this in the canonical way, i.e. just package the properties files, edit *.ini and manifest.webapp files, change languages.json and (until bug 802383 is fixed) settings/index.html. In parallel, we should continue working on the performance in bug 809600. It should not, however, stop us from getting the localized builds out to the localizers and testers.

A couple of implementation notes: RelEng has two entry points to creating Gaia profiles. For desktop nightlies we call make in gecko's b2g dir, and eventually end up here: https://mxr.mozilla.org/mozilla-central/source/b2g/gaia/Makefile.in#39 For unagi builds we call build.sh and eventually end up here (the install-gaia target depends on 'profile', so that gets called): https://github.com/mozilla-b2g/B2G/blob/master/flash.sh#L182 We need to pass along at least two pieces of information the Gaia build system (pointer to a languages.json file, base directory for l10n repos). If the Gaia build system receives these directly through env vars or some other means, we don't need to touch b2g/gaia/Makefile.in or flash.sh. If the Gaia build systems wants that information through Makefile vars, we probably need to forward that info along through those two files. Either way is fine for Releng.

Ben -- I won't have time to work on this today, sorry. I'm busy getting all of the strings ready for the final string freeze (which was supposed to be yesterday). Instead, let me try to document how this should work. 1. which locales to build ========================= Let's use shared/resources/languages.json as the list of locales to build instead of passing them to the make target. In the git repo, let's have three versions of the file: - languages.json with (en-US, ar, fr, zh-TW) - this file makes Gaia work out of the box when you just clone the repo - languages-dev.json with (en-US, ar, fr, zh-TW, es, pt-BR) - this file will be used to produce builds like the one in bug 812831, bug 812833 and bug bug 812834. - langauges-all.json with all locales - https://l10n.etherpad.mozilla.org/b2g-locales-list has the list; the file will be used for builds from bug 812836. Until bug 802383 is fixed, we'll need to also edit apps/settings/index.html and add one line per locale from langauges-*.json, like so: <li> <span data-l10n-id="language">Language <select name="language.current"> <option value="ar">العربية</option> <option value="en-US">English (US)</option> <option value="fr">Français</option> + <option value="pl">Polski</option> <option value="zh-TW">正體中文</option> </select> </span> </li> 2. shared l10n resources ======================== For each locale we're building, we'll need to copy the shared properties files from $LOCALE_BASEDIR/$LOCALE/shared/*/*.properties (where $LOCALE_BASEDIR is a directory with all the clones of h.m.o/gaia-l10n/*) to shared/locales/*/*.$LOCALE.properties in the git clone. E.g. https://hg.mozilla.org/gaia-l10n/fr/file/779974489458/shared/date/date.properties should end up overwriting https://github.com/mozilla-b2g/gaia/blob/master/shared/locales/date/date.fr.properties Notice the file name change: we're adding the locale code before the extension. For each *.ini file in shared/locales, we need to add a section with an @import rule to it. The URL of the @import rule should be the same as the one for the en-US in that file. 3. apps & properties files ========================== For each locale and for each app, we need to copy the properties files from $LOCALE_BASEDIR/$LOCALE/$APP/$APP.properties to apps/$APP/locales/$APP.$LOCALE.properties in the git clone. The communications apps make this a bit tricky. They have multiple levels of directories where you can find the properties files. It might be easier to first look for all *.ini files, parse them and this way know exactly where to copy things to. See https://github.com/mozilla-b2g/gaia/tree/master/apps/communications and dialer/locales/, ftu/locales/, contacts/locales/ & contacts/locales/fb/ For each *.ini file in apps/$APP/locales/**, we need to add a section with an @import rule to it. The URL of the @import rule should be the same as the one for the en-US in that file. 4. apps & manifest.webapp ========================= Lastly, we need to add localized names and descriptions of all apps to the manifest.webapp files. The localized contents will be available in Mercurial (the files haven't landed yet): $LOCALE_BASEDIR/$LOCALE/$APP/manifest.properties We need to put them in manifest.webapp which is a JSON file. For instance (see https://github.com/mozilla-b2g/gaia/blob/master/apps/browser/manifest.webapp#L15). { ... "locales": { ... "fr": { "name": "Navigateur", "description": "Navigateur Web Gaia" }, ... }, ... } The communications apps make this harder, again. We have a single manifest.webapp for all of them, and the 'locales' key is under each of the 'entry_points', such like 'dialer', 'contacts' etc. See https://github.com/mozilla-b2g/gaia/blob/master/apps/communications/manifest.webapp#L10 5. package it all ================= `make profile` will take all of these copied files and will output a profile with zipped apps which include the localizations. We'll probably then need to `git checkout -- .` so that we discard the local modifications to the *.ini files, manifest.webapp files and `git clean` to remove the copied *.properties.

It looks like for the 2) & 3) we could use some of jhford's code he wrote for bug 810448: https://github.com/mozilla-b2g/gaia/pull/6411/files https://github.com/jhford/gaia/blob/402e8cac99aae6de74de4660628183e758403109/build/translate-html.py I'll work on a script tomorrow.

Here's what I came up with on the plane :) A few comments: 1. The list of available locales are: languages.json, languages-dev.json and languages-all.json. I'm not sure if we should always hardcode the path to languages.json and then copy languages-*.json into it during build. Would pointing to the right file be a better solution? 2. The make target takes LOCALE_BASEDIR which should be a tree to clones of hg.mozilla.org/gaia-l10n. There's nothing in this patch that actually touches Mercurial, but I'm assuming this is a job for mozharness, not make. 3. Bug 802383 has been triaged as blocking-basecamp+ and has a patch, so I didn't put in the logic from step 1 about apps/settings/index.html (see comment 4) 4. We're leaving the working directory very dirty after running this script. Should I add another target that cleans all changes to *.properties, *.ini, and manifest.webapp files?

Comment on attachment 684641 [details] [diff] [review] WIP patch Review of attachment 684641 [details] [diff] [review]: ----------------------------------------------------------------- ::: build/multilocale.py @@ +154,5 @@ > + manifest["locales"][locale] = {} > + manifest["locales"][locale].update(strings["default"]) > + f.close() > + with open(manifest_file, 'w') as o: > + json.dump(manifest, o, encoding="utf-8", indent=2) I've had some troubles with Unicode here. The original manifest.webapp files are decoded Unicode, and I wasn't able to get json.dump to preserve this. Instead, I end up with encoded Unicode, like so: - "zh-TW": { - "name": "網路瀏覽器", - "description": "Gaia 網路瀏覽器" - } + "zh-TW": { + "name": "\u7db2\u8def\u700f\u89bd\u5668", + "description": "Gaia \u7db2\u8def\u700f\u89bd\u5668" + }, This works fine, but looks ugly in the code.

(In reply to Staś Małolepszy :stas (traveling) from comment #7) > Comment on attachment 684641 [details] [diff] [review] > WIP patch > > Review of attachment 684641 [details] [diff] [review]: > ----------------------------------------------------------------- > > ::: build/multilocale.py > @@ +154,5 @@ > > + manifest["locales"][locale] = {} > > + manifest["locales"][locale].update(strings["default"]) > > + f.close() > > + with open(manifest_file, 'w') as o: > > + json.dump(manifest, o, encoding="utf-8", indent=2) > > I've had some troubles with Unicode here. The original manifest.webapp > files are decoded Unicode, and I wasn't able to get json.dump to preserve > this. Instead, I end up with encoded Unicode, like so: > > - "zh-TW": { > - "name": "網路瀏覽器", > - "description": "Gaia 網路瀏覽器" > - } > + "zh-TW": { > + "name": "\u7db2\u8def\u700f\u89bd\u5668", > + "description": "Gaia \u7db2\u8def\u700f\u89bd\u5668" > + }, > > This works fine, but looks ugly in the code. That seems OK since these files aren't going to be checked in. Not ideal, but OK.

Comment on attachment 684641 [details] [diff] [review] WIP patch Review of attachment 684641 [details] [diff] [review]: ----------------------------------------------------------------- This looks like it's 99% of the way there, awesome! ::: Makefile @@ +173,5 @@ > + for appdir in $(GAIA_APP_SRCDIRS); do \ > + targets="$$targets --target $$appdir"; \ > + done; \ > + python $(CURDIR)/build/multilocale.py \ > + --config $(CURDIR)/shared/resources/languages.json \ I think languages.json needs to be replaced by a variable (otherwise we can't do builds with different sets of locales). That variable could default to languages.json, though. ::: build/multilocale.py @@ +1,1 @@ > +#!/usr/bin/env python I'm not going to pretend that I can review this file well. One thing I did notice while testing was that I can't build a non-multilocale profile after building a multilocale one. This may be a limitation of not using an objdir. I do see a 'purge' target, but it doesn't appear to be cleaning up files generating at build time. I did a diff of a profile created without your patch, and a profile created with your patch using languages-dev.json. It _appeared_ to work fine, but I couldn't make either profile work with the desktop b2g builds so I can't be sure. ::: shared/resources/languages-dev.json @@ +4,5 @@ > + "es" : "Español", > + "fr" : "Français", > + "pt-BR" : "Português (do Brasil)", > + "zh-TW" : "正體中文 (繁體)" > +} Any reason not to just put these in languages.json? Seems like this set is generally more useful than the current set in that file.

Hm, so I did a build with languages-all.json and ended up with a profile that appeared to have all languages in it. However, when running with a Desktop build they don't all seem to be available: https://people.mozilla.com/~bhearsum/sattap/e4d8de5e.png Dunno if this is a bug with this patch or a general b2g bug.

(In reply to Ben Hearsum [:bhearsum] from comment #8) > > This works fine, but looks ugly in the code. > > That seems OK since these files aren't going to be checked in. Not ideal, > but OK. True. I also tested just to be sure that it's displayed correctly in Gaia and it was fine. (In reply to Ben Hearsum [:bhearsum] from comment #9) > ::: Makefile > @@ +173,5 @@ > > + for appdir in $(GAIA_APP_SRCDIRS); do \ > > + targets="$$targets --target $$appdir"; \ > > + done; \ > > + python $(CURDIR)/build/multilocale.py \ > > + --config $(CURDIR)/shared/resources/languages.json \ > > I think languages.json needs to be replaced by a variable (otherwise we > can't do builds with different sets of locales). That variable could default > to languages.json, though. Right, I'll fix this in a new patch. > ::: shared/resources/languages-dev.json > @@ +4,5 @@ > > + "es" : "Español", > > + "fr" : "Français", > > + "pt-BR" : "Português (do Brasil)", > > + "zh-TW" : "正體中文 (繁體)" > > +} > > Any reason not to just put these in languages.json? Seems like this set is > generally more useful than the current set in that file. My intention here is to make it possible for people to simply clone mozilla-b2g/gaia and have it work right away. The default set of locales in git is (ar, en-US, fr, zh-TW) so that's also what the default languages.json should have. In fact, for any builds that we want to be doing, we shouldn't use the default languages.json. Just -dev or -all. (In reply to Ben Hearsum [:bhearsum] from comment #10) > Hm, so I did a build with languages-all.json and ended up with a profile > that appeared to have all languages in it. However, when running with a > Desktop build they don't all seem to be available: > https://people.mozilla.com/~bhearsum/sattap/e4d8de5e.png > > Dunno if this is a bug with this patch or a general b2g bug. You probably hit bug 802383. Until it's fixed, some changes are needed in apps/settings/index.html which my script doesn't do. What you can do right now, however, is just add one of the extra locales to apps/settings/index.html and see if it works. Something like this should be enough: <li> <span data-l10n-id="language">Language <select name="language.current"> <option value="ar">العربية</option> <option value="en-US">English (US)</option> <option value="fr">Français</option> + <option value="pl">Polski</option> <option value="zh-TW">正體中文</option> </select> </span> </li> Again, once bug 802383 is fixed, this will not be needed.

This patch addresses the feedback comments. It also adds a new target, multilocale-clean, which removes local modifications made to *.properties, *.ini and manifest.webapp files. It's not ideal as there still is a risk of destroying someone's work (as in, you might have other local edits in a manifest.webapp file besides the one made by the multilocale target). The real solution is to use an objdir, but for now, this should be good enough.

Comment on attachment 684796 [details] [diff] [review] Add LOCALES_LIST variable and multilocale-clean target Review of attachment 684796 [details] [diff] [review]: ----------------------------------------------------------------- > (In reply to Ben Hearsum [:bhearsum] from comment #10) > > Hm, so I did a build with languages-all.json and ended up with a profile > > that appeared to have all languages in it. However, when running with a > > Desktop build they don't all seem to be available: > > https://people.mozilla.com/~bhearsum/sattap/e4d8de5e.png > > > > Dunno if this is a bug with this patch or a general b2g bug. > > You probably hit bug 802383. Until it's fixed, some changes are needed in > apps/settings/index.html which my script doesn't do. > > What you can do right now, however, is just add one of the extra locales to > apps/settings/index.html and see if it works. Something like this should be > enough: > > <li> > <span data-l10n-id="language">Language > <select name="language.current"> > <option value="ar">العربية</option> > <option value="en-US">English (US)</option> > <option value="fr">Français</option> > + <option value="pl">Polski</option> > <option value="zh-TW">正體中文</option> > </select> > </span> > </li> > > Again, once bug 802383 is fixed, this will not be needed. Hmm, even if I do this before creating the profile I still can't get other languages to show up in the first-run screen or in Settings. I'm going to assume that this is PEBKAC though. I looked through some of the application.zip files and they certainly appeared to have all of the strings in them. A few comments below but this looks good to me in general, assuming the aforementioned issue is my fault and note the code's. Also, I'm not an appropriate person for an "official" review though...so I'll stick to feedback+. ::: Makefile @@ +161,4 @@ > > # Generate profile/ > > +profile: multilocale applications-data preferences app-makefiles test-agent-config offline extensions install-xulrunner-sdk profile/settings.json Should multilocale-clean be called here? @@ +168,5 @@ > > +.PHONY: multilocale > +multilocale: > +ifneq ($(LOCALE_BASEDIR),) > + @echo "Enable locales specified in shared/resources/languages.json..." Should probably use $(LOCALES_FILE) in here, otherwise the message could be a lie! @@ +181,5 @@ > + @echo "Done" > +endif > + > +.PHONY: multilocale > +multilocale-clean: Typo, should be .PHONY: multilocale-clean?

Pointer to Github pull-request

Comment on attachment 684808 [details] Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/6613 Kaze: can you help me find the right person to review this?

Comment on attachment 684808 [details] Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/6613 Staś: the Gaia part looks good, and I can review it if you squash your commits. My main objection is, could you write this part in JS instead of Python please? We’re getting rid of the Python dependency, at the moment there’s only one Python file left that we rely on (build/install-gaia.py). I won’t block on this but it would be very apprecitated.

(In reply to Fabien Cazenave [:kaze] from comment #16) > Comment on attachment 684808 [details] > Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/6613 > > Staś: the Gaia part looks good, and I can review it if you squash your > commits. Err, do you want me to squash the commits before you review? I usually rebase once I get an r+. > My main objection is, could you write this part in JS instead of Python > please? We’re getting rid of the Python dependency, at the moment there’s > only one Python file left that we rely on (build/install-gaia.py). I won’t > block on this but it would be very apprecitated. I wish I had more time on my hands to comply with this request, but given the current circumstances, I'd much rather avoid putting more work into this, given that the Python version work as expected. Our whole build system is written in Python, too, so I expected it to be a more natural choice. File handling via the os (especially os.walk), shutil and fnmatch libs is, IMHO, easier done in Python than in JS. I filed bug 815227 to keep track of this request. I suggest we move on with the Python version for now.

(In reply to Staś Małolepszy :stas (traveling) from comment #17) > Our whole build > system is written in Python, too, so I expected it to be a more natural > choice. File handling via the os (especially os.walk), shutil and fnmatch > libs is, IMHO, easier done in Python than in JS. > Sure, but we’re trying to make Firefox OS as easy to hack on as possible for everyone, which includes Windows users — hence the current Python removal effort. However, as long as Python is not required to build Gaia with the default locales, we can cope with this for the short term. Thanks for opening bug 815227.

(In reply to Fabien Cazenave [:kaze] from comment #18) > However, as long as Python is not required to build Gaia with the > default locales, we can cope with this for the short term. My thoughts exactly. Thanks.

Stas, this isn't quite working for me. I think something is overwriting languages.json after the multilocale target gets run: ➜ gaia git:(master) ✗ wc -l shared/resources/languages.json 6 shared/resources/languages.json ➜ gaia git:(master) ✗ wc -l shared/resources/languages-all.json 46 shared/resources/languages-all.json ➜ gaia git:(master) ✗ make multilocale LOCALE_BASEDIR=`pwd`/l10n LOCALES_FILE=shared/resources/languages-all.json Enable locales specified in shared/resources/languages-all.json... Done cp shared/resources/languages-all.json shared/resources/languages.json ➜ gaia git:(master) ✗ wc -l shared/resources/languages.json 46 shared/resources/languages.json ➜ gaia git:(master) ✗ make profile LOCALE_BASEDIR=`pwd`/l10n LOCALES_FILE=shared/resources/languages-all.json <snip> ➜ gaia git:(master) ✗ wc -l shared/resources/languages.json 6 shared/resources/languages.json When I run the resulting profile in a desktop nightly build I only see ar, en, fr, and zh-TW.

I think I'm hitting some other bugs here: 1) When I first launch a build with this ne wprofile I immediately get a white screen after unlocking. If I hold "home" to bring up the task manager, I see FTU running with at least 6 languages (huzzuh). I seem to be able to click through the FTU even though I can't see it. 2) After the FTU finishes I end up with a taskbar + background. Holding home down to bring up the task manager shows nothing running at all. I don't know if this a problem with multilocale or with something else. I don't know how to fully verify these multilocale builds in the face of these two bugs -- I can't interact with the FTU's list of locales nor access the Settings app.

(In reply to Ben Hearsum [:bhearsum] from comment #20) > Stas, this isn't quite working for me. I think something is overwriting > languages.json after the multilocale target gets run: > ➜ gaia git:(master) ✗ wc -l shared/resources/languages.json > 6 shared/resources/languages.json > ➜ gaia git:(master) ✗ wc -l shared/resources/languages-all.json > 46 shared/resources/languages-all.json > ➜ gaia git:(master) ✗ make multilocale LOCALE_BASEDIR=`pwd`/l10n > LOCALES_FILE=shared/resources/languages-all.json > Enable locales specified in shared/resources/languages-all.json... > Done > cp shared/resources/languages-all.json shared/resources/languages.json > ➜ gaia git:(master) ✗ wc -l shared/resources/languages.json > > 46 shared/resources/languages.json > ➜ gaia git:(master) ✗ make profile LOCALE_BASEDIR=`pwd`/l10n > LOCALES_FILE=shared/resources/languages-all.json > > <snip> > > ➜ gaia git:(master) ✗ wc -l shared/resources/languages.json > > 6 shared/resources/languages.json This is because make profile depends on the offline target which invokes multilocale-clean but only if LOCALE_BASEDIR is not empty. The

(In reply to Ben Hearsum [:bhearsum] from comment #21) > I think I'm hitting some other bugs here: > 1) When I first launch a build with this ne wprofile I immediately get a > white screen after unlocking. If I hold "home" to bring up the task manager, > I see FTU running with at least 6 languages (huzzuh). I seem to be able to > click through the FTU even though I can't see it. I noticed this too. It seems to be a Gecko bug, actually. See bug 796421. > 2) After the FTU finishes I end up with a taskbar + background. Holding home > down to bring up the task manager shows nothing running at all. I don't know > if this a problem with multilocale or with something else. Are you able to swipe to the sides? Can you try running the multilocale in a gecko 19-based desktop build from http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/latest-mozilla-central/ and seeing if the issues persist?

Comment on attachment 684808 [details] Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/6613 NOTE: If blocking-basecamp+ is set, just land it for now. [Approval Request Comment] Bug caused by (feature/regressing bug #): removal of [es] and [pt-BR] User impact if declined: no testable build for the targeted audience Testing completed: manual Risk to taking this patch (and alternatives if risky): medium, though this patch should be neutral when building a standard Gaia (= without [es] and [pt-BR] locales)

Comment on attachment 684808 [details] Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/6613 It seems clear to me that this works is needed in order to resolve bug 809253. a=me.

I did a multilocale profile build today, tested it with an OS X desktop build, and it looked great -- I had all locales available in both the FTU and Settings app, and switching between them worked fine as far as I could tell. Getting this landed gets us a lot closer to bugs 812836, 812835, 812834, and 812833 being fixed.

https://github.com/mozilla-b2g/gaia/commit/22a5d847ddad5cee714d6c537b568420dd909208 Awesome, thanks so much!

Just found a small problem with the multilocale-clean target -- it uses "git" to clean things up. Our automated builds will be using an hg mirror. I think we shouldn't rely on a specific VCS in this code. If there's no easy way to do this without "git", let's just take it out since it was a nice-to-have, not a requirement?

Pointer to Github pull-request

Comment on attachment 686148 [details] Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/6688 Thanks fr catching this, Ben. What do you think about this proposed patch? I figure that for Mercurial, we can clean the repo more aggressively, because releng will be the only consumer of it. Hence, `hg upate --clean` and `rm -rf`.

Comment on attachment 686148 [details] Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/6688 It seems like the best thing to do would to clean up without using a VCS to do it. If that's not possible, this seems ok.

(In reply to Ben Hearsum [:bhearsum] from comment #31) b > It seems like the best thing to do would to clean up without using a VCS to > do it. If that's not possible, this seems ok. How would you do it? Sorry if I'm missing something.

Comment on attachment 686148 [details] Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/6688 Kazé, can you take a look at this?

(In reply to Staś Małolepszy :stas (traveling) from comment #32) > (In reply to Ben Hearsum [:bhearsum] from comment #31) > b > > It seems like the best thing to do would to clean up without using a VCS to > > do it. If that's not possible, this seems ok. > > How would you do it? Sorry if I'm missing something. I'm not sure, I was hoping you would know :). As I said, if we can't this seems fine for now.

Comment on attachment 686148 [details] Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/6688 [Approval Request Comment] Bug caused by (feature/regressing bug #): n/a User impact if declined: we can't produce multilocale builds Testing completed: manual Risk to taking this patch (and alternatives if risky): none (Makefile only)

This blocks 5 other basecamp bugs.

Comment on attachment 686148 [details] Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/6688 Removing the approval-gaia-master? request. This landed as bb+.