Zibi Braniecki [:zbraniecki][:gandalf] Reporter
	Description • 6 years ago

Currently, we build Firefox with a single locale (en-US) and then we do "repacks" which take the en-US Firefox and repackage it with another single locale.

On top of that we have a concept of repackaging Firefox with many locales which is used in production by Firefox for Android and is mostly done but not used for Desktop Firefox.

We also have ability to build language packs which bundle the same resources for desktop or android Firefox that a repack puts into the package and allow users to add them at runtime.

We want to start using those language packs more but the challenge is that they can't cover non-gecko parts, in particular, an installer.

For that reason, in bug 1444016, we'd like to enable ability to have an installer packaged with an arbitrary list of locale resources, that is separate from the list of resources we package Firefox with, and then use langpacks at runtime.

In this bug I'd like to focus on separating the list of locales we use to decide on what locales to package into the installer from the list of locales we package into the product.

Currently, if I'm not mistaken, we use two build time variables:

AB_CD - which is the "default locale" - it's the locale we build and package in, we use in `update.locale` to request updates for etc.

MOZ_CHROME_MULTILOCALE is a list of BCP47 (+ja-JP-mac) language tags we can use to perform a multi-locale repack which will package those locales into the product.

I'd like to, in the future, use AB_CD less and less, eventually keeping it only as "this is the default locale of the build", and use MOZ_CHROME_MULTILOCALE and more and more with "and those are locales packaged into it".

It's totally fine if the list of those locales is 1.

Now, for this bug, we'd need a separate list for the installer, sth like INSTALLER_MULTILOCALE?

I wouldn't mind renaming them to:

 - DEFAULT_LOCALE // "en-US"
 - CHROME_LOCALES // "en-US fr de pl"
 - INSTALLER_LOCALES "en-US fr de pl it es zh-CN"

Another thought is that maybe we should consider storing them in some config file?

p.s. that may actually be expanded in the future - we may want to package different list of locales for devtools, for example.

	(not currently active) Ted Mielczarek
	Comment 1 • 6 years ago

I can't really speak to the multilocale setup, I have only tenuous knowledge of it. For the installer, I would think we'd need to hash out what the implementation of multiple locales in the installer looks like before being able to usefully decide anything here, right?

	Zibi Braniecki [:zbraniecki][:gandalf] Reporter
	Comment 2 • 6 years ago

Matt - who's the right person to advice on how the installer expects the locales to be packaged? I don't have any understanding of how the installer handles locale selection and where it expects the resources to be stored.

We'd need that to design the model that we can implement on the build side

	Molly Howell (she/her) [:mhowell]
	Comment 3 • 6 years ago

I'm... not very familiar with any of this either. The only recent change that I've seen in this procedure is bug 1385227/bug 1436662, so I'll redirect to :Pike. Unfortunately he's on PTO for a few more days, but I've just been looking into it myself to try and have something useful to tell you, and every time I see something new it disproves something that I thought I knew. So maybe you don't want my advice. :)

	Axel Hecht [:Pike]
	Comment 4 • 6 years ago

TBH, this is almost full circle, all that's missing is me asking zibi what he really wants ;-)

To drill this down just a notch: In NSIS, what options do we have for multi-lingual UI? Could we create a mockup or dummy installer with such a UI, to see if it's a useful experience?

Related question, do we have metrics in the current installer binaries that tell us how good their conversion rates are?

The build and automation questions arising from that would be a secondary step, I guess.

	Molly Howell (she/her) [:mhowell]
	Comment 5 • 6 years ago

(In reply to Axel Hecht [:Pike] from comment #4)
> TBH, this is almost full circle, all that's missing is me asking zibi what
> he really wants ;-)

:D

> To drill this down just a notch: In NSIS, what options do we have for
> multi-lingual UI? Could we create a mockup or dummy installer with such a
> UI, to see if it's a useful experience?

NSIS provides a localization system with multi-language support built-in. As long as we can make string table files for each language in the format it expects (a job currently done for a single language by preprocess-locale.py [0]), then we can just tell that system at runtime what language we want to use. We can select a language automatically based on the OS settings or show a dialog box to ask the user for their preference (which I would recommend against for the stub installer, but it could be appropriate for the full installer).

> Related question, do we have metrics in the current installer binaries that
> tell us how good their conversion rates are?

It depends on what you mean by conversion rate; we know how many times the stub installer is run and how many of those attempts result in a successful installation and first run [1], and we can break that down by locale (I don't think we're running a query right now for that, but we have the data). Tracking conversations all the way back through the web site is trickier and would involve correlating those stub installer pings with attribution telemetry, which to my knowledge nobody has bothered to do.

[0] https://searchfox.org/mozilla-central/rev/3abf6fa7e2a6d9a7bfb88796141b0f012e68c2db/toolkit/mozapps/installer/windows/nsis/preprocess-locale.py
[1] https://sql.telemetry.mozilla.org/queries/3648#7201

	Nick Alexander :nalexander [he/him] (PTO until Montreallhands August 21, 2023)
	Comment 6 • 6 years ago

Amendments first, thoughts second.

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #0)
> Currently, we build Firefox with a single locale (en-US) and then we do
> "repacks" which take the en-US Firefox and repackage it with another single
> locale.

For Android, we should stop doing this, since we have no reason to believe it's valuable.

> On top of that we have a concept of repackaging Firefox with many locales
> which is used in production by Firefox for Android and is mostly done but
> not used for Desktop Firefox.

For Android, this is quite different from repackaging.  I consistently use the terminology "single-locale repacks" and "multi-locale builds" for Android to emphasize that these multi-locale artifacts are the result of a single internally consistent build process, and the single-locale artifacts are not internally consistent.  (We made them as consistent as possible on Android by turning them into stripped down builds.  Repacking is nonsense on Android.)

> We also have ability to build language packs which bundle the same resources
> for desktop or android Firefox that a repack puts into the package and allow
> users to add them at runtime.
> 
> We want to start using those language packs more but the challenge is that
> they can't cover non-gecko parts, in particular, an installer.
> 
> For that reason, in bug 1444016, we'd like to enable ability to have an
> installer packaged with an arbitrary list of locale resources, that is
> separate from the list of resources we package Firefox with, and then use
> langpacks at runtime.
> 
> In this bug I'd like to focus on separating the list of locales we use to
> decide on what locales to package into the installer from the list of
> locales we package into the product.
> 
> Currently, if I'm not mistaken, we use two build time variables:
> 
> AB_CD - which is the "default locale" - it's the locale we build and package
> in, we use in `update.locale` to request updates for etc.

I would say that AB_CD is many things.  Externally, it maps to something like what you say, for single-locale repacks.  Internally, it's the build system's "here's what I'm thinking about right now" scratchpad.

> MOZ_CHROME_MULTILOCALE is a list of BCP47 (+ja-JP-mac) language tags we can
> use to perform a multi-locale repack which will package those locales into
> the product.

This is mostly correct.  Note that Android muddies the waters by _also_ using AB_CD=multi to signal that we're in a multi-locale build.  I have no idea why these two parallel tracks evolved, but we should stop using AB_CD=multi.

> I'd like to, in the future, use AB_CD less and less, eventually keeping it
> only as "this is the default locale of the build", and use
> MOZ_CHROME_MULTILOCALE and more and more with "and those are locales
> packaged into it".

I support this.  Note that AB_CD generally _doesn't_ mean the "default locale of the build" at any place where you see it in the source tree.

> It's totally fine if the list of those locales is 1.
> 
> Now, for this bug, we'd need a separate list for the installer, sth like
> INSTALLER_MULTILOCALE?
> 
> I wouldn't mind renaming them to:
> 
>  - DEFAULT_LOCALE // "en-US"
>  - CHROME_LOCALES // "en-US fr de pl"
>  - INSTALLER_LOCALES "en-US fr de pl it es zh-CN"

I support this, and think we should "just do it": let's turn MOZ_CHROME_MULTILOCALE -> MOZ_UI_LOCALES or MOZ_PRODUCT_LOCALES now, before its usage grows.  I don't want to use "chrome", since that's both another product and means a thousand things at Mozilla.

> Another thought is that maybe we should consider storing them in some config
> file?

The root of all configuration files in the current system is mozconfig.  The best way to make something available everywhere -- build time, package time, tools time -- is mozconfig.  We shouldn't be passing this list around in secret automation scripts (https://searchfox.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/l10n/multi_locale_build.py#188), we should be setting it up front at multi-locale build configure time.

> p.s. that may actually be expanded in the future - we may want to package
> different list of locales for devtools, for example.

Really, lots of features ship with different sets of locales.  On day one, for example, Pocket shipped with a very restricted set of locales.  (Maybe it wasn't part of the product directly at that point.)

Conclusion: I support renaming all of these variables.  AB_CD and all of the details around need to be simplified.  I doubt it will be hard to produce a multi-locale NSIS using MOZ_INSTALLER_LOCALES _if we have them at build time_, although that Makefile.in is crufty (as all l10n-aware makefiles are).  But Ted and I have been making progress on these crufty Makefile.in files...

	Axel Hecht [:Pike]
	Comment 7 • 6 years ago

(In reply to Nick Alexander :nalexander from comment #6)
> The root of all configuration files in the current system is mozconfig.  The
> best way to make something available everywhere -- build time, package time,
> tools time -- is mozconfig.  We shouldn't be passing this list around in
> secret automation scripts
> (https://searchfox.org/mozilla-central/source/testing/mozharness/mozharness/
> mozilla/l10n/multi_locale_build.py#188), we should be setting it up front at
> multi-locale build configure time.

I quite strongly disagree on this one, given that changing mozconfig basically requires you to recompile the world. And that's a horrible developer experience.

We should make adding localizations to your local testing easier instead of harder.

I'm not sure that "my local builds have 10 languages" is a good development default either, so maybe even doing defaults on mozconfig might be a non-helpful choice.

	Justin Wood (:Callek)
	Comment 8 • 6 years ago

(In reply to Axel Hecht [:Pike] from comment #7)
> (In reply to Nick Alexander :nalexander from comment #6)
> > The root of all configuration files in the current system is mozconfig.  The
> > best way to make something available everywhere -- build time, package time,
> > tools time -- is mozconfig.  We shouldn't be passing this list around in
> > secret automation scripts
> > (https://searchfox.org/mozilla-central/source/testing/mozharness/mozharness/
> > mozilla/l10n/multi_locale_build.py#188), we should be setting it up front at
> > multi-locale build configure time.
> 
> I quite strongly disagree on this one, given that changing mozconfig
> basically requires you to recompile the world. And that's a horrible
> developer experience.
> 
> We should make adding localizations to your local testing easier instead of
> harder.
> 
> I'm not sure that "my local builds have 10 languages" is a good development
> default either, so maybe even doing defaults on mozconfig might be a
> non-helpful choice.

I should note, I have initially held this view, but I've come around to feel, summarized:

* L10n should be a first class citizen in the build system. And as such the build system should retain knowledge of locales going into the build itself, upfront where possible.

The reasons or this change in heart are multiple, let me enumerate some of the big ones:

* Configure rerun times have drastically reduced over time, in the old days it was a huge part of local development time, today its much smaller. Additionally the more cruft we can cull out of the old-style configure (including some of these l10n supporting bits) the faster it can become
* Unpacking, and repacking can be fragile, even more so when we're talking doing so to replace localized content, across platform and application types.
* Artifact builds are a thing and have a much more modern mindspace around 'how do I unpack this thing, so our build system knows about it'
* I don't like unique codepaths, so if we can reuse artifact logic to expand our archives that is better!

This can enable us to do (in the future) '--with-locale="de"' in a normal mozconfig and any build (artifact or not) could build with de, we could also in theory do --with-locale="en-US" --with-langpack="de,es,en-CA" and the main build will be english but we'd produce langpacks for de, es and en-CA. The related logic around artifact builds or not can still apply, we might even make it so that you can use an already expanded build as if it was an artifact build (e.g. you build yourself and THEN you want to do an artifact build)

Bottom line, I think this approach helps open us up to many future possibilities in areas we'd like to see this logic go, where the current approach merely keeps us stagnated and "off to the side" of the main build system improvements that greatly assist developer ergonomics.

	(not currently active) Ted Mielczarek
	Comment 9 • 6 years ago

We're planning to continue iterating on the work we started in bug 1447932. When Nick and I were working on that patch series we clobbered and rebuilt quite a bit and it turns out that with an artifact build it almost doesn't matter because the whole process takes about 30 seconds. These kinds of things are only possible when we free ourselves from the constraints of the existing recursive make build, and things like "we must be able to override values on the make commandline such that they can only be handled in Makefile rules" are precisely such constraints.