Closed Bug 1353337 Opened 7 years ago Closed 4 years ago

Add not yet locally existing languages to locale picker

Categories

(Firefox for Android Graveyard :: General, enhancement, P5)

Product:
Firefox for Android Graveyard
Component:
General
Platform:
All
Android
Type:
enhancement

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: sebastian, Unassigned)

References

Details

For downloadable language resources we need a way to let a user pick a language for which we do not have any resources yet. Ideally we mix in languages that are in the catalog and which match the local configuration (e.g. can be downloaded).
My preferred interaction for this is if DLC exposed a function that L10nRegistry can call to get the locales available via DLC. Then the L10nRegistry will compare it with the locales it has locally and produce the "getAvailableLocales".

It would be nice if DLC was able to also "update" the list of available locales for L10nRegistry (so, download new list, compare to the old one, if differs, fire "L10nRegistry.updateAvailableSources").
(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #1)
> My preferred interaction for this is if DLC exposed a function that
> L10nRegistry can call to get the locales available via DLC.

Okay, this shouldn't be a problem.

> Then the L10nRegistry will compare it with the locales it has locally
> and produce the "getAvailableLocales".

Okay. Somehow we need to know that the selected language requires a download and kick off DLC. Is this something you think L10nRegistry would do too?
(In reply to Sebastian Kaspari (:sebastian) from comment #2)
> Okay. Somehow we need to know that the selected language requires a download
> and kick off DLC. Is this something you think L10nRegistry would do too?

Actually I could pre-emptively start DLC whenever the language changes: This way DLC will have the opportunity to see if there's any content for the current locale in the catalog.
> Okay. Somehow we need to know that the selected language requires a download and kick off DLC. Is this something you think L10nRegistry would do too?

My understanding is - DLC operates on "requested locales" - if requested locales changes, we fire "intl:requested-locales-changed" event and then DLC can read LocaleService.getRequestedLocales and download the ones it has.

Once the download is completed it registers locales in L10nRegistry and that triggers language negotiation which may end up with a new locale.

Does it make sense?
I put some more thinking into this, my current proposal is:

 - DLC provides 'GetLocallyAvailableLocales'
 - DLC provides 'GetAllAvailableLocales'
 - Settings use the latter to show all languages that can be selected
 - LocaleService uses the former to negotiate locales between available and requested
 - When user selects a locale that is not yet locally available, DLC triggers a download
 - When a new/updated locale is downloaded, DLC triggers "dlc:locally-available-locales-changed'
 - LocaleService reacts to that event and retrigers language negotiation
 - If the result language change is different from the previous one it triggers 'intl:app-locales-changed'
 - Localization API then invalidates cache to use new languages from L10nRegistry which loads them from DLC

:sebastian, let me know if it sounds sane to you
Flags: needinfo?(s.kaspari)
(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #5)
>  - DLC provides 'GetLocallyAvailableLocales'
>  - DLC provides 'GetAllAvailableLocales'

If this translates to "get list of locales installed from catalog" and "get list of all locales in catalog" then this sounds good. I want to keep DLC itself independent from the things the content is about - it only knows where to get content, where to download it to and what other components to notify.

>  - Settings use the latter to show all languages that can be selected

I haven't looked into where Settings gets the list from currently - but this would be current list + languages in catalog, right? This should be straight-forward.

>  - LocaleService uses the former to negotiate locales between available and
> requested
>  - When user selects a locale that is not yet locally available, DLC
> triggers a download

Yeah, I'm probably going to always trigger a DLC task ("study") to see whether any content should be downloaded now (a locale switch might make us want to download content that we weren't interested in before - for other use cases too.)

>  - When a new/updated locale is downloaded, DLC triggers
> "dlc:locally-available-locales-changed'

Sounds good. We already send an event after downloading fonts - so this should be easy. Note that DLC is completely independent from Gecko and can run in the background without the app being open. So we might download a locale and can't LocaleService at this time because Gecko is not running. So the next time Gecko starts LocaleService should be able to pick up the new language on its own. Is this a problem?


Two questions from my side:
* Do we already know how to package those "locales" we are talking about? Would it be possible to get one of those packages for testing. I'd like to upload it to our staging server.
* Do you have any preferences where DLC should put those files? Content currently in the catalog is a gz compressed and will be extracted by DLC to a location specific for the type of content.
Flags: needinfo?(s.kaspari)
> I haven't looked into where Settings gets the list from currently - but this would 
> be current list + languages in catalog, right? This should be straight-forward.

Are we talking about picking in Settings > Language?

If so, the list there is the locales with which the build was compiled: BrowserLocaleManager.getPackagedLocaleTags(Context).

If we're altering the definition of which locales are included in our APK to include different sets of Gecko locales vs Android locales, then if the Gecko locales are a subset of the Android locales -- which they must be, because the Android locales aren't swappable, and they define the primary UI -- then that makes life a little easier: an entry in the picker is either completely installed or only the Android side is installed.

I think it would be confusing to offer locales for which no Android-side equivalent exists.
> Are we talking about picking in Settings > Language?
> If so, the list there is the locales with which the build was compiled: BrowserLocaleManager.getPackagedLocaleTags(Context).

Yes, of course in case of two sources of locales (Java locales and Gecko locales) we'll want the list of available locales to be a subset of java locales for which we have gecko locales available.

> So the next time Gecko starts LocaleService should be able to pick up the new language on its own. Is this a problem?

no, that's easy. I was thinking about ability to renegotiate languages when the app is open, without the need to restart.

> * Do we already know how to package those "locales" we are talking about? Would it be possible to get one of those packages for testing. I'd like to upload it to our staging server.

No, we don't. In the roadmap document we talked about the need to change things in the build system to not package those files in the .apk, but instead add ability to generate a DLC package with them.
If I remember correctly you suggested that you'll take it and try to assign it to someone on your team or yourself.
If that changed, please, let me know.

> * Do you have any preferences where DLC should put those files? Content currently in the catalog is a gz compressed and will be extracted by DLC to a location specific for the type of content.

I need to be able to access those files either via file:// protocol or resource:// protocol from within Gecko. I know that in Gecko we usually keep the resources inside .jar package (zip) and resource:// protocol transparently accesses them.
Not sure if that would work here. If you unpack them and add a Source (AddonSource) to L10nRegistry with the basePath to some file:// directory, that will work for me as well.
After an IRC conversation with :rnewman, my updated proposal strategy is:

Scenario for manual change of locale:
==================

DLC <---> Java

 - Fennec UI shows available locales (intersection of locales gecko and java have) and downloadable locales (intersection of locales java has and DLC can download for Gecko)
 - If user clicks on download button, DLC downloads a new locale
 - Once the locale is downloaded, DLC triggers language renegotiation in Java which end up with new locale on top (because it's now available in gecko and in java and requested by user)

Java <--> Gecko

 - When starting a process, Java communicates the list of negotiated locales to Gecko
 - When language negotiation in Java returns an updated list of locales, Java sends an updated list of locales to Gecko

DLC <---> Gecko

 - When starting Gecko process, DLC registers its langpack sources in L10nRegistry
 - When DLC downloads a new locale it registers it via L10nRegistry:addSource
 - When DLC updates a locale it triggers L10nRegistry::updateSource

==================

In the scenario where user just installed Fennec, the post-installation step looks into what locales OS use and triggers download of corresponding resources from DLC. The rest is the same.

The major change here is that we hand over language negotiation to where it belongs - Java code in Fennec, and we use Gecko's LocaleService as a client that just follows what parent process negotiated.

That means that there's less work on the DLC <--> Gecko side (for Fennec, for Firefox Desktop we'll have to do the "DLC <--> Java" part in "DLC <--> Gecko" as well), but more in "DLC <--> Java".

:sebastian, does it sound good?
Flags: needinfo?(s.kaspari)
I'll add the client-server dual nature of LocaleService in bug 1348042.
(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #8)
> No, we don't. In the roadmap document we talked about the need to change
> things in the build system to not package those files in the .apk, but
> instead add ability to generate a DLC package with them.
> If I remember correctly you suggested that you'll take it and try to assign
> it to someone on your team or yourself.
> If that changed, please, let me know.

Yes, we should have someone else working on that. With me working less on Fennec going forward it makes a lot of sense to have this critical part owned by someone else - I do not know much about this part so far anyways. I thought the L10N team might have opinions on that - afaik those files are not even part of mozilla-central, so packaging and uploading them might be possible without interacting with the build system?


(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #9)
> DLC <---> Java
> 
>  - Fennec UI shows available locales (intersection of locales gecko and java
> have) and downloadable locales (intersection of locales java has and DLC can
> download for Gecko)
>  - If user clicks on download button, DLC downloads a new locale
>  - Once the locale is downloaded, DLC triggers language renegotiation in
> Java which end up with new locale on top (because it's now available in
> gecko and in java and requested by user)

Sounds good to me! Just to clarify: We switch the language immediately after selecting, right? So the Android UI uses the new language instantly? Or do we defer switching until everything is available?


> Java <--> Gecko
> 
>  - When starting a process, Java communicates the list of negotiated locales
> to Gecko
>  - When language negotiation in Java returns an updated list of locales,
> Java sends an updated list of locales to Gecko

Can you help me understand what "negotiated locales" / "language negotiation" means here?


> DLC <---> Gecko
> 
>  - When starting Gecko process, DLC registers its langpack sources in
> L10nRegistry
>  - When DLC downloads a new locale it registers it via L10nRegistry:addSource
>  - When DLC updates a locale it triggers L10nRegistry::updateSource

Sounds good.
Flags: needinfo?(s.kaspari)
@joe, wesley: For this "downloadable translations" story (bug 1347802) help from the Fennec / core team is needed. With "Focus" going into another iteration I do not have much time to take this over. Does the core team have the resources to support this? Of course I'd help mentoring this to transfer knowledge/ownership to the core team.
Flags: needinfo?(whuang)
Flags: needinfo?(jcheng)
> Just to clarify: We switch the language immediately after selecting, right? So the Android UI uses the new language instantly? Or do we defer switching until everything is available?

I believe we came to conclusion with :rnewman that we should switch only when the new DLC locale is available.

So:

1)
  requstedLocales: ['en-US'],
  availableLocalesInGecko: ['en-US'],
  availableLocalesInJava: ['en-US', 'es'],
  availableLocalesInDLC: ['es']
  appLocales: ['en-US'] // <-- negotiated locale list

2) User opens preferences and sees "en-US", and "es" with some mark indicating that "es" would have to be downloaded
3) User drags "es" to the "selected locales" area to the top
4) DLC triggers download, and requestedLocales gets updated:

  requstedLocales: ['es', 'en-US'],
  availableLocalesInGecko: ['en-US'],
  availableLocalesInJava: ['en-US', 'es'],
  availableLocalesInDLC: ['es']
  appLocales: ['en-US'] // <-- negotiated locale list

5) DLC finishes download, informs LocaleService in Java, the new  list:

  requstedLocales: ['es', 'en-US'],
  availableLocalesInGecko: ['es', 'en-US'],
  availableLocalesInJava: ['en-US', 'es'],
  availableLocalesInDLC: ['es']
  appLocales: ['es', 'en-US'] // <-- negotiated locale list

6) Java updates its locale UI
7) Java LocaleService updates Gecko LocaleService with the new list of app locales
8) Gecko updates its locale UI

> Can you help me understand what "negotiated locales" / "language negotiation" means here?

Sure. Language negotiation is the logic that takes two lists:
 - sorted locales requested by the user (requested-locales)
 - unsorted available locales (available-locales)

and creates a new result list of locales that are available according to the request from the user.

Here is the C++ implementation of the algorithm we use: http://searchfox.org/mozilla-central/source/intl/locale/LocaleService.cpp#256
And here's JS: https://github.com/projectfluent/fluent.js/blob/master/fluent-langneg/src/matches.js#L6
Thanks! This was super helpful.

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #13)
> > Just to clarify: We switch the language immediately after selecting, right? So the Android UI uses the new language instantly? Or do we defer switching until everything is available?
> 
> I believe we came to conclusion with :rnewman that we should switch only
> when the new DLC locale is available.

Oh, okay. This sounds like it requires some kind of UX so that users understand why "nothing happens" immediately (not sure if some indicator is enough)?
(In reply to Sebastian Kaspari (:sebastian) from comment #12)
> @joe, wesley: For this "downloadable translations" story (bug 1347802) help
> from the Fennec / core team is needed. With "Focus" going into another
> iteration I do not have much time to take this over. Does the core team have
> the resources to support this? Of course I'd help mentoring this to transfer
> knowledge/ownership to the core team.
Core team's backlog is growing bigger and bigger now.
Which is good but we'll need some iterations (from sprint planning) to make all these good stuff happen.
I'm putting this one blocking bug 1341545 and we'll see if we have available bandwidth.
From what I can see now, there are several topline features occupying our April~June,
Blocks: 1341545
Flags: needinfo?(whuang)
(In reply to Sebastian Kaspari (:sebastian) from comment #14)

> Oh, okay. This sounds like it requires some kind of UX so that users
> understand why "nothing happens" immediately (not sure if some indicator is
> enough)?

I think the idea was to behave a little like an app store/music player: rather than having a list of locales, have a list of locales with "(Download)" buttons on the right. Tap, download happens (with progress spinner, even); tap on a downloaded locale to switch to it.

Yes, UX is required here, though it should be relatively straightforward.
Flags: needinfo?(jcheng)
Re-triaging per https://bugzilla.mozilla.org/show_bug.cgi?id=1473195

Needinfo :susheel if you think this bug should be re-triaged.
Priority: -- → P5
We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → INCOMPLETE
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.