When bug 1035333 gets landed, we should QA the new translation experiment on the staging server and sign it off for a push to production. We should ensure that the three locales correctly receive the experiment and that translation work on them.

I'll take this on as I did the QA work for the German translation experiment in bug # 1022411, this should be similar.

Because the minimum version is set to fx 32.0 in the manifest which is still currently on Aurora, there's no way of testing each locale experiment unitl fx beta becomes fx32. In the mean time, I went through the following test cases: Used the following staging server for the test cases below: - https://telemetry-experiment-dev.allizom.org/firefox-manifest.json Minimum BuildID Test Cases: For all of the locales, I used the following build and ensured that none of the translation experiments are being installed: - http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b5/win32/pl/ - http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b5/win32/tr/ - http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b5/win32/vi/ - 1406042627426 Browser.Experiments.Experiments DEBUG ExperimentEntry #2::isApplicable() - id=fx-translation-pl-beta32@mozilla.org - test 'minBuildID' failed - 1406043217984 Browser.Experiments.Experiments DEBUG ExperimentEntry #3::isApplicable() - id=fx-translation-tr-beta32@mozilla.org - test 'minBuildID' failed - 1406043777541 Browser.Experiments.Experiments DEBUG ExperimentEntry #4::isApplicable() - id=fx-translation-vi-beta32@mozilla.org - test 'minBuildID' failed Channel Test Cases: For all of the locales, I used the following build and ensured that none of the translation experiments are being installed: - http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora-l10n/firefox-33.0a2.pl.win32.installer.exe - http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-07-22-03-02-01-mozilla-central-l10n/firefox-34.0a1.tr.win32.installer.exe - http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-07-22-03-02-01-mozilla-central-l10n/firefox-34.0a1.vi.win32.installer.exe - 1406044827446 Browser.Experiments.Experiments TRACE Experiments #0::evaluateExperiments() - added EXPERIMENT_ACTIVATION to TelemetryLog: ["REJECTED","fx-translation-pl-beta32@mozilla.org","channel"] - 1406044827447 Browser.Experiments.Experiments TRACE Experiments #0::evaluateExperiments() - added EXPERIMENT_ACTIVATION to TelemetryLog: ["REJECTED","fx-translation-tr-beta32@mozilla.org","channel"] - 1406045284487 Browser.Experiments.Experiments TRACE Experiments #0::evaluateExperiments() - added EXPERIMENT_ACTIVATION to TelemetryLog: ["REJECTED","fx-translation-vi-beta32@mozilla.org","channel"] Locale Test Cases: For all of the locales, I used the following build and ensured that none of the translation experiments are being installed: - http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b9/win32/be/ - http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b9/win32/zh-TW/ - http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b9/win32/ro/ - http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b9/win32/kk/ - 1406047391222 Browser.Experiments.Experiments DEBUG ExperimentEntry #2::isApplicable() - id=fx-translation-pl-beta32@mozilla.org - test 'locale' failed - 1406047690068 Browser.Experiments.Experiments DEBUG ExperimentEntry #3::isApplicable() - id=fx-translation-tr-beta32@mozilla.org - test 'locale' failed - 1406047952019 Browser.Experiments.Experiments DEBUG ExperimentEntry #4::isApplicable() - id=fx-translation-vi-beta32@mozilla.org - test 'locale' failed Minimum Version Test Cases: - http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b9/win32/pl/ - http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b9/win32/tr/ - http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b9/win32/vi/ - 1406049104829 Browser.Experiments.Experiments DEBUG ExperimentEntry #2::isApplicable() - id=fx-translation-pl-beta32@mozilla.org - test 'minVersion' failed - 1406050147629 Browser.Experiments.Experiments DEBUG ExperimentEntry #3::isApplicable() - id=fx-translation-tr-beta32@mozilla.org - test 'minVersion' failed - 1406049935704 Browser.Experiments.Experiments DEBUG ExperimentEntry #4::isApplicable() - id=fx-translation-vi-beta32@mozilla.org - test 'minVersion' failed For all the build test cases above, I also went through the following: - ensured that the translation experiment wasn't appearing under about:addons - ensured that the translation experiment wasn't appearing under about:support - ensured that the translation experiment appeared as experiments.activeExperiment;false under about:config - ensured that extensions.bootstrappedAddons;{} - ensured that browser.translation.ui.show wasn't appearing or is set as browser.translation.ui.show;false - ensured that the translation experiments are still not installed after fx is restarted As mentioned above, the text portion of testing will be based on each experiment once beta merges into fx 32 (currently beta => fx 31)

Used the following as the staging server: - https://telemetry-experiment-dev.allizom.org/firefox-manifest.json **** pl locale testing: **** Went through the test cases listed below using the following build: - ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/32.0b1-candidates/build1/win32/pl/ Websites used: - http://www.thenews.pl/ - http://www.warsawvoice.pl/WVpage/pages/index.php - ensured that the experiment stays active/enabled after restarting fx several times - ensured extensions.bootstrappedAddons;{} - ensured experiments.translation-beta32-pl.isTestGroup was created and set to either true/false - ensured browser.translation.detectLanguage;true was created - ensured browser.translation.ui.show matched the value of experiments.translation-beta32-pl.isTestGroup - ensured that when experiments.translation-beta32-pl.isTestGroup;false, visiting a pl website doesn't launch the translation toolbar - ensured that when experiments.translation-beta32-pl.isTestGroup;true, visiting a pl website show's the translation toolbar - ensured that the translation toolbar worked without any issues (translated English into Polish) - ensured that the experiment appears disabled under about:addons once it's removed/expired - ensured that the experiment appears as disabled under about:support once it's removed/expired - ensured that experiments.translation-beta32-pl.isTestGroup doesn't change values once the experiment has been removed/expired - ensured browser.translation.detectLanguage;flase once the experiment has been removed/expired - ensured browser.translation.ui.show;false once the experiment has been removed/expired - ensured that the experiment can't be installed the second time around once it's been removed/expired - ensured that the experiment stays disabled when restarting fx **** tr locale testing: **** Went through the test cases listed below using the following build: - ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/32.0b1-candidates/build1/win32/tr/ Websites used: - https://www.google.ca/search?q=Microsoft+Translate+tr&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:tr:official&client=firefox-beta&channel=sb&gfe_rd=cr&ei=qEnPU6D-J4SN8Qe42IGAAg - ensured that the experiment stays active/enabled after restarting fx several times - ensured extensions.bootstrappedAddons;{} - ensured experiments.translation-beta32-tr.isTestGroup was created and set to either true/false - ensured browser.translation.detectLanguage;true was created - ensured browser.translation.ui.show matched the value of experiments.translation-beta32-tr.isTestGroup - ensured that when experiments.translation-beta32-tr.isTestGroup;false, visiting a tr website doesn't launch the translation toolbar - ensured that when experiments.translation-beta32-tr.isTestGroup;true, visiting a tr website show's the translation toolbar - ensured that the translation toolbar worked without any issues (translated English into Turkish) - ensured that the experiment appears disabled under about:addons once it's removed/expired - ensured that the experiment appears as disabled under about:support once it's removed/expired - ensured that experiments.translation-beta32-tr.isTestGroup doesn't change values once the experiment has been removed/expired - ensured browser.translation.detectLanguage;flase once the experiment has been removed/expired - ensured browser.translation.ui.show;false once the experiment has been removed/expired - ensured that the experiment can't be installed the second time around once it's been removed/expired - ensured that the experiment stays disabled when restarting fx **** vi locale testing: **** Went through the test cases listed below using the following build: - ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/32.0b1-candidates/build1/win32/vi/ Websites used: - http://www.vide.vi - ensured that the experiment stays active/enabled after restarting fx several times - ensured extensions.bootstrappedAddons;{} - ensured experiments.translation-beta32-vi.isTestGroup was created and set to either true/false - ensured browser.translation.detectLanguage;true was created - ensured browser.translation.ui.show matched the value of experiments.translation-beta32-vi.isTestGroup - ensured that when experiments.translation-beta32-vi.isTestGroup;false, visiting a vi website doesn't launch the translation toolbar - ensured that when experiments.translation-beta32-vi.isTestGroup;true, visiting a vi website show's the translation toolbar - ensured that the translation toolbar worked without any issues (translated English to Vietnamese) - ensured that the experiment appears disabled under about:addons once it's removed/expired - ensured that the experiment appears as disabled under about:support once it's removed/expired - ensured that experiments.translation-beta32-vi.isTestGroup doesn't change values once the experiment has been removed/expired - ensured browser.translation.detectLanguage;flase once the experiment has been removed/expired - ensured browser.translation.ui.show;false once the experiment has been removed/expired - ensured that the experiment can't be installed the second time around once it's been removed/expired - ensured that the experiment stays disabled when restarting fx

Potential Issues: Issue #1: The translation toolbar won't appear when the experiment is initially installed, you need to completely restart fx, example (this affects pl, tr, vi): - translation experiment has been installed and experiments.translation-beta32-pl.isTestGroup;true - visit http://www.thenews.pl (translation toolbar will not appear no matter how many times you visit the website) - restart fx - visit the same website again and the translation toolbar will start appearing without any issues I've reproduced this on all three locales using different websites. I'm not sure if this is expected behavior but I would personally expect the translation toolbar to work as soon as the experiment was installed without having to restart fx. Issue #2: (attached image) The "Experiment" title for the container under about:addons is not being translated under the vi locale. It's being translated in the other two locales (pl, tr). I'm not sure if this is because "Experiment" is difficult to translate in Vietnamese or perhaps it was just missed?

Issue #1: upon installation, the translation toolbar should start appearing on every new tab created after installation. It's expected that it won't appear for existing tabs, even if you navigate to a different website on that tab. Can you verify that that's what you were seeing? Issue #2: Yeah it's just missing strings, shouldn't be a problem for us since it's secondary UI. They did a sprint to translate all translation-related string but those ones were probably missed as they are not specific to translation and are part from the broader experiments system.

(In reply to :Felipe Gomes (away Jul 23 - Aug 13) from comment #5) > Issue #1: upon installation, the translation toolbar should start appearing > on every new tab created after installation. It's expected that it won't > appear for existing tabs, even if you navigate to a different website on > that tab. Can you verify that that's what you were seeing? Indeed that's what was happening. I went through the three locales once again and can confirm that the translation toolbar correctly slides into view when opening the target websites on new tabs/windows. > Issue #2: Yeah it's just missing strings, shouldn't be a problem for us > since it's secondary UI. They did a sprint to translate all > translation-related string but those ones were probably missed as they are > not specific to translation and are part from the broader experiments system. Created bug #1042728 to address the missing translations in vi. Felipe, I guess that's pretty much it! I'll start working on bug #1018200. Can you mark this as Resolved so I can close this as verified?

Sure thing