We need to produce a telemetry experiment which: - targets all users of the `de` locale on Aurora 32 - upon activation, the experiment should: - decide from a 50%/50% draw if the user will be part of the test or control group - for test group: set browser.translation.detectLanguage to true set browser.translation.ui.show to true - for control group: only set browser.translation.detectLanguage to true - upon deactivation, the experiment should: - clear both prefs to false

More details on the description in bug 1021921

Hi Felipe, are you taking Bug 1022411 during the current iteration?

Marco: yeah I'm taking it. I'll provide a point estimate soon as I get to know a bit more of the work involved here. But it's probably gonna be a 3 or a 5.

Added to Iteration 33.1

There's probably not much more to it than this. Not requesting feedback yet as I need to ask around how to test this locally and how to test deployment.

This works on a basic level, however i'm getting into some strange problems. After forcing a manifest update, I get the experiment correctly loaded and installed. Going to the Experiments tab of AOM correctly shows the calculated endTime ("21 days remaining"), and the install() code from my bootstrap.js ran fine. However, the extension also ends up marked as "for removal", and if you go into the detailed view, it says "... will be removed when Aurora restarts". After a restart, though, the extension was disabled (and not removed), and it says "Complete (NaN days ago)". A second restart removes the extension. See these screenshots for details: http://imgur.com/a/rHPA8 This was testing using a local Aurora build. I also tried using Nightly, and the results are similar. However, on Nightly, during shutdown, an exception is thrown [see it in the next comment], and the extension never gets disabled/removed. I don't know if the differences between Aurora/Nightly are due to the state of the profile being used to test. (I've started both tests with clean profiles) The extension code is trivial (just setting some prefs), so I don't think it could be causing these problems. Any help debugging it is appreciated. The attached patch includes the .xpi and the manifest needed for the server.

Exception thrown on Nightly during shutdown: 1402957344714 addons.manager ERROR Exception calling provider shutdown: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIObserverService.removeObserver]" nsresult: "0x80004005 (NS_ERROR_FAILURE)" location: "JS frame :: resource:///modules/experiments/Experiments.jsm :: this.Experiments.PreviousExperimentProvider.prototype<.shutdown :: line 2087" data: no] Stack trace: this.Experiments.PreviousExperimentProvider.prototype<.shutdown()@resource:///modules/experiments/Experiments.jsm:2087 < callProvider()@resource://gre/modules/AddonManager.jsm:194 < AMI_unregisterProvider()@resource://gre/modules/AddonManager.jsm:849 < AMP_unregisterProvider()@resource://gre/modules/AddonManager.jsm:2321 < Experiments.Experiments.prototype._unregisterWithAddonManager()@resource:///modules/experiments/Experiments.jsm:500 < Experiments.Experiments.prototype.uninit<()@resource:///modules/experiments/Experiments.jsm:446 < TaskImpl_run()@resource://gre/modules/Task.jsm:282 < TaskImpl_handleResultValue()@resource://gre/modules/Task.jsm:338 < TaskImpl_run()@resource://gre/modules/Task.jsm:290 < TaskImpl()@resource://gre/modules/Task.jsm:247 < createAsyncFunction/asyncFunction()@resource://gre/modules/Task.jsm:224 < Barrier.prototype<._wait()@resource://gre/modules/AsyncShutdown.jsm:565 < Barrier.prototype<.wait()@resource://gre/modules/AsyncShutdown.jsm:531 < Spinner.prototype.observe()@resource://gre/modules/AsyncShutdown.jsm:337 < <file:unknown>

(In reply to :Felipe Gomes from comment #6) > However, the extension also ends up marked as "for removal" Sounds vaguely related to bug 1010397 / bug 612168? (In reply to :Felipe Gomes from comment #7) > Exception thrown on Nightly during shutdown: That's bug 1012466.

(In reply to :Felipe Gomes from comment #6) > However, the extension also ends up marked as "for removal", and if you go > into the detailed view, it says "... will be removed when Aurora restarts". That's fine and due to how we sync with the addon manager (experiment addons get "pending disable", after startup we recheck and re-activate them if applicable). I thought i hid all the "pending" parts in the AM though - can you file on how to get those? > After a restart, though, the extension was disabled (and not removed), and > it says "Complete (NaN days ago)". A second restart removes the extension. This sounds weird. What does the manifest look like and does the log have any interesting info? Relevant prefs: * experiments.logging.dump;true * experiments.logging.level;0

(In reply to :Gavin Sharp (email gavin@gavinsharp.com) from comment #8) > (In reply to :Felipe Gomes from comment #6) > > However, the extension also ends up marked as "for removal" > > Sounds vaguely related to bug 1010397 / bug 612168? Those are only about manually clicking "remove" in the addon manager (and the related "undo" behavior).

(In reply to :Gavin Sharp (email gavin@gavinsharp.com) from comment #8) > (In reply to :Felipe Gomes from comment #7) > > Exception thrown on Nightly during shutdown: > > That's bug 1012466. That bug shouldn't cause the behavior mentioned here. We re-check the experiments on/after startup and should re-enable it if applicable.

Comment on attachment 8440471 [details] [diff] [review] First ver >+function install(data, reason) { >+ let testGroup = (Math.random() > 0.5); >+ In case of any future upgrade, this will run again and may cause the user to switch groups. I think you should do all this in startup() and store this value in an experiment-specific pref, e.g. let inTestGroup; try { inTestGroup = Services.prefs.getBoolPref(TRANSLATION_EXPERIMENT_GROUP); } catch (e) { inTestGroup = (Math.random() > 0.5); Services.prefs.setBoolPref(TRANSLATION_EXPERIMENT_GROUP, inTestGroup); } >+ Services.prefs.setBoolPref(PREF_SHOWUI, testGroup); >+ Services.prefs.setBoolPref(PREF_DETECTLANGUAGE, true); >+ >+ Services.prefs.setCharPref(PREF_CLIENTID, "AAA"); >+ Services.prefs.setCharPref(PREF_APIKEY, "BBB"); These should not be set as user prefs. Set them as default prefs in startup() so that we don't have to worry about resetting them at the end of the experiment: let defaults = Services.prefs.getDefaultBranch(""); defaults.setBoolPref(PREF_SHOWUI, testGroup); ... >diff --git a/chrome.manifest b/chrome.manifest Did you include this file to suppress a warning? It seems unnecessary.

(In reply to :Felipe Gomes from comment #3) > Marco: yeah I'm taking it. I'll provide a point estimate soon as I get to > know a bit more of the work involved here. But it's probably gonna be a 3 or > a 5. Hi Felipe, checking in to see if you've had enough time to review and provide an estimate.

Hi Marco, yeah. Setting p=5.

In summary: - the problem about my experiment getting in a strange state was because I was using a local build which doesn't have telemetry enabled by default. I was forcing the Experiments service to initialize and run, which got the experiment installed and everything.. but some other maintenance/startup tasks that properly checked for toolkit.telemetry.enabled wouldn't run and reactive it after restarts. I filed bug 1026850 to handle this better in the UI - regarding the add-on being marked as pending uninstall - it's the expected behavior of the experiment system, but it shouldn't be displayed to the user as it was. Filed bug 1026853. - The exception thrown is indeed bug 1012466, but it's not causing any problems. Added more details to that bug with my log.

(In reply to Benjamin Smedberg [:bsmedberg] from comment #12) > Comment on attachment 8440471 [details] [diff] [review] > First ver > > >+function install(data, reason) { > >+ let testGroup = (Math.random() > 0.5); > >+ > > In case of any future upgrade, this will run again and may cause the user to > switch groups. I think you should do all this in startup() and store this > value in an experiment-specific pref, e.g. > > let inTestGroup; > try { > inTestGroup = Services.prefs.getBoolPref(TRANSLATION_EXPERIMENT_GROUP); > } catch (e) { > inTestGroup = (Math.random() > 0.5); > Services.prefs.setBoolPref(TRANSLATION_EXPERIMENT_GROUP, inTestGroup); > } I had this! Then wondered if it was necessary and removed it. I'll put it back. > >+ Services.prefs.setBoolPref(PREF_SHOWUI, testGroup); > >+ Services.prefs.setBoolPref(PREF_DETECTLANGUAGE, true); Setting these prefs on startup won't be good. The DETECTLANGUAGE pref is checked on tab creation (specifically, when the framescript of that tab loads), so a toggle of the pref only takes effect for new tabs. There would be a race between SS and the experiment getting activated, and if SS ran first the restored tabs wouldn't be running language detection. > >+ > >+ Services.prefs.setCharPref(PREF_CLIENTID, "AAA"); > >+ Services.prefs.setCharPref(PREF_APIKEY, "BBB"); > > These should not be set as user prefs. Set them as default prefs in > startup() so that we don't have to worry about resetting them at the end of > the experiment: But the defaultBranch is not saved to dis.. oh, that's clever! Yeah I will do that for the API keys prefs and set them on startup. I guess in the case of an upgrade, since uninstall/install runs again, there's no good way to clear the EXPERIMENT_GROUP pref after the experiment finishes, right? (because we can't use uninstall). So we'll leave that pref behind, but that sounds ok. Bug 1017806 will probably cover that for future experiments!

Add-on with the suggested changes made. TBD is the clientid/apikey, and add-on name/description in english and german.

Sorry for the churn, added strictCompatibility to the install.rdf for extra nicety.. The patch also includes the icon.png and icon64.png images, which I took from the hotfixes add-ons (the icon is the latest FF logo)

Comment on attachment 8442268 [details] [diff] [review] Add-on Ah, didn't notice you had r+ before attaching a new version.. I'll carry the r+. If I shouldn't use strictCompatibility for some reason, speak up!

Do we even use the icons for experiment addons?

This is the patch for the deployment server. It includes experiment.xpi with the current add-on built, but that file should be replaced with a new one when we rebuild the add-on with the API keys and sign the xpi. startTime/endTime/maxActiveSeconds TBD.

The icon is used if present, but I just noticed the default also looks nice, it uses the Experiments icon. So I suppose it is not necessary. Here's an screenshot comparing both.

Comment on attachment 8442274 [details] [diff] [review] Telemetry server patch No .xpi signing is necessary for experiments.

KaiRo, Archaeopteryx: the experiment add-on which will enable the trial needs a name and a description that shows up in the add-ons manager. (See the attached image in this bug to see what it looks like, but basically it's the same thing as an add-on, except in a different pane of the add-ons manager). Would you be able to provide a translation for it, so I can include a localized description in the add-on? Here is what we've settled for in English: Name: Automatic Translation Description: Read more of the web by automatically translating sites written in different languages.

Name: Übersetzungsfunktion Description: Stellt die Möglichkeit, Seiten übersetzen zu lassen, zur Verfügung KaiRo: What's your opinion? I dropped the "automatic" part because people could expect that the page gets translated without interaction by the user which isn't true.

Archeopteryx, I agree on the "Name", but my proposal for the "Description" would go in a different, more literal, direction. Description: Erweitert die Verständlichkeit des Web durch automatisierte Übersetzung von Seiten in andreren Sprachen. I don't think the connotation of "automatic" is different between English and German, and all in all it's still machine translation. For me "automatic" in this context mostly means "expect weirdness and potentially low quality" and we should not entirely remove that. :p

How do you like this? Name: Übersetzungsfunktion Description: Automatische Übersetzung von in anderen Sprachen geschriebenen Seiten

Sounds OK, after all it's just an experiment for beta users, we shouldn't spend too much time on its description ;-)

Hi Florin, can a contact be assigned for QA verification of this bug.

Requesting final approval from release-drivers to ship this experiment to the Aurora population.

Go ahead with Aurora test.

Guide for verifying this bug, using an Aurora 32 German build: Set the following prefs (some of them do not exist and should be created): - experiments.manifest.uri => https://telemetry-experiment-dev.allizom.org/ - experiments.manifest.cert.checkAttributes => false - experiments.logging.dump => true - experiments.logging.level => 0 Run in the Browser Console: Cu.import("resource:///modules/experiments/Experiments.jsm"); Experiments.instance().updateManifest(); Expected behavior: - Users of Aurora 32, from the German locale ("de"), with a build equal or newer than 2014-06-21, should get the translation experiment installed. It will show up as a new entry in the Add-ons manager, under a "Experiments" section. What to verify: - Verify that the expected behavior works, and the following prefs are set: browser.translation.detectLanguage => true experiments.translation-aurora32-de.isTestGroup => either true or false browser.translation.ui.show => same value as the isTestGroup pref - Verify that the experiment is not installed on other conditions, e.g. - Users of a channel which is not Aurora - Users of Aurora but not German - Users of German Aurora with a build older than 2014-06-21 - Also it would be nice to close Firefox, move the computer time ahead to Jul 28 or later, re-open, and verify that the experiment will uninstall, the prefs will be reset (except for the isTestGroup pref) Notes: to re-run tests using the same profile, remove the "experiments.json" file from the profile folder check if language detection is working but don't use actual translation lots of times.. Just a small check one or two times to make sure it works, with a small page: http://people.mozilla.org/~fgomes/trtest.htm this is because we don't want to spend the translation quota doing tests Aurora builds from 2014-06-27 have the experiments system broken (see bug 1031326) so it won't be possible to verify them with those builds until that bug is re-uplifted

Felipe, I've been trying to install the experiment using "experiments.manifest.uri;https://telemetry-experiment-dev.allizom.org/" on several different builds but I keep getting the following error every single time: 1403931206980 Browser.Experiments.Experiments ERROR Experiments #0::_loadManifest - failure to fetch/parse manifest (continuing anyway): SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data [Log.jsm:760] However when I use "experiments.manifest.uri;https://telemetry-experiment-dev.allizom.org/firefox-manifest.json", the experiment installs without any issues. These are the builds that I've reproduced the error with while using "experiments.manifest.uri;https://telemetry-experiment-dev.allizom.org/" - http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-06-22-00-40-04-mozilla-aurora-l10n/ [Received Error] - http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-06-23-00-40-02-mozilla-aurora-l10n/ [Received Error] - http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-06-24-00-40-01-mozilla-aurora-l10n/ [Received Error] - http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-06-25-00-40-01-mozilla-aurora-l10n/ [Received Error] - http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-06-26-00-40-01-mozilla-aurora-l10n/ [Received Error] In the mean time, I'll keep going through this ticket using the "experiments.manifest.uri;https://telemetry-experiment-dev.allizom.org/firefox-manifest.json" method and make sure there's no other issues. If this is an issue, I'll re-test everything using "experiments.manifest.uri;https://telemetry-experiment-dev.allizom.org/" which shouldn't take that long. (BTW, I'm testing on a VM Win 8.1 and restore the machine to a brand new state every single time)

Build Testing: * Installed firefox-32.0a2.de.win32.installer [20140619004025] and attempted to install the experiment, received the following error message: ** 1403934742076 Browser.Experiments.Experiments DEBUG ExperimentEntry #1::isApplicable() - id=fx-translation-de-aurora32@mozilla.org - test 'minBuildID' failed * Installed firefox-32.0a2.pl.win32.installer [20140626004001] and attempted to install the experiment, received the following error message: ** 1403939085556 Browser.Experiments.Experiments DEBUG ExperimentEntry #0::isApplicable() - id=fx-translation-de-aurora32@mozilla.org - test 'locale' failed * Installed firefox-32.0a2.sq.win32.installer [20140626004001] and attempted to install the experiment, received the following error message: ** 1403939646726 Browser.Experiments.Experiments DEBUG ExperimentEntry #0::isApplicable() - id=fx-translation-de-aurora32@mozilla.org - test 'locale' failed * Installed firefox-33.0a1.en-US.win32.installer [20140627030212] and attempted to install the experiment, received the following error message: ** 1403940065991 Browser.Experiments.Experiments DEBUG ExperimentEntry #0::isApplicable() - id=fx-translation-de-aurora32@mozilla.org - test 'channel' failed * Installed Firefox Setup 31.0b4 [20140623175014] and attempted to install the experiment, received the following error message: ** 1403940462472 Browser.Experiments.Experiments DEBUG ExperimentEntry #0::isApplicable() - id=fx-translation-de-aurora32@mozilla.org - test 'channel' failed For all of the builds mentioned above, I also went through the following test cases: * ensured that the experiment container wasn't created under about:addons * ensured that the experiment wasn't appearing under about:support * ensured that experiments.activeExperiment;false * ensured that experiments.translation-aurora32-de.isTestGroup wasn't created * ensured that browser.translation.detectLanguage;false * ensured that extensions.bootstrappedAddons;{} Experiment Testing: Used the following build to test the experiment: * installed correctly: 1403941090939 Browser.Experiments.Experiments DEBUG Experiments #0::evaluateExperiments() - activating experiment fx-translation-de-aurora32@mozilla.org * ensured experiments.activeExperiment;true * ensured that once the experiment is disabled/uninstalled, experiments.activeExperiment;false * ensured experiments.translation-aurora32-de.isTestGroup was created * ensured browser.translation.detectLanguage;true * ensured that once the experiment is disabled/uninstalled, browser.translation.detectLanguage;false * ensured browser.translation.ui.show matched the value of experiments.translation-aurora32-de.isTestGroup * ensured that the experiment appeared under about:addons * ensured that once the experiment is disabled/uninstalled, it appears as inactive * ensured that the experiment is being listed under about:support as active:true * ensured that a disabled experiment is being listed under about:support as active:false * ensured the experiment is still enabled after restarting fx several times * ensured that a disabled experiment isn't re-enabled once fx restarts * moved the machines date forward a few days and ensured that the experiment is displaying the correct amount of days remaining * ensured that once the 29 days have past, the experiment was disabled * ensured that visiting http://people.mozilla.org/~fgomes/trtest.htm while the experiment was enabled displayed a translation menu * translated http://people.mozilla.org/~fgomes/trtest.htm a few times to English without any issues * ensured http://people.mozilla.org/~fgomes/trtest.htm worked on several different tabs at the same time * ensured that once the experiment has been removed/disabled, the translation menu doesn't appear anymore Possible Issues: - Disabling/Enabling the experiments.enabled;true will completely disable the current experiment even though there are 29 remaining Felipe, other then the above issue and the issue I mentioned in comment # 34, everything appears to be working! Please let me know if there's anything that I missed.

Hey Kamil, thanks for the thorough testing. Glad that you managed to test with the alternate URL. I realized that my instruction was incomplete about the experiments.manifest.uri pref. The part to be changed is only the base of the URL, while the rest needs to be kept similar to the default. The final value for the pref should be: experiments.manifest.uri => https://telemetry-experiment-dev.allizom.org/manifest/v1/firefox/%VERSION%/%CHANNEL% I think it's just needed a quick recheck that with that value the system works, no need to reverify everything because those two URLs are actually equivalent.

(In reply to Kamil Jozwiak [:kjozwiak] from comment #35) > Possible Issues: > > - Disabling/Enabling the experiments.enabled;true will completely disable > the current experiment even though there are 29 remaining I don't know if this behavior is intentional or not by the experiments system. But in any case it is unrelated to this specific experiment. gfritzche, do you know?

I went through a few builds and made sure that the experiment is being installed with the new manifest link from comment #36: * http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-06-25-00-40-01-mozilla-aurora-l10n/ (made sure that the translation worked) * http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-06-26-00-40-01-mozilla-aurora-l10n/ (made sure that the translation worked) Also made sure that the experiment wasn't being installed on incorrect locales/channels while using the new manifest link from comment #36: * firefox-32.0a2.de.win32.installer [20140619004025], received the correct 'minBuildID' error * firefox-32.0a2.ach.win32.installer [20140625004001], received the correct test 'locale' failed error * firefox-33.0a1.en-US.win32.installer [20140626030201], received the correct test 'channel' failed error Felipe, the link from comment #36 doesn't have a certification correct? You have to use "experiments.manifest.cert.checkAttributes;false" to get the experiment installed (just double checking). I'm guessing the certification portion will kick in once the experiment is moved into the main telemetry staging server? Other than comment #37, everything looks good!

(In reply to :Felipe Gomes from comment #37) > (In reply to Kamil Jozwiak [:kjozwiak] from comment #35) > > Possible Issues: > > > > - Disabling/Enabling the experiments.enabled;true will completely disable > > the current experiment even though there are 29 remaining > > I don't know if this behavior is intentional or not by the experiments > system. But in any case it is unrelated to this specific experiment. > gfritzche, do you know? That is intentional. Disabling the system has to immediately stop every experiment and we never re-enable experiments.

(In reply to Kamil Jozwiak [:kjozwiak] from comment #38) > Felipe, the link from comment #36 doesn't have a certification correct? You > have to use "experiments.manifest.cert.checkAttributes;false" to get the > experiment installed (just double checking). I'm guessing the certification > portion will kick in once the experiment is moved into the main telemetry > staging server? For security reasons the certificate for the main manifest server is pinned (bug 979474), it should be enough to set experiments.manifest.cert.requireBuiltin to false.

Update minBuildI to 2014-06-30 due to bug 1031326, and bump endDate to Aug 1 to have 30+ days of test window.

minBuildId/endDate bump: http://hg.mozilla.org/webtools/telemetry-experiment-server/rev/fdef65492439

Hi Kamil, the desktop iteration ends on Monday. Is it possible to have this bug verified by then.

Went through verification with the German locale using the following build: - http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora-l10n/ Quickly went through the experiment and ensured it was installed and worked as expected. This has also been tested via comment #34, comment #35 and comment #38. Other things that I quickly checked: * ensured that when Telemetry is disabled via the "Options", the experiment isn't being installed * ensured that when the telemetry experiment is installed, disabling Telemetry via "Options" disables the experiment