[Experiment] Pref-Flip Experiment: Add-on Blocklisting using Remote Settings Fx 71.0 Beta
Categories
(Shield :: Shield Study, task, P3)
Tracking
(firefox71+ fixed)
People
(Reporter: experimenter, Assigned: esmyth)
References
()
Details
User Story
Experiment Type: Pref Flip Experiment What is the preference we will be changing extensions.blocklist.useXML What are the branches of the experiment and what values should each branch be set to? - Treatment XML blocklist cohort 50%: Value: true Members of this cohort will use the current blocklist implementation based on the daily download of a large XML file. - Treatment RemoteSettings blocklist cohort 50%: Value: false Members of this cohort will use the proposed blocklist implementation based on RemoteSettings which provides differential JSON-formatted updates. What version and channel do you intend to ship to? 20% of Beta Firefox 71.0 Are there specific criteria for participants? Prefs: extensions.blocklist.enabled == True Experiments: Any additional filters: Countries: all Locales: all What is your intended go live date and how long will the experiment run? Oct 28, 2019 - Nov 11, 2019 (14 days) What is the main effect you are looking for and what data will you use to make these decisions? What is the main effect you are looking for and what data will you use to make these decisions? What metrics are you using to measure success No significant difference (< 5%) of Remote Settings blocklist efficacy when compared to the XML-based implementation. No significant increase (≤ 5%) in crash rate in: - overall crash reports - OOM crash reports - shutdown crashes No significant regressions (≤ 5%) on the following telemetry probes: - timestamps.first_paint - timestamps.about_home_topsites_first_paint - FX_PAGE_LOAD_MS_2, - FX_TAB_SWITCH_COMPOSITE_E10S_MS At least 2.5% improvement to browser startup time measured by: - timestamps.first_paint Do you plan on surveying users at the end of the experiment? No. Who is the owner of the data analysis for this experiment? cdowhygelund@mozilla.com Will this experiment require uplift? False QA Status of your code: Signed-off as GREEN. For more information see https://experimenter.services.mozilla.com/experiments/add-on-blocklisting-using-remote-settings/#comment495. PI-160 (https://moz-pi-test.atlassian.net/servicedesk/customer/portal/9/PI-160) Link to more information about this experiment: https://experimenter.services.mozilla.com/experiments/add-on-blocklisting-using-remote-settings/
Add-on Blocklisting using Remote Settings
We believe that migrating the add-on blocklist from a static XML file to use Remote Settings will result in a 2.5-7.5% improvement to browser startup time. We also need to validate that the change will not break the addon blocklist functionality particularly via crashes/failure to load the blocklist or false negatives for blocklist checks.
Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.
More information: https://experimenter.services.mozilla.com/experiments/add-on-blocklisting-using-remote-settings/
[Tracking Requested - why for this release]: Given risks associated with this change, our risk mitigation plan requires experimental validation. FF70b is the first beta period available following the approval of the risk assessment and mitigation plan.
Comment 2•5 years ago
|
||
(In reply to esmyth from comment #1)
F70b is the first beta period available following the approval of the risk assessment and mitigation plan.
Do you mean to have this experiment in 70beta or 71beta?
71 beta. I must've been thinking about something else when I wrote that comment.
Updated•5 years ago
|
Comment 4•5 years ago
|
||
Add-on Blocklisting using Remote Settings
Beta 71
We have finished testing the Add-on Blocklisting using Remote Settings experiment.
QA’s recommendation: GREEN - SHIP IT
Reasoning:
- We haven’t found any new issues and the only known issue is a Normandy related one that is unlikely to be encountered by users.
Testing Summary:
- TestRail: Test suite
Tested Platforms:
- Windows 10 x64
- Ubuntu 16 x64
- MacOS 10.15
Tested Firefox versions:
- Firefox Nightly 71
- Firefox Beta 71
Regards,
Robert Martin
Reporter | ||
Updated•5 years ago
|
Reporter | ||
Updated•5 years ago
|
Updated•5 years ago
|
Description
•