[Experiment] Pref-Flip Experiment: Holdback Study for Quantumbar v2 Fx 82.0 to 85.0 Release
Categories
(Shield :: Shield Study, task, P3)
Tracking
(firefox82+ fixed, firefox83+ fixed, firefox84+ fixed)
People
(Reporter: experimenter, Assigned: nraketic)
References
(Depends on 1 open bug, )
Details
(Whiteboard: [no-nag])
User Story
Delivery Type: Pref Flip Experiment
What is the preference we will be changing
browser.urlbar.experiment.update2
What are the branches of the experiment and what values should
each branch be set to?
- Treatment Holdback Group, No Update 50%:
Value: false
Holdback group that does not get v2 update
- Control QuantumBar v2 Group 50%:
Value: true
Group that gets v2 update
What version and channel do you intend to ship to?
10% of Release Firefox 82.0 to 85.0
Are there specific criteria for participants?
Prefs:
Experiments:
Any additional filters:
Countries: all
Locales: all
What is your intended go live date and how long will the experiment run?
Nov 10, 2020 - Feb 22, 2021 (104 days)
What is the main effect you are looking for and what data will you use to
make these decisions?
We will measure `search_counts`, `n_search_alias`, `n_searchmode`, and `search_engagement`, which is `sap/(organic + sap)`.
We want to know the search volume impact of the new features for the address bar. Search engagement and QuantumBar searchmode engagement (proxy for revenue) are the KPIs.
# Metrics
- `search_counts`
- `searchmode_counts`
- `search_counts_by_sap`
- `search_engagement` = sap / (organic + sap)
- Amazon search mode count by entry (oneoffs, typed) for treatment branch
- Amazon search alias count for control branch
Special Note:
- Analysis should be mindful of the TopSites study that will be running at the same time as this study.
- From Engineering: `urlbar.tips.tabtosearch_onboard-shown` , `urlbar.picked.searchmode.keywordoffer`, `urlbar.picked.searchmode.topsites_urlbar`, `urlbar.picked.searchmode.tabtosearch`, and `urlbar.picked.searchmode.tabtosearch_onboard` are broken in Fx 83. All the other urlbar.picked.searchmode.* probes are fine
Who is the owner of the data analysis for this experiment?
tbrooks@mozilla.com
Will this experiment require uplift?
False
QA Status of your code:
Link to more information about this experiment:
https://experimenter.services.mozilla.com/experiments/holdback-study-for-quantumbar-v2/
Holdback Study for Quantumbar v2
Holdback Study - Release QuantumBar v2 to 90% of users and 5% treatment, 5% control for a holdback study
Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.
Data Science Issue: https://jira.mozilla.com/browse/DO-364
More information: https://experimenter.services.mozilla.com/experiments/holdback-study-for-quantumbar-v2/
|
||
Updated•4 years ago
|
Holdback Study - Release QuantumBar v2
Firefox 82 to Firefox 85
We have finished testing the [PI-843] Holdback Study - Release QuantumBar v2 experiment on Firefox 82 and 83. Considering the fact that the experiment will be live until Firefox 85, for a more accurate assessment, we will retest the experiment on Firefox 84 and 85 builds closest to RC / Firefox Release RC build and send a new sign off email for each version.
Quality status: GREEN - SHIP IT
Why is this experiment green?
Delivery mechanism:
- During testing we have encountered one new issue:
- Bug 1675796 - The profile remains enrolled in the "QuantumBar v2 Group" branch after manually disabling the feature from about:config
- This is an edgecase because users would have to manually disable the feature using the "about:config" page. However, QA is concerned that this might also affect the experiment’s telemetry data since the feature will disabled but they will still be enrolled in the experiment.
Product specific:
- Since the features functionality was not in scope, we only verified that the features are enabled/disabled accordingly by the experiment and have found no issues during testing.
- The QuantumBar v2 features were tested and signed off as GREEN by the Desktop teams (PI-715 and PI-735).
Testing Summary:
- Test Runs: Test Rail;
Tested Platforms:
- Windows 10 x64,
- macOS 10.15.6;
- Ubuntu 20.04 x64;
Tested Firefox versions:
- Firefox Release 82.0.2 en-US (Build ID: 20201027185343)
- Firefox Release 82.0.2 de (Build ID: 20201027185343)
- Firefox Release 82.0.2 fr (Build ID: 20201027185343)
- Firefox Beta 83.0b9 en-US (Build ID: 20201105203649)
- Firefox Beta 83.0b9 de (Build ID: 20201105203649)
- Firefox Beta 83.0b9 fr (Build ID: 20201105203649)
|
||
Comment 2•4 years ago
|
||
[Tracking Requested - why for this release]:
|
||
Updated•4 years ago
|
|
Reporter | |
Updated•4 years ago
|
|
|
Reporter | |
Comment 3•4 years ago
|
||
|
||
Updated•4 years ago
|
|
||
Updated•4 years ago
|
Holdback Study - Release QuantumBar v2
Firefox 82 to Firefox 85
We have finished testing the “[PI-887] - Holdback Study - Release QuantumBar v2” experiment on Firefox 84.
The study is already live, but since it's running until Firefox 85, we are running our test suite on the latest builds of each Beta cycle to make sure that there are no new issues between Firefox versions.
Quality status: GREEN
Why is this experiment green?
- We haven’t found any issues during testing.
Testing Summary:
Tested Platforms:
- Windows 10 x64;
- macOS 10.16.6;
- Ubuntu 20.0 x64;
Tested Firefox versions:
- Firefox Release 82.0.3 en-US (Build ID: 20201108180448)
- Firefox Release 82.0.3 de (Build ID: 20201108180448)
- Firefox Release 82.0.3 fr (Build ID: 20201108180448)
- Firefox Beta 84.0b7 en-US (Build ID: 20201201213706)
- Firefox Beta 84 RC de build (Build ID: 20201207203640)
- Firefox Beta 84 RC fr build (Build ID: 20201207203640)
Holdback Study - Release QuantumBar v2
Firefox 82 to Firefox 85
We have finished testing the “[PI-909] - Holdback Study - Release QuantumBar v2 for Firefox 85” experiment.
The study is already live, but since it's running until Firefox 85, we are running our test suite on the latest builds of each Beta cycle to make sure that there are no new issues between Firefox versions.
Quality status: GREEN
Why is this experiment green?
- We haven’t found any issues during testing.
Testing Summary:
Tested Platforms:
- Windows 10 x64;
- macOS 10.16.6;
- Linux Mint 20 6x4;
Tested Firefox versions:
- Firefox Beta 85.0b8 fr (Build ID: 20210112185508)
- Firefox Beta 85.0b9 de build (Build ID: 20210114193053)
- Firefox Beta 85 RC en-US build (Build ID: 20210118153634)
|
|
Reporter | |
Updated•4 years ago
|
|
|
||
Updated•4 years ago
|
Description
•