Experimenter Bot Reporter
	Comment 5 • 5 years ago

    Experiment Type: Pref Flip Study

    What is the preference we will be changing

browser.newtabpage.activity-stream.discoverystream.config

    What are the branches of the study and what values should
    each branch be set to?

- Treatment control 33%:

Value: {"enabled":false,"show_spocs":true,"layout_endpoint":"https://getpocket.com/v3/newtab/layout?version=1&consumer_key=40249-e88c401e1b1f2242d9e441c4&layout_variant=basic"}

Control branch, no changes to existing experience
        
- Treatment default 33%:

Value: {"api_key_pref":"extensions.pocket.oAuthConsumerKey","enabled":true,"show_spocs":true,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=basic"}

Replication of control branch with new discovery-stream API
        
- Treatment complex 34%:

Value: {"api_key_pref":"extensions.pocket.oAuthConsumerKey","enabled":true,"show_spocs":true,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=66-beta-2-complex"}

This is a more complex layout with a hero element on top and multiple topic rows below.
        

    What version and channel do you intend to ship to?

15% of Beta Firefox 66.0

    Are there specific criteria for participants?

Locales:
* en-US only

Geographic regions:
* US only

Prefs:
* please exclude any users with non-default values in the following prefs:
- `browser.newtabpage.enable`
- `browser.startup.homepage`
- `browser.newtabpage.activity-stream.showSearch`
- `browser.newtabpage.activity-stream.feeds.topsites`
- `browser.newtabpage.activity-stream.feeds.section.topstories`
- `browser.newtabpage.activity-stream.feeds.section.highlights`

Studies:
* please exclude any profiles that have been enrolled in: 
    - activity-stream-pkt-newtab-beta66smoketest1    
    - experiment link: https://experimenter.services.mozilla.com/experiments/activity-stream-pkt-newtab-beta66smoketest1/ 

Any additional filters:
* please exclude any users that have the Mozilla Dark Theme activated: `['environment']['addons']['activeAddons']['theme']['id']` is `'firefox-compact-dark@mozilla.org'`
    

ADDITIONAL NOTE: the code for this experiment is not currently landed in Beta66 yet. There will be an uplift landing the code on Friday. There will be an additional filter to target only those users who have updated with the latest code (will update in comments)

    What is your intended go live date and how long will the study run?

Feb 25, 2019 - Mar 27, 2019 (30 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will
you use to make these decisions? What metrics are you using to measure success

We are looking for non-regression in browser usage metrics (days used, retention, hours, active hours, uris, and searches). We are looking for increase in AS and pocket engagement metrics. We are looking for non-regressions in AS performance metrics. 

Do you plan on surveying users at the end of the study? Yes/No.
Strategy and Insights can help create surveys if needed
 
No.

    Who is the owner of the data analysis for this study?

shong@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

The currently public version of Beta does not have all the code we need for this experiment. It will be uplifted on Friday, and users on Monday will have the code. 

However, for QA purposes, please follow the instructions provided here: 

https://docs.google.com/document/d/1iySmNA2tBl3ApLuJtP7UBlPwNx9sH1SmK-jG6exvMQ0/edit#heading=h.s177b01wzbij

Under the section: tinderbox build (relman has provided a tinderbox build with the changed uplifted before the official uplift date for us to QA on)

    Link to more information about this study:

https://experimenter.services.mozilla.com/experiments/activity-stream-pkt-newtab-beta66complexandperformance2/