Experiment Type: Pref Flip Study What is the preference we will be changing browser.newtabpage.activity-stream.discoverystream.config What are the branches of the study and what values should each branch be set to? - Treatment control 25%: Value: "{\"api_key_pref\":\"extensions.pocket.oAuthConsumerKey\",\"enabled\":false,\"show_spocs\":true,\"layout_endpoint\":\"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=basic\"}" Control, user gets current newtab/homepage experience (activitystream). Discoverystream is not used. - Treatment above-7-below-1 25%: Value: "{\"api_key_pref\":\"extensions.pocket.oAuthConsumerKey\",\"enabled\":true,\"show_spocs\":true,\"layout_endpoint\":\"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=66-release-2-above-7-below-1\"}" This is a more complex layout with a hero element on top and multiple topic columns below. - Treatment complex 25%: Value: "{\"api_key_pref\":\"extensions.pocket.oAuthConsumerKey\",\"enabled\":true,\"show_spocs\":true,\"layout_endpoint\":\"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=66-beta-2-complex\"}" Complex: pocket section has been expanded with multiple rows of content. - Treatment basic 25%: Value: "{\"api_key_pref\":\"extensions.pocket.oAuthConsumerKey\",\"enabled\":true,\"show_spocs\":true,\"layout_endpoint\":\"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=basic\"}" "Basic" variant of discoverystream, replicates the current experience, but does not include highlights section. What version and channel do you intend to ship to? 0.5% of Release Firefox 66.0 Are there specific criteria for participants? Locales: en-US Geographic regions: US Prefs: * please exclude any users with non-default values in the following prefs: - `browser.newtabpage.enable` - `browser.startup.homepage` - `browser.newtabpage.activity-stream.showSearch` - `browser.newtabpage.activity-stream.feeds.topsites` - `browser.newtabpage.activity-stream.feeds.section.topstories` - `browser.newtabpage.activity-stream.feeds.section.highlights` Studies: * N/A Any additional filters: * please exclude any users that have the Mozilla Dark Theme activated: `['environment']['addons']['activeAddons']['theme']['id']` is `'firefox-compact-dark@mozilla.org'` Please only include profiles in release that are 66 and 67. If they are FF68 and above, exclude them from the experiment. Note on population size: again, the purpose of this experiment is to check that everything related to the pref is working correctly on release. Hence, we don't care too much about the power of the experiment. As such, I'm targeting about 12,000 users in each branch, which should be 0.5% of the eligible population (roughly estimated here: https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/64292/command/65044) What is your intended go live date and how long will the study run? Mar 25, 2019 - May 06, 2019 (42 days) What is the main effect you are looking for and what data will you use to make these decisions? What is the main effect you are looking for and what data will you use to make these decisions? What metrics are you using to measure success We are not looking for a specific effect, but we are looking to make sure that the metrics we will use are reporting as expected and in a reasonable range. Docs with metrics we will test here: * Topline Success Metrics: https://docs.google.com/document/d/1iMRp13mAXVPp4Am58HIRpWybIbVGT_1w9IbvDywZIUA/edit# * Dashboard and metrics of interest breakdown: https://docs.google.com/document/d/16wAKJ2d_ALLhwM5Os1DwVrkdwilCyw_AwrKWCaNmTLc/edit Do you plan on surveying users at the end of the study? Yes/No. Strategy and Insights can help create surveys if needed No Who is the owner of the data analysis for this study? shong@mozilla.com Will this experiment require uplift? False QA Status of your code: QA guidelines here: https://docs.google.com/document/d/1iySmNA2tBl3ApLuJtP7UBlPwNx9sH1SmK-jG6exvMQ0/edit#heading=h.s177b01wzbij Link to more information about this study: https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtabh12019-release66smoketest1/