[Experiment] Pref-Flip Experiment: activity-stream-pocket-new-tab-hero-layout-test-1624064 Fx 77.0 to 78.0 Release
Categories
(Shield :: Shield Study, task, P3)
Tracking
(firefox77+ fixed)
People
(Reporter: experimenter, Assigned: digitalnewslab)
References
()
Details
(Whiteboard: [no-nag])
User Story
Delivery Type: Pref Flip Experiment What is the preference we will be changing browser.newtabpage.activity-stream.discoverystream.config What are the branches of the experiment and what values should each branch be set to? - Treatment Hero Layout 50%: Value: {"api_key_pref":"extensions.pocket.oAuthConsumerKey","collapsible":true,"enabled":true,"show_spocs":true,"hardcoded_layout":false,"personalized":true,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=hero-layout","unused_key":"unused_2"} New Hero layout with hero element above 6 rows of tiles. - Control 3 Columns 7 Rows 50%: Value: {"api_key_pref":"extensions.pocket.oAuthConsumerKey","collapsible":true,"enabled":true,"show_spocs":true,"hardcoded_layout":true,"personalized":true,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=basic","unused_key":"unused_2"} Default New Tab layout with 3 Columns and 7 Rows of tiles. What version and channel do you intend to ship to? 1% of Release Firefox 77.0 to 78.0 Are there specific criteria for participants? Prefs: browser.newtabpage.enabled OR (we want AT LEAST one of these to be default) browser.startup.homepage AND (we want ALL of the below to be default) browser.newtabpage.activity-stream.showSearch AND browser.newtabpage.activity-stream.feeds.topsites AND browser.newtabpage.activity-stream.feeds.section.topstories AND browser.newtabpage.activity-stream.feeds.section.highlights Experiments: None Countries: United States of America (US) Locales: English (US) (en-US) What is your intended go live date and how long will the experiment run? Jun 09, 2020 - Jul 07, 2020 (28 days) What is the main effect you are looking for and what data will you use to make these decisions? If we see higher engagement with Hero Layout we will make it the default New Tab layout. Our primary success metrics will be (1) the % of each variant that engaged with Pocket recs at least once during the test and (2) the click rate on the 1st position Pocket rec. This latter metric is important, as it allows us to understand whether showing Syndicated content in the Hero position can send more traffic there. Do you plan on surveying users at the end of the delivery? No. Strategy and Insights can help create surveys if needed Who is the owner of the data analysis for this experiment? kirill@getpocket.com Will this experiment require uplift? False QA Status of your code: QA Status: GREEN. For more information, see: https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-hero-layout-test-1624064/#comment872 Link to more information about this experiment: https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-hero-layout-test-1624064/
activity-stream-pocket-new-tab-hero-layout-test-1624064
Pocket New Tab study comparing engagement with stories and SPOCs between hero layout and Control.
Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.
Data Science Issue: https://jira.mozilla.com/browse/DO-352
More information: https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-hero-layout-test-1624064/
Updated•5 years ago
|
Comment 2•4 years ago
|
||
New Tab Layouts - Release 77 Test - 6/9
Firefox Release 77
We have finished testing the [PI-569] New Tab Layouts - Release 77 Test - 6/9 experiment.
Quality status: GREEN-SHIP IT
Why is this experiment green?
- During testing we have encountered 4 known issues. However, the issues are not considered of high impact for the end user experience.
Testing Summary:
- Test Suite: TestRail
Tested Platforms:
- Windows 10 x64
- Mac 10.15
- Ubuntu 18.04 x64
Tested Firefox versions:
- Firefox 77 RC (en-US)
Best regards,
Robert Martin
Ecosystem QA
Updated•4 years ago
|
Reporter | ||
Updated•4 years ago
|
Reporter | ||
Comment 3•4 years ago
|
||
Updated•4 years ago
|
Description
•