Closed Bug 1634726 Opened 5 years ago Closed 4 years ago

[Experiment] Pref-Flip Experiment: activity-stream-pocket-new-tab-newsletter-signup-study-1616997 Fx 78.0 to 79.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox77+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox77 + fixed

People

(Reporter: experimenter, Assigned: digitalnewslab)

References

(
URL
)

Details

(Whiteboard: [no-nag])

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

browser.newtabpage.activity-stream.discoverystream.config

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment Email Snippet 50%:

Value: None

New Tab layout with newsletter signup snippet
        
- Control Default 3x7 Layout 50%:

Value: None

New Tab layout without a newsletter signup snippet
        

    What version and channel do you intend to ship to?

1% of Release Firefox 78.0 to 79.0

    Are there specific criteria for participants?

Prefs:

browser.newtabpage.enabled
OR (we want AT LEAST one of these to be default)
browser.startup.homepage

AND (we want ALL of the below to be default)

browser.newtabpage.activity-stream.showSearch
AND
browser.newtabpage.activity-stream.feeds.topsites
AND
browser.newtabpage.activity-stream.feeds.section.topstories
AND
browser.newtabpage.activity-stream.feeds.section.highlights

Experiments: None
Countries: United States of America (US) 

Locales: English (US) (en-US) 

    What is your intended go live date and how long will the experiment run?

Jul 07, 2020 - Aug 04, 2020 (28 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

If we see a high CTR on the email newsletter sign up snippet with no significant negative impact on stories we will introduce this element to New Tab. Our primary success metric is the % of each variant that clicks a Pocket rec at least once during the experiment.


Do you plan on surveying users at the end of the delivery?  No.
Strategy and Insights can help create surveys if needed

    Who is the owner of the data analysis for this experiment?

kirill@getpocket.com

    Will this experiment require uplift?

False

    QA Status of your code:

QA Status: GREEN. For more information, see: https://experimenter.services.mozilla.com/experiments/new-tab-email-newsletter-signup-snippet/#comment869

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/new-tab-email-newsletter-signup-snippet/

activity-stream-pocket-new-tab-newsletter-signup-study-1616997

A newsletter signup snippet below the first row of Pocket tiles in New Tab

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DO-352
More information: https://experimenter.services.mozilla.com/experiments/new-tab-email-newsletter-signup-snippet/

[Tracking Requested - why for this release]:

tracking-firefox77: --- → ?

Pocket New Tab - Email Signup Text Component [Experiment] - Release 77 - 6/9
Firefox Release 77

We have finished testing the [PI-589] Pocket New Tab - Email Signup Text Component [Experiment] - Release 77 - 6/9 experiment.

Quality status: GREEN - SHIP IT

Why is this experiment green?

  • No issues were found during testing.

Testing Summary:

Tested Platforms:

  • Windows 10 x64
  • Mac 10.15
  • Ubuntu 18.04 x64

Tested Firefox versions:

  • Firefox 77 RC (build 1) en-US
Whiteboard: [no-nag]
User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: activity-stream-pocket-new-tab-newsletter-signup-study-1616997 → [Experiment] Pref-Flip Experiment: activity-stream-pocket-new-tab-newsletter-signup-study-1616997 Fx 78.0 to 79.0 Release
Start Date: 2020-07-07 End Date: 2020-08-04
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
status-firefox77: affected → fixed
User Story: (updated)
You need to log in before you can comment on or make changes to this bug.