What is your basic hypothesis? We want to understand what the CTR impact of showing more rows of Pocket Recommendations in New Tab is. Additionally, we want to understand the changes in engagement on other sections of the New Tab. We think that by observing these changes, we can be better informed about further monetization efforts in the Pocket section, as well as better understand how more content can influence other monetizable sections on the page. What is the preference we will be changing? browser.newtabpage.activity-stream.section.topstories.rows What are the branches of the study and what values should each branch be set to? Control: 1 Portal: 4 What percentage of users do you want in each branch? 1% of en-US, US release users 50% in Control 50% in Portal What Channels and locales do you intend to ship to? 1% Release, version >= 62. The test should run for English users in the US only. What is your intended go live date and how long will the study run? Start: Monday, November 19, 2018 End: Monday, December 10, 2018 Are there specific criteria for participants? The test should run for English users in the US only with Pocket Recs enabled. What is the main effect you are looking for and what data will you use to make these decisions? We are looking for CTR and engagement changes across various sections of New Tab and will look at the following metrics: 1. Firefox 3-Week Retention 2. Click through rate (CTR) on each item across rows, aggregate row CTR, aggregate CTR on the Pocket Recommendations section 3. % of users disabling Pocket in New Tab 4. Search Volume 5. Top Sites Engagement 6. Highlights Engagement Who is the owner of the data analysis for this study? Kirill Demtchouk, with support from Su-Young Hong Will this experiment require uplift? No. QA Status of your code: QA Request has been submitted Additional Materials: PHD: https://docs.google.com/document/d/1gVHkAYI4ne6Ol6W429bSLwp3uIzMtiAoTXaJwd_pBxo/

Given that this is a pref already togglable in the UI this seems fine from a code review standpoint. peer review +

Hey, can we modify the this experiment to have: * 1 week enrollment period (Nov 19th to Nov 25th) * 3 weeks study period (each enrolled profile remains enrolled in the study for 3 weeks) Thanks!

Spoke offline with Tushar and Kenny, there was a bug that was found (https://bugzilla.mozilla.org/show_bug.cgi?id=1504165) that causes multiple sponsored content cards to be shown when we increase the number of rows. The fix for this is in Nightly (but not in Beta or Release yet). As such we'd like to change the experiment to run on Nightly. Is it possible to target this experiment on: * Channel: nightly * Country: US * Locale: en-US * pref: pocket enabled * version: 65.0a1 (and higher) * build ID: 20181107220128 (and higher) And is it possible to sample 100% of the eligible profiles? (not sure what our max enrollment policy is for this for nightly) * 50% control * 50% treatment Duration * 4 weeks enrollment period * open study period (keep enrolled profiles in the study indefinitely until we end the study) We're aware that nightly is not representative of the release population and the population size of nightly is limited (so any browser usage metrics will be under-powered). The experiment is exploratory and we're mostly interested in the pocket card CTR metrics.

Adding in Rosanne / Ciprian to weigh in on Science / QA signoff. PHD updated: https://docs.google.com/document/d/1gVHkAYI4ne6Ol6W429bSLwp3uIzMtiAoTXaJwd_pBxo/

[Shield] Pref Flip Study: Pocket Content Targeted: Firefox Beta 64, Firefox Nightly 64 We have finished testing the Pref Flip Study: Pocket Content shield study experiment. QA’s recommendation: GREEN - SHIP IT Reasoning: - All blocking issues have been fixed and verified on the latest Nightly 65.0a1 and Beta 64.0b9. As such, the study can be launched on the Nightly and Beta channels. Testing Summary: - Verified that the sponsored cards are correctly displayed. - Verified that no more than 1 sponsored card is displayed at the time. - Verified that the sponsored content can be enabled/disabled from the “about:preferences#home” page. - Verified that the study is not affected at browser update. - Verified that the sponsored content is displayed while the browser is in the safe mode. - Verified that the sponsored content can be accessed. Tested Platforms: - Windows 10 x64; - Mac 10.13.3 - Arch Linux 4.16.6 x64 Tested Firefox versions: - Firefox Beta 64.0b9 - Firefox Nightly 65.0a1 Regards, Marius

Science R+

[Tracking Requested - why for this release]: This has changed from a study on Release to a study on Nightly. Adjusting accordingly.

Ryan, just a head's up that this study will launch in Nightly on Monday, November 19th.

After talking with Tushar and given that the initial blocking issue for the Release channel has been fixed in 63.0.3, we have done another round of testing for the Pref Flip Study: Pocket Content shield study experiment on Firefox Release 63.0.3. QA’s recommendation: GREEN - SHIP IT Reasoning: - The blocking issue has been fixed and verified in the latest Firefox Release 63.0.3 and as such we can run this study in the release Firefox channel. - No new issues or regressions have been found during testing. Testing Summary: - Verified that the sponsored cards are correctly displayed. - Verified that no more than 1 sponsored card is displayed at the time. - Verified that the sponsored content can be enabled/disabled from the “about:preferences#home” page. - Verified that the study is not affected at browser update. - Verified that the sponsored content is displayed while the browser is in the safe mode. - Verified that the sponsored content can be accessed. Tested Platforms: - Windows 10 x64; - Mac 10.13.3 - Arch Linux 4.16.6 x64 Tested Firefox versions: - Firefox Release 63.0.3

[Tracking Requested - why for this release]: This has flipped back to a study on Release from Nightly.

following up on the switch back from Release to nightly, please run experiment on following specs below: Targeting: * channel = 'release' * locale = 'en-US' * country = 'US' * pref: pocket enabled * version = '60.0.3' and higher Dates: * Start Date: November 26th, 2018 * Enrollment Period: 1 week * Study Period: 4 weeks (keep subjects in the experiment for 4 weeks after they've been enrolled) Sampling: * 1% of release total * 50/50 split between test and control branches

(In reply to Su-Young Hong from comment #11) > following up on the switch back from Release to nightly, please run > experiment on following specs below: >... > * version = '60.0.3' and higher You meant 63.0.3, right?

Su, Tushar mentioned starting this study this Monday, the 19th if possible, pending Pascal's signoff.

(In reply to Pascal Chevrel:pascalc from comment #12) > (In reply to Su-Young Hong from comment #11) > > following up on the switch back from Release to nightly, please run > > experiment on following specs below: > >... > > * version = '60.0.3' and higher > > You meant 63.0.3, right? Yes, my mistake, 63.0.3

(In reply to Marnie Pasciuto-Wood [:marnie] from comment #13) > Su, Tushar mentioned starting this study this Monday, the 19th if possible, > pending Pascal's signoff. Hi Marnie, yes, that sounds good to me. Nov 19th if possible, otherwise Nov 26th.

I reviewed the study and it looks good to me, so a+ from a relman perspective. Note though that with a one week enrollment and a 4 weeks study, it's likely that enrolled users will have updated to 64 during the study (ship date is December 11) which may affect the conditions of your study if there are changes to about:home in 64 that affect user engagement.

We're live on this. Thanks all!

hey, can we confirm that the experiment slug is "pref-flip-pocket-content-rows-1504077"? I'm monitoring enrollment and it looks like there was only 1 profile enrolled yesterday. I'll check again tomorrow for today's data, but this seems off. https://sql.telemetry.mozilla.org/queries/60232/source https://sql.telemetry.mozilla.org/queries/60231/source

I think our version targeting was inaccurate. Great catch Su, we're fixing now.

Recipe 640 disabled, recipe 648 published as replacement.

Hi Josh, Just wanted to confirm, the new enrollment sampling will exclude profiles that were enrolled in the treatment branch of the original study, is that correct? So the new recipe should be the same as the old, except for the following differences: - a new shield slug that includes the string "activity-stream" - sampling (for both branches) excludes all profiles that were given the treatment from the original study

The slug has been corrected but we did not realize there was a requirement that previous clients be excluded - can you cope with it post-hoc?

Also Su or Tushar, should this push out your end date? Right now I have this study ending 12/24/18.

:jgaunt, we can't because we can't identify pocket interactions from the previous experiment. if we don't exclude profiles enrolled in the old experiment, we could end up with users in the control branch who saw 4 rows previously (or users in the treatment branch who got the treatment a week before the experiment 'started') :( Just so we're on the same page, is the new recipie enrolling users in a new experiment? or are they taking users from the old experiment and 'reenrolling' them in the new experiment? :marnie, yes and no. this experiment should have a 1 week enrollment period (so enroll new profiles from when we turn on to 1 week after), and for those enrolled, stay in the experiment for 4 weeks (so depends on their date of enrollment). does that work?

Both new and old recipes use stable sampling, which selects from the release audience as a whole. Although it is more unlikely this way that the same 1% will have the pref flipped again by the new recipe as the old one (as opposed to bucket sampling which segments the audience before sampling within selected segments), it is not impossible. I was presuming you could find clients with the old slug in the experiments table and exclude those from analysis in case they surfaced in the new sample. I realize that's not ideal, but I don't think impossible? Maybe I've got a blind spot there.

Right right, I agree, normally, that would be an option. But in this case it's an issue because the `assa_impression_stats_daily` data, which is where we calculate the pocket CTR from has this client side-filter condition on the `shield_id` / experiment slug field. If "activity-stream" isn't somewhere in an experiment slug, it won't report it in that data. So the pocket clicks data from clients in the previous experiment (which didn't have the "activity-stream" string) won't report that it belonged to that experiment. So we won't be able to filter those profiles (those enrolled in our new experiment that also were in the last experiment) for the pocket clicks data[1] in our new experiment. [1] And there's no way to backfill that experiment info because the filtering happens client side, and we can't join from other data sources because this dataset uses a anonymized CID for legal purposes)

Sounds like this case warrants special treatment. We killed recipe 648 and launched 649 in its place with specific filter expression clauses to exclude anyone targeted by the prior 2 recipes. The data for this one will land under this slug: pref-flip-activity-stream-content-rows-2-1504077

awesome, thank you josh

update: this study has run it's course and we can end this study now (end enrollment, end participation)

Thanks Su - Recipe 648 has been disabled. Enrollment and participation have concluded.