This study has exactly the same parameters and expectations of the beta 63 study (#1491379) it is derived from, except the sizing should be something more reasonable for the release channel.

:matt_g how should we size this study (does 1% of en-US seem reasonable/powerful enough?) The slug id for this study should be *prefixed with*: prefflip-activity-stream-cfr-release-63

That seems reasonable, but bmiroglio owns the final population size. I'll let him comment.

:ilana r? for science :k88hudson r? for peer :mcoman r? for QA :jcristau r? for relman :osunick r? for VP

Science: R+

1% is more than powerful enough--with the following note: To calculate retention we'll need ~40k users per branch *minimum*. A 1% sample will likely hit that target in a day or two, so we should ensure to stop continually enrolling users into the study after 3-4 days, to give us some padding as we'll need to filter out some users before analysis (as is standard with all experiments).

Peer: R+

[Tracking Requested - why for this release]:

redirect to Pascal as he's looking after 63

Approved.

This has relman approval.

Activity Stream Contextual Feature Recommender - 63 Release Channel Targeted: Firefox Release 63 We have finished testing the Activity Stream Contextual Feature Recommender experiment. QA’s recommendation: RED - DON'T SHIP Reasoning: - We have found a blocker (Bug 1501656) that affects the “Experiment 2” branch. The recommendation for the “Facebook” and “Reddit” websites will be displayed only two times per add-on instead of three. Testing Summary: - Full Functional test suite: https://goo.gl/G3dUFj; Tested Platforms: - Windows 10 x64 - Mac 10.13.3 - Arch Linux 4.16.6 x64 Tested Firefox versions: - Firefox 63

Thanks, Marius. We have identified the bug and have a patch in the works, which we intend to ask for uplift into Beta 64. :mcoman, because the study is split into two experiment branches (one with freq_cap=1 and one with freq_cap=3), I propose we run the study with a single experiment branch where freq_cap=1. Because the undesired behaviour happens for freq_cap values > 1, could we get an r+ for an experiment with the single branch?

Activity Stream Contextual Feature Recommender - 63 Release Channel Targeted: Firefox Release 63 We have finished testing the Activity Stream Contextual Feature Recommender experiment. QA’s recommendation: GREEN - SHIP IT Reasoning: - Considering the fact that the “Experiment 2” branch, which is affected by Bug 1501656, will not be part of this shield study, and the fact that the user enrollment will be performed through “Normandy” and it cannot be tested by us, the study is good to go from the QA point of view. Testing Summary: - Full Functional test suite: https://goo.gl/G3dUFj; Tested Platforms: - Windows 10 x64 - Mac 10.13.3 - Arch Linux 4.16.6 x64 Tested Firefox versions: - Firefox 63

We took a look at the telemetry, and the pings for Reddit and Facebook addons are being double counted for the recommendation. Let's put this study on hold for now.

We have landed a patch to fix and have requested uplift to beta 64 and ride-along to release 63 if possible (bug #1501656)

r? :mcoman. 63.0.1 Release includes the fix for Bug #1501656, and so we should now be able to launch the experiment as originally specified with both cohorts (freq caps 1 and 3). Could we get a quick re-test of the original?

Hi Tim, first of all, in order to test this study on a short period of time we would have to run our test suites only on two OS's, more precisely on Windows 10 x64 and Mac 10.13.3 and even in this case we'll need at least two days. That means we can test the CFR shield study, send the sign-off and launch on Tuesday morning (6th November 2018) your time. Also, if this is ok for you, could you please send a reply to the original PI e-mail in order to announce the change of plans.

(In reply to Pascal Chevrel:pascalc from comment #10) > This has relman approval. Hi Pascal, this time NI for review for running this study on release 63 in the coming week. I am told QA is in progress.

Provided QA give their green light after the re-test of the study for tomorrow, this is a+ for relman.

Hi Pascal, from Tim's email it seems that the launch date of the "CFR" shield study was postponed to November 12, in order to provide enough time for the QA to test the study.

Hey folks, I wanted to give QA enough time to properly re-test the feature, and a Nov 12 launch date still gives us lots of time to gather data from the study. If, on the other hand, the QA team is done sooner, then let's by all means launch the study sooner.

Activity Stream Contextual Feature Recommender - 63 Release Channel Targeted: Firefox Release 63 We have finished testing the Activity Stream Contextual Feature Recommender shield study experiment. During testing we have found the following new issues: Bug 1505450 - The "CFR" recommendation is wrongly displayed on "Facebook" and "YouTube" websites if a similar add-on is installed and the website is focused when the is browser restarted Bug 1506135 - The CFR recommandation is wrongly displayed on an untargeted tab if you quickly switch to it from a targeted one QA’s recommendation: YELLOW - SHIP CONDITIONALLY Reasoning: - We do not consider the two issues as blockers for the Study. However, we recommend fixing them as quickly as possible because they could affect a small percent of users. Testing Summary: - Full Functional test suite: https://goo.gl/G3dUFj; Tested Platforms: - Windows 10 x64; - Mac 10.13.3 - Arch Linux 4.16.6 x64 Tested Firefox versions: - Firefox release 63.0.1

Pascal, QA gave this a Yellow. How do you feel about shipping this study on Monday?

(In reply to Marnie Pasciuto-Wood [:marnie] from comment #23) > Pascal, QA gave this a Yellow. How do you feel about shipping this study on > Monday? Yes it is ok for me.

This study is now live with the following details: * 1% of en_US Release 63+ * Slug = pref-flip-activity-stream-cfr-release-63 (note the additional "-" to conform with the standard. This is different than what tspurway posted above) * User preference branch * Branches: [ { "ratio": 1, "slug": "Control", "value": "[]" }, { "ratio": 1, "slug": "Cohort1", "value": "[{\"id\":\"cfr\", \"cohort\": \"one_per_day\", \"frequency\": {\"custom\": [{\"period\": \"daily\", \"cap\": 1}]}, \"type\":\"local\",\"localProvider\":\"CFRMessageProvider\",\"enabled\":true}]" }, { "ratio": 1, "slug": "Cohort2", "value": "[{\"id\":\"cfr\", \"cohort\": \"three_per_day\", \"frequency\": {\"custom\": [{\"period\": \"daily\", \"cap\": 1}]}, \"type\":\"local\",\"localProvider\":\"CFRMessageProvider\",\"enabled\":true}]" } ]

We've closed this study. Thanks

Are you keeping this bug open to post results of the study later?