Telemetry and branch generation for trailhead first run
Categories
(Firefox :: Messaging System, enhancement, P1)
Tracking
()
People
(Reporter: k88hudson, Assigned: k88hudson)
References
Details
(Keywords: github-merged)
Updated•6 years ago
|
Assignee | ||
Comment 1•6 years ago
|
||
See https://docs.google.com/document/d/1t3dDeaAKwLpH39a2LZ7xYzX4n17w-IGiTUpk0xU9LdY/edit#heading=h.eeup78jxxkio for notes on a possible strategy for this.
Assignee | ||
Updated•6 years ago
|
Assignee | ||
Updated•6 years ago
|
Assignee | ||
Updated•6 years ago
|
Assignee | ||
Comment 2•6 years ago
|
||
Is it ok for us to include all users for 67.5 in a branch for this experiment, or do we need to exclude some users?
Comment 3•6 years ago
|
||
This is in the context of activity stream needing to do its own experiment user selection at startup where we do have a "control" group and if a user wouldn't be in the experiment would have the same experience as the "control" group anyway. So if the user would experience the same thing anyway, would it be okay to just part of a much larger "control" group?
Comment 4•6 years ago
|
||
I checked with Sunah Suh, and she said that as long as the type
parameter used with calling setExperimentActive
is unique, there won't be any problems with tagging every user, as if they were in a really big control group.
Comment 5•6 years ago
|
||
https://github.com/mozilla/activity-stream/pull/4980
We'll have a primary experiment (interrupts) and secondary (triplets) with most people getting a default experience (no experiment data).
Default experience would show the new "Join" modal followed by 1 of the 4 triplets:
- triplet "supercharge" = mobile firefox, sync, firefox monitor
- triplet "payoff" = firefox monitor, facebook container, firefox send
- triplet "multidevice" = mobile firefox, pocket, send tabs
- triplet "privacy" = private browsing, tracking protection, firefox lockwise
The primary experiment has 5 possible "interrupts":
- control / existing blue fxa messaging
- join / new purple "Join Firefox"
- sync / new purple but with existing fxa messaging
- nofirstrun / nothing - looks like about:newtab
- cards / no modal straight to triplet
The secondary experiment has 4 possible "triplets" as described above and shown after "join"
We will land with a default trailhead.firstrun.branches
pref value so nightly users will always see join-privacy. For testing, this value can be changed requiring restarting firefox and visiting about:welcome. For uplift to 67, the pref value will be set to "control" so everyone gets the existing experience there. Then for later uplift don't set the pref and experimentation will pick the appropriate branches.
For debugging, after visiting about:welcome and getting an experiment picked, one can go to about:newtab#devtools and confirm the picked branches.
Comment 8•6 years ago
|
||
The code is landing in nightly with equal probability of a user picked for 1) no experiment, 2) interrupts experiment or 3) triplet experiment. This picking happens when trailhead.firstrun.branches
is set to blank/nothing/deleted. While we have the equal probability sampling, for testing, one can set app.normandy.user_id
to these values to end up with a desired experiment variant (instead of creating a new profile and randomly getting some variant):
0 {"experiment":"","interrupt":"join","triplet":"multidevice"}
20 {"experiment":"","interrupt":"join","triplet":"payoff"}
2 {"experiment":"","interrupt":"join","triplet":"privacy"}
10 {"experiment":"","interrupt":"join","triplet":"supercharge"}
12 {"experiment":"interrupts","interrupt":"cards","triplet":"supercharge"}
11 {"experiment":"interrupts","interrupt":"control"}
7 {"experiment":"interrupts","interrupt":"join","triplet":"supercharge"}
1 {"experiment":"interrupts","interrupt":"nofirstrun"}
5 {"experiment":"interrupts","interrupt":"sync","triplet":"supercharge"}
15 {"experiment":"triplets","interrupt":"join","triplet":"multidevice"}
3 {"experiment":"triplets","interrupt":"join","triplet":"payoff"}
46 {"experiment":"triplets","interrupt":"join","triplet":"privacy"}
9 {"experiment":"triplets","interrupt":"join","triplet":"supercharge"}
(These user_id values will be different when we have different ratios in release)
For example, set app.normandy.user_id
pref to "0" (string pref) and trailhead.firstrun.branches
to empty string and restarting then going to about:welcome should show "join" interrupt with "multidevice" triplet.
Similarly, setting to "12" should show no modal and "supercharge" triplets. Also, going to about:telemetry -> Environment Data -> Experiments should show:
activity-stream-firstrun-trailhead-interrupts.branch cards
activity-stream-firstrun-trailhead-interrupts.type as-firstrun
For just testing the various combinations of interrupts and triplets, one can set trailhead.firstrun.branches
to <interrupt>-<triplet>, e.g., "cards-multidevice" or "control" (as there's no new triplet). This also requires a restart after setting the pref.
Comment 9•6 years ago
|
||
Also, when doing the branches = blank/nothing/deleted and user_id (or just randomly generated) to verify if about:telemetry shows the active experiment…
This should persist after restart and show the same active experiment.
And this should only happen for en/de/fr locales while other locales should always see the existing/control. (Again emphasizing a clean profile will by default show about:welcome specified in trailhead.firstrun.branches
for any locale so make sure to set that pref to blank/nothing/deleted to allow experimentation testing.)
Comment 10•6 years ago
|
||
Comment 11•6 years ago
|
||
I have verified this issue with the latest Firefox Nightly (68.0a1 Build ID - 20190507214514) installed, on Windows 10 x64, Arch Linux and Mac 10.14.4. I can confirm that all the elements form the "First Run" page are displayed according to the provided documentation and the telemetry pings are successfully sent.
Comment 12•6 years ago
|
||
Marking verified as per bug 1550098 Comment 13 for bugs that were status-firefox68=verified
Updated•5 years ago
|
Description
•