User story now updated to reflect the fact we target Release 52 to run the tests.

funnelcake 105: control Funnelcake 106: variation Start date: 3/7 End date: 3/25

We're having to delay this by 2 releases because of a flash bug (bug 1336517). A work-around is being implemented with FF54 (bug 1340276) so FF54 is the sooner we can run the funnelcake. User story field now updated.

Eric R + :jpetto: here's another traffic cop experiment. We need to make sure this experiment doesn't overlap the targeting conditions of the search retention test in bug 1363499.

Will the Traffic Cop use the same style of download links as we've previously used with Optimizely ? For this funnelcake that would be https://download.mozilla.org/?product=firefox-stub-f105&os=win&lang=en-US https://download.mozilla.org/?product=firefox-stub-f106&os=win64&lang=en-US ie add a -fNNN suffix on the product, and serve a full installer even though it's named like a stub. This does have the nice feature that www.mozilla.org doesn't need any changes if a point release comes along.

(In reply to Nick Thomas [:nthomas] from comment #5) > Will the Traffic Cop use the same style of download links as we've > previously used with Optimizely ? For this funnelcake that would be The download links for funnelcakes are controlled by bedrock, not Optimizely or Traffic Cop, so yes, the links will follow the pattern of past experiments. As we now know this experiment cannot target en-US users (based on the concurrent experiment in bug 1363499), we need to know which locale/locales to target for this experiment. We currently do not have the ability in bedrock to target all locales except those specified. May be easiest to just target the next most active locale/locales? We should also decide on what percentage of visitors will get a funnelcake. We can do 50% per if we want to get numbers quickly, or any other whole number per if we want to draw out the experiment a bit.

(In reply to Jon Petto [:jpetto] from comment #6) > As we now know this experiment cannot target en-US users (based on the > concurrent experiment in bug 1363499), we need to know which locale/locales > to target for this experiment. We currently do not have the ability in > bedrock to target all locales except those specified. May be easiest to just > target the next most active locale/locales? OK. I recommend we test Germany and Canada because they are our other tier-1 desktop user markets. > We should also decide on what percentage of visitors will get a funnelcake. > We can do 50% per if we want to get numbers quickly, or any other whole > number per if we want to draw out the experiment a bit. We serve about 1.2M installs/day to Win64 OS users in Germany + Canada. We want 120K users in funnelcake (60K control + 60K experiment). Serving the funnelcake to 2% of Win64 OS installs in Germany and Canada should spread our 120K installs over a 5-7 day period. In summary: I think we should serve the funnelcake to 2% of Win64 OS installs in Germany and Canada until we hit 120K funnelcake installs.

(In reply to Chris Peterson [:cpeterson] from comment #7) > (In reply to Jon Petto [:jpetto] from comment #6) > > As we now know this experiment cannot target en-US users (based on the > > concurrent experiment in bug 1363499), we need to know which locale/locales > > to target for this experiment. We currently do not have the ability in > > bedrock to target all locales except those specified. May be easiest to just > > target the next most active locale/locales? > > OK. I recommend we test Germany and Canada because they are our other tier-1 > desktop user markets. Sounds good > > > We should also decide on what percentage of visitors will get a funnelcake. > > We can do 50% per if we want to get numbers quickly, or any other whole > > number per if we want to draw out the experiment a bit. > > We serve about 1.2M installs/day to Win64 OS users in Germany + Canada. We > want 120K users in funnelcake (60K control + 60K experiment). Serving the > funnelcake to 2% of Win64 OS installs in Germany and Canada should spread > our 120K installs over a 5-7 day period. > > In summary: I think we should serve the funnelcake to 2% of Win64 OS > installs in Germany and Canada until we hit 120K funnelcake installs. It looks more like 30k daily per https://sql.telemetry.mozilla.org/queries/4853/source#9911 Chis, can you point me to where you got the 1.2M from? Also just to confirm - Are we talking about the 'de' and 'ca' locales? I see very little 'ca' locale installs so either I get it wrong or canadian users get en-us.

Romain's guess is most likely accurate. Either through explicit choice or following presets, many English speakers get the en-US build of Firefox - from the UK to India. (I'm passing on this info third-hand - I do not have stats.) We can certainly target de and ca as the locales, but ca may not give as much traffic as expected. For reference, here are the locales we can target: https://github.com/mozilla/bedrock/blob/master/bedrock/settings/base.py#L131

Per https://sql.telemetry.mozilla.org/queries/3663/source#7245 it looks like the addition of pt-BR, es-ES, fr, de and ru get us 110k installs daily. Running 22% across all for 5 days would get us 121k downloads - does that sound OK?

(In reply to Romain Testard [:RT] from comment #8) > It looks more like 30k daily per > https://sql.telemetry.mozilla.org/queries/4853/source#9911 > Chis, can you point me to where you got the 1.2M from? Sorry. I was looking at monthly installs and misread the data as daily! <:) (In reply to Romain Testard [:RT] from comment #10) > Per https://sql.telemetry.mozilla.org/queries/3663/source#7245 it looks like > the addition of pt-BR, es-ES, fr, de and ru get us 110k installs daily. > Running 22% across all for 5 days would get us 121k downloads - does that > sound OK? Sounds good: 22% of pt-BR, es-ES, fr, de and ru until we get 120K downloads

Just to confirm, we are talking about the locale derived from geoip, aka a reasonable approximation for actual location of the user ? And we know that those users are already using an en-US browser, say from the user-agent, so we serve them an en-US funnelcake ? Or to put it another way, do we need to generate funnelcake builds with localizations other than en-US ?

*I'm* talking about the locale in the URL, not geolocation. The locale in the URL has no solid tie to a user's actual location. Geolocation is something we can do, but I'm not sure how many visitors we're going to get with de as their top accept-language that are located within the US *and* want to download an en-US build. (If a user hits mozilla.org/de/firefox/new, they *probably* want a de build of the product.) It sounds like we probably do need funnelcake builds for each locale we're going to target. We should have :cpeterson confirm.

(In reply to Jon Petto [:jpetto] from comment #13) > *I'm* talking about the locale in the URL, not geolocation. The locale in > the URL has no solid tie to a user's actual location. That's OK. > It sounds like we probably do need funnelcake builds for each locale we're > going to target. We should have :cpeterson confirm. Yes. I forgot that, since we're no longer testing en-US, we will need separate funnelcake builds for each locale.

(In reply to Chris Peterson [:cpeterson] from comment #11) > Sounds good: 22% of pt-BR, es-ES, fr, de and ru until we get 120K downloads I'll generate builds for that over in bug 1309844. It might be worth checking with cmore if a full 7 days is preferable to some proportion of a week. That's the impression I got from previous studies.

Chris, can you please confirm if you prefer 7 full days? Also do you or Jon know if we have a staging system for www.mozilla.org + TrafficCop to simulate handing them out. We're consider testing that the distribution works as expected before shipping but I'm unsure if this is typically done on funnelcake or if we just typically trust the system.

We can put the experiment on a demo server prior to launch to verify all is working as intended.

That would be great, we're planning launch tomorrow. We although need to make sure Grover could do the testing - Grover can you please confirm to validate with Jon it's worth setting-up the demo server?

(In reply to Romain Testard [:RT] from comment #16) > Chris, can you please confirm if you prefer 7 full days? > Also do you or Jon know if we have a staging system for www.mozilla.org + > TrafficCop to simulate handing them out. We're consider testing that the > distribution works as expected before shipping but I'm unsure if this is > typically done on funnelcake or if we just typically trust the system. We can (and should) launch mid-week, but I would prefer to have two full weeks from Sunday through Saturday. Experiment start: 6/20 Measurement start: 6/25 (Sunday) Measurement end: 7/8 (Saturday) That would give two full weeks for measurement and if we run into some problems or delays with the launch, we have until 6/25 to get them resolved.

Can someone confirm that we are targeting only Windows 64bit systems? Do we need an additional check around a Windows version (e.g. 7/8/8.1/10), or will detecting 64bit be sufficient? Are we targeting only non-Firefox visitors, or should we target users running any browser?

Hm, there may not be a 100% reliable way to detect 64bit Windows in JavaScript (which we are limited to). Looks like checking navigator.userAgent for either "WOW64" or "Win64" is the best we can do, though that may not catch all, e.g. 64bit Chrome users. (However, I can't really test this as I don't have the hardware.) How critical (if at all) is the 64bit check?

(In reply to Jon Petto [:jpetto] from comment #20) > Can someone confirm that we are targeting only Windows 64bit systems? Yes. We are only targeting users running Win64 OS. To minimize experiment variables, we want to compare 32-bit and 64-bit Firefox on the same OS architecture. > Do we need an additional check around a Windows version (e.g. 7/8/8.1/10), > or will detecting 64bit be sufficient? Firefox only supports Windows 7 and later, so that Windows version check is adequate for this Funnelcake test. > Are we targeting only non-Firefox visitors, or should we target users > running any browser? Is there any reason to exclude Firefox visitors? (In reply to Jon Petto [:jpetto] from comment #21) > Hm, there may not be a 100% reliable way to detect 64bit Windows in > JavaScript (which we are limited to). Looks like checking > navigator.userAgent for either "WOW64" or "Win64" is the best we can do, > though that may not catch all, e.g. 64bit Chrome users. (However, I can't > really test this as I don't have the hardware.) Checking the User-Agent string should be adequate. If it includes "WOW64" or "Win64", then the user is running Win64 OS. On my 64-bit Windows 10 machine, Chrome's and Edge's User-Agent strings include "Win64" and IE11's includes "WOW64", so we should be covered. > How critical (if at all) is the 64bit check? It is critical. We don't want to serve the 64-bit installer to a user on 32-bit Windows OS because it wouldn't run!

Ok, if we only support Windows 7 and later, I don't think we need anything more than the 64bit check. I ran a quick test on my Vista VM and was prompted to download 52 ESR, so we should be good here. I did a test on my Win 10 VM, and it is reporting as 64bit. Chrome and Edge both report Win64, and IE 11 reports WOW64, so we should be good here as well. I only ask about excluding Firefox users as that's been a standard request on previous funnelcakes. I don't see a reason we need to exclude Firefox users here, but I am not the funnelcake experiment master. :)

The experiment is now up on a demo: https://bedrock-demo-jpetto.us-west.moz.works/firefox/new/ If you hit with en-US in the URL, you will receive a search retention funnelcake. If you hit with pt-BR, fr, es-ES, de, or ru in the URL, you will have a 22% chance of getting either ?f=105 or ?f=106. Note that the no-Firefox restriction is currently on the demo, so use either Edge, IE, or Chrome. You must be on Windows 64bit and have DNT disabled.

> Is there any reason to exclude Firefox visitors? We normally exclude Firefox downloading Firefox because for many of the experiments, we want to minimize the paveover installs as we are normally targeting a hypotheses that is better when they get the normal full new user experience. Does installing Firefox win64 over top of Firefox win32 matter to you or if win64 to win64 paveover happens? Since people do pageovers so often, we exclude Firefox browsers from the targeting conditions to minimize that impact. It still happens that someone downloads Firefox from Chrome when they have an existing profile, but it is only a small minority of users.

(In reply to Chris More [:cmore] from comment #25) > > Is there any reason to exclude Firefox visitors? > > We normally exclude Firefox downloading Firefox because for many of the > experiments, we want to minimize the paveover installs as we are normally > targeting a hypotheses that is better when they get the normal full new user > experience. In that case, excluding Firefox downloading Firefox sounds OK. If we are decreasing the size of our eligible user population by excluding paveover installs, we will need to watch our download numbers to make sure our throttled rollout is reaching enough users. Paveover installs might suggest that the user has some weird configuration problem we would like to avoid in our tests. :) > Does installing Firefox win64 over top of Firefox win32 matter to you or if > win64 to win64 paveover happens? Let's exclude Firefox downloading Firefox to avoid these corner cases.

(In reply to Jon Petto [:jpetto] from comment #24) > The experiment is now up on a demo: > > https://bedrock-demo-jpetto.us-west.moz.works/firefox/new/ > > If you hit with en-US in the URL, you will receive a search retention > funnelcake. If you hit with pt-BR, fr, es-ES, de, or ru in the URL, you will > have a 22% chance of getting either ?f=105 or ?f=106. > > Note that the no-Firefox restriction is currently on the demo, so use either > Edge, IE, or Chrome. > > You must be on Windows 64bit and have DNT disabled. Hi Jon, We tried the link provided in Comment 24, but it's leading to a 404 page not found error. Can you provide us any assistance on the matter?

(In reply to Grover Wimberly IV [:Grover-QA] from comment #27) > (In reply to Jon Petto [:jpetto] from comment #24) > > The experiment is now up on a demo: > > > > https://bedrock-demo-jpetto.us-west.moz.works/firefox/new/ > > > > If you hit with en-US in the URL, you will receive a search retention > > funnelcake. If you hit with pt-BR, fr, es-ES, de, or ru in the URL, you will > > have a 22% chance of getting either ?f=105 or ?f=106. > > > > Note that the no-Firefox restriction is currently on the demo, so use either > > Edge, IE, or Chrome. > > > > You must be on Windows 64bit and have DNT disabled. > > Hi Jon, > > We tried the link provided in Comment 24, but it's leading to a 404 page not > found error. Can you provide us any assistance on the matter? The URL from comment 24 is working for me. Which URL are you testing?

(In reply to Jon Petto [:jpetto] from comment #28) > > We tried the link provided in Comment 24, but it's leading to a 404 page not > > found error. Can you provide us any assistance on the matter? > > The URL from comment 24 is working for me. Which URL are you testing? I configured my Chrome to use es-ES and then opened: https://bedrock-demo-jpetto.us-west.moz.works/firefox/new/ which redirected to: https://bedrock-demo-jpetto.us-west.moz.works/es-ES/firefox/new/?f=106 I then clicked the "Descarga gratuita" download button, which opened: https://bedrock-demo-jpetto.us-west.moz.works/es-ES/firefox/new/?scene=2&f=106 which redirected to: https://download.mozilla.org/?product=firefox-stub-f106&os=win&lang=es-ES which returns "404 page not found".

> https://download.mozilla.org/?product=firefox-stub-f106&os=win&lang=es-ES Nick, the bouncer links for the 64-bit Funnelcake installers (f106) are returning 404. QA is now testing the Funnelcake installers from bug 1309844 comment 34, but they are blocked testing the bouncer logic.

106 should be using product=win64; 105 uses product=win (which means win32).

Testing funnelcake build on Windows 7 32 bits OS in VirtualBox, the first run URL contains "f-105" string. The build is downloaded from a direct link [1], not from the bouncer. Here is screen capture:https://testing-1.tinytake.com/sf/MTcxMjUyOF81NjgyMzI3 [1] https://download.mozilla.org/?product=firefox-stub-f105&os=win&lang=ru

(In reply to Abe - QA (:Abe_LV) from comment #32) > Testing funnelcake build on Windows 7 32 bits OS in VirtualBox, the first > run URL contains "f-105" string. > The build is downloaded from a direct link [1], not from the bouncer. This sounds like a bug. 32-bit Windows OS users should be excluded from this Funnelcake experiment, so they should not receive a funnelcake "f=105" or "f=106" URL.

Also, using a Chrome private browsing ("Incognito") window, I receive a funnelcake f=105 or f=105 URL after about 4-5 tries, as expected. (I had to close and open a new private browsing window each time). But I never receive a funnelcake f=105 or f=106 URL in IE or Edge. Checking the User-Agent string for "Win64" or "WOW64" should work in any browser, so I'm not sure why IE and Edge don't seem to work for me. Chrome User-Agent string: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.32 Safari/537.36 IE11 User-Agent string: Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko Edge User-Agent string: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063

(In reply to Chris Peterson [:cpeterson] from comment #34) > Also, using a Chrome private browsing ("Incognito") window, I receive a > funnelcake f=105 or f=105 URL after about 4-5 tries, as expected. (I had to > close and open a new private browsing window each time). But I never receive > a funnelcake f=105 or f=106 URL in IE or Edge. Jon told me on IRC that IE and Edge do not always clear cookies from private browsing windows, so I will need to manually clear my cookies every time before loading https://bedrock-demo-jpetto.us-west.moz.works/firefox/new/ to make sure I have a chance to be randomly selected for the Funnelcake.

You can also open dev tools and run `document.cookie`. If you see "experiment_win64_funnelcake=" anywhere in the resulting string, you were opted in to the experiment. The string would be "experiment_win64_funnelcake=novariation" if you were not selected for either funnelcake (which has a 78% chance of happening).

(In reply to Chris Peterson [:cpeterson] from comment #33) > This sounds like a bug. 32-bit Windows OS users should be excluded from this > Funnelcake experiment, so they should not receive a funnelcake "f=105" or > "f=106" URL. I watched the screencast and the download.mozilla.org link for a win32 bit build is used, rather than the bedrock server. (It also has a firstrun with ?f=105, rather than ?f-105). This would only be a concern (from a user selection point of view) if the download started at the bedrock server, or www.mozilla.org once we get to production.

(In reply to Nick Thomas [:nthomas] from comment #37) > (In reply to Chris Peterson [:cpeterson] from comment #33) > > This sounds like a bug. 32-bit Windows OS users should be excluded from this > > Funnelcake experiment, so they should not receive a funnelcake "f=105" or > > "f=106" URL. > > I watched the screencast and the download.mozilla.org link for a win32 bit > build is used, rather than the bedrock server. (It also has a firstrun with > ?f=105, rather than ?f-105). This would only be a concern (from a user > selection point of view) if the download started at the bedrock server, or > www.mozilla.org once we get to production. OK. I see now. The screencast shows the f=105 funnelcake installer is downloaded because a direct installer URL was entered instead of jpetto's staging server web page https://bedrock-demo-jpetto.us-west.moz.works/firefox/new/.

(In reply to Jon Petto [:jpetto] from comment #36) > You can also open dev tools and run `document.cookie`. If you see > "experiment_win64_funnelcake=" anywhere in the resulting string, you were > opted in to the experiment. > > The string would be "experiment_win64_funnelcake=novariation" if you were > not selected for either funnelcake (which has a 78% chance of happening). Good news! I was finally able to be selected for the funnelcake using IE and Edge. I had to use a non-private browsing window and manually clear my cookies. I never got it to work in IE's or Edge's private browsing windows. For some reason, document.cookie is always unset (document.cookie == "") on https://bedrock-demo-jpetto.us-west.moz.works/firefox/new/ in IE's and Edge's private browsing window, even though the same page sets a cookie in a non-private browsing window and other websites can set cookies in the private browsing window.

Hey everyone. /firstrun/ URL *should* be f=105 and not f-105. Not sure if that was a typo or not in the config. We've always had f=XXX as a firstrun parameter. As for the download.mozilla.org bouncer link for funnelcakes, they always append -fXXX to the product name.

where "f" is the parameter and the value being of XXX being the specific funnelcake ID of the cohort.

Commits pushed to master at https://github.com/mozilla/bedrock https://github.com/mozilla/bedrock/commit/5cee4f9d705d9b6dc8f7e8ec0d64f1e52b0d9d02 [fix bug 1309847] Win64 funnelcake experiment. https://github.com/mozilla/bedrock/commit/a43757faccd9765be863b7ef1bbe4674aa09d9be Merge pull request #4922 from jpetto/bug-1309847-win64-funnelcake-experiment [fix bug 1309847] Win64 funnelcake experiment.

Jon, we have enough users enlisted in our win64 funnelcake. You can stop distributing the 105 and 106 funnelcake installers whenever is convenient for you.

I hadn't realized we were still distributing 54.0-based funnelcakes, and we could have been providing 54.0.1-based builds since 29 June. Users will get an update to 54.0.1 so there's just a window of exposure to the bugs fixed in the point release.

The experiment has been shut off [1], and a PR filed to remove the associated bedrock code [2]. [1] https://github.com/mozmar/www-config/pull/22 [2] https://github.com/mozilla/bedrock/pull/4972

Commits pushed to master at https://github.com/mozilla/bedrock https://github.com/mozilla/bedrock/commit/0cbe476964deacb6a777c0757f2c03fe5f79f29c [fix bug 1309847] Remove Win64 funnelcake experiment. https://github.com/mozilla/bedrock/commit/3291d143aa59030911d7bfb9835bfada8f6da00b Merge pull request #4972 from jpetto/bug-1309847-stop-win64-funnelcake-experiment [fix bug 1309847] Remove Win64 funnelcake experiment.