clarkbw, do we want telemetry of people clicking on the button as well as switching views from the doorhanger?

This would be great to have, but I don't want to block on it.

kghim, we can add probes to see how many people select which items from the menu, but that won't tell us how many people are set to Enhanced vs Classic vs Blank. Similarly we can measure the number of clicks on "Learn more" and "Privacy notice" but what are we trying to figure out from each of those measurements? I.e., what would we change based on the number of clicks of Learn more?

(In reply to Ed Lee :Mardak from comment #3) > kghim, we can add probes to see how many people select which items from the > menu, but that won't tell us how many people are set to Enhanced vs Classic > vs Blank. Similarly we can measure the number of clicks on "Learn more" and > "Privacy notice" but what are we trying to figure out from each of those > measurements? I.e., what would we change based on the number of clicks of > Learn more? I would like to know the difference in attitudes towards advertising (if present) on first run vs returning users. My assumption is the latter would be more interested in the learn more/privacy notice once they're used to how the new tab currently works (as speed dial).

For new users vs existing users, we might be able to look at existing probes perhaps relating to profile age. Although I don't think that type of query is supported on https://telemetry.mozilla.org/ so we'll have to ask someone (Metrics?) to analyze that. I'll poke around on #telemetry

mgrimes (cc'd) should be able to help out with something like a sentiment analysis

rvitillo provided some comments from #telemetry: I see, well, bucketing is a good start; you would want to ensure that you have the same number of users for each “profile age bucket”. Also, you probably should select users with similar characteristics, i.e. that use FF on a daily basis for instance. you could also take a certain set of users and for those users compare their behavior on their first runs vs their late behavior you have to write your own analysis job for that right now we have a map-reduce framework and an experimental setup with Spark which allows you to run SQL-like queries on telemetry datasets here is a pointer for the map-reduce framework: http://mreid-moz.github.io/blog/2013/11/06/current-state-of-telemetry-analysis/ and one for Spark: http://robertovitillo.com/2014/11/29/a-telemetry-api-for-spark/ in a week or so we are probably going to be able to let you run custom analysis jobs from a IPython notebook in your browser, so it’s going to be much easier to do this sort of stuff At least for the last item, perhaps mruttley can help out with creating python custom analysis.

Happy to help. Adding Ilana here as well. What questions exactly are you trying to answer? Knowing that will help us frame our work.

Chiming in, seconding some of Mardak's earlier questions. What do you plan on changing given the outcomes of these statistics? Pushing the "learn more" to a greater extent for existing users, for example? Unfortunately, I don't think telemetry has profile age (I haven't been able to find it, though I could be wrong), but you could get that from FHR. Adding probes to telemetry will only let you see actions on a session by session basis. I actually think a qualitative study might be more useful here, but learning more about what you're trying to do will help me answer this more effectively.

kghim: comment 8 comment 9

(In reply to Ed Lee :Mardak from comment #10) > kghim: comment 8 comment 9 Ilana, Matt_G, we're in the exploration/planning stages of new tab experiences relating to Tiles. From Bill Selman's user research feedback, the current cog wheel options are hard to understand and the language is confusing. First thing I'd like to figure out is whether the users are even aware of the cog wheel and if they are they actively switching between options. Secondly, I'd like to know the difference of the cog wheel usage between first run (inexperienced) users and experienced (history tiles) users. This will provide some direction on auto/user-opt-in programs and creating separate modes/views of new tab. Goal: obtain CTR for each option listed below: 1) Enhanced 2) Classic 3) Blank 4) What is this page? If #4 is selected, obtain CTR for the following links: 4a) Learn more 4b) Privacy notice Additionally, the stats above from 1st run experience users and history (frecency) tiles users are desired.

Ilana, Matt_G, did my above comment help in understanding how we can proceed? I'd be happy to do a vidyo on this.

Meeting scheduled.

The code that needs to be added is probably along the lines of https://hg.mozilla.org/mozilla-central/file/tip/browser/base/content/newtab/page.js#l216 Where we just remember when enhanced/classic are clicked with a simple telemetry probe.

Reviving this bug to track new cog wheel options design for the Suggested Tiles release. Insert telemetry probes to track: - Cog wheel click: on/off - Show your top sites: on/off (on by default) - Include suggested sites: on/off (on by default) - Show blank page: on/off (off by default) - Learn about New Tab: (off by default) Screen shot attached for reference.

Last we discussed, measuring the interaction rates on the menu items is not useful as it's a sample of the menu and doesn't capture what you're trying to get at of how many users should be seeing suggestions. The probe that gets us that data is similar to the one that reports back how many users have new tab enabled (i.e., showing blank or not). We should only need one probe to measure if suggestions are turned on or not. This will give us a percentage of users who have newtab enabled. And another percentage of users who can see suggestions.

(In reply to Ed Lee :Mardak from comment #17) > We should only > need one probe to measure if suggestions are turned on or not. This will > give us a percentage of users who have newtab enabled. And another > percentage of users who can see suggestions. Yes, since suggestions are on by default, the important stat to track here is negative reactions towards suggestions: how many users turn off suggestions and how many users turn on show blank page.

> Yes, since suggestions are on by default, the important stat to track here > is negative reactions towards suggestions: how many users turn off > suggestions and how many users turn on show blank page. Is this stored as a pref anywhere? If we want the stat that :Mardak mentioned - how many people aren't seeing suggestions - it's going to be a very hard (and sort of implausible, frankly) to have to traverse back in the history to see if they've ever hit the gear. What you'd want is to track what the preference is set to on that particular day to see what the breakdown is for a particular period. Now, if you're interested in how many people are actively tuning these preferences every day, that's another issue, and this probe captures that perfectly.

(In reply to Ilana from comment #19) > Is this stored as a pref anywhere? Indeed! Simple enough patch. > Now, if you're interested in how many people are actively tuning these > preferences every day, that's another issue, and this probe captures that > perfectly. Yup, we can calculate rates from the trend of on/off-ness of suggestions on a daily basis.

ni? per https://wiki.mozilla.org/Firefox/Data_Collection (In reply to Pulsebot from comment #21) > https://hg.mozilla.org/integration/fx-team/rev/455e500b9b17 This telemetry probe adds one just like the existing NEWTAB_PAGE_ENABLED except for enhanced/suggestions on/off.

f+ I'd really like to get a feedback request on these before they land.

(In reply to Benjamin Smedberg [:bsmedberg] from comment #24) > I'd really like to get a feedback request on these before they land. Got it. And will request f? instead of ni?. Thanks!

verified on Nightly - 41.0a1 (2015-05-26): Steps to verify: 1. Went to http://telemetry.mozilla.org 2. Searched for "newtab" and went to newtab_page_enhanced under nightly 41 3. Verified that data is being collected with line graph showing activity and distribution showing values for 0 and 1. 4. Verification link: http://telemetry.mozilla.org/#filter=nightly%2F41%2FNEWTAB_PAGE_ENHANCED%2Fsaved_session%2FFirefox&aggregates=Submissions&evoOver=Builds&locked=true&sanitize=true&renderhistogram=Graph

Nightly41: http://telemetry.mozilla.org/#filter=nightly%2F41%2FNEWTAB_PAGE_ENHANCED%2Fsaved_session%2FFirefox&aggregates=multiselect-all!Submissions&evoOver=Builds&locked=true&sanitize=true&renderhistogram=Graph Aurora40: http://telemetry.mozilla.org/#filter=aurora%2F40%2FNEWTAB_PAGE_ENHANCED%2Fsaved_session%2FFirefox&aggregates=multiselect-all!Submissions&evoOver=Builds&locked=true&sanitize=true&renderhistogram=Graph