The background is bug 1437524. We have an a11y problem in the address bar because our TAB usage (coming from ancient Firefox) doesn't obey most common HIGs, tab should not cycle through results. We'd like to change the tab behavior to be consistent with the search bar, but we don't know how many users we may break. Dave, could you please suggest a kind of measurement that allows us to figure out the percentage of involved users?

In bug 1334615 we added probes for the method used to select a result from the dropdown. The case where a user uses the keyboard to move down the list to a result and presses <Enter> is counted as `enterSelection`. We talked about this informally as "using the arrow keys to move down the list" because that's the most obvious way to cycle through it. Do these counts include all methods of navigating the list using the keyboard: <Tab>, <Alt> + arrow, etc? I suggest adding another category to this histogram, eg. `tabEnterSelection`, which counts cases where the user pressed <Tab> at least once to move down the list on their way to making a selection. Is this feasible? If it's not much more work, we could also add `arrowEnterSelection` to count cases where the user cycled through results using the arrow keys exclusively before pressing <Enter> to select. We'd assume this corresponds to the vast majority of `enterSelection` cases, but it might not be bad to measure it directly. To answer the question in bug 1437524, I think it's sufficient to count search events where <Tab> was used, as opposed to, say, the number of <Tab> keypresses. I don't think we gain much from the additional granularity. If a user is accustomed to using the arrow keys, I expect they'd be unlikely to press <Tab> by accident while the dropdown is open, unless they are expecting a different behaviour (eg. jump to the one-off buttons, as in the other bars, or jump to a different UI element).

I'll see if I can figure this out.

(In reply to Dave Zeber [:dzeber] from comment #1) > I suggest adding another category to this histogram, eg. > `tabEnterSelection`, which counts cases where the user pressed <Tab> at > least once to move down the list on their way to making a selection. Is this > feasible? > > If it's not much more work, we could also add `arrowEnterSelection` to count > cases where the user cycled through results using the arrow keys exclusively > before pressing <Enter> to select. We'd assume this corresponds to the vast > majority of `enterSelection` cases, but it might not be bad to measure it > directly. Just to clarify, should these events be registered both in enterSelection AND tab/arrowEnterSelection, or should they only registered in one of the categories (thus enterSelection would end up only accounting for cases where the user didn't arrow nor tab before pressing enter)?

Since enterSelection is only set when something else than the heuristic result is picked, if we'd decide to split enterSelection (instead of double counting) I'd expect it to become zero. I guess this may break ongoing analysis or dashboards?

Good question. AFAIK, changing the definition of enterSelection won't break any existing analyses. I'm not really sure what the best approach is. If we go with the double-counting, we preserve the original definition of enterSelection, but summing the values would no longer give the total number of result selections (although this is still available from other histograms, eg. FX_URLBAR_SELECTED_RESULT_TYPE). However, if we split enterSelection, that might make more sense for future consumers of the histogram, and would avoid relying on people seeing the documentation on the double-counting behaviour. I guess there's always the backup option of a new histogram.

I think I will split EnterSelection then, if the change is not going to break ongoing analysis. I will still retain the "EnterSelection" category, even if ideally it should be zero, I suspect code/add-ons may still set a selection without user interaction, and it will be a good checksum. So in the end we should have EnterSelection, TabEnterSelection (tab used at least once), ArrowEnterSelection (only arrows/pageup/pagedown used to select)

Dave, I suppose we don't need a data review if we just add categories to an already approved histogram?

A few questions/comments (and sorry if they've already been discussed): (1) I think we might want to capture the case where both tab and arrows are used before pressing enter, like "tabAndArrowEnterSelection". It doesn't seem right to me that if the behavior is tab, it gets stuck there for that selection. And I think it might be helpful to break out that case anyway. (2) What if the user tabs/arrows to a result but then clicks a result? Should we capture the initial tab/arrow? I would guess that doesn't happen much, but who knows. (3) Are we interested in how many results the user tabs/arrows over? I wonder if people are more likely to tab/arrow down to the first result much more than others, for example. That seems interesting, but maybe out of scope for this bug? What do you think?

I'll post my opinion regarding those questions: (In reply to Drew Willcoxon :adw from comment #10) > (1) I think we might want to capture the case where both tab and arrows are > used before pressing enter, like "tabAndArrowEnterSelection". As Dave said, a user not used to a certain behavior is unlikely to do it by mistake. I'm not sure the mixed case is really interesting, also considered the question we are trying to answer "How many users cycle with tab?". Even a single tab is a sign of some habit. The only interesting false positive would be the user who pressed tab thinking to move to the next widget, but it's likely a minority. > (2) What if the user tabs/arrows to a result but then clicks a result? > Should we capture the initial tab/arrow? I don't think it's an interesting case to answer our current question. it was more likely in the past when mouseover was selecting to mix mouseover and keyboard, not it doesn't seem to make sense for any interaction. > (3) Are we interested in how many results the user tabs/arrows over? Again, we care about an habit, and one tab usage is enough to identify that.

Comment on attachment 8955536 [details] Bug 1437803 - Add telemetry to understand how many users use TAB to cycle through Address Bar results. https://reviewboard.mozilla.org/r/224688/#review231018 r+, although mainly looking at Histograms.json. I'll wait for adw to give final review on the JS code.

(In reply to Marco Bonardo [::mak] from comment #9) > Dave, I suppose we don't need a data review if we just add categories to an > already approved histogram? I don't think so, but I will confirm.

Comment on attachment 8955536 [details] Bug 1437803 - Add telemetry to understand how many users use TAB to cycle through Address Bar results. https://reviewboard.mozilla.org/r/224688/#review231136

There's a small problem with using the expando on event, since looks like we have some calls of handleCommand not passing any event. Thus event can be null. I'll just add an optional argument to _recordUrlOrSearchbarSelectedResultMethod and stop using the event expando for now. A couple product code cases passing a null event are "paste&go" and "drop". These likely just finish up accounted into the "Enter" case. https://searchfox.org/mozilla-central/search?q=%5C.handleCommand%5C%28%28null%7C%5C%29%29&regexp=true&path= Maybe in the future we'll care about counting these differently?

Pushed by mak77@bonardo.net: https://hg.mozilla.org/integration/autoland/rev/d06ed3c943eb Add telemetry to understand how many users use TAB to cycle through Address Bar results. r=adw,dzeber

Data review is recommended here, since the original histogram never went through the process. We are requesting data review from rrayborn for the FX_URLBAR_SELECTED_RESULT_METHOD histogram, initially landed in bug 1334615, and updated here. --- 1) What questions will you answer with this data? What are the different input mechanisms used to select a result in the URLbar dropdown (eg. mouse click, arrow keys, etc.) and how often are they used? 2) Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? In general, we want to optimize the UX for interacting with the Awesomebar, as this supports both our users' needs and our product goals. One aspect of this is understanding how users are physcially selecting results. In this bug, we are specifically interested in how many users use <Tab> to navigate between results, as we are thinking about changing that functionality. 3) What alternative methods did you consider to answer these questions? Why were they not sufficient? The best way to do this is instrumenting UX interactions directly. 4) Can current instrumentation answer these questions? Prior to landing the FX_URLBAR_SELECTED_RESULT_METHOD histogram in bug 1334615, no. 5) List all proposed measurements and indicate the category of data collection for each measurement. Measurements are made using the FX_URLBAR_SELECTED_RESULT_METHOD histogram. Details are given in Histograms.json (https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Histograms.json#l6737), bug 1334615, and in this bug. This is all Category 2 data. 6) How long will this data be collected? Choose one of the following: The search team plans to monitor this data indefinitely (:javaun/:dzeber). 7) What populations will you measure? Full population (opt-out/default-on histogram). 8) If this data collection is default on, what is the opt-out mechanism for users? Standard Telemetry opt-out. 9) Please provide a general description of how you will analyze this data. Related to this bug, we will compute the proportion of users who use <Tab> to cycle through dropdown results. In general, we will compute the distributions of input mechanism preference, and possibly use this field as a filter for other search analyses (eg. restricting to users who never use the arrow keys, etc.) 10) Where do you intend to share the results of your analysis? Internally to Mozilla.

Better to track this, since it's missing a data review, if it doesn't arrive in time we'll have to backout.

(In reply to Dave Zeber [:dzeber] from comment #20) r+ (Generally data review requests are made as attachments with request for general review. I think our documentation needs to be clear on that, will fix) 1) Is there or will there be **documentation** that describes the schema for the ultimate data set available publicly, complete and accurate? Yes, Telemetry 2) Is there a control mechanism that allows the user to turn the data collection on and off? (Note, for data collection not needed for security purposes, Mozilla provides such a control mechanism) Provide details as to the control mechanism available. Yes, Telemetry opt-out 3) If the request is for permanent data collection, is there someone who will monitor the data over time?** Expires in 63, dzeber is monitoring. 4) Using the category system of data types (https://wiki.mozilla.org/Firefox/Data_Collection) on the Mozilla wiki, what collection type of data do the requested measurements fall under? Category 2 5) Is the data collection request for default-on or default-off? On with Telemetry 6) Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)? No 7) Is the data collection covered by the existing Firefox privacy notice? Yes, Telemetry policies 8) Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No) (If yes, set a todo reminder or file a bug if appropriate)** No, this is informing near-term design decisions

Thank you very much, resolving again since we have all the requirement in place.