Tapping on the screen to click on a link or a button requires much more effort to hit the target than when clicking with the mouse. We should be able to detect nearby links/clickable elements when a tap is missed and decide about correcting it or letting it go missed (depending on the distance for example)

Mobile is working on adding code that would increase the bounding box used for hit-testing elements. You would use CSS to specify the increase in the bounding box. Something like padding, where you can set the top/right/bottom/left parts.

Writing down some of the things talked on IRC and checking madhava's list of things we would want to do (https://wiki.mozilla.org/Mobile/Projects/SmartTap). So basically we have two alternatives: 1) using CSS to add an invisible bounding box and change the hittest to consider this new box: - consider the tap not exactly centered: yes by using different top/left/right/bottom values - different areas for different type of elements: yes - consider zoom level: yes (if it's exposed to css?) - :visited links: yes, but no advanced data based on frecency When there are overlaps, the chosen element would be defined by indirect things from the layout, such as the z-order of elements nearby 2) using elementsFromPointWithArea (from bug 489127) to get a list of possible targets: - consider the tap not exactly centered: yes, by offsetting the target area - different areas for different types of elements: no (? not directly) - consider zoom level: yes - visited and frencecy of links: yes The good thing about this approach is that we would get a list of possible elements, and then filtering/sorting can be performed in C++/JS as complexly as wanted

Also, I said on IRC that I thought that we should never correct a click if it would originally hit something. But I've put some more thought into it and this use case (from Gmail) proves it's not the case. A tap on the point marked should click, I believe, in the gtalk bar, not in the e-mail behind it. A good disambiguation for this case would be vingtetun's idea of considering the area size of the intersecting rect, which could be possible to do with the approach #2.

The more I think of it, the more I think approach #2 should do the trick. I can already think of other use case (e.g: a marquee selection where we pass a rect and get all elements under this rect that should be selected) where I would like to have such an API while with approach #1 I don't really think of anything else except click disambiguation in fennec.

WIP. Very rough first version. It works well for some cases and really bad for others. I have to work out some issues with overlapping vs ordering, and also perform some disambiguation based on frame type and distance.

Got some good progress on this patch. I've fixed some positioning problems, such that the code that deals with overlapping actually works now. Still, the best improvement so far came from using info about the depth level of elements. Still it's far from perfect, so there's more work ahead

Roc might have some suggestions what is the best way to do this. (And at least he is the one who should review the final patch.)

nodesFromPointWithArea is definitely the right way to go. > - different areas for different types of elements: no (? not directly) effectively you can, by postprocessing the list. This is kind of hacky. It would be nicer to keep this code out of Gecko and simply provide an extension point, possibly an XPCOM service that we look up. It would also be nice to have the interface just use DOM nodes so the code doesn't look at frames directly.

Thanks! My current plan was to use GetFramesForArea during CheckForAndDispatchClick (or somewhere similar) in presshell, and choose the best frame there and retarget the click. Also, just a note that here I'm interested in everything on the browser's chrome too, not only web content (one of my main use cases is to make the tab close button easier to hit). > This is kind of hacky. It would be nicer to keep this code out of Gecko and > simply provide an extension point, possibly an XPCOM service that we look up. > It would also be nice to have the interface just use DOM nodes so the code > doesn't look at frames directly. This is what was done in Fennec, since they already had all the necessary machinery on the front-end to redispatch events to content. For the logic I'm building at the moment, I'm trying hard to avoid anything that takes into account the frame type or content type. They would certainly help on disambiguation since there's no reliable concept of a "clickable" frame, but it looks to me that creating such special cases (e.g. "give priority to nsButtonBoxFrame or a frame with mContent == nsHTMLAnchorElement") is usually a bad thing to do, since it gives bad bias to cases not considered, among other things. Thoughts?

Special casing certain element types doesn't sound bad to me. For XUL you might even want to add an attribute to elements that gives them priority.

Just bringing in some data to illustrate the approach I'm leaning to, and to use it to plan and fine-tune the heuristics. The file shows various tab bar situations and the information retrieved from each X/Y. From the looks of it and some experimentation I've done this week, for XUL looks like that giving priority to certain frame types is the way to go, and also the frame depth information seems promising. At the end of the file there's my goal for the tab bar hit area. For HTML I'll post more data I have soon. The intersecting area and frame depth look like the winners. To avoid special cases for html (since anything can be made clickable and styled), I'll see if finding onclick event listeners helps. Other data that isn't described here but I've used to some effect on the patch posted is checking overlaps between elements. I don't know if that you will remain at the end or not, it might not be needed after all. At the moment I'm trying to nail down the heuristics. After I finish that I'll go about seeing the API and where this code should live, probably converting from frames to elements following roc's suggestion.

Data for <ul> <li><a>Link</a></li> <li><a>Link</a></li> </ul>

(forcing utf-8 on text file, let's see if attachment looks better now) Data for <ul> <li><a>Link</a></li> <li><a>Link</a></li> </ul>

Data for: <a>Link</a> / <a>Link</a>

Isn't this a dup of the bug roc just fixed. (Trying to find the bug#)