Closed Bug 1282089 Opened 8 years ago Closed 4 years ago

Touch simulation doesn't support mobile gestures

Categories

(DevTools :: Responsive Design Mode, enhancement, P3)

enhancement

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bigben, Assigned: mtigley)

References

(Blocks 2 open bugs)

Details

(Whiteboard: [rdm-reserve])

Attachments

(1 file, 2 obsolete files)

There are a lot of gestures commonly used on mobile devices to favor more direct forms of interactions. They've become so intuitive that you probably use them without thinking about it: - Double tap to zoom - Drag/flick to scroll - Press and hold to display options (a bit like right click) Of course, some of them are hard to reproduce with a mouse. I don't really have a scenario with expected/actual results, but I believe we should support this in our RDM touch simulation mode. Chrome does, and it would really enhance the "mobile feeling" of the tool.
I've looked up the current code for touch simulation, and almost everything is done in the simulator-content script [1]. This JS file is loaded in the viewport by the RDM code using the message manager [2]. The script basically captures every mouse event, and translates them into touch events when the simulator is enabled. We could detect specific events like double click = double tap = force the viewport to zoom, but maybe this is already done somewhere in the codebase, for mobile devices, and we could just trick the platform into thinking this viewport is really on a mobile device. Perhaps we should use other functions like InjectTouchEvent to force that behaviour? [3] I'm not sure what the right approach here. [1]: https://dxr.mozilla.org/mozilla-central/source/devtools/shared/touch/simulator-content.js [2]: https://dxr.mozilla.org/mozilla-central/source/devtools/shared/touch/simulator.js#28
Oops, forgot the link related to my previous comment. [3]: https://dxr.mozilla.org/mozilla-central/source/dom/ipc/TabParent.cpp#2826
Whiteboard: [multiviewport][triage]
Priority: -- → P3
Whiteboard: [multiviewport][triage] → [multiviewport] [reserve-rdm]
Depends on: 1288187
Attached patch Part 1: Fix platform InjectTouchEvent function (obsolete) (deleted) — Splinter Review
Okay, here's my latest progress on bringing real touch simulation in RDM. As mentioned in previous comments, the goal here is to use the platform just like on a mobile device, instead of trying to imitate the mobile behaviour with a frame script. There are 3 parts in this patch: - Part 1: Vivien Nicolas fixed injectTouchEvent for me, it's an old API used for B2G and provides an easy way to test "real" touch events. However, it's deprecated and we should use SendNativeTouchPoint as soon as it's available on all platforms (see bug 1288187, I'm discussing this with kats). For obscure thread reasons, injectTouchEvent tends to crash a lot, so I had to add a 500ms delay between the mouse events and the resulting touch actions (hopefully this will be removed when we use SendNativeTouchPoint). - Part 2 adds an attribute in the docshell to manage the meta-viewport preference tab by tab, just like dom.w3c_touch_events.enabled in Bug 970346. This was needed because setting the preference for the whole browser breaks most of the UI. - Part 3 mostly mods the touch simulator in the RDM to use this new method. There is an invisible div above the iframe that I called the "touch proxy", and it captures every mouse event to call injectTouchEvent when needed. I set/remove the meta-viewport docshell attribute, and a few other preferences. A lot of useless code in the simulator script is also removed in this part. This is still WIP, but it already does some cool things: you can double tap to zoom, drag to scroll, long press to open contextual menu, etc. The touch proxy thing also resolves Bug 1282084 (no more hover events), and the meta-viewport preference fixes a lot of websites that weren't rendered properly in RDM before. We can also add fluffing easily with this approach (accept clicks even when there aren't exactly on the target), using ui.touch.radius preferences, and this would justify Bug 1271728. All suggestions/advices are welcome!
Assignee: nobody → bchabod
Status: NEW → ASSIGNED
Depends on: 1289432
Flags: qe-verify?
Since Benoit's internship has ended, I am going to assume he may not be working on this anymore. If that's incorrect, feel free to pick it up again!
Assignee: be.chabod → nobody
Status: ASSIGNED → NEW
Flags: qe-verify? → qe-verify+
QA Contact: mihai.boldan
According to comment 6 it seems Benoit was pretty close to finish the patch. Based on the duplicates, people on CC and questions in forums and other pages, I assume this is a feature many people are missing. Would someone of the team be willing to finish this up? Sebastian
giving that more tablets devices are being used, this feature is getting more desirable.
Flags: needinfo?(jryans)
Please avoid using needinfo (and Bugzilla generally) just to state an opinion on a feature's importance. This is one feature of thousands that the DevTools team prioritizes. We regularly revisit our feature backlog to decide what to work on next. We do still want feedback though, and Discourse[1] is a good place for that, along with Twitter, IRC, Slack, etc. [1]: https://discourse.mozilla.org/c/devtools
Flags: needinfo?(jryans)
Product: Firefox → DevTools

Mobile touch simulation will be an important feature for RDM this year. So I'll be picking this up to begin investigation on what needs to be done to get Benoit's work landed.

Assignee: nobody → mtigley
Status: NEW → ASSIGNED
Comment on attachment 8773304 [details] [diff] [review] Part 1: Fix platform InjectTouchEvent function This is in the tree in nsDOMWindowUtils::SendTouchEventCommon. First introduced in Bug 603008, pre-dating this patch. That seems to indicate that the existing pathway is adequate.
Attachment #8773304 - Attachment is obsolete: true
Comment on attachment 8773305 [details] [diff] [review] Part 2: Allow per-document control of the meta-viewport preference Meta viewport is enabled if and only if touch simulation is enabled, since Bug 1290420 landed. This patch is no longer needed to force meta viewport, since any touch gestures will, by definition, have touch simulation enabled.
Attachment #8773305 - Attachment is obsolete: true

In order to resolve this, this bug will depend on already existing smaller bugs and we can solve it from those. For example, Part 3 introduces a possible fix to hover states, which we can fix from Bug 1282084.

Depends on: 1282084
Assignee: mtigley → nobody
Status: ASSIGNED → NEW
Flags: qe-verify+
Priority: P3 → P2
QA Contact: mihai.boldan
Whiteboard: [multiviewport] [reserve-rdm] → [rdm-mvp]
Assignee: nobody → mtigley
Status: NEW → ASSIGNED
Priority: P2 → P1

Currently, the touch-simulator is using out of date methods to create and dispatch touch events. Now that SendNativeTouchPoint is available on all platforms, we can now use the platform API to simulate scrolling/swiping as they would appear on a mobile device. To see where these touch points are being created in the RDM viewport, I've been using https://patrickhlauke.github.io/touch/tracker/multi-touch-tracker-pointer-hud.html.

Using SendNativeTouchPoint, I've been observing a noticeable x and y offset of the created touch point from where the mouse has been clicked. So I'll be continuing more investigation on why this is.

Assignee: mtigley → nobody
Status: ASSIGNED → NEW
Priority: P1 → P2
Priority: P2 → P3
Whiteboard: [rdm-mvp] → [rdm-reserve]
Type: defect → enhancement

If simulated touch events had event.pointerType === 'touch' instead of "mouse", this would allow developing and testing touch scroll/gestures libraries using Firefox. Obviously this will be the case when this problem is resolved... or is it worth opening a dedicated issue?

Depends on: 1489653, 1621206
Depends on: 1621781
No longer depends on: 1621206

Now that support for mobile scrolling gestures in RDM has landed, RDM now has basic support for mobile gestures. The following gestures are supported (as of Firefox Nightly 79):

• Double tap to zoom (Bug 1621781)
• Swipe/scroll (Bug 1621781)
• Long press to open the contextual menu
• Pinch to zoom (on trackpads) thanks to work in the Bug 1620055 meta.

I believe we can close this issue now. If there are additional issues relating to simulating mobile touch gestures in RDM, we can continue to file them separately against the Bug 1401304 (rdm-touch) meta.

Assignee: nobody → mtigley
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: