Open Bug 1340930 Opened 8 years ago Updated 2 years ago

Add secure overlay API

Categories

(WebExtensions :: General, enhancement, P3)

Product:
WebExtensions
Component:
General
Type:
enhancement

Tracking

(Not tracked)

People

(Reporter: ntim, Unassigned)

References

Details

Attachments

(2 files)

First (probably incomplete) proposal for the API
(deleted), text/plain
Details
findbar_overlay.png
(deleted), image/png
Details
It'd be nice if developers could inject stuff over the page content without spamming the website DOM. The inspector and the findbar overlay does that using its CanvasAnonymousFrameHelper. How about exposing something similar to WebExtensions?
capability to just do a pop up above the page in content - not attached to button in bar. doesn't require editing the page to display. shane, kris, and luca sounded positive on this idea.
Whiteboard: [design-decision-needed]
There are pros and cons to this, i like the upside that UI could be overlayed without touching the page dom. Worth more discussion.
Useful for custom panels and toolbars that cannot be implemented properly if using content scripts (because special pages don't have those). I can imagine a download manager using this functionality.
webextensions: --- → ?
Hi Tim and Geoffrey, this has been added to the agenda for the May 9 WebExtensions API triage meeting. Would you be able to join us? Wiki: https://wiki.mozilla.org/Add-ons/Contribute/Triage Agenda: https://docs.google.com/document/d/1q7UD3DxsT5z0lO3EbOyF9Iln32Wg_e9LYMdHZ80BEb4/edit#
Flags: needinfo?(aswan)
We discussed this today and the consensus was that this seems like a good idea in principle but that it needs to be more precisely defined. Enumerating some specific use cases (with more detail than eg comment 3) would be helpful but even better would be writing a concrete proposal for what this API might look like. None of the Mozilla staff has time available to do that right now. Tim, would you be willing to take a pass at it? (Whoops, tried to needinfo? Tim and he's blocking requests. This is now a jump ball, if anybody is interested in working on a proposal, please comment here)
Flags: needinfo?(aswan)
Andrew Swan: I described 2 usecases in bug 1364404 (sorry, didn't find this bug while searching for similar issues, also I thought Bug 1344573 was the latest bug on this) >concrete proposal for what this API might look like You talking about an interface specification, so name classes, methods, parameters and such? I could give that a try. Are there any resources on guide-lines or rules to follow when doing this?
(In reply to Lusito from comment #6) > >concrete proposal for what this API might look like > You talking about an interface specification, so name classes, methods, > parameters and such? I could give that a try. Are there any resources on > guide-lines or rules to follow when doing this? That's the idea, yes. Unfortunately we don't much in the way of written documentation for this. Some basic principles are: - The api should be organized around a high-level operation that extensions want to perform rather than exposing some specific existing Firefox/gecko capability - The api should be fully asynchronous (though I don't think that's going to be a big factor here) - It should be clear to users who install extensions that use this API exactly what the extension is doing. This can be via something like permission prompts at install time, clearly distinguishing visual elements created by extensions, etc And of course we want something that can be implemented in a performant and maintainable way. I (and I suspect others too) can give you feedback and we can do this iteratively...
I've attached my first naive API specification. Of course, it's not complete yet, but it should be good to get some initial feedback I think. I tried to take the tabs and windows API as inspiration. Things to be specified/discussed: - How the javascript in the panels communicates with other parts. I think the existing APIs should be enough, not 100% sure tho. - the details for the CreateTypes - maybe a way to hide/show panels would be good? (instead of destroying/recreating them) - should panels be tab-specific or window-specific, or maybe allow both?
Oh, just noticed.. the positional attributes refer to screen pixels, those should obviously be document pixels (or window pixels?)
Shane, you probably have the best context for this, can you look at the proposal?
Flags: needinfo?(mixedpuppy)
Before even going deep into proposals and such, can we lay out some requirements first ? I had many proposals in mind, but I also had some main questions before getting any of those out: - Do we want to allow watching for webpage mouse events while the overlay is opened ? (eg. like the Developer Tools inspector overlay, or the findbar overlay) - Do we want to allow multiple overlays at once ? (eg. DevTools Rulers highlighter + Measuring tool, both shown at once) AFAIK, the proposal in comment 8 doesn't seem to cover the first issue. A good way to solve the first issue would be to use the CanvasFrameAnonymousContentHelper [0] API rather than using a frame. The main issue would be concealing overlays from other add-ons, since they would all belong to the same doc. Some possible ways to solve this: - Do platform work to allow multiple CanvasFrame - Separate each overlay in its own div: <CanvasFrame> <div id="_overlay_addon_1"> # add-ons do not see this ... # only add-on 1 sees this </div> <div id="_overlay_addon_2"> # add-ons do not see this ... # only add-on 2 sees this </div> </CanvasFrame> and somehow conceal all possible parentNode references. Or instead of using the CanvasFrame, we could use iframe, but then we would need to support some kind of special behaviour for pointer-events on iframes. [0]: https://dxr.mozilla.org/mozilla-central/source/devtools/docs/tools/highlighters.md#117
>Do we want to allow watching for webpage mouse events while the overlay is opened You can do that with content-scripts already now, can't you? you just need to send the info to the panel, but that is covered using sendMessage. > Do we want to allow multiple overlays at once ? When two different add-ons want to show an overlay, that should be possible (e.g. a Download panel add-on, an inspector add-on, etc.) So since that should be allowed, I don't see why one add-on should not be able to have multiple overlays.. The only problem I see here is if overlays overlap. Either we allow the user to get all overlay rectangles and arrange their overlays accordingly, or we only take the positional arguments as suggestions, pushing the overlays away from each other to ensure all are visible. Ok, nevermind.. the more I think about it, your idea of this seems completely different from my idea. You want to overlay stuff directly into the web-page (like element highlighting), whereas I want to overlay stuff on the window/tab (like toolbars and tooltips, see the screenshot on bug 1364404). So my panel would stay fixed to the window and not transparent, while yours would stay at the position of the website and might be transparent. Not sure if these 2 ideas can be combined at all, so maybe I'll keep my proposal in my ticket?
About your goal: Shane Caraveo (:mixedpuppy) said the following on bug 1344573: >Given the direction we're headed, the easy issue to point at with detached panels is lack of identifiable source by the user. A panel comes up, is it Firefox, the web page, or an addon? Not sure how that is possible with simple overlay tools like the devtools ruler and measuring tools, there is no way to add an icon or something to everything without making it really ugly. Two ideas to solve this: 1. Maybe it would be possible to toggle a mode similar to the responsive design mode, i.e. a frame would appear around the webpage (making the page smaller tho), with add-on icon and name, as well as an option to close the mode. In this case, I'd say at max one mode should be allowed at the same time, but of course one add-on could add multiple overlaid drawings onto its own mode. 1. Alternatively, instead of a frame, a toolbar would appear above the page (with add-on icon, name and close button), moving the page down a bit, which would allow for multiple toolbars to be opened. Once a toolbar is closed, the overlay would be closed too.
(In reply to Lusito from comment #12) > >Do we want to allow watching for webpage mouse events while the overlay is opened > You can do that with content-scripts already now, can't you? you just need > to send the info to the panel, but that is covered using sendMessage. When the overlay is opened, mouse events from the webpage are blocked by the overlay right ? Unless we can make the mouse events magically pass through somehow. > Ok, nevermind.. the more I think about it, your idea of this seems > completely different from my idea. You want to overlay stuff directly into > the web-page (like element highlighting), whereas I want to overlay stuff on > the window/tab (like toolbars and tooltips, see the screenshot on bug > 1364404). > So my panel would stay fixed to the window and not transparent, while yours > would stay at the position of the website and might be transparent. > Not sure if these 2 ideas can be combined at all, so maybe I'll keep my > proposal in my ticket? It's not *entirely* different, they both serve the same purpose, but I'm just trying to find a solution to the mouse events issue. Technically if you have an iframe overlaying over the page content, the page content can't be clicked, and pointer-events would not work between different frames.
>Unless we can make the mouse events magically pass through somehow. Imho that would be ugly depending on the use-case.. imagine a non-transparent element, do you still want the mouse to pass through. But as said, I think the concepts of these 2 ideas are completely different.. you want to do more than overlay a panel.. you want to draw lines, text, boxes, etc, but also allow the mouse to still pass through these drawings. As said, the mozilla team wants the user to see where the stuff comes from that he/she sees, which would be very difficult with your approach if you don't add something like I suggested in comment #13
Maybe create a panel (for your tools) using: https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/devtools.panels And add a new API to add new buttons to the top-right to toggle (and support) these kinds of overlays. It seems to me the best idea is to attach this kind of overlay to the devtools. Or do you have a use-case which is not meant for web-design?
(In reply to Lusito from comment #15) > >Unless we can make the mouse events magically pass through somehow. > Imho that would be ugly depending on the use-case.. imagine a > non-transparent element, do you still want the mouse to pass through. Ideally, you'd be able to control what's pass through and what's not. (In reply to Lusito from comment #16) > Maybe create a panel (for your tools) using: > https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/devtools.panels > And add a new API to add new buttons to the top-right to toggle (and > support) these kinds of overlays. > > It seems to me the best idea is to attach this kind of overlay to the > devtools. Or do you have a use-case which is not meant for web-design? This goes beyond devtools, look at the findbar overlay for example.
(In reply to Tim Nguyen :ntim (not accepting requests until 1st June) from comment #17) > (In reply to Lusito from comment #15) > > >Unless we can make the mouse events magically pass through somehow. > > Imho that would be ugly depending on the use-case.. imagine a > > non-transparent element, do you still want the mouse to pass through. > Ideally, you'd be able to control what's pass through and what's not. FYI, devtools allow this kind of control using pointer-events: none/auto on different elements.
>This goes beyond devtools, look at the findbar overlay for example. What is the findbar overlay? If you're talking about the ctr+f seachbar, then that's a different thing imho (it actually pushes the viewport up and reduces its size, so it's not really an overlay)
Andrew Swan, could you maybe explain what has been discussed in the meeting? Or rather what idea was in your minds when discussing it. Just so we are on the same page about the proposal.
Flags: needinfo?(aswan)
Attached image findbar_overlay.png (deleted) —
I think we're on the same page here. When you hit Cmd/Ctrl+F and search for something, an overlay pops up highlighting all matches+the current match.
Flags: needinfo?(core)
Ah, i see.. no idea how to tell the user which add-on this overlay belongs to in this case. Seems to me like this should be some special kind of highlighting API tho instead of a you can draw anything you like and make it clickable type of API.
Flags: needinfo?(core)
I updated bug 1364404 with the next iteration of my proposal for my overlayPanels API, adding a mockup image for what I have in mind. If you still think this should be one API, maybe you can explain how you think this could be done?
(In reply to Lusito from comment #24) > I updated bug 1364404 with the next iteration of my proposal for my > overlayPanels API, adding a mockup image for what I have in mind. > If you still think this should be one API, maybe you can explain how you > think this could be done? The overlayPanels API you drafted looks good. From my standpoint, your overlayPanels API solves both bug 1364404 and this bug.
Could you guys also port sdk/panel to webextension?
(In reply to Lusito from comment #20) > Andrew Swan, could you maybe explain what has been discussed in the meeting? > Or rather what idea was in your minds when discussing it. > Just so we are on the same page about the proposal. Just that this bug seemed very broad and that it would be easier to make a decision about moving forward with a more specific proposal. I don't have the necessary background to evaluate whether your proposal can be implemented in a safe/performant/maintainable/etc way, hopefully somebody like Shane can take a look. Then feedback to the user is another important area, you offer some good ideas in comment 13. If there is consensus around those items, I think this should go back to the community triage for approval but it would be a much more focused decision at that point and I would expect it to be approved if there aren't big concerns in the areas covered above.
Flags: needinfo?(aswan)
Just keep in mind, that the API proposal I attached is not for this ticket, since I thought this ticket was about something else. Currently there is no API proposal for this. I tried removing the file, but there seems to be no such option. Tim Nguyen, I'd suggest you should create an API proposal since you are most aware of what you actually need.
(In reply to Lusito from comment #29) > Just keep in mind, that the API proposal I attached is not for this ticket, > since I thought this ticket was about something else. Currently there is no > API proposal for this. I tried removing the file, but there seems to be no > such option. > > Tim Nguyen, I'd suggest you should create an API proposal since you are most > aware of what you actually need. Seems like your API could also cover my usecase with a few changes.
I haven't read/digested everything here yet, I'm sorry I didn't chime in sooner. This bug is getting entwined/confusing with a couple others... One reaction I have from looking at the image ntim postaed is that a transparent layer over content is not what I was considering or expecting, and not something I am supportive of. I'm good with pursuing a visible overlay (tab-specific modal dialog/panel) with UI that makes it obvious what extension is in control. I commented more on bug 1364404.
webextensions: ? → ---
Flags: needinfo?(mixedpuppy)
My usecase for this is implementing something akin to a screen-keyboard below the page. Ideally resizable on the y-axis. Currently this can only be implemented with an iframe in the top level document. It is very fragile because it has to battle with the layout of the content page.
Severity: normal → enhancement
Priority: -- → P5
Can we retriage this in light of bug 1280235 and bug 1340930?
Flags: needinfo?(mconca)
Just be clear of what this bug is about in case there was any confusion: allowing add-ons to inject content into web pages without that content being visible to web pages. Exactly like how the developer tools highlighters work.
Shadow dom with closed roots would provide comparable functionality, would it not?
I think we're more likely to allow add-ons to inject content into a layer that appears over web pages, rather than inject it into the same document.
(In reply to Wil Clouser [:clouserw] from comment #34) > Can we retriage this in light of bug 1280235 and bug 1340930? In the first triage (before my time) it looks like this was thought to be a good idea, but details needed flushing out. That still feels to be the case, specifically around how to handle user notification for an extensions doing this (security/privacy concerns). Is there a specific outcome you are hoping for, Wil, by requesting another triage?
Flags: needinfo?(mconca)
My goal is to find a webextension-compatible solution to bug 1389707. I think this should be re-triaged because bug 1280235 recently landed and would allow us to build this API in a protected space. If this bug is re-triaged and given a higher priority, we can move forward with a fix for the blocking bugs and land something in 6-12 months. If it's re-triaged and wontfix'd it signals to the Screenshots and Devtools teams that they won't be able to be web extensions with their current feature sets. Either way, it gets us out of the P5 purgatory. :)
Flags: needinfo?(mconca)
(In reply to Wil Clouser [:clouserw] from comment #39) > My goal is to find a webextension-compatible solution to bug 1389707. > > I think this should be re-triaged because bug 1280235 recently landed and > would allow us to build this API in a protected space. Why is this API limited to Mozilla? Third-party extension authors also require this feature.
I think the web extensions team wants to move carefully with this API and starting it out as mozilla-only would let them stay flexible and test options without committing to supporting one implementation forever.
When will the API be available non-mozilla consumers? Is it possible to make something like Origin Trials?
Blocks: 1424870
As this feature enables several Mozilla system components to move entirely to WebExtensions, I'm bumping the priority up to P3. I'll be doing some work to capture the requirements into a PRD-style document that can be reviewed and agreed upon.
Flags: needinfo?(mconca)
Priority: P5 → P3
Whiteboard: [design-decision-needed] → [design-decision-needed] [needs-follow-up]
I'd like to help move this bug along. I don't think I can resolve the design issues, but please needinfo me if I can help with implementation.
This feature request is too complicated for the normal design-decision-needed process, so I am removing that tag. Instead, a full-fledged PRD is being developed to try and capture the needed requirements. This PRD will be reviewed by engineering, product, and security.
Whiteboard: [design-decision-needed] [needs-follow-up]
Product: Toolkit → WebExtensions

As it seems some bugs are being closed cause "there has been very little interest" on them i want to confirm that i'm still interested on this bug.

I really want to use this API as soon as it is implemented. Currently I have to expose extensions UI to web pages that is so bad from security/privacy/fingerprinting perspective.

I know that currently a lot of work is going into project Fission. I just want to list some edge cases for the future, which should be regarded in the design process of this API.

This API is required for security, privacy and overall extension functionality / reliability reasons. Here is why:

The only way to display anything above the current website securely is by injecting an iframe. However this still doesn't work on for example pure SVG pages and theoretically the website itself could remove the iframe simply by using a DOM Observer. Moreover handling these iframes is now getting even harder with fission, because the content script can no longer directly access the iframes content.

Others stated that this API would not require to detect mouse events or events in general, because they can be detected via content scripts. I disagree on that. While developing and maintaining Gesturefy (a mouse gesture extension) I came across several limitations. Content script event listeners for example cannot detect any mouse events like "click" on a video element or disabled input elements regardless how you attach the listener to the window or document. Since the element picker is able to do this, it probably isn't relying on content scripts. Another big problem are iframes, which make it hard to detect events across the entire website. While content scripts can somehow workaround iframes by forwarding events to one another, the core content script limitations still apply.

In general I think the dev-tools inspector serves as a good example in marking the boundaries of this API.

Removing the reference to 1389707, as we aren't dependent on a fix here.

Severity: normal → S3

This has been mentioned several times, but having to inject elements to the DOM makes the use of extensions fingerprintable.
Having an overlay accessible by webextension seems like a good solution in that regard.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: