Open
Bug 480347
Opened 16 years ago
Updated 2 years ago
AccessibleComponent_getAccessibleAtPoint does not work if menu is displayed
Categories
(Core :: Disability Access APIs, defect)
Tracking
()
NEW
M1
People
(Reporter: sam.quiring, Unassigned)
References
(Depends on 1 open bug, Blocks 1 open bug)
Details
(Keywords: access)
Attachments
(1 file, 3 obsolete files)
(deleted),
patch
|
Details | Diff | Splinter Review |
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.5) Gecko/2009012320 Minefield/3.0.5
I'm trying to workaround the fact that Minefield delivers a useless "redundant object" to the Gtk:GtkWidget:button-press-event event listener. By the way, is there a better event to be listening for?
As a user I click on the "Help" menu, causing the menu to be displayed -- my event hander correctly figures out that Menu(Help) is the GUI item being clicked.
As a user I next click on the "About Minefield" menu item. I get the correct mouse coordinates and use an algorithm involving AccessibleComponent_getAccessibleAtPoint() to find the GUI item under the mouse. Unfortunately, this routine does not appear to be aware of the Help drop-down menu that is being displayed on top of the web page -- so it finds the GUI item in the web page under the "About Minefield" menu item.
I'm not sure how to work around this.
Reproducible: Always
Steps to Reproduce:
1. Set up an event handler for Gtk:GtkWidget:button-press-event
2. When an event comes in, use AccessibleComponent_getAccessibleAtPoint to drill to the topmost GUI item.
3. as a user click on "Help", causing the Help Menu to appear
4. as a user click on the "About Minefield" menu item.
Actual Results:
The topmost GUI item will be from the webpage underneath the "About Minefield" menu item, instead of the Menu item.
Expected Results:
AccessibleComponent_getAccessibleAtPoint() should know that a menu window is being displayed and that this menu window is above the web page (Z-order), and therefore higher priority.
What event AT-SPI handler(s) should I set up so that I can figure out what menu item the user clicked on? I'm very open to suggestions.
Reporter | ||
Comment 1•16 years ago
|
||
The email on this should be Sam.Quiring@windriver.com, not sam.quiring@windriver.com. If I knew how to change it myself, I would.
Reporter | ||
Comment 2•16 years ago
|
||
I tried this on FF 3.0.1 and FF 3.0.6 and I am seeing the same problems there.
Updated•16 years ago
|
Component: Disability Access → Disability Access APIs
Keywords: access
Product: Firefox → Core
QA Contact: disability.access → accessibility-apis
Target Milestone: --- → M1
Comment 3•16 years ago
|
||
Ginn, any ideas?
Currently, from a11y API, we don't know which window/widget is above, and which window/widget is behind.
Even if the window is minimized, the state of a11y object is still showing/visible.
I don't know if it is expected result.
For your case, can you use "focus event" and "focused state"?
Comment 5•16 years ago
|
||
Thanks Ginn, how does the at-spi fill out the button-press-event information? If we expose the z-order of the FF gui somehow could you fix this in at-spi?
The problem of getAccessibleAtPoint is on all platforms.
The code is in
http://mxr.mozilla.org/mozilla-central/ident?i=GetDeepestChildAtPoint
I think at-spi or IA2 doesn't have button-press-event.
So basically AT client should not listen to it, at least for Firefox.
Reporter | ||
Comment 7•16 years ago
|
||
Gtk made up their own event: Gtk:GtkWidget:button-press-event. Mozilla is at liberty to do this as well - AT-SPI supports vendor-specific events. If you go this route, I urge you to have an event, button-click, representing the fact that the user both pushed and released the mouse button on the same object.
In the meantime, what event or sequence of events should the FF/Minefield AT client listen for so it can determine unambiguously:
1) the user has clicked a GUI item (left mouse push, left mouse release)
2) the GUI item clicked by the user
?
I do not think 'focus event' nor 'focused state' will work. When the menu comes up (after clicking on 'Help'), the user is going to slide his mouse over each item to get to the bottommost, "About Mozilla Firefox". As he does this, each item gets the focus. So, unless I am misunderstanding what you mean, I do not think you can use focus to say "the user clicked on a menu item".
Comment 8•16 years ago
|
||
(In reply to comment #0)
> As a user I next click on the "About Minefield" menu item. I get the correct
> mouse coordinates and use an algorithm involving
> AccessibleComponent_getAccessibleAtPoint() to find the GUI item under the
> mouse.
...
> Actual Results:
> The topmost GUI item will be from the webpage underneath the "About Minefield"
> menu item, instead of the Menu item.
Before I look into this one, are you sure the menu (and the menu item) are still shown when you call AccessibleComponent_getAccessibleAtPoint?
Reporter | ||
Comment 9•16 years ago
|
||
Am I sure? Pretty sure. Here's the experiment I just ran to be sure:
My key event handler (as previously documented) is for Gtk:GtkWidget:button-press-event. Since it triggers on the "press" and not the "click", I can press the mouse button but not release it and get my code to execute while keeping the menu open. Here are the steps of my experiment:
1. start listening for events
2. click on Firefox's Help menu
3. The Help menu drops down
4. move mouse down to "About Mozilla Firefox"
5. see all the focus events for each item in the menu
6. press left mouse down on "About Mozilla Firefox" but do not let it up, just sit there.
7. Move mouse (left button still pressed) off menu, then release left button.
8. Help Menu is still being displayed - it never went away
In the log I can see that Gtk:GtkWidget:button-press-event was delivered. My code begins with the Firefox application (because it contains the mouse) and uses AccessibleComponent_getAccessibleAtPoint to find the gui item under the mouse. I found the web page item (a form) that was under the "About Mozilla Firefox" menu item. The menu never showed up.
That is as sure as I can be.
Updated•16 years ago
|
Assignee: nobody → david.bolter
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Comment 10•16 years ago
|
||
I spoke briefly with Neil Deakin and it looks like we'll need to look at popups first using the nsXULPopupManager, like is done in PresShell::HandleEvent. (I have recreated the bug on Linux via Accerciser's console).
Comment 11•16 years ago
|
||
(In reply to comment #6)
> I think at-spi or IA2 doesn't have button-press-event.
Actually, at-spi has mouse:button:* events, but the only information is pointer x,y coordinate.
David,
It sounds good for looking at popups first.
It may solve the problem for AccessibleComponent_getAccessibleAtPoint() API.
But it may not solve the use case.
We need 3 parameters for AccessibleComponent_getAccessibleAtPoint(), component accessible, x, y.
How can we find the right "component accessible" to query?
If we have 2 browser window and have overlaps and we got a screen (x,y), how do we know which window should be asked for AccessibleComponent_getAccessibleAtPoint?
We still need get Z-order work?
Also if "Gtk:GtkWidget:button-press-event" gives "redundant accessible", which is produced be gail, we can do nothing about it, it's out of our control.
Comment 12•16 years ago
|
||
(In reply to comment #11)
> (In reply to comment #6)
> > I think at-spi or IA2 doesn't have button-press-event.
> Actually, at-spi has mouse:button:* events, but the only information is pointer
> x,y coordinate.
>
> David,
> It sounds good for looking at popups first.
> It may solve the problem for AccessibleComponent_getAccessibleAtPoint() API.
>
> But it may not solve the use case.
>
> We need 3 parameters for AccessibleComponent_getAccessibleAtPoint(), component
> accessible, x, y.
> How can we find the right "component accessible" to query?
>
> If we have 2 browser window and have overlaps and we got a screen (x,y), how do
> we know which window should be asked for
> AccessibleComponent_getAccessibleAtPoint?
> We still need get Z-order work?
>
> Also if "Gtk:GtkWidget:button-press-event" gives "redundant accessible", which
> is produced be gail, we can do nothing about it, it's out of our control.
Yeah, hmmm, Ginn let's we treat these as separate bugs?
Do you the popup work (this bug) is probably needed anyways?
Comment 13•16 years ago
|
||
Comment 14•16 years ago
|
||
Attachment #365187 -
Attachment is obsolete: true
Updated•16 years ago
|
Attachment #365189 -
Attachment is obsolete: true
Comment 15•16 years ago
|
||
Here's the plan:
davidb: Enn: what do you recommend?
Enn: probably to add a new interface or modify nsIMenuRollup or something
Enn: when does this need to be called?
Enn: do you have access to a presshell?
davidb: Enn: anytime, by AT (indirectly)
davidb: Enn: yep
Enn: davidb: if you have a presshell, I'd just add a method there
davidb: Enn: ok i'll give that a shot thanks
Enn: davidb: and perhaps refactor it into the existing code which checks popups
Comment 16•16 years ago
|
||
(In reply to comment #11)
> (In reply to comment #6)
> David,
> It sounds good for looking at popups first.
> It may solve the problem for AccessibleComponent_getAccessibleAtPoint() API.
>
> But it may not solve the use case.
I think it will because XUL popups are not their own windows AFAIK.
>
> We need 3 parameters for AccessibleComponent_getAccessibleAtPoint(), component
> accessible, x, y.
> How can we find the right "component accessible" to query?
I'm not sure, what are you passing in for this Sam?
>
> If we have 2 browser window and have overlaps and we got a screen (x,y), how do
> we know which window should be asked for
> AccessibleComponent_getAccessibleAtPoint?
> We still need get Z-order work?
Probably, if so, then that's a spin-off piece to this.
>
> Also if "Gtk:GtkWidget:button-press-event" gives "redundant accessible", which
> is produced be gail, we can do nothing about it, it's out of our control.
What does gail use to get the accessible?
Updated•16 years ago
|
Status: ASSIGNED → NEW
Comment 17•16 years ago
|
||
I spent some time on this today, but I'd like to know what those redundant objects Sam describes are all about before spending much more.
(BTW I'm not endorsing the approach I took in this WIP, we might what an nsPresShell GetTopFrameFor(PRInt32 screenx, PRInt32 screeny) instead, or something else entirely.)
Reporter | ||
Comment 18•16 years ago
|
||
I call AccessibleComponent_getAccessibleAtPoint() with each direct child of the
Firefox Application node that meets the requirements below (if I'm doing this
wrong, let me know).
1. role == SPI_ROLE_FRAME, SPI_ROLE_DIALOG, or SPI_ROLE_ALERT
2. Accessible_isComponent(child)
3. AccessibleComponent_contains(childComponent, mouse_x, mouse_y)
The general algorithm builds a list of these, but for Firefox I've never seen
more than one direct child, the "frame". Here's an example log:
app 19/19 (n:Firefox,d:,r:application)[subs: Application] (children: 1)
child 1/1 (n:Alan Turing - Google Search - Mozilla Firefox,d:,r:frame)
Box[407,49,979,642][subs: Component Document] (children: 15)
child 1/1 - attr 1/3 event-from-input:false
child 1/1 - attr 2/3 tag:window
child 1/1 - attr 3/3 id:main-window
Firefox is the 19th child of the 19 children of the desktop. The application
node has one child with role name "frame" and attributes as shown. That
"frame" is the Accessible * object passed to ..._getAccessibleAtPoint(). The
(x, y) coordinates are the mouse coordinates.
If it were possible to deliver the correct Accessible *object via the
Gtk:GtkWidget:button-press-event that would be excellent for me.
Comment 19•16 years ago
|
||
Sam are you able to build FF with this patch applied to test it out?
Attachment #365528 -
Attachment is obsolete: true
Reporter | ||
Comment 20•16 years ago
|
||
I have never built Firefox before. I do not have the source downloaded nor do I know how to download it. I do not know how to apply patches. I'll gladly download a binary from somewhere and try it out.
Comment 21•16 years ago
|
||
Firing up a tryserver build (sorry for delay).
Comment 22•16 years ago
|
||
(In reply to comment #20)
> I'll gladly
> download a binary from somewhere and try it out.
Should be here now/soon: https://build.mozilla.org/tryserver-builds/2009-05-14_17:16-dbolter@mozilla.com-480347/
Comment 23•16 years ago
|
||
Sam how do this build work for you?
Reporter | ||
Comment 24•16 years ago
|
||
I am not working on this at the moment, which means it is not easy to find time to download and test. I will try it in the next 2-3 days.
Comment 25•14 years ago
|
||
Fernando, can you check if this is still issue?
Comment 26•14 years ago
|
||
It is.
I can check any element coordinate with this python script:
def callback(event):
global count
if event is None:
return
if type(event) == pyatspi.event.DeviceEvent:
return
print event
reg = pyatspi.Registry
reg.registerEventListener(callback, "mouse:abs")
reg.start()
So mouse over any ff element, I get the screen absolute coords. With them, I got to accerciser and do:
a = acc.queryComponent().getAccessibleAtPoint(x,y,0)
while a is not None:
a = a.queryComponent().getAccessibleAtPoint(x,y,0)
print a
So I get the final child contained a t that point.
It works fine for some elements, but sometimes it enters a loop returning itself as a child. So probably that is another bug.
Checking for menus, I can confirm that that it gives you the element below the menu.
Updated•14 years ago
|
Assignee: bolterbugz → fherrera
Updated•13 years ago
|
Blocks: boundsa11y
Updated•13 years ago
|
Assignee: fherrera → nobody
Updated•2 years ago
|
Severity: normal → S3
You need to log in
before you can comment on or make changes to this bug.
Description
•