User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0

Steps to reproduce:

This started happening a few days ago
Using the latest versions of Firefox Nightly
When I go to the GMail web site
The web page is presented
Nothing on the web page can be spoken by NVDA using mouse over
Fortunately, it still works fine using the regular Firefox

Actual results:

I use the mouse to navigate / hover over content on the web page and nothing is spoken by NVDA
I have tried to restart NVDA, the computer, and update Firefox Nightly but it does not help

Actually, it also seems to be a problem with other Google web pages
Google contacts, voice, calendar, drive, documents...

Expected results:

Usually when i use the mouse and hover over content on the GMail web page the content is spoken by NVDA

The Bugbug bot thinks this bug should belong to the 'Core::Disability Access APIs' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

I can confirm this. It doesn't happen on all websites, but Gmail is definitely busted. I'm not sure why yet, but I've confirmed it's specific to the caching implementation.

Morgan, would you be able to look into this? Thoughts from my investigation so far:

1. Calling ChildAtPoint on the document with a point inside a text leaf inside one of the cells in the Gmail message list returns a div very high in the ancestor chain containing most of the document. It has position: relative.
2. Calling ChildAtPoint on the div with the same point returns the div itself.
3. Calling ChildAtPoint on the parent of the text leaf with the same point returns null.
4. The text leaf (nor any of its ancestors) does not have the offscreen state, which suggests this isn't due to them missing from the viewport cache altogether.
5. I didn't verify this, but all of this leads me to think that the viewport cache is not in hit testing order in this case. That is, I think (at least some) ancestors appear in the viewport cache before their descendants.

Note that I tested this on two machines and got similar results, so hopefully, this isn't that weird issue I was seeing a while back with inconsistent viewport caches related to the document node.

I did a regression range and here are the results:
DEBUG : Found commit message:
Bug 1798621 - P2: Add a11y-no-cache test variant to CI. r=taskgraph-reviewers,ci-and-tooling,jmaher

Differential Revision: https://phabricator.services.mozilla.com/D161347

I've been trying to write a test case that reproduces this issue, but I'm having a hard time. Modelling this after what I see in the DOM inspector when I click on a the subject text of an unread email.
:Jamie do you have a way of reproducing this in a reduced test case right now ?
I think you're probably right that the issue here is the relative position, but what I have here passes with the cache on:

addAccessibleTask(
`
<div id="wrapper" style="position:relative; top: 40px; left: 40px;">
  <table id="table">
    <tr id="rowOne">
      <td id="cellOne">cellOne</td>
      <td id="cellTwo">cellTwo</td>
    </tr>
    <tr id="rowTwo">
      <td id="cellThree">cellThree</td>
      <td id="cellFour">cellFour</td>
    </tr>
  </table>
</div>
`,
async function(browser, accDoc) {
  const cellOne = findAccessibleChildByID(accDoc, "cellOne");
  const wrapper = findAccessibleChildByID(accDoc, "wrapper");
  const dpr = await getContentDPR(browser);
  // We want to hittest the text leaf inside of cellOne
  const [x, y] = Layout.getBounds(cellOne.firstChild, dpr);
  const [containerX, containerY] = Layout.getBounds(accDoc, dpr);
  // The point passed below will be made relative to accDoc, so subtract
  // off accDoc's screen coords here so we'll end up with a point in
  // cellOne's text leaf
  await testChildAtPoint(
    dpr,
    x + 3 - containerX,
    y + 3 - containerY,
    accDoc,
    wrapper, // direct child
    cellOne.firstChild // deepest child
  );
},
{
  iframe: true,
  remoteIframe: true,
  // Ensure that all hittest elements are in view.
  iframeAttrs: { style: "width: 600px; height: 600px; padding: 10px;" },
}
);

oh wait, rereading.

you're saying the div getting returned has position:relative -- I saw position:relative on the table and thought that's what you were referencing. lemme try that on the container instead/in addition.

EDIT: edited the above to what I think is a more appropriate test case, but still doesn't fail

I don't have a distilled test case, unfortunately. Also, I'm by no means certain that position: relative has anything to do with it. It just stood out to me, which is why I mentioned it.

Not everything on the Gmail page is failing to hit test. A little more digging suggests that everything below this node (let's call it h):
let h = document.querySelector("[role=navigation]").parentNode;
ends up returning this ancestor (let's call it r):
let r = document.querySelector('body > div > div > div[style="position: relative;"]');
That would in turn suggest that r is placed earlier in the viewport cache than h for some reason, but not all of r's descendants are in the wrong place, only h and its descendants.

The other thing that comment 8 suggests is that the table is irrelevant to this problem. For example, hit testing the Compose button or the labels also fails.

See the code comments for explanation.

:Jamie I'm not sure if this is related, but when playing around with the gmail test case I reduced, I found a problem with our offscreen state. We don't report the text in the following test as on screen (ie. it doesn't show up in the viewport cache) and so we hittest it wrong too.
I phab'd your patch locally and this issue still reproduces there for me (as I'd anticipate because it's an issue with the viewport cache not containing the text instead of not iterating the viewport cache completely). Just sharing in case it's helpful for this in some way. I can file a separate bug for this.

addAccessibleTask(
  `
<svg width="24px" id="svgArrow" height="24px" viewBox="0 0 24 24"><path d="M7 10l5 5 5-5z"></path></svg><div style="display:inline-block;" id="supportContainer"><a role="button" id="support" tabindex="0" aria-label="Support">y</a></div><div style="display:inline-block;" id="settingsContainer"><a role="button" tabindex="0" id="settings" aria-label="Settings">x</a></div>
 `,
  async function(browser, accDoc) {
    const support = findAccessibleChildByID(accDoc, "support");
    const supportContainer = findAccessibleChildByID(
      accDoc,
      "supportContainer"
    );
    await untilCacheOk(
      () => testVisibility(support, false, false),
      "support should be on screen and visible"
    );
    await untilCacheOk(
      () => testVisibility(supportContainer, false, false),
      "support container should be on screen and visible"
    );
    await untilCacheOk(
      () => testVisibility(support.firstChild, false, false),
      "support text should be on screen and visible"
    );
  },
  {
    iframe: true,
    remoteIframe: true,
    // Ensure that all hittest elements are in view.
    iframeAttrs: { style: "width: 600px; height: 600px; padding: 10px;" },
  }
);

As discussed on Matrix, that svg case is weird, but removing the svg doesn't seem to impact the semi-reduced Gmail test case. Let's spin that off into another bug.

In case it is any easier to test/debug
I wanted to make sure you saw my note

Actually, it also seems to be a problem with other Google web pages
Google contacts, voice, calendar, drive, documents...

Jamie's got a patch for this, so shifting the assignee

https://hg.mozilla.org/mozilla-central/rev/8735086656d3

While the latest version is a major improvement
109.0a1 (2022-12-07) (64-bit)
There remain regressions as compared to the previous working version of Nightly
As well as the current version of the regular Firefox

For example
The left hand panel of folders/labels is no longer spoken with mouse over
Mouse over text in text boxes like this one is also not speaking the correct line

(In reply to 4RJames from comment #17)

While the latest version is a major improvement
109.0a1 (2022-12-07) (64-bit)
There remain regressions as compared to the previous working version of Nightly
As well as the current version of the regular Firefox

For example
The left hand panel of folders/labels is no longer spoken with mouse over
Mouse over text in text boxes like this one is also not speaking the correct line

Hmm I can't get this to reproduce successfully with VO (testing on release firefox)
Does something like this test case exercise the issue you're seeing?
data:text/html,hello<textarea>world%0agoodbye</textarea>

I'm sorry, I don't understand your test case...

I didn't know if I should start additional bugs or add info to this one.

I have noticed the following:

Using Firefox Nightly with GMail/NVDA and mouse cursor

• top line content not spoken
• left panel content not spoken
• compose edit box content under panel spoken/bleeds through
• selectors not spoken
• newer, older, toggle filters not spoken
• star and action icons on inbox lines not spoken

I can reproduce the labels issue at least.

Morgan, I looked into this a bit and am noticing that with the cache enabled, when the Gmail inbox is open, the links for the labels on the left side all get the same y coordinate. With the cache disabled, this doesn't happen; the y coordinate is different for each link, as you'd expect for a vertical list. So, I suspect this is less about the viewport cache and more about bounds being wrong. Are you able to reproduce that?

It also occurred to me while putting my son to bed (totally normal, right?) that perhaps the reason we're seeing weirdness with the viewport cache (necessitating the hacky patch here) is that when we're building the viewport cache, we call GetAccessibleOrContainer, not GetAccessible. That means that if a particular content node doesn't have an Accessible (e.g. a div we pruned from the a11y tree) but does have a frame, we'll insert the container in its place. There might have been other descendants of that container that do have Accessibles, though. In that case, we've just put the container ahead of some of its descendants!

I'm not quite sure how to reproduce that, but maybe something like this (haven't verified yet):

data:text/html,<div role="main"><p>hi</p><div style="width: 10px; height: 10px;"></div><p>bye</p>

In this case, the inner div doesn't get an Accessible because it has no semantic value. The frames will probably be something like:
[paragraph for hi, inner div, paragraph for bye, outer div]
But because the inner div has no Accessible and we fall back to its container, the Accessibles will be:
[paragraph for hi, outer div, paragraph for bye]

I'm wondering whether we should be calling GetAccessible instead of GetContainerAccessible. I worry that might cause us to miss some Accessibles, but I can't come up with a case where that would happen.

Regardless of what we do here, I think we'll need to file separate bugs, but I'm dropping this all here until we figure out a path forward.

(In reply to James Teh [:Jamie] from comment #20)

It also occurred to me while putting my son to bed (totally normal, right?) that perhaps the reason we're seeing weirdness with the viewport cache (necessitating the hacky patch here) is that when we're building the viewport cache, we call GetAccessibleOrContainer, not GetAccessible.

I filed bug 1805876 for this. We still need to deal with the y coordinate being broken for Gmail labels, though.

clearing ni? since we discussed this on zoom and I filed follow up bugs (see above) for the issues we could reproduce.

4RJames, the issue where items in the compose dialog are not read (and items behind the dialog bleed through) should be fixed by bug 1805876, which is in the latest Firefox Nightly. The issue with the labels list is still under investigation over in bug 1806026.

The latest Firefox Nightly has improved things
However, not all of these mouse over regressions are resolved
While I focussed on GMail I mentioned the other Google web apps
I'm noticing similar regressions with others as well
Like when trying to use Linked In...
While using this edit box the mouse cursor and line spoken are one line off...
Do you think this is a general problem that when resolved will address all of these regressions?
For now I'm counting on the regular Firefox but fear this will eventually happen there...
I appreciate all your hard work on this matter!

(In reply to 4RJames from comment #24)

The latest Firefox Nightly has improved things

Not sure if you tested the nightly that had this, but the issue with the labels list should also be resolved now. That only happened a few hours ago.

While I focussed on GMail I mentioned the other Google web apps

What issues are you experiencing in other Google apps? Please file these as separate bugs.

Like when trying to use Linked In...
While using this edit box the mouse cursor and line spoken are one line off...

Please file a separate bug for this with steps to reproduce.

Do you think this is a general problem that when resolved will address all of these regressions?

In this area, some fixes resolve multiple problems, some fixes resolve only a single problem. Multiple fixes will almost certainly be required to address all of the problems. Unfortunately, the web is complex and almost every site does things differently in some way or another.

For now I'm counting on the regular Firefox but fear this will eventually happen there...

This will move to Firefox release, yes. This is a massive re-architecture which is necessary to fix many outstanding problems, including significant performance problems. While I can understand that issues like this are frustrating, it's worth noting that this re-architecture also fixes other problems (including some with mouse tracking) that existed in the old architecture. We're doing our best to address any reported regressions as quickly as we can. Thanks for your patience and the time you've taken to report these issues.

I have confirmed the reproduction of this issue in Nightly v109.0a1 (2022-11-20) and verified the fix in Nightly v110.0a1 (2022-12-29) and Beta v109.0b7. Furthermore, I've also confirmed it does not occur in Firefox Release v108.0.1. The Gmail web app works properly with NVDA on mouseover.

Thank you for all the efforts and explanations!

Using Firefox Nightly with GMail/NVDA and mouse cursor
Not Fixed - top line content not spoken
Fixed - left panel content not spoken
Fixed - compose edit box content under panel spoken/bleeds through
Not Fixed - selectors not spoken
Not Fixed - newer, older, toggle filters not spoken
Not Fixed - star and action icons on inbox lines not spoken

Using edit boxes like this one, the mouse cursor does not always read the line it is on and reads another line...
However, the arrow keys always read the correct line.

I'll do my best to report the regressions that remain.

Thanks for the additional details.

(In reply to 4RJames from comment #27)

Not Fixed - top line content not spoken

If I press c to compose a message in Gmail, tab to the message body, and type the following:

Line 1
Line 2
Line 3

Routing the mouse to each line reports the line as expected here. Can you provide alternative steps to reproduce which show the problem you're experiencing?

Not Fixed - selectors not spoken

What do you mean by selectors? Do you mean the check boxes in the inbox that allow you to select individual messages?
For me, moving the mouse to these reports "check box" (if I have Report role when mouse enters object enabled in NVDA -> Preferences -> Settings -> Mouse). That's not entirely helpful - it should report the label - but this is a bug in NVDA, not Firefox. This is the same in stable builds of Firefox. Are you seeing something different here in stable builds?

Not Fixed - newer, older, toggle filters not spoken
Not Fixed - star and action icons on inbox lines not spoken

Similar to the above, NVDA reports "button" for me in this case. The failure to report the label is due to the same NVDA bug. Again, this occurs in stable Firefox as well.

When I use the physical mouse and hover over your three lines of text
Only the first one is spoken...
This is the case here in your comment as well as the GMail message I received...
It doesn't seem to matter if I come from the top or bottom, only line 1 is spoken
I don't understand that issue...

In this comment edit box something similar to the GMail compose is going on...
If the edit box content is scrolled such that the top line is visible everything works with the mouse reading the line under the cursor.
However, when there are more lines than fit in the edit box things change.
When the top line of text is not showing in the edit box the mouse does not read the line under the cursor and another line is read.

I just discovered this because this comment edit box is rather small...
Perhaps this is what made things less obvious in other edit boxes...

I think this edit box and others are similar like GMail and LI message edit boxes.

It also is different based on whether I scroll up to the top line with the mouse wheel or use the up arrow on the keyboard...
If I go all the way up with the arrow key the mouse cursor does not read the correct line under the cursor
However, if I scroll up to the top using the mouse wheel it works correctly.

Yes, by selector I mean the check boxes to select individual messages and I think you are correct these work the same with nightly and stable.

Thank you for your help and patience!

(In reply to 4RJames from comment #29)

When I use the physical mouse and hover over your three lines of text
Only the first one is spoken...

My test involved typing this into the Gmail compose box line by line and then moving the mouse. That seems to work for me. However, I can reproduce the problem you're seeing when reading the three lines as they appear in this Bugzilla bug. I'll file a separate bug for that shortly.

If the edit box content is scrolled such that the top line is visible everything works with the mouse reading the line under the cursor.
However, when there are more lines than fit in the edit box things change.
When the top line of text is not showing in the edit box the mouse does not read the line under the cursor and another line is read.

Morgan, is it possible we don't handle scroll offsets correctly with regard to the calculation of relative text bounds? I don't think we have a test for that.

(In reply to James Teh [:Jamie] from comment #30)

(In reply to 4RJames from comment #29)

When I use the physical mouse and hover over your three lines of text
Only the first one is spoken...

My test involved typing this into the Gmail compose box line by line and then moving the mouse. That seems to work for me. However, I can reproduce the problem you're seeing when reading the three lines as they appear in this Bugzilla bug. I'll file a separate bug for that shortly.

If the edit box content is scrolled such that the top line is visible everything works with the mouse reading the line under the cursor.
However, when there are more lines than fit in the edit box things change.
When the top line of text is not showing in the edit box the mouse does not read the line under the cursor and another line is read.

Morgan, is it possible we don't handle scroll offsets correctly with regard to the calculation of relative text bounds? I don't think we have a test for that.

I don't think we handle scroll offsets on hypertexts themselves, no. We should do that :) I think the bug you filed covers that, so will dig in there.

(In reply to Morgan Reschenberg [:morgan] from comment #31)

I don't think we handle scroll offsets on hypertexts themselves, no. We should do that :) I think the bug you filed covers that, so will dig in there.

Bug 1808828 covers the y coordinate being incorrect for subsequent lines inside <pre><code>, but I didn't think that had anything to do with scrolling?

Regarding scrolling, here's a test case:
data:text/html,<textarea style="height: 10px;">a%0ab%0ac
If you focus the textarea and move the caret to the last line, the textarea scrolls visually, but the bounds for the characters don't update. I don't think bug 1808828 covers that, but perhaps I'm missing something?

I wanted to share another observation...

While composing messages in GMail
The default composition edit box is a smaller window that includes a horrizontal scroll bar
That scroll bar goes away if using a full screen composition window as it is wider...
When that scroll bar is present it seems to have some affect on the mouse cursor reading content
When the mouse cursor is placed near the bottom of the edit box near that scroll bar the content spoken is the font type
It is as if somehow it thinks the cursor is below the scroll bar and reading from the formatting tool bar

Which is also interesting as you cannot read the content of the formatting tool bar when you place the mouse cursor over it's elements...
In fact the only thing accessible to the mouse cursor in that area is the send button.

(In reply to 4RJames from comment #34)

When the mouse cursor is placed near the bottom of the edit box near that scroll bar the content spoken is the font type

Does this happen in Firefox stable?
In Nightly, if you mouse over the font type itself (in the correct location on the screen), does the font type get reported?

Which is also interesting as you cannot read the content of the formatting tool bar when you place the mouse cursor over it's elements...

What does NVDA report in that case? Note that these buttons trigger the same NVDA bug as the selection and star icons, so you wouldn't get any label reported in this case.

Does this happen in Firefox stable?
No, it does not

In Nightly, if you mouse over the font type itself (in the correct location on the screen), does the font type get reported?
No, it does not
However, it does in stable

FYI - The LI related regressions are reported here:
https://bugzilla.mozilla.org/show_bug.cgi?id=1809082

(In reply to James Teh [:Jamie] from comment #33)

If you focus the textarea and move the caret to the last line, the textarea scrolls visually, but the bounds for the characters don't update.

We use text leaf nodes to get text bounds. BoundsWithOffset already applies the scroll offset for ancestors, so I would've thought it should handle this already. However, it seems that the textarea cache doesn't have scroll-position. I wonder if there's a frame between the textarea frame and the text leaf frame that we're missing?

I would like to follow up with additional information.

When reading GMail messages
There are sometimes suggested responses
These are presented below the last message and above the reply, reply all, and forward buttons
These are buttons which have the suggested response text in the button.

For some reason when I mouse over these response buttons, they are spoken as the buttons below them instead of the response text...

Although, it seems to depend on how the mouse over approaches the responses as sometimes they are correctly spoken.
I think they are incorrect when the mouse comes down to the response buttons and sometimes correct when the mouse comes up to the response buttons.

When they incorrectly say the below button labels it is dependant on how many buttons are presented.
Sometimes there is only a reply and other times there is a reply all...

The problem I described in my previous comment still exists

In addition, I have recently discovered

When addressing a new message
When I start typeing the recipiant address
The list of matching addresses is presented
The entries on this list are not accessible to the mouse
They used to be...
The other day they were not accessible when using the arrow keys
Today when using:
115.0a1 (2023-05-11) (64-bit)
They are accessible using the arrow keys but still not the mouse

When using:
115.0a1 (2023-05-16) (64-bit)

When I search in sent items
The suggestions that pop up
are no longer read by NVDA when using the mouse

In addition to the above regressions in my previous comments
When using nightly version:
115.0a1 (2023-06-03) (64-bit)

When the subject field has a suggested auto completion it is not accessible to NVDA.
I could not read it using teh arrow keys or the mouse...
This used to be possible.

The GMail suggestions are still not accessible to NVDA using the mouse

The reply suggestions are almost accessible using the mouse if you play around with different approaches/positions
However, if you position the mouse over the suggested reply button it often reads the corresponding button below like reply, reply all, forward

The email address suggestions are not accessible using the mouse
Nothing is spoken when the mouse moves over the suggested email addresses on the list
However, if you use the arrow keys they are spoken by NVDA