This failure happens intermittently on "Linux 18.04 x64 debug" if I fix bug 1596165. I will add a timer to be able to land bug 1596165.

The entire message is:
[task 2021-03-24T18:06:54.706Z] 18:06:54 INFO - TEST-UNEXPECTED-FAIL | accessible/tests/browser/tree/browser_searchbar.js | { editcombobox: [ entry: [ ] , combobox list: [ ] , grouping: [ ] ] } is expected to have a child at index 2 : { grouping: [ ] }, original tested: [ 'searchbar@id="searchbar" node', address: [object XULElement] ], [Exception... "Component returned failure code: 0x80070057 (NS_ERROR_ILLEGAL_VALUE) [nsIArray.queryElementAt]" nsresult: "0x80070057 (NS_ERROR_ILLEGAL_VALUE)" location: "JS frame :: chrome://mochitests/content/a11y/accessible/tests/mochitest/common.js :: testAccessibleTree :: line 564" data: no] -

We should probably wait for a show a11y event after opening the panel. However, this test will need to change anyway after bug 1699053 because the panel will get removed from the a11y tree when it is hidden. I suggest the test should:

1. Watch for both an a11y show and the popupshown.
2. Open the popup.
3. Wait for the promises in 1).
4. Test the tree. It should include the results grouping.
5. Watch for both an a11y hide and the popuphidden.
6. Close the popup.
7. Wait for the promises in 5).
8. Test the tree. It should not include the results grouping.

Assigning to you Morgan since you're working on bug 1699053.

(In reply to James Teh [:Jamie] from comment #2)

We should probably wait for a show a11y event after opening the panel. However, this test will need to change anyway after bug 1699053 because the panel will get removed from the a11y tree when it is hidden. I suggest the test should:

1. Watch for both an a11y show and the popupshown.
2. Open the popup.
3. Wait for the promises in 1).
4. Test the tree. It should include the results grouping.
5. Watch for both an a11y hide and the popuphidden.
6. Close the popup.
7. Wait for the promises in 5).
8. Test the tree. It should not include the results grouping.

Assigning to you Morgan since you're working on bug 1699053.

Huh. A few things:

I tried your steps above (which make sense!) on my patch. The test runs fine the first time it is launched with --verify, but the next test run in the suite stalls waiting for a show event. We get the popupshown, but no a11y EVENT_SHOW. I'm using Promise.all to test that both occur. When I run with the event logger, I don't see a show event on the second run. Strangely enough, I see a hide event on the panel after we get the popupshown even though the panel is visually open and stays open 🤔

To check if this was an artifact of my patch, I also upped to central and changed just the first event promise to wait for both a popupshown and a11y show. The same thing happens -- we get the popupshown but we don't get a11y show, and instead we get an a11y hide.

I can get the test to work in --verify if I wait for popupshown in the beginning and both popuphidden and a11y hidden at the end. The hide events are fired correctly for all subsequent test runs, but the show events are only right in the first. Is there something I'm missing here? We should have show events for the panel on subsequent tests, right?

Anyway, investigating that rn

You're not missing anything. The results you're seeing with your patch seem bizarre and unexpected to me also.

To clarify, when you run without your patch, I assume you don't see a11y hide events for popuphidden? The Accessible doesn't get removed without your patch, so I wouldn't expect to see a hide for popuphidden.

After waiting for and receiving an a11y hide, it might also be worth checking ok(!isAccessible(panel), "Notification panel not in a11y tree"), where panel is the notification panel.

I do wonder whether this could be related to the "hidden" attribute check you added in the XULMap for panel. That's probably not the source of the problem, but it's the part of the patch I'm least certain about, so might be worth investigating that if you run out of ideas.

(In reply to James Teh [:Jamie] from comment #4)

You're not missing anything. The results you're seeing with your patch seem bizarre and unexpected to me also.

To clarify, when you run without your patch, I assume you don't see a11y hide events for popuphidden? The Accessible doesn't get removed without your patch, so I wouldn't expect to see a hide for popuphidden.

After waiting for and receiving an a11y hide, it might also be worth checking ok(!isAccessible(panel), "Notification panel not in a11y tree"), where panel is the notification panel.

I think I've fixed this with the latest patch on phab for bug 1699053 :) should've cancelled this NI after I added the comment there on my other patch.

Fixed as part of bug 1699053.