Closed
Bug 441974
Opened 17 years ago
Closed 16 years ago
Several tests fail when running in conjunction with other mochitests, succeed when running stand-alone
Categories
(Core :: Disability Access APIs, defect)
Core
Disability Access APIs
Tracking
()
VERIFIED
FIXED
mozilla1.9.1
People
(Reporter: MarcoZ, Assigned: MarcoZ)
References
(Depends on 1 open bug, Blocks 1 open bug)
Details
(Keywords: access, verified1.9.1)
Attachments
(3 files, 1 obsolete file)
(deleted),
patch
|
surkov
:
review+
|
Details | Diff | Splinter Review |
(deleted),
patch
|
surkov
:
review+
beltzner
:
approval1.9.1+
|
Details | Diff | Splinter Review |
(deleted),
patch
|
MarcoZ
:
review+
|
Details | Diff | Splinter Review |
This is a spin-off of bug 424195. Several Mochitests that succeed when running stand-alone.
1. Make a release or debug build with the enable_tests option.
2. Launch runtests.py with the --a11y switch.
3. In the test harness screen, click the test_bug368835.xul file.
Result: All tests pass.
4. Return to the main harness screen.
5. Click "Run all tests".
6. Observe as some of the tests for test_bug368835.xul fail.
Same result for (maybe):
test_bug420863.html
test_nsOuterDocAccessible.html
test_nsIAccessibleText.html
Notice that clicking the "Run tests" several times and watching the results, they appear randomly, except for the test_table_indexes.html, which will be dealt with in bug 437980, and the test_nsIAccessibleImage.html, which gets fixed in bug 439057. All others ordinarily pass or have todo items, but when running all together, the above candidates tend to fail.
Any idea why?
Assignee | ||
Comment 1•17 years ago
|
||
On IRC on the #developers today, Shawn recommended that our seemingly random failures could have something to do with the tests requiring a certain state when they start running.
The one thing that is identical in all tests is this: In order to be able to retrieve accessibles for any given DOM element, we have to enable the nsIAccessibleRetrieval service:
http://mxr.mozilla.org/mozilla-central/source/accessible/tests/mochitest/test_aria_activedescendant.html?force=1#14,
http://mxr.mozilla.org/mozilla-central/source/accessible/tests/mochitest/test_aria_activedescendant.html?force=1#17,
and
http://mxr.mozilla.org/mozilla-central/source/accessible/tests/mochitest/test_aria_activedescendant.html?force=1#34.
So, whenever a new test file starts, we do a call to this getService method.
If there were a way to discover whether the nsIAccessibleRetrieval service is already active, I'd be happy to use that method to avoid unnecessary calls.
Any thoughts?
Comment 2•17 years ago
|
||
That's likely not your problem.
Assignee | ||
Comment 3•16 years ago
|
||
Just talked to Ted M. on IRC, and he suggested to take a look at this under Valgrind (Windows) or Purify (Linux).
Anyone familiar with these tools?
Assignee | ||
Comment 4•16 years ago
|
||
sayrer, any chance you could take a stab at this? Ted M. suggested you might be able to find something.
Assignee | ||
Comment 5•16 years ago
|
||
Clint, Sayrer, any chance you'd be able to take a stab at this this week? Would be very much appreciated!
Comment 6•16 years ago
|
||
Could this have something to do with the test being run inside an iframe vs. as the toplevel document?
Assignee | ||
Comment 7•16 years ago
|
||
Hi Ted, this is the bug we were discussing yesterday about the a11y random failures.
Assignee | ||
Comment 8•16 years ago
|
||
This fixes the tests all except for test_textattrs.html, which has 6 failures on the input type text having to do with different default fonts.
Lukas, do these tests also work on the staging if you apply this patch?
Assignee: nobody → marco.zehe
Status: NEW → ASSIGNED
Attachment #334860 -
Flags: review?(surkov.alexander)
Comment 9•16 years ago
|
||
Comment on attachment 334860 [details] [diff] [review]
Fix some of the tests to pass when running in conjunction with others
><html><body><pre style="word-wrap: break-word; white-space: pre-wrap;">diff --git a/accessible/tests/mochitest/test_bug368835.xul b/accessible/tests/mochitest/test_bug368835.xul
>--- a/accessible/tests/mochitest/test_bug368835.xul
>+++ b/accessible/tests/mochitest/test_bug368835.xul
>@@ -100,22 +100,22 @@
> gTreeInvalidatedCount++;
>
> switch (gTreeInvalidatedCount) {
> case 1:
> TreeInvalidatedHandlerHelper(aEvent, 0, 5, null, null,
> "nsITreeBoxObject::rowCountChanged");
iirc TreeInvalidatedHandlerHelper takes null because aEvent.getData() should return null if there is no requested data. So does it return empty string in this case?
>- testThis("emptyLink", 93, 4, "");
>+ testThis("emptyLink", 93, 4, null);
>
> // see bug 428954: No name wanted for internal frame
>- is(outerDocAcc.name, "", "Wrong name for internal frame!");
>+ is(outerDocAcc.name, null, "Wrong name for internal frame!");
>
because it's a bit tricky where we use empty string or null then please comment here why do we return null
would be nice to look one more patch :)
Attachment #334860 -
Flags: review?(surkov.alexander)
Assignee | ||
Comment 10•16 years ago
|
||
I simply took my output from the logs the tests produced and transformed them into expected values. So, the event.getData calls really return 0 instead of null, and in the other cases return null instead of "". Bug 451533 was actually spun off of this work because I saw that we're not always truncating the aName parameter, but this doesn't help those tests. Some of them still return null instead of "".
Assignee | ||
Comment 11•16 years ago
|
||
Lukas, can you give us the lines that show failures from a the staging machine on Linux? I'd like to see if we're getting the same or different failures there. Thanks!
Comment 12•16 years ago
|
||
Marco: fyi, Lukas is on vacation and offline until approx 08 sept.
Robcee or myself can try to help, but we dont have context. Can you clarify which lines of which logs are you looking for?
Assignee | ||
Comment 13•16 years ago
|
||
(In reply to comment #12)
> Robcee or myself can try to help, but we dont have context. Can you clarify
> which lines of which logs are you looking for?
Basically a current run of the whole a11y test suite, all lines with "unexpected" in them, from the Linux staging box.
Assignee | ||
Comment 14•16 years ago
|
||
Attachment #334860 -
Attachment is obsolete: true
Attachment #337930 -
Flags: review?(surkov.alexander)
Updated•16 years ago
|
Attachment #337930 -
Flags: review?(surkov.alexander) → review+
Assignee | ||
Comment 15•16 years ago
|
||
Pushed in changeset:
http://hg.mozilla.org/mozilla-central/rev/ffc759affe23
Status: ASSIGNED → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
Comment 16•16 years ago
|
||
With this checkin, I'm still getting two test failures in my build (from just a little bit ago) on Windows:
32 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_b
ug420863.html | Error thrown during test: uncaught exception: [Exception... "Com
ponent returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIAccessible.numAc
tions]" nsresult: "0x80004005 (NS_ERROR_FAILURE)" location: "JS frame :: chrom
e://mochikit/content/a11y/accessible/test_bug420863.html :: doTest3 :: line 97"
data: no] - got 0, expected 1
833 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_
nsIAccessible_focus.html | No focused state for element with ID aria-link.
Comment 17•16 years ago
|
||
Reopening, as there are still a couple of failing tests reported in comment#16.
Updated•16 years ago
|
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment 18•16 years ago
|
||
So I tried turning on the a11y tests in moz2 unittests.
Here is what the results were:
Win32 (7 tests fail consistently):
*** 851 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_nsIAccessible_focus.html | No
focused state for element with ID aria-link2.
*** 919 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area1 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
*** 935 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area2 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
*** 980 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area4 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
*** 998 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area5 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
*** 1017 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area6 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
*** 1048 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area7 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
On Linux (22 fails consistent):
79:*** 32 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_bug420863.html | Error thrown
during test: uncaught exception: [Exception... "Component returned failure
code: 0x80004005 (NS_ERROR_FAILURE) [nsIAccessible.numActions]" nsresult:
"0x80004005 (NS_ERROR_FAILURE)" location: "JS frame ::
chrome://mochikit/content/a11y/accessible/test_bug420863.html :: doTest3 ::
line 97" data: no] - got 0, expected 1
153:*** 112 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong posinset
on menu_item1.1 - got "", expected "1"
154:*** 113 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong setsize
on menu_item1.1 - got "", expected "1"
156:*** 115 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong posinset
on menu_item1.2 - got "", expected "1"
157:*** 116 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong setsize
on menu_item1.2 - got "", expected "3"
159:*** 118 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong posinset
on menu_item1.4 - got "", expected "2"
160:*** 119 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong setsize
on menu_item1.4 - got "", expected "3"
162:*** 121 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong posinset
on menu_item2 - got "", expected "3"
163:*** 122 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong setsize
on menu_item2 - got "", expected "3"
419:*** 385 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html | Wrong
screen y coordinate for linkedImage! - got 587, expected 588
441:*** 407 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html | Wrong
screen y coordinate for linkedImageWithAlt! - got 713, expected 714
463:*** 429 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html | Wrong
screen y coordinate for linkedImageWithTitle! - got 839, expected 840
485:*** 451 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html | Wrong
screen y coordinate for linkedImageEmptyAlt! - got 965, expected 966
507:*** 473 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html | Wrong
screen y coordinate for linkedImageEmptyAltAndTitle! - got 1091, expected 1092
563:*** 530 ERROR TEST-UNEXPECTED-PASS |
chrome://mochikit/content/a11y/accessible/test_nsIAccessibleTable_1.html |
columnHeader should not throw
876:*** 850 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_nsIAccessible_focus.html | No
focused state for element with ID aria-link.
942:*** 920 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area1 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
958:*** 936 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area2 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
1003:*** 981 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area4 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
1021:*** 999 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area5 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
1040:*** 1018 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area6 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
1071:*** 1049 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area7 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
Assignee | ||
Comment 19•16 years ago
|
||
OK, I believe I fixed the ones Lukas is seeing on Windows by disabling, temporarily fixing with hard-coded values, or marking as known fail. See dependent bugs for more details on these.
I remember having seen that "uncaught exception" error once, and Ted seems to see it, too, but I can't reproduce that right now any longer. The root cause of that seems to be that removing the event listener for the "click" event somehow invalidates the accessible.
Anyone got an idea why the offsets for those failing y coordinates on Linux are all off by 1? Are the offsets on Linux 1-based, not 0-based or something?
Comment 20•16 years ago
|
||
Marco, any chance we can just disable / exclude the failing tests in comment#18, while debugging continues? We've a whole bunch of unittest systems migration work on hold because of this. :-(
Assignee | ||
Comment 21•16 years ago
|
||
I've wallpaper-fixed the test failures Lukas provided from the Windows and Linux logs by mostly disabling/protecting against them and providing bug numbers for these specific failures.
Lukas or John, could you run them again on the staging machines and see if you get any additional failures?
Comment 22•16 years ago
|
||
Marco - I am waiting on some VMs to put on m-c staging (see bug 455433 and 455431) at which point I will be able to run these on m-c again.
Is it possible for you to look at the tests that are failing in 1.9 at this time and disable any tests there that are failing?
Assignee | ||
Comment 23•16 years ago
|
||
Lukas, could you provide me with a full failure log when you run the a11y tests on 1.9 staging on both Windows and Linux? I don't trust my failures I'm getting locally, so would like to use your output to wall-paper-fix/disable them.
Assignee | ||
Comment 24•16 years ago
|
||
This patch does two things:
1. It corrects test cases for select and select@sioze elements which have different names after bug 455482 was checked in. These tests should have been fixed as part of that bug, but weren't.
2. Disable checking for attributeChanged events on anything but Windows. Linux fails with this test.
Attachment #349938 -
Flags: review?(surkov.alexander)
Comment 25•16 years ago
|
||
Comment on attachment 349938 [details] [diff] [review]
Fix last remaining mochitest failures
>- is(gA11yEventObserver.mTextAttrChangedEventCounter, 2,
>- "Wrong count of 'text attribute changed' events for " + ID);
>+ if (navigator.platform == "Win32")
>+ is(gA11yEventObserver.mTextAttrChangedEventCounter, 2,
>+ "Wrong count of 'text attribute changed' events for " + ID);
>+ else
>+ todo(gA11yEventObserver.mTextAttrChangedEventCounter, 2,
>+ "Wrong count of 'text attribute changed' events for " + ID);
If it's fail on linux and mac both then please file bug for this (if we don't have it already).
Attachment #349938 -
Flags: review?(surkov.alexander) → review+
Assignee | ||
Comment 26•16 years ago
|
||
Done, bug 466629.
Updated•16 years ago
|
Attachment #349938 -
Flags: approval1.9.1?
Comment 27•16 years ago
|
||
Comment on attachment 349938 [details] [diff] [review]
Fix last remaining mochitest failures
a191=beltzner
Attachment #349938 -
Flags: approval1.9.1? → approval1.9.1+
Comment 28•16 years ago
|
||
Status: REOPENED → RESOLVED
Closed: 16 years ago → 16 years ago
Resolution: --- → FIXED
Comment 29•16 years ago
|
||
We tried running a11y in production this morning, and had a bunch of a11y tests fail again. Unclear why some tests are failing like this, and MarcoZ/joduinn unable to figure it out, so we backed a11y out from production.
Reopening. :-(
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment 30•16 years ago
|
||
I had a very quick look at one of the failing tests:
http://mxr.mozilla.org/mozilla-central/source/accessible/tests/mochitest/test_events_caretmove.html?force=1
I notice the use of "this" to declare variables in the function definitions. Variables declared via "this.foo" inside a function will not be function local. In these cases "this" is most likely the window object (global namespace). I wasn't sure if this was known/intended? If you want function local scope use "var foo"...
I'm not familiar with the mochitest framework... but I'm wondering if there is something funny going on with the global namespace getting cluttered?
Comment 31•16 years ago
|
||
Comment 32•16 years ago
|
||
Surkov, of course you are correct, thanks! I see they are indeed been used as object constructors. Please disregard comment #30. I'm used to a different test harness.
Assignee | ||
Comment 34•16 years ago
|
||
John, can we get this patch onto both a Windows and Linux staging server and run a build cycle with a11y enabled there to see if this improves the situation? Since I cannot reproduce any of this locally, we'd have to actually run it in that environment to get accurate results.
Assignee | ||
Comment 35•16 years ago
|
||
CC'ing Bob to see if he has any expertise to share!
Comment 36•16 years ago
|
||
Run on m-c staging with the patch applied locally, a11y on win32 isn't crashing out anymore. Both Linux and Windows just have the one tests that is failing/passing unexpectedly:
Win32
*** 1007 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Wrong count of 'text attribute changed' events for area8 - got 0, expected 2
Linux
*** 1007 ERROR TEST-UNEXPECTED-PASS | chrome://mochikit/content/a11y/accessible/test_textattrs.html | 2 - Wrong count of 'text attribute changed' events for area8
Comment 37•16 years ago
|
||
the patch i'm referring to is the "one more fix" one in the attachments for this bug.
Comment 38•16 years ago
|
||
I stand corrected, the windows build just passed a11y tests with this patch applied locally. So it's just the unexpected pass on linux. Marco - if you want to approve this and get it checked in, I can enable a11y tests tomorrow.
Comment 39•16 years ago
|
||
Ok, now the linux build just passed a11y tests and the Windows build failed that single test same as in comment 36.
Not sure what's happening here.
Comment 40•16 years ago
|
||
I've set them to output to http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest
You want to look at Linux moz2-linux-slave09 dep unit test and WINNT 5.2 bm-win2k3-unittest-01 dep unit test
Assignee | ||
Updated•16 years ago
|
Attachment #352504 -
Flags: review?(marco.zehe) → review+
Assignee | ||
Comment 41•16 years ago
|
||
Comment on attachment 352504 [details] [diff] [review]
one more fix
Thanks Lukas! I'm sure the test fluke with that particular test can be easily resolved.
Alex, please take a look at that particular test and see if you can correct it. Check it in as a bustage fix.
Comment 42•16 years ago
|
||
(In reply to comment #36)
> Run on m-c staging with the patch applied locally, a11y on win32 isn't crashing
> out anymore. Both Linux and Windows just have the one tests that is
> failing/passing unexpectedly:
>
> Win32
>
> *** 1007 ERROR TEST-UNEXPECTED-FAIL |
> chrome://mochikit/content/a11y/accessible/test_textattrs.html | Wrong count of
> 'text attribute changed' events for area8 - got 0, expected 2
>
> Linux
>
> *** 1007 ERROR TEST-UNEXPECTED-PASS |
> chrome://mochikit/content/a11y/accessible/test_textattrs.html | 2 - Wrong count
> of 'text attribute changed' events for area8
I guess it's bug 466629
Assignee | ||
Comment 43•16 years ago
|
||
This patch landed along with the fix for bug 448744. In addition, bug 466629 landed, which fixes Lukas' above reported random orange. I believe we're good! I don't see any failures when I run these locally.
Status: REOPENED → RESOLVED
Closed: 16 years ago → 16 years ago
Resolution: --- → FIXED
Comment 44•16 years ago
|
||
Yes, we are definitely seeing green in both m-c and 1.9.1 Linux staging, and in m-c Win32. I have to see a build of 1.9.1 Win32 just to verify - but this looks really promising.
Assignee | ||
Comment 45•16 years ago
|
||
Lukas, we didn't push anything to m-1.9.1 yet, this was all exclusively m-c work. So, m-1.9.1 is still subject to the same randomness that we've been seeing all along.
Assignee | ||
Comment 46•16 years ago
|
||
Alexander, logs on the mozilla-1.9.1 staging servers show similar failures as the "One more fix" patch fixes. However this patch does not apply at all to the 1.9.1 branch, since there were other changes to common.js which cause it not to apply.
Can you create a patch for mozilla-1.9.1 that fixes the problems?
Assignee | ||
Comment 47•16 years ago
|
||
I created a smaller patch that just contains the changes to test_events_caretmove.html and pushed that to mozilla-1.9.1. The rest of that patch applies to features that were explicitly added to mozilla-central. We'll see if this was enough to fix the Windows timeouts we're seeing on mozilla-1.9.1.
Keywords: fixed1.9.1
Comment 48•16 years ago
|
||
MarcoZ, would be nice to have patch attached or to have changeset at least.
Assignee | ||
Comment 49•16 years ago
|
||
(In reply to comment #48)
> MarcoZ, would be nice to have patch attached or to have changeset at least.
Oops, sorry! Here's the changeset: http://hg.mozilla.org/releases/mozilla-1.9.1/rev/37d44cd6a2c5
Assignee | ||
Comment 50•16 years ago
|
||
Tests were enabled on December 18, 2008, and have been running continuously since.
Keywords: fixed1.9.1 → verified1.9.1
Assignee | ||
Comment 51•16 years ago
|
||
Same goes for mozilla-central AKA trunk.
Status: RESOLVED → VERIFIED
You need to log in
before you can comment on or make changes to this bug.
Description
•