Requesting that the a11y Mochitests be run regularly to automatically catch possible regressions. There aren't that many yet, but those we have are already good for several situations. The parameter for runtests.py and runtests.pl is --a11y .

re-classifying and prioritizing. OK to take this.

Marco: I ran this on a local build yesterday just to see if it's working. The harness came up as it should, but there were no tests showing. A couple of questions: 1) Do I need to turn on any special configure options? 2) Are there any tests checked in? thanks!

(In reply to comment #2) > 1) Do I need to turn on any special configure options? You need to run the runtests.pl or runtests.py with the --a11y switch. Then, the "Accessibility testharness" page comes up. When you click the "Run tests" button, you usually get 479 or so passes, a bunch of todo's, and hopefully no failures. :-) > 2) Are there any tests checked in? Most definitely! They are stored in mozilla/accessible/tests/mochitest.

One thing that just came to my mind: Are you trying this on Mac? If so, that may be why you're not able to see/run them. 1. Mac accessibility is practically non-existent (for supporting the Universal Access APIs, that is), so these tests are currently only making sense to run on Win and Linux. 2. The build system may not pick the a11y testfiles up because of that very reason.

I am indeed running on a Mac. I'll try rerunning this under linux and windows. I guess it's safe to say that we won't need to add these tests to the Mac unittest boxes then?

(In reply to comment #5) > I guess it's safe to say that > we won't need to add these tests to the Mac unittest boxes then? Right! Until we have full Mac accessibility, which I currently see for Mozilla2 rather than Gecko 1.9.1, we don't need to clutter up the Mac unittest boxes with these.

Since mochitest make tests for Gecko stuff code only then it make sense to run them on Mac too. Just build should be built with enabled accessibility. Also, I think it's important to make builds with enabled accessibility to make sure we don't broke code on Mac.

(In reply to comment #7) > Since mochitest make tests for Gecko stuff code only then it make sense to run > them on Mac too. Just build should be built with enabled accessibility. Also, I > think it's important to make builds with enabled accessibility to make sure we > don't broke code on Mac. Alex, thanks for the heads-up! Is there a special option that should be set that is not required on other platforms? For example, on Linux and Win, accessibility comes with the standard build options, I don't need to enable it explicitly. Could you give Rob some pointers?

Yes, that's option is ac_add_options --enable-accessibility. I use it when I build firefox on mac.

good advice. Thanks for the help, Alex and Marco!

with the attached .mozconfig file, I ran mac and linux. The mac run completed with: *** 874 INFO Passed: 705 *** 875 INFO Failed: 60 *** 876 INFO Todo: 78 the linux run completed with: *** 875 INFO Passed: 707 *** 876 INFO Failed: 60 *** 877 INFO Todo: 77 Do these numbers look right or am I missing something? Failures seems high. See other attachments for mac and linux error logs.

(In reply to comment #11) > Do these numbers look right or am I missing something? Failures seems high. The last 9 errors (in tableinsane2 and tableinsane4) have already been filed as bug 437980. The Mac error at the top is very strange. It almost appears as if that particular property is not exposed to JS on Mac. Did this one come up everytime? The top Linux problems in test_bug368835.xul may be fixed by bug 427841, but I am not sure. Also, this is something I've seen come and go, and it almost never happens if the test is run alone and not as part of the full run. The wrong X and Y coordinates for images are very bizarre. If I run that test stand-alone, the images appear at the spots I specify in the mochitest. If the test runs within all others, the images appear in a different place. Any recommended course of action other than leaving these tests out?

(In reply to comment #14) > The wrong X and Y coordinates for images are very bizarre. If I run that test > stand-alone, the images appear at the spots I specify in the mochitest. If the > test runs within all others, the images appear in a different place. Any > recommended course of action other than leaving these tests out? Filed bug 439057 for this issue.

first crack at automation. Successful run on staging on mac and windows.

for the record, staging currently lives on: http://tinderbox.mozilla.org/showbuilds.cgi?tree=UnitTest

Comment on attachment 326991 [details] [diff] [review] [checked in] a11y buildbot configs staging You import copy, but then don't seem to use it. r+ otherwise.

Comment on attachment 326991 [details] [diff] [review] [checked in] a11y buildbot configs staging Checking in master.cfg; /cvsroot/mozilla/tools/buildbot-configs/testing/unittest-stage/master.cfg,v <-- master.cfg new revision: 1.11; previous revision: 1.10 done Checking in mozbuild.py; /cvsroot/mozilla/tools/buildbot-configs/testing/unittest-stage/mozbuild.py,v <-- mozbuild.py new revision: 1.3; previous revision: 1.2 done

Marco, Rob: 1) What o.s. are we running these tests on? Comment#4 talks about not being on mac, but comment#11 talks about running on mac. 2) If we disable the Failed+ToDo tests, could we start running the Passing tests in production?

(In reply to comment #20) > 1) What o.s. are we running these tests on? Comment#4 talks about not being on > mac, but comment#11 talks about running on mac. I originally assumed that the accessibility code was not running at all on the Mac. But I was wrong about that: The platform-independent code is executed once someone asks for accessibility on the Mac, or turns it on as described above by Surkov. So the platform independent Mochitests (which are currently all of them) should definitely also run on Mac. > 2) If we disable the Failed+ToDo tests, could we start running the Passing > tests in production? I think so. Note however, that the more exciting stuff is taking place in mozilla-central, not on the 1.9.0 branch. So, we should get them running on something running off of mozilla-central. That's where testing needs to catch regressions, and if something is good or has baked there for a while, it can then be backported to branch if needed.

(In reply to comment #21) > (In reply to comment #20) > > 2) If we disable the Failed+ToDo tests, could we start running the Passing > > tests in production? > > I think so. Note however, that the more exciting stuff is taking place in > mozilla-central, not on the 1.9.0 branch. So, we should get them running on > something running off of mozilla-central. That's where testing needs to catch > regressions, and if something is good or has baked there for a while, it can > then be backported to branch if needed. > Marco, can you disable those failing a11y tests? We'd like to enable the tests that *do* pass on production asap.

(In reply to comment #22) > Marco, can you disable those failing a11y tests? We'd like to enable the tests > that *do* pass on production asap. I have a fix for bug 439057 in the pipeline that will make those tests pass. bug 437980 we don't have a patch yet, so I'd need to disable all these table_indexes tests (the whole file), since the tests are part of a loop that also has passing ones. As for general performance: bug 441974 may also be visible on production or staging machines. On my system, it causes some tests to fail randomly when running alltogether, but passing whenever I run the file in question stand-alone. If you or Lucas have any idea on that, please comment on that bug. Clint said he'd also take a look.

hi Marco; 1) Any ETA on when the patch can be landed, and other failing tests disabled? We'd really like to get this into production for 1.9. 2) I note the patch above is for cvs. Do you have a similar patch incoming for these tests on hg?

(In reply to comment #24) > hi Marco; > > 1) Any ETA on when the patch can be landed, and other failing tests disabled? > We'd really like to get this into production for 1.9. Patch for bug 439057 landed a few days ago on HG, fixing those big numbers of failing tests. Leaves the table_indexes.html to be disabled, and the other bug 441974 to work out why some tests fail randomly. I'm desperately waiting on feedback from Clint and Sayrer on that one. > 2) I note the patch above is for cvs. Do you have a similar patch incoming for > these tests on hg? Rob, can you speak more to that?

However, I didn't find the mozbuild.py file at all in this repo. Would appreciate some help on this!

http://mxr.mozilla.org/mozilla/source/tools/buildbot-configs/testing/unittest-stage/mozbuild.py However, there is a patch waiting to come in here - https://bugzilla.mozilla.org/show_bug.cgi?id=440164 that you should look at before making changes to the current revision of mozbuild.py I'll try to get that checked in today.

Comment on attachment 331052 [details] [diff] [review] [backed out] master.cfg changes for build/buildbot-configs/mozilla2-unittest Looks fine, assuming the A11y buildsteps are defined somewhere else (mozbuild.py, buildbotcustom, ...). Is there another patch forthcoming for those?

(In reply to comment #28) > Looks fine, assuming the A11y buildsteps are defined somewhere else > (mozbuild.py, buildbotcustom, ...). > > Is there another patch forthcoming for those? Do you have any hints of where I could look for this? I synced with http://hg.mozilla.org/build/buildbot-configs, but didn't find a suitabe mozilla2/Firefox file where I could put that second half of the original patch I converted. config.py is present, but it contains totally different stuff than the ones that were in CVS. So any hints as to where these other steps need to be defined would be greatly appreciated!

(In reply to comment #27) > http://mxr.mozilla.org/mozilla/source/tools/buildbot-configs/testing/unittest-stage/mozbuild.py I'm actually talking Mercurial here, not CVS trunk. Joduinn asked whether we could get the tests running on the Mozilla-Central tinderboxen.

For the mercurial staging and production, we use buildbotcustom steps.py as seen here: http://mxr.mozilla.org/mozilla/source/tools/buildbotcustom/unittest/steps.py (we still check out our configs from cvs for now, there is no repo in hg yet for buildbot) Steps.py replaces mozbuild.py

Ok, this adds the steps to steps.py which is what buildbotcustom looks to in the moz2 setup. I'll request a win32 box for moz2 staging, for now we can try this on the linux box that's already running on staging.

steps.py got updated by bug 448416, so this is a new patch to work with the updated version.

Checking in tools/buildbotcustom/unittest/steps.py; /cvsroot/mozilla/tools/buildbotcustom/unittest/steps.py,v <-- steps.py new revision: 1.12; previous revision: 1.11 done Checked in on Lukas' behalf. Thanks!

Pushed master.cfg changes to build/buildbot-configs in changeset: http://hg.mozilla.org/build/buildbot-configs/index.cgi/rev/4af357980258

(In reply to comment #35) > Pushed master.cfg changes to build/buildbot-configs in changeset: > http://hg.mozilla.org/build/buildbot-configs/index.cgi/rev/4af357980258 So, the big block of failing tests are disabled, bug#441974 filed to track other intermittent test failures and now this patch is landed. Whats left to do in this bug before we have a11y tests running on mozilla-central?

I don't know if we're still waiting for a checkin or something, but the a11y tests are still throwing warnings on both staging 1.9 and staging m-c - until I see some green, these can't move to production.

marco, ted: until bug#441974 is resolved, can we just mark those specific tests as "expected fail"? At least that would get the remaining passing tests to show green, and we could then enable those into production. Right now, we have nothing in production. :-(

marco, ted: any update? This is a Q3 goal, and we're only weeks away from end of quarter.

Comment on attachment 331052 [details] [diff] [review] [backed out] master.cfg changes for build/buildbot-configs/mozilla2-unittest I backed this change out. It never made it into production and is blocking other changes we want to take. We can reland and deploy (in close succession!) when the time is right.

So I tried turning on the a11y tests in moz2 unittests. Here is what the results were: Win32 (7 tests fail consistently): *** 851 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_nsIAccessible_focus.html | No focused state for element with ID aria-link2. *** 919 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area1 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)" *** 935 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area2 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)" *** 980 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area4 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)" *** 998 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area5 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)" *** 1017 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area6 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)" *** 1048 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area7 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)" On Linux (22 fails consistent): 79:*** 32 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_bug420863.html | Error thrown during test: uncaught exception: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIAccessible.numActions]" nsresult: "0x80004005 (NS_ERROR_FAILURE)" location: "JS frame :: chrome://mochikit/content/a11y/accessible/test_bug420863.html :: doTest3 :: line 97" data: no] - got 0, expected 1 153:*** 112 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong posinset on menu_item1.1 - got "", expected "1" 154:*** 113 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong setsize on menu_item1.1 - got "", expected "1" 156:*** 115 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong posinset on menu_item1.2 - got "", expected "1" 157:*** 116 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong setsize on menu_item1.2 - got "", expected "3" 159:*** 118 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong posinset on menu_item1.4 - got "", expected "2" 160:*** 119 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong setsize on menu_item1.4 - got "", expected "3" 162:*** 121 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong posinset on menu_item2 - got "", expected "3" 163:*** 122 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong setsize on menu_item2 - got "", expected "3" 419:*** 385 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html | Wrong screen y coordinate for linkedImage! - got 587, expected 588 441:*** 407 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html | Wrong screen y coordinate for linkedImageWithAlt! - got 713, expected 714 463:*** 429 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html | Wrong screen y coordinate for linkedImageWithTitle! - got 839, expected 840 485:*** 451 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html | Wrong screen y coordinate for linkedImageEmptyAlt! - got 965, expected 966 507:*** 473 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html | Wrong screen y coordinate for linkedImageEmptyAltAndTitle! - got 1091, expected 1092 563:*** 530 ERROR TEST-UNEXPECTED-PASS | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleTable_1.html | columnHeader should not throw 876:*** 850 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_nsIAccessible_focus.html | No focused state for element with ID aria-link. 942:*** 920 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area1 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)" 958:*** 936 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area2 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)" 1003:*** 981 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area4 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)" 1021:*** 999 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area5 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)" 1040:*** 1018 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area6 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)" 1071:*** 1049 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area7 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)"

Here is the output of errors on 1.9 staging: Win32 pgo build (11 fails): 95:*** 35 ERROR FAIL | Error thrown during test: uncaught exception: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIAccessible.numActions]" nsresult: "0x80004005 (NS_ERROR_FAILURE)" location: "JS frame :: chrome://mochikit/content/a11y/accessible/test_bug420863.html :: doTest3 :: line 97" data: no] | got 0, expected 1 | chrome://mochikit/content/a11y/accessible/test_bug420863.html 683:*** 638 ERROR FAIL | Wrong name for internal frame! | got null, expected "" | chrome://mochikit/content/a11y/accessible/test_nsOuterDocAccessible.html 807:*** 763 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 1, column = 0 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 811:*** 767 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 1, column = 1 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 815:*** 771 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 1, column = 2 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 835:*** 791 ERROR FAIL | tableinsane2: cell index from object attributes of cell accessible isn't corrent. | got 8, expected 9 | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 844:*** 800 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 4, column = 3 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 848:*** 804 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 4, column = 4 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 867:*** 823 ERROR FAIL | tableinsane4: Can't get cell accessible at row = 1, column = 0 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 871:*** 827 ERROR FAIL | tableinsane4: Can't get cell accessible at row = 1, column = 1 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 875:*** 831 ERROR FAIL | tableinsane4: Can't get cell accessible at row = 1, column = 2 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html Win32 Build (only 10 error fails): 685:*** 638 ERROR FAIL | Wrong name for internal frame! | got null, expected "" | chrome://mochikit/content/a11y/accessible/test_nsOuterDocAccessible.html 809:*** 763 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 1, column = 0 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 813:*** 767 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 1, column = 1 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 817:*** 771 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 1, column = 2 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 837:*** 791 ERROR FAIL | tableinsane2: cell index from object attributes of cell accessible isn't corrent. | got 8, expected 9 | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 846:*** 800 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 4, column = 3 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 850:*** 804 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 4, column = 4 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 869:*** 823 ERROR FAIL | tableinsane4: Can't get cell accessible at row = 1, column = 0 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 873:*** 827 ERROR FAIL | tableinsane4: Can't get cell accessible at row = 1, column = 1 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 877:*** 831 ERROR FAIL | tableinsane4: Can't get cell accessible at row = 1, column = 2 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html Linux (24 error fail): 60:*** 12 ERROR FAIL | Wrong 'startrow' of 'treeInvalidated' event on nsITreeBoxObject::invalidateColumn | got 5, expected null | chrome://mochikit/content/a11y/accessible/test_bug368835.xul 61:*** 13 ERROR FAIL | Wrong 'endrow' of 'treeInvalidated' event on nsITreeBoxObject::invalidateColumn | got 5, expected null | chrome://mochikit/content/a11y/accessible/test_bug368835.xul 62:*** 14 ERROR FAIL | Wrong 'startcolumn' of 'treeInvalidated' event on nsITreeBoxObject::invalidateColumn | got null, expected 0 | chrome://mochikit/content/a11y/accessible/test_bug368835.xul 63:*** 15 ERROR FAIL | Wrong 'endcolumn' of 'treeInvalidated' event on nsITreeBoxObject::invalidateColumn | got null, expected 0 | chrome://mochikit/content/a11y/accessible/test_bug368835.xul 64:*** 16 ERROR FAIL | Wrong 'startrow' of 'treeInvalidated' event on nsITreeBoxObject::invalidateRow | got 0, expected 1 | chrome://mochikit/content/a11y/accessible/test_bug368835.xul 65:*** 17 ERROR FAIL | Wrong 'endrow' of 'treeInvalidated' event on nsITreeBoxObject::invalidateRow | got 0, expected 1 | chrome://mochikit/content/a11y/accessible/test_bug368835.xul 66:*** 18 ERROR FAIL | Wrong 'startcolumn' of 'treeInvalidated' event on nsITreeBoxObject::invalidateRow | got 0, expected null | chrome://mochikit/content/a11y/accessible/test_bug368835.xul 67:*** 19 ERROR FAIL | Wrong 'endcolumn' of 'treeInvalidated' event on nsITreeBoxObject::invalidateRow | got 0, expected null | chrome://mochikit/content/a11y/accessible/test_bug368835.xul 251:*** 211 ERROR FAIL | Wrong screen y coordinate for linkedImage! | got 560, expected 561 | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html 273:*** 233 ERROR FAIL | Wrong screen y coordinate for linkedImageWithAlt! | got 686, expected 687 | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html 295:*** 255 ERROR FAIL | Wrong screen y coordinate for linkedImageWithTitle! | got 812, expected 813 | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html 317:*** 277 ERROR FAIL | Wrong screen y coordinate for linkedImageEmptyAlt! | got 938, expected 939 | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html 339:,10001:0*** 299 ERROR FAIL | Wrong screen y coordinate for linkedImageEmptyAltAndTitle! | got 1064, expected 1065 | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html 671:*** 639 ERROR FAIL | Wrong name for internal frame! | got null, expected "" | chrome://mochikit/content/a11y/accessible/test_nsOuterDocAccessible.html 795:*** 764 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 1, column = 0 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 799:*** 768 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 1, column = 1 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 803:*** 772 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 1, column = 2 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 823:*** 792 ERROR FAIL | ta,170:0bleinsane2: cell index from object attributes of cell accessible isn't corrent. | got 8, expected 9 | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 832:*** 801 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 4, column = 3 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 836:*** 805 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 4, column = 4 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 855:*** 824 ERROR FAIL | tableinsane4: Can't get cell accessible at row = 1, column = 0 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 859:*** 828 ERROR FAIL | tableinsane4: Can't get cell accessible at row = 1, column = 1 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html 863:*** 832 ERROR FAIL | tableinsane4: Can't get cell accessible at row = 1, column = 2 | | chrome://mochikit/content/a11y/accessible/test_table_indexes.html

All disabled/protected. Please try another run, thanks!

Thank you Marco, looking good on 1.9 - I see at least one green run on Linux, and Win32 (non-pgo) - will keep an eye on it while we wait for VMs to put on moz2 staging. Sorry for the delay, I hope we'll have this done within the week.

Lukas: any update?

Ok, so I ran the a11y step on qm-moz2-unittest01 slaves and the results so far are: Linux - green! Win32 - http://pastebin.mozilla.org/545432 Marco - what do you think of the leaks on windows? Any ideas what could be causing this?

Also - for further update, the 1.9 accessibility tests have all passed successfully at least once. I believe we can go ahead and enable these on production 1.9 later this week.

(In reply to comment #46) > Ok, so I ran the a11y step on qm-moz2-unittest01 slaves and the results so far > are: > > Linux - green! > Win32 - http://pastebin.mozilla.org/545432 > > Marco - what do you think of the leaks on windows? Any ideas what could be > causing this? I don't see any leaks there. Did you miss them in your paste, or am I missing something?

Sorry - here is the leak info from a build where the only orange was on the a11y test step: nsTraceRefcntImpl::DumpStatistics: 876 entries TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 15868 bytes during test execution (should have leaked no more than 0 bytes) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of AtomImpl with size 16 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of ThreadShutdownObserver with size 12 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of imgLoader with size 24 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of mozStorageConnection with size 72 bytes each (144 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of mozStorageFunctionGetUnreversedHost with size 8 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of mozStorageService with size 24 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 48 instances of mozStorageStatement with size 40 bytes each (1920 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 13 instances of nsAccessNode with size 20 bytes each (260 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 13 instances of nsAccessible with size 72 bytes each (936 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsBaseAppShell with size 60 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsBaseURLParser with size 8 bytes each (16 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsBasicEncoder with size 12 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsCategoryManager with size 76 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsCollation with size 8 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsCollationWin with size 16 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsCommandManager with size 56 bytes each (112 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsDOMStorage with size 64 bytes each (128 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsDSURIContentListener with size 32 bytes each (64 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsDateTimeFormatWin with size 36 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsDefaultURIFixup with size 24 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsDeque with size 52 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsDocLoader with size 144 bytes each (288 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsDocShell with size 448 bytes each (896 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsDocShell::InterfaceRequestorProxy with size 12 bytes each (24 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsDocumentCharsetInfo with size 20 bytes each (40 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsDownloadManager with size 56 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 13 instances of nsHTMLImageAccessible with size 164 bytes each (2132 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 13 instances of nsHTMLImageAccessibleWrap with size 172 bytes each (2236 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsLayoutHistoryState with size 48 bytes each (96 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsLoadGroup with size 4 bytes each (8 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 3 instances of nsLocalFile with size 88 bytes each (264 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsLocale with size 16 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsNavBookmarks with size 192 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsNavHistory with size 808 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsObserverService with size 44 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsPersistentProperties with size 76 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsPrefBranch with size 52 bytes each (104 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsPrefService with size 36 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 4 instances of nsRect with size 16 bytes each (64 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsRunnable with size 8 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsSHEntry with size 140 bytes each (280 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 6 instances of nsStandardURL with size 172 bytes each (1032 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 193 instances of nsStringBuffer with size 8 bytes each (1544 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsStringBundle with size 32 bytes each (64 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsSystemPrincipal with size 32 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 4 instances of nsTArray_base with size 4 bytes each (16 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsThread with size 68 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsThreadPool with size 56 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsTimerImpl with size 48 bytes each (96 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 66 instances of nsVoidArray with size 4 bytes each (264 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 6 instances of nsWeakReference with size 12 bytes each (72 bytes total) TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsWebBrowserFind with size 48 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsWebNavigationInfo with size 16 bytes TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsWebShell with size 472 bytes each (944 bytes total) started: 2008-10-06 09:32:20.082000 finished: 2008-10-06 09:32:40.762000 program finished with exit code 0

So there's finally a pretty solid moz2-unittest staging setup now, and the a11y tests are being run on them. Right now, the linux tests are failing 37 tests and the win32 tests are throwing timeout exceptions. Someone want to take a look? These slaves are reporting to Tinderbox tree=UnitTest Linux: moz2-linux-slave09 moz2-linux-slave10 Win32: moz2-win32-slave09 moz2-win32-slave10

Just checked in again - one of the linux boxes is experiencing some pretty erratic results in the a11y tests, between 16 and 60 fails depending on the build, while the other is holding steady at 36 fails (and one instance of 37 fails).

(In reply to comment #50) > So there's finally a pretty solid moz2-unittest staging setup now, and the a11y > tests are being run on them. > Right now, the linux tests are failing 37 tests and the win32 tests are > throwing timeout exceptions. > Someone want to take a look? Marco, Ted: Who can best look at these failing a11y tests? The linux test *used* to pass green, so we feel their setup is good. The win32 timeout exception is recent, without us changing anything, so we're not sure whats causing that - any ideas?

The results of these test (logs, etc.) can be found at: http://tinderbox.mozilla.org/showbuilds.cgi?tree=UnitTest

Just did a clobber on the two win32 boxes. Is someone able to look at the exception that is being caused in a11y on win32? Also, there are 22 tests failing in the Linux run, so can those be disabled for now so we can attempt to get this on production?

Marco or Surkov- Can one of you disable those 22 tests so we can make progress this this bug (deploying to production) and resolving those failures in parallel? Thx!

Lukas, can you send me just the failing lines of those logs for both Windows and Linux? I am not able to reproduce these 22 failures you are seeing, for whatever reason, and I am also having trouble reading those Tinderbox pages. So it would be very helpful if you could get me the logs so I can disable the tests for now. This is all mozilla-central, right?

(In reply to comment #56) > Lukas, can you send me just the failing lines of those logs for both Windows > and Linux? I am not able to reproduce these 22 failures you are seeing, for > whatever reason, and I am also having trouble reading those Tinderbox pages. Are you able to reproduce the problem if you run the entire suite? > So it would be very helpful if you could get me the logs so I can disable the > tests for now. Thanks Marco, it would be really great to deal with this, and finally get those suites running in production! (In reply to comment #54) > Is someone able to look at the exception that is being caused in a11y on win32? Any ideas about these win32 exceptions?

This is what the most recent Linux test fails reported. Marco, I hope this is what you need to be able to disable certain Linux tests.

and here's the windows output from the most recent a11y fails.

Looking at these logs, I see very strange things output already on the first two lines of each file: 1. On Windows, a global array variable is claimed not to be defined, even though its definition is clearly visible a few lines above the first use. In a different file, a constant is claimed to be used (and undefined) that appears nowhere within that file. 2. On Linux, the nsISupports interface is claimed not to be defined. Huh? All NSIAccessible* interfaces inherit from nsISupports and work fine. I am not aware that anything within the a11y module could have caused this dramatic kindof failure. In chatting with Lukas, we decided to run the a11y tests manually to see if the output is different. Question: Did any other unit test parts have this dramatic kind of failure recently, for example after turning on JIT, and were there specific steps that needed to be taken when this happened?

So I ran these manually on a windows build. Didn't run the other test suites first. The a11y suite runs, up to chrome://mochikit/content/a11y/accessible/test_textboxes.xul it reports passing 932, failing 80 and todo 78 (if --close-when-done is included in the command) it reports passing 932, failing 135 and todo 78 (if no --close-when-done) and then when you close the window, you get the leak output as well as a memory reference error. (see attached)

So on the linux build, running the a11y tests also stops short at the test_textboxes.xul test, Passed 804, Failed 76, Todo 77.

Per irc, marco landed patch to disable the hanging "test_testboxes" suite, which should get us past these known failing tests. If this test run finds no other problems, then we'd enable this "as is" into production immediately. bug#465755 filed to track debugging those specific test/code failures. We'll continue tracking infrastructure setup in this bug.

Marco: On http://tinderbox.mozilla.org/showbuilds.cgi?tree=UnitTest, I just noticed a11y 518/0/79 ...is that success? :-) I also noticed only win32 unittest running... where did the linux unittests go?

John, I still have two patched files waiting to be committed. With all the branching, tagging etc. going on, I decided to wait until the dust cleared. This didn't seem to be the case throughout all Friday, so I didn't make a move yet.

(In reply to comment #66) > John, I still have two patched files waiting to be committed. With all the > branching, tagging etc. going on, I decided to wait until the dust cleared. > This didn't seem to be the case throughout all Friday, so I didn't make a move > yet. Marco; can you post your two patches to bug#441974 and get them reviewed asap? The tree is going to be closed for another few days while we create the beta2 builds. However, if your changes are only touching tests, and are already r+'d, then I'll see if we might be allowed to land.

I'm trying, but the person who usually reviews such patches is involved in his ph.d process and is on and off rather randomly.

John, I got review. What's the next step in this special case?

(In reply to comment #69) > John, I got review. What's the next step in this special case? Marco: I see you've already landed bug#441974 on mozilla-central. Cool. Next step is for Lukas to verify if your fix commented out the last of the failing tests, and if the a11y suite is now passing cleanly. If so, then we could finally be ready to enable a11y in production.

From today's dev meeting, now is a good time to enable a11y in production. However, this afternoon, we decided to not enable today because of the unrelated ESX server woes causing fun on the tree. Instead, planning to enable a11y test suite early tomorrow morning. ToDo: 1) joduinn to post patch uncommenting existing a11y step in moz2-unittest/master.cfg. This will be for win2 and linux, on both mozilla-1.9.1 and mozilla-central. 2) joduinn to reconfig master.cfg and watch to make sure it goes smmothly. 3) Marco to watch tests in a11y suite be run per build. In case any specific a11y test fails, Marco will land patch to disable that specific test, and work with developers to correct. Marco/Lukas/anyone: anything else we need to do?

see previous comment in bug for details.

Comment on attachment 352249 [details] [diff] [review] re-enable a11y for linux, win32 on both mozilla-central and mozilla-1.9.1 Requesting that this be backed out and the a11y tests disabled again. We are getting a lot of strange failures on both platforms that is impossible to deal with in the short term.

OK, I have reverted the local change (!!) which applied attachment 352249 [details] [diff] [review], and reconfig'd moz2-master.

(In reply to comment #74) > OK, I have reverted the local change (!!) which applied attachment 352249 [details] [diff] [review], and > reconfig'd moz2-master. Thanks Nick. I've reopened bug#441974, to figure out why some tests within the a11y suite started failing again.

We're going to turn these on in production tomorrow - stay tuned.

a11y test suite is now running in production on both 1.9.1 and m-c with success all around. Closing.

Moving closed Future bugs into Release Engineering in preparation for removing the Future component.