We should enable them now that we can almost can. We deployed Mozilla Build to all XP slaves in bug 549458#c38. In bug 563036 we deployed VC 2005 SP1 redist which enables us to run optimized unit tests. The Debug CRT will be deployed in bug 562459. We will have to verify that we don't hit bug 592806 which so it seems it was specific to staging slaves (after re-imaging the unit tests run).

I will be now be pushing this to completion. I expect this to be done in the next 2-4 weeks.

All XP test machines already have the "vc2005sp1_redist" already installed (https://bugzilla.mozilla.org/show_bug.cgi?id=563036#c17). I am running all test jobs through staging and the optimized jobs are succeeding. We could enable opt XP unit tests next week if we want to.

dbaron: would opt WinXP tests be enough to let us shut off running tests on debug Win2K3 build slaves?

Yes.

Marking as blocking the "and disable them on" half of bug 614956, then.

Thanks philor for updating the bug. WRT to comment 2. I left on Friday happy seeing two test runs go green but today I was dissapointed to see purple runs on staging; I thought the machine got wedged but to my surprise there might be something else going on. Only mochitest is failing to some odd reason. All the other test suites run to completion. I will try to run it manually when I get a chance and see what's going on. FTR here is one of the mochitest logs: python mochitest/runtests.py --appname=firefox/firefox.exe --utility-path=bin --extra-profile-file=bin/plugins --certificate-path=certs --autorun --close-when-done --console-level=INFO --symbols-path=symbols --total-chunks=5 --this-chunk=4 --chunk-by-dir=4 in dir C:\talos-slave\mozilla-central_xp_test-mochitests-4\build (timeout 1200 secs) watching logfiles {} argv: ['python', 'mochitest/runtests.py', '--appname=firefox/firefox.exe', '--utility-path=bin', '--extra-profile-file=bin/plugins', '--certificate-path=certs', '--autorun', '--close-when-done', '--console-level=INFO', '--symbols-path=symbols', '--total-chunks=5', '--this-chunk=4', '--chunk-by-dir=4'] ... args: ['C:\\talos-slave\\mozilla-central_xp_test-mochitests-4\\build\\bin\\xpcshell.exe', '-g', 'C:\\talos-slave\\mozilla-central_xp_test-mochitests-4\\build\\firefox', '-v', '170', '-f', './httpd.js', '-e', "const _PROFILE_PATH = 'c:\\\\docume~1\\\\cltbld\\\\locals~1\\\\temp\\\\tmpg1lozh';const _SERVER_PORT = '8888'; const _SERVER_ADDR ='127.0.0.1';", '-f', './server.js'] INFO | runtests.py | Server pid: 336 args: ['C:\\mozilla-build\\Python25\\python.exe', 'C:\\talos-slave\\mozilla-central_xp_test-mochitests-4\\build\\mochitest\\pywebsocket/standalone.py', '-p', '9988', '-w', 'C:\\talos-slave\\mozilla-central_xp_test-mochitests-4\\build\\mochitest', '-l', 'C:\\talos-slave\\mozilla-central_xp_test-mochitests-4\\build\\mochitest\\websock.log', '--log-level=debug'] INFO | runtests.py | Websocket server pid: 3764 INFO | runtests.py | Running tests: start. args: ['C:\\talos-slave\\mozilla-central_xp_test-mochitests-4\\build\\bin\\certutil.exe', '-N', '-d', 'c:\\docume~1\\cltbld\\locals~1\\temp\\tmpg1lozh', '-f', 'c:\\docume~1\\cltbld\\locals~1\\temp\\tmpg1lozh\\.crtdbpw'] Traceback (most recent call last): File "mochitest/runtests.py", line 759, in <module> File "mochitest/runtests.py", line 756, in main File "mochitest/runtests.py", line 615, in runTests File "C:\talos-slave\mozilla-central_xp_test-mochitests-4\build\mochitest\automation.py", line 818, in runApp certificateStatus = self.fillCertificateDB(profileDir, certPath, utilityPath, xrePath) File "C:\talos-slave\mozilla-central_xp_test-mochitests-4\build\mochitest\automation.py", line 485, in fillCertificateDB status = self.Process([certutil, "-N", "-d", profileDir, "-f", pwfilePath], env = env).wait() File "C:\talos-slave\mozilla-central_xp_test-mochitests-4\build\mochitest\automation.py", line 173, in __init__ universal_newlines, startupinfo, creationflags) File "c:\mozilla-build\Python25\lib\subprocess.py", line 593, in __init__ errread, errwrite) File "c:\mozilla-build\Python25\lib\subprocess.py", line 793, in _execute_child startupinfo) WindowsError: [Error 22] This application has failed to start because the application configuration is incorrect. Reinstalling the application may fix this problem command timed out: 1200 seconds without output remoteFailed: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion. ]

I installed the CRT manually and it worked. staging-opsi still was reporting the old talos-r3-xp-002 (*sigh*) which got re-imaged back in September. I will have to fix the entry for that slave, deploy through OPSI and try again.

The XP machines don't have the _dumbwin32proc.py patch as documented in: * https://wiki.mozilla.org/ReferencePlatforms/Win32#Downgrade_twisted_and_apply_process_group_patch * https://wiki.mozilla.org/ReferencePlatforms/Test/Win7_64-bit#Buildbot.2FTalos_toolchain I have also compared locally on C:\mozilla-build\python25\Lib\site-packages\twisted for winXP, win7 and win7x64 slaves. The last two have _dumbwin32proc.py patched unlike on WinXP slaves. I have also verified that twisted\spread\banana.py is already patched in all 3 platforms (as seen in http://hg.mozilla.org/build/twisted/raw-rev/0b441f0f68b4).

Comment on attachment 499063 [details] [diff] [review] twisted/internet/_dumbwin32proc.py - be able to kill processes on XP machines Changing the review to feedback as I still have to verify that the patch does fix the issue (it most likely will as it is the same thing that is used for all the other windows platforms).

Comment on attachment 499063 [details] [diff] [review] twisted/internet/_dumbwin32proc.py - be able to kill processes on XP machines This patch is not good enough. There is more than that. Unless you have any strong objection I will pass the patch review to coop when it is fixed.

This is what I am currently running on talos-r3-xp-00[1-4]. It matches what win7 slaves are running (both are running twisted 10.1.0). So far I am capable to "stop build" the jobs.

This is how I am deploying it: wget -Osubproc.patch --no-check-certificate https://bug614955.bugzilla.mozilla.org/attachment.cgi?id=499149 cd C:\mozilla-build\python25\Lib\site-packages C:\mozilla-build\msys\bin\patch -p1 < "C:\Documents and Settings\cltbld\subproc.patch" STATUS UPDATE: ############## * we can enable optimized unit tests for XP since the CRT was deployed long ago *but*: * we don't have the twisted patch on the XP slaves as we do for win7, win7x64 and w2k3 slaves PLAN: ##### * step1. Deploy twistd patch - If the attached patch is the right thing I would like to deploy it tomorrow if coop_buildduty agrees and the whole run on staging goes well over night. * step2. Enable opt unit tests for XP - If no fall-outs happen during Wednesday then we can enable optimized unit tests for XP on Thursday's downtime NOTE: If any of the above fails to stick this will have to wait until January as I am going on holidays this Friday.

I am not having too much luck. The test runs are being able to be stopped but the machine is not being able to reboot. The machine gets into reboot mode but it gets stopped with a Window saying: "cmd.exe - Application error; The application failed to initialize properly (0x0000142)" Once you press "OK" the machine is capable to reboot. From looking at the twistd.log the last command being executed is: > self.process.signalProcess(self.KILL) http://mxr.mozilla.org/build/source/buildbot/slave/buildslave/runprocess.py#728 I am also comparing the diff between what we run on the slaves and twisted's 10.1.0 file (http://twistedmatrix.com/trac/browser/tags/releases/twisted-10.1.0/twisted/internet/_dumbwin32proc.py) and I can only see a difference: https://bug543626.bugzilla.mozilla.org/attachment.cgi?id=451277 Which is the first patch that I attached which is the same that is used on win7 and win7x64 slaves. There is something odd for me. Why is it that I needed a second patch? If twisted 10.1.0 is what is installed on the win7 as well as the WinXP machines why did I need the second patch to match the code on win7 and WinXP? Not sure If I have over looked something. I guess I should go deep into the patch and try to understand why is there the need of a patch to begin with. 2010-12-21 14:43:44-0800 [Broker,client] startCommand:shell [id 20802] 2010-12-21 14:43:44-0800 [Broker,client] ShellCommand._startCommand 2010-12-21 14:43:44-0800 [Broker,client] python tools/buildfarm/maintenance/count_and_reboot.py -f ../reboot_count.txt -n 1 -z 2010-12-21 14:43:44-0800 [Broker,client] in dir C:\talos-slave\test\. (timeout 1200 secs) 2010-12-21 14:43:44-0800 [Broker,client] watching logfiles {} 2010-12-21 14:43:44-0800 [Broker,client] argv: ['python', 'tools/buildfarm/maintenance/count_and_reboot.py', '-f', '../reboot_count.txt', '-n' , '1', '-z'] 2010-12-21 14:43:44-0800 [Broker,client] environment: {'TMP': 'C:\\DOCUME~1\\cltbld\\LOCALS~1\\Temp', 'COMPUTERNAME': 'TALOS-R3-XP-003', 'MOZ_N O_REMOTE': '1', 'USERDOMAIN': 'TALOS-R3-XP-003', 'TACFILE': '"c:\\talos-slave\\buildbot.tac" ', 'COMMONPROGRAMFILES': 'C:\\Program Files\\Common Files', 'PROCESSOR_IDENTIFIER': 'x86 Family 6 Mod el 23 Stepping 10, GenuineIntel', 'PROGRAMFILES': 'C:\\Program Files', 'PROCESSOR_REVISION': '170a', 'SYSTEMROOT': 'C:\\WINDOWS', 'PATH': 'C:\\m ozilla-build;C:\\mozilla-build\\msys\\bin;C:\\mozilla-build\\msys\\local\\bin;C:\\mozilla-build\\Python25;C:\\mozilla-build\\Python25\\Scripts;C :\\mozilla-build\\hg;C:\\mozilla-build\\7zip;C:\\mozilla-build\\upx203w;C:\\WINDOWS\\System32;C:\\WINDOWS;', 'NO_EM_RESTART': '1', 'BB_BUILDBOT' : '"C:\\mozilla-build\\python25\\Scripts\\buildbot" ', 'XPCOM_DEBUG_BREAK': 'warn', 'TACSCRIPT': '"c:\\tools\\buildbot-helpers\\buildbot-tac.py" ', 'TEMP': 'C:\\DOCUME~1\\cltbld\\LOCALS~1\\Temp', 'PROCESSOR_ARC HITECTURE': 'x86', 'ALLUSERSPROFILE': 'C:\\Documents and Settings\\All Users', 'SESSIONNAME': 'Console', 'HOMEPATH': '\\Documents and Settings\\ cltbld', 'USERNAME': 'cltbld', 'LOGONSERVER': '\\\\TALOS-R3-XP-003', 'PROMPT': '$P$G', 'COMSPEC': 'C:\\WINDOWS\\system32\\cmd.exe', 'BOOTMODE': 'BKSTD', 'PATHEXT': '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH', 'PASSWORD': '"secret" ', 'CLIENTNAME': 'Console', 'FP_NO_HOST_CHECK': 'NO', 'WINDIR': 'C:\\WINDOWS', ' HOMEDRIVE': 'C:', 'APPDATA': 'C:\\Documents and Settings\\cltbld\\Application Data', 'TYPE': '~0,5 ', 'SYSTEMDRIVE': 'C:', 'HOSTNAME': 'talos-r3-xp-003 ', 'NUMBER_OF_PROCESSORS': '2', 'PWD': 'C:\\talos-slave\\test', 'PROCESSOR_LEVEL': '6', 'BB _PYTHON': '"C:\\mozilla-build\\python25\\Scripts\\..\\python"', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'CONTROLFILE': '"c:\\buildbot-tac.control" ', 'OS': 'Windows_NT', 'USERPROFILE': 'C:\\Document s and Settings\\cltbld'} 2010-12-21 14:43:44-0800 [Broker,client] closing stdin 2010-12-21 14:43:44-0800 [Broker,client] using PTY: False 2010-12-21 14:44:14-0800 [-] Received SIGBREAK, shutting down. 2010-12-21 14:44:14-0800 [-] stopCommand: halting current command <buildbot.slave.commands.base.SlaveShellCommand instance at 0x01359F08> 2010-12-21 14:44:14-0800 [-] command interrupted, killing pid 2744 2010-12-21 14:44:14-0800 [-] trying process.signalProcess('KILL')

This is what happens: * after we add attachment 499149 [details] [diff] [review] the unit test *step* can be aborted with "stop build" (which is not possible before) * therefore, with the patch, we can reach the reboot step which does not finish Anyone has any thoughts on why this could be happening? I won't be able to deploy anything this week and will have to wait until I am back on January. Tomorrow I will put my focus on the reboot issue. Here is the code that gets called: http://hg.mozilla.org/build/tools/file/tip/buildfarm/maintenance/count_and_reboot.py By the end of tomorrow I will put xp-003 and xp-004 in production by reverting first attachment 499149 [details] [diff] [review] from them.

This avoids the preemption of a reboot like on screenshot from attachment 499341 [details] to happen. What are your thoughts on it? This has been working on staging; can you think of any problems that this could introduce?

Comment on attachment 499589 [details] [diff] [review] [tools] force reboot and wait 0 seconds to start reboot I think some things (eg, debugger dialogs) could still hang the system up, but if this helps us avoid the hangups you're seeing, great!

Comment on attachment 499589 [details] [diff] [review] [tools] force reboot and wait 0 seconds to start reboot Sorry, thought I had reviewed this before I left.

Comment on attachment 499589 [details] [diff] [review] [tools] force reboot and wait 0 seconds to start reboot This worked on staging. Let's take it.

Comment on attachment 499589 [details] [diff] [review] [tools] force reboot and wait 0 seconds to start reboot http://hg.mozilla.org/build/tools/rev/c342e0b44940 This change will be picked up immediately by the following platforms: * w2k3, xp, win7 & win7x64

I found the root cause of XP machines not being able to kill unit test jobs. It wasn't just the twisted patch but XP in itself with the PATH that we set for unit test jobs cannot kill the jobs because taskkill.exe cannot find framedyn.dll. The reboot patch allowed us to reboot (disregarding any messages) but looking closer to the output of the runs they were not really orange as I thought. They turned orange because they timed out and they timed out because XP with the PATH given would be unable to reach framedyn.dll. After the timeout happened buildbot would be able to kill the step because of the twisted patch. This patch solves the root cause. Without including Wbem in the PATH I can't even run the jobs manually. Here is the article by Microsoft explaining that Wbem has to be in the PATH: http://support.microsoft.com/kb/319114 These are the steps to reproduce manually: > set PATH=C:\mozilla-build;C:\mozilla-build\msys\bin;C:\mozilla-build\msys\local\bin;C:\mozilla-build\Python25;C:\mozilla-build\Python25\Scripts;C:\mozilla-build\hg;C:\mozilla-build\7zip;C:\mozilla-build\upx203w;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32;C:\WINDOWS;C:\WINDOWS\System32\Wbem > wget --progress=dot:mega -N http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1294139437/firefox-4.0b9pre.en-US.win32.zip > unzip -o firefox-4.0b9pre.en-US.win32.zip > wget --progress=dot:mega -N http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1294139437/firefox-4.0b9pre.en-US.win32.tests.zip > unzip -o firefox-4.0b9pre.en-US.win32.tests.zip bin* certs* mochitest* > python mochitest/runtests.py --appname=firefox/firefox.exe --utility-path=bin --extra-profile-file=bin/plugins --certificate-path=certs --autorun --close-when-done --console-level=INFO --symbols-path=symbols --chrome

Comment on attachment 501130 [details] [diff] [review] [buildbotcustom] include C:\Windows\System32\wbem into unit tests jobs' PATH Good find. Do we need/want to revert the other patch?

(In reply to comment #23) > Comment on attachment 501130 [details] [diff] [review] > include C:\Windows\System32\wbem into unit tests jobs' PATH > > Good find. Do we need/want to revert the other patch? No, we don't want to revert it. This patch only avoids us having to deploy the twisted patch. The reboot patch deals with the "cmd.exe - Application Error" attachment 499341 [details] which is different than the "taskkill.exe - cannot find framedyn.dll".

With attachment 501130 [details] [diff] [review] and this patch we are ready to enable XP optimized unit tests.

Comment on attachment 501130 [details] [diff] [review] [buildbotcustom] include C:\Windows\System32\wbem into unit tests jobs' PATH checked-in to "default": http://hg.mozilla.org/build/buildbotcustom/rev/67d99371f50f

Comment on attachment 501130 [details] [diff] [review] [buildbotcustom] include C:\Windows\System32\wbem into unit tests jobs' PATH Merged to production-0.8: http://hg.mozilla.org/build/buildbotcustom/rev/164c9740d9af

Comment on attachment 501318 [details] [diff] [review] [buildbot-configs] enable XP optimized unit tests Landed on: "default" - http://hg.mozilla.org/build/buildbot-configs/rev/8dfecf4da9f1 "production" - http://hg.mozilla.org/build/buildbot-configs/rev/c75c494415bf

Reconfigured scheduler masters and test masters. buildbot-configs: 164c9740d9af buildbotcustom: c75c494415bf

I have announced this on the mailing lists. What is left on this bug? * file the perma-oranges we find * see what is needed to enable the debug unit tests We have to deploy the debug CRT (https://bugzilla.mozilla.org/show_bug.cgi?id=562459#c29) to be able to run debug unit tests. The problem is that currently we have 20 slaves that can't receive packages through OPSI (bug 620517). We could deploy this manually but I might have to end up be unlucky and have to deal with it :'( Those 20 slaves also have the problem that it takes them a lot of time to login. This occurs because it has a timeout for trying to reach the network device that they can't reach. This means that those slaves will always take them longer to come back online. Adding more load with debug unit tests could increase the wait times if it takes them that long to come back online. We'll see.

Reftest failures are all filed, everything but reftest is green (or known intermittent orange), and ready for you to unhide it.

For copy/paste into wiki pages purposes let me re-write the perma-oranges filed: PERMA-ORANGES (all of them reftests): * bug 623405 - anim-text-rotate-01.svg & dynamic-text-04.svg * bug 623423 - editor/xul/empty-1.xul, editor/xul/autocomplete-1.xul, editor/xul/emptyautocomplete-1.xul, editor/xul/number-3.xul, editor/xul/number-5.xul, editor/xul/numberwithvalue-1.xul * bug 623454 - reftests/bugs/580863-1.html * bug 623450 - layout/reftests/bugs/579323-1.html ** bug 623452 - layout/reftests/bugs/579985-1.html (depends on bug 623450) ** bug 623456 - layout/reftests/bugs/581317-1.html (depends on bug 623450) ** bug 623460 - reftests/ogg-video/encoded-aspect-ratio-1.html (depends on bug 623450) ** bug 623403 - gradient-live-01*.svg (depends on bug 623450) Already fixed: * bug 623459

I never uploaded this screenshot. It shows how a "taskkill.exe cannot find framedyn.dll" prompt appears and blocks buildbot from finishing that step. attachment 501130 [details] [diff] [review] fixes this problem. That is, buildbot needs to set for each step 'C:\\WINDOWS\\System32\\Wbem' as part of the PATH. In case taskkill.exe has to be called for that step to kill any processes that the step might have created (IIUC). Here is the article by Microsoft explaining that Wbem has to be in the PATH: http://support.microsoft.com/kb/319114

STATUS UPDATE: * all test suites for XP are now green * philor has disabled w2k3 from the tbox pages (he should be announcing soon)

Resuming this work. Let's add debug unit tests for XP.

I am having trouble getting the XP debug unit tests to run. Luckily ted will be able to give me a hand with this early next week.

Yay! The XP debug unit test run on staging went through and we only have 3 test suites that contain perma-oranges. TODO: File bugs for these oranges. Rev3 WINNT 5.1 mozilla-central debug test mochitest-other: http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1299276935.1299283630.13684.gz 7558 ERROR TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/layout/base/test/chrome/test_leaf_layers_partition_browser_window.xul | Leaf layers should form a non-overlapping partition of the browser window Rev3 WINNT 5.1 mozilla-central debug test reftest: http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1299283826.1299286283.25568.gz REFTEST TEST-UNEXPECTED-FAIL | file:///C:/talos-slave/test/build/reftest/tests/layout/reftests/svg/tspan-rotate-01.svg | image comparison (==) REFTEST TEST-UNEXPECTED-FAIL | file:///C:/talos-slave/test/build/reftest/tests/layout/reftests/svg/tspan-rotate-02.svg | image comparison (==) REFTEST TEST-UNEXPECTED-FAIL | file:///C:/talos-slave/test/build/reftest/tests/layout/reftests/svg/tspan-rotate-03.svg | image comparison (==) REFTEST TEST-UNEXPECTED-FAIL | file:///C:/talos-slave/test/build/reftest/tests/layout/reftests/svg/tspan-rotate-04.svg | image comparison (==) REFTEST TEST-UNEXPECTED-FAIL | file:///C:/talos-slave/test/build/reftest/tests/layout/reftests/svg/tspan-rotate-05.svg | image comparison (==) REFTEST TEST-UNEXPECTED-FAIL | file:///C:/talos-slave/test/build/reftest/tests/layout/reftests/svg/tspan-rotate-06.svg | image comparison (==) Rev3 WINNT 5.1 mozilla-central debug test mochitests-2/5: http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1299273603.1299275641.14918.gz&fulltext=1 6366 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/scriptaculous/test_Scriptaculous.html | testInPlaceEditor - 31 assertions, 2 failures, 0 errors 6367 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/scriptaculous/test_Scriptaculous.html | testInPlaceEditor - 31 assertions, 2 failures, 0 errors

I run it a second time. (In reply to comment #37) > Rev3 WINNT 5.1 mozilla-central debug test mochitest-other: > http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1299276935.1299283630.13684.gz > 7558 ERROR TEST-UNEXPECTED-FAIL | > chrome://mochitests/content/chrome/layout/base/test/chrome/test_leaf_layers_partition_browser_window.xul > | Leaf layers should form a non-overlapping partition of the browser window > This is the only one failure that has repeated from the previous run. http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1299543394.1299546049.14379.gz

Comment on attachment 517753 [details] [diff] [review] [buildbot-configs] enable XP debug unit tests checked-in to default without the code from another patch. http://hg.mozilla.org/build/buildbot-configs/rev/9dd3078a9ac4 This will be picked up with Thursday morning's scheduled reconfig.

This got enabled today. And the suites are looking green! (except M1 & Mo) http://hg.mozilla.org/build/buildbot-configs/rev/f1a4d01c8caa

Is it known that Mo is hanging in mochitest-browser-chrome ? It generates 50M of log, at which point we clamp the output, but the job continues at least 20 hours. Lots of js stacks from Places in the logs I think.

I'd expect that to be (did expect that it was, in the logs I glanced at) ye olde bug 631447 - does that do the same things when it hits non-XP?

Sort of difficult to tell frequency, since they also expanded bug 629610 into browser_focus_steal_from_chrome.js, which kills the run before bug 631447 has a chance to overflow the log. Between them, those two are not actually quite permanent, though, since http://tbpl.mozilla.org/?tree=MozillaTry&noignore=1&rev=a1cb76b70af6 did manage to succeed in browser-chrome (the only thing on m-c or try which has, so far).

For Mo we have: - bug 640889 - chrome -> 7558 ERROR TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/layout/base/test/chrome/test_leaf_layers_partition_browser_window.xul | Leaf layers should form a non-overlapping partition of the browser window - no bug? - browser-chrome -> TEST-UNEXPECTED-FAIL | chrome://mochitests/content/browser/dom/tests/browser/browser_focus_steal_from_chrome_during_mousedown.js | Exited with code -1073741819 during test run For M1 we have (which is green on the last changeset): - bug 640887 36170 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | Test expected to fail, but passed: conformance/canvas-test.html I am thinking of disabling browser-chrome for Windows XP until we figure it out on staging. philor what do you say? I know it is intermittent but quite frequent and the slaves get stuck until someone reboots the machine. Is this right nthomas? We would also need the twistd patch at some point to be able to stop the jobs that get into a similar state.

I am rebooting XP machines that are getting stuck with mochitest-other jobs. I will put a patch as soon as I can.

Seems reasonable to me - mak has a handle on it, but it's not a test-only patch so we're stuck waiting for the tree to reopen (and then we'd still be stuck waiting for people to merge it around to every tree).