Seems to be an error with file permissions or the talos python script. Odd that they all went at the same time, doesn't appear to be anything changed on that front (config or script) since yesterday. Tree is closed until this is resolved. Traceback (most recent call last): File "run_tests.py", line 345, in ? test_file(arg) File "run_tests.py", line 315, in test_file res, browser_dump, counter_dump = ttest.runTest(browser_config, tests[test]) File "c:\talos-slave\vista-trunk-nochrome\talos\ttest.py", line 230, in runTest shutil.rmtree(temp_dir) File "C:\Python24\lib\shutil.py", line 163, in rmtree rmtree(fullname, ignore_errors, onerror) File "C:\Python24\lib\shutil.py", line 163, in rmtree rmtree(fullname, ignore_errors, onerror) File "C:\Python24\lib\shutil.py", line 168, in rmtree onerror(os.remove, fullname, sys.exc_info()) File "C:\Python24\lib\shutil.py", line 166, in rmtree os.remove(fullname) OSError: [Errno 13] Permission denied: 'c:\\users\\mozill~1\\appdata\\local\\temp\\tmpcqt51m\\profile\\Cache\\_CACHE_001_'

There is nothing in the IT support document at http://wiki.mozilla.org/Buildbot/IT_Talos_Support_Document that covers this situation. Handing off to QA.

joduinn has been paged.

alice has been paged as well.

joduinn's on the phone with me right now, and alice is on IRC and sounds like she's got it tracked down already.

OK, branch is fine it's only the 5 trunk talos boxes affected by this.

The errors that talos is throwing indicate that it is attempting to clean up a profile directory that is still locked and in use. In this case, since all the machines went down at once on the same build, I'd assume that something was checked in that caused the browser to either a) close slower or b) close in an unclean fashion. I've checked in a talos patch that I was working on in bug 416911 that should handle/mask this problem by having talos do a better job of killing browser processes at the end of testing - but I still believe that there was a browser regression.

I got to see some of the winxp machines run tests and found the problem. A dialog is being raised during Ts: Internet Security A script from "file://" is requesting enhanced abilities that are UNSAFE and could be used to compromise your machine or data:" From talk on irc it seems that we tightened up security around file:// links. That change is interfering with the operation of talos tests that use file load.

I've made local changes to all the production talos config files to set security.fileuri.strict_origin_policy : false That should override the new security setting and allow talos to complete tests.

Filed bug 424594 to have talos comply with new security restrictions.

These machines are all green now due to the temporary fix in comment #7. Follow up with bug 424594 for talos complying with new file:// security.