These failures started on 2011-09-05 (Wednesday). On this push: https://tbpl.mozilla.org/php/getParsedLog.php?id=6697342&tree=Mozilla-Beta ...there was a green old_tp run at 13:00 that day: https://tbpl.mozilla.org/php/getParsedLog.php?id=6689229&tree=Mozilla-Beta ...but by 21:56 it was red on the same push, same platform: https://tbpl.mozilla.org/php/getParsedLog.php?id=6697342&tree=Mozilla-Beta and has been perma-red ever since. Example log: https://tbpl.mozilla.org/php/getParsedLog.php?id=6717876&tree=Mozilla-Beta Rev3 MacOSX Leopard 10.5.8 mozilla-beta talos old_tp on 2011-10-06 20:49:54 PDT for push 570515d49188 ========= Started 'python run_tests.py ...' failed (results: 2, elapsed: 27 secs) ========== python run_tests.py --noisy 20111006_2012_config.yml in dir /home/cltbld/talos-slave/test/../talos-data/talos/ (timeout 21600 secs) watching logfiles {} argv: ['python', 'run_tests.py', '--noisy', '20111006_2012_config.yml'] environment: CVS_RSH=ssh DISPLAY=:0 G_BROKEN_FILENAMES=1 HISTCONTROL=ignoreboth HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=talos-r3-fed-063.build.mozilla.org LANG=en_US.UTF-8 LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld MAIL=/var/spool/mail/cltbld MOZ_CRASHREPORTER_NO_REPORT=1 MOZ_NO_REMOTE=1 NO_EM_RESTART=1 PATH=/home/cltbld/bin:/tools/buildbot-0.8.0/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin PWD=/home/cltbld/talos-slave/talos-data/talos SHELL=/bin/bash SHLVL=1 SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass TERM=xterm USER=cltbld XPCOM_DEBUG_BREAK=warn _=/home/cltbld/bin/python using PTY: False RETURN:s: talos-r3-fed-063 RETURN:id:20111006165549 RETURN:<a href = "http://hg.mozilla.org/releases/mozilla-beta/rev/8f6cce824102">rev:8f6cce824102</a> talos-r3-fed-063: Started Thu, 06 Oct 2011 20:12:33 Running test tp5: Started Thu, 06 Oct 2011 20:12:33 Screen width/height:1600/1200 colorDepth:24 Browser inner width/height: 1024/650 NOISE: [Exception... "Component returned failure code: 0x80520012 (NS_ERROR_FILE_NOT_FOUND) [nsIFileInputStream.init]" nsresult: "0x80520012 (NS_ERROR_FILE_NOT_FOUND)" location: "JS frame :: chrome://pageloader/content/pageloader.js :: plLoadURLsFromURI :: line 605" data: no] NOISE: __startSecondTimestamp1317957170603__endSecondTimestamp Traceback (most recent call last): File "run_tests.py", line 540, in <module> test_file(arg, screen, amo) File "run_tests.py", line 485, in test_file browser_dump, counter_dump, print_format = mytest.runTest(browser_config, test) File "/home/cltbld/talos-slave/talos-data/talos/ttest.py", line 290, in runTest cm = self.cmanager.CounterManager(self._ffprocess, browser_config['process'], counters) File "/home/cltbld/talos-slave/talos-data/talos/cmanager_linux.py", line 150, in __init__ self.primaryPid = self.ffprocess.GetPidsByName(process)[-1] IndexError: list index out of range program finished with exit code 1 elapsedTime=26.898262 TinderboxPrint:s: talos-r3-fed-063 TinderboxPrint:id:20111006165549 TinderboxPrint:<a href = "http://hg.mozilla.org/releases/mozilla-beta/rev/8f6cce824102">rev:8f6cce824102</a> ======== Finished 'python run_tests.py ...' failed (results: 2, elapsed: 27 secs) ========

This is probabaly a fallout from the PGO changes last night for releases. A mozconfig[1] change landed at Oct 06 at 17:44:15 -0700 A better way to look at this is: https://tbpl.mozilla.org/?tree=Mozilla-Beta&noignore=1&jobname=old_tp mbrubeck thanks for filing this. jmaher, could you please give me a hand at looking at the error? old_tp is the only suite that started failing. As mbrubeck says: * The last round that went green was on Oct. 6th at 15:04 PDT * The first one that went red was Oct. 6th at 21:56 PDT - retriggered for the same changeset This cannot be seen anywhere else as we don't run old_tp anywhere else anymore. Removing "randomorange" dependency as this is a "permared". [1] http://hg.mozilla.org/build/buildbot-configs/rev/876f80347ed7

Hidden on tbpl since apparently nobody's going to stop pushing just because the tree is covered with unstarred red, and all we're doing is training them that it's fine to do so.

odd, I ensured I had the latest talos and pageloader bits from hg, and ran tp5 (without mozafterpaint- old_tp5 parity) against a mozilla-beta build. This ran with no problems. I even copied the command lines from the tbpl log for a failing instance and just changed the location of the firefox binary as well as the graph servers. There has to be something wrong in the way we have things setup, but really odd that this is only old_tp5.

(In reply to Joel Maher (:jmaher) from comment #3) > odd, I ensured I had the latest talos and pageloader bits from hg, ... Did you take into account that the tip of the repo has changes since the last update on production ? True of talos at least.

maybe tp5 pageset isn't download and/or unzipped? this error in pageloader.js is for when we are trying to open the tp5.manifest file.

We disabled old_tp for beta a couple of weeks ago.