MozillaTry is currently not displaying available info on several Talos boxes. The error does not appear to be on the side of the builders as other Talos machines with the same configuration are successfully adding their information. Missing columns: qm-pubuntu-try01 qm-pubunut-try02 qm-pxp-try01 Several other columns were missing but reed resurrected them through running a cleaning script.

cls, I'll get you another tarball.

More missing columns: qm-pxp-try02 qm-pvista-try01 qm-pvista-try02

I'm not sure why I'd need another tarball. The next time the problem occurs, do: tail -200 build.dat | perl -e 'while(<STDIN>) { chomp; ($e,$b,$n) = split(/\|/,$_); $t=localtime($b); print "$t -- $n\n";}' If you see a bunch of dates older than 14 hrs or so, then you're hitting bug 225735 .

Well, it is still happening right now - as those 6 columns are not being displayed. Is there anything that can be done to get them back on board?

[root@dm-webtools02 MozillaTry]# tail -200 build.dat | perl -e 'while(<STDIN>) { chomp; ($e,$b,$n) = > split(/\|/,$_); $t=localtime($b); print "$t -- $n\n";}' Thu Mar 26 04:04:06 2009 -- Try server linux hg builder Thu Mar 26 01:52:39 2009 -- Try linux hg unittest builder Thu Mar 26 04:05:09 2009 -- Try server linux hg builder Thu Mar 26 02:04:19 2009 -- Try linux hg unittest builder Thu Mar 26 01:53:39 2009 -- Try win32 hg unittest builder Thu Mar 26 04:17:41 2009 -- Try server win32 hg builder Thu Mar 26 04:19:53 2009 -- Try server linux hg builder Thu Mar 26 02:16:39 2009 -- Try linux hg unittest builder Thu Mar 26 04:23:55 2009 -- Try server win32 hg builder Thu Mar 26 01:53:12 2009 -- Try win32 hg unittest builder Wed Mar 25 23:07:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try01 Thu Mar 26 04:36:50 2009 -- Try server win32 hg builder Thu Mar 26 01:54:36 2009 -- Try win32 hg unittest builder Wed Mar 25 15:59:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01 Wed Mar 25 16:26:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01 Wed Mar 25 15:59:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02 Wed Mar 25 16:26:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02 Thu Mar 26 03:18:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try02 Thu Mar 26 03:18:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try01 Thu Mar 26 04:52:09 2009 -- Try server mac hg builder Thu Mar 26 04:52:10 2009 -- Try macosx hg unittest builder Thu Mar 26 03:19:00 2009 -- Try server mac hg builder Thu Mar 26 04:54:03 2009 -- Try server win32 hg builder Thu Mar 26 04:00:00 2009 -- Try server win32 hg builder Wed Mar 25 13:55:00 2009 -- WINNT 5.1 talos try qm-pxp-try02 Wed Mar 25 13:55:00 2009 -- WINNT 6.0 talos try qm-pvista-try01 Wed Mar 25 13:55:00 2009 -- WINNT 5.1 talos try qm-pxp-try01 Wed Mar 25 13:55:00 2009 -- WINNT 6.0 talos try qm-pvista-try02 Wed Mar 25 14:52:00 2009 -- Linux talos try qm-pubuntu-try01 Wed Mar 25 14:52:00 2009 -- Linux talos try qm-pubuntu-try02 Thu Mar 26 04:19:53 2009 -- Try server linux hg builder Thu Mar 26 05:09:06 2009 -- Try server linux hg builder Thu Mar 26 05:20:07 2009 -- Try linux hg unittest builder Thu Mar 26 03:56:49 2009 -- Try server linux hg builder Thu Mar 26 05:22:50 2009 -- Try server linux hg builder Thu Mar 26 04:04:06 2009 -- Try server linux hg builder Thu Mar 26 05:26:09 2009 -- Try linux hg unittest builder Thu Mar 26 04:05:09 2009 -- Try server linux hg builder Thu Mar 26 03:19:00 2009 -- Try macosx hg unittest builder Wed Mar 25 16:26:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01 Wed Mar 25 19:25:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01 Wed Mar 25 16:26:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02 Wed Mar 25 19:25:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02 Thu Mar 26 03:18:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try02 Thu Mar 26 03:18:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try01 Wed Mar 25 14:52:00 2009 -- Linux talos try qm-pubuntu-try01 Wed Mar 25 14:52:00 2009 -- Linux talos try qm-pubuntu-try02 Wed Mar 25 13:14:00 2009 -- Linux talos try qm-pubuntu-try02 Wed Mar 25 13:14:00 2009 -- Linux talos try qm-pubuntu-try01 Thu Mar 26 05:59:27 2009 -- Try server linux hg builder Thu Mar 26 05:09:06 2009 -- Try server linux hg builder Wed Mar 25 15:59:00 2009 -- WINNT 5.1 talos try qm-pxp-try01 Wed Mar 25 15:59:00 2009 -- WINNT 5.1 talos try qm-pxp-try02 Wed Mar 25 15:59:00 2009 -- WINNT 6.0 talos try qm-pvista-try02 Wed Mar 25 15:59:00 2009 -- WINNT 6.0 talos try qm-pvista-try01 Wed Mar 25 15:59:00 2009 -- WINNT 5.1 talos try qm-pxp-try02 Wed Mar 25 15:59:00 2009 -- WINNT 5.1 talos try qm-pxp-try01 Wed Mar 25 15:59:00 2009 -- WINNT 6.0 talos try qm-pvista-try01 Wed Mar 25 15:59:00 2009 -- WINNT 6.0 talos try qm-pvista-try02 Thu Mar 26 04:52:10 2009 -- Try macosx hg unittest builder Thu Mar 26 06:17:37 2009 -- Try win32 hg unittest builder Thu Mar 26 04:23:55 2009 -- Try server win32 hg builder Wed Mar 25 19:25:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01 Wed Mar 25 23:07:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01 Wed Mar 25 19:25:00 2009 -- WINNT 5.1 talos try qm-pxp-try02 Wed Mar 25 19:25:00 2009 -- WINNT 6.0 talos try qm-pvista-try01 Wed Mar 25 19:25:00 2009 -- WINNT 6.0 talos try qm-pvista-try02 Wed Mar 25 19:25:00 2009 -- WINNT 5.1 talos try qm-pxp-try01 Thu Mar 26 04:51:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try01 Thu Mar 26 04:51:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try02 Thu Mar 26 04:52:09 2009 -- Try server mac hg builder Thu Mar 26 06:32:00 2009 -- Try server mac hg builder Thu Mar 26 06:32:00 2009 -- Try macosx hg unittest builder Wed Mar 25 19:25:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02 Wed Mar 25 23:07:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02 Thu Mar 26 04:51:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try01 Thu Mar 26 04:51:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try02 Thu Mar 26 06:37:00 2009 -- Try macosx hg unittest builder Thu Mar 26 06:37:01 2009 -- Try server mac hg builder Wed Mar 25 13:14:00 2009 -- Linux talos try qm-pubuntu-try02 Wed Mar 25 13:14:00 2009 -- Linux talos try qm-pubuntu-try01 Thu Mar 26 06:40:45 2009 -- Try server win32 hg builder Thu Mar 26 04:54:03 2009 -- Try server win32 hg builder Wed Mar 25 13:55:00 2009 -- Linux talos try qm-pubuntu-try02 Wed Mar 25 13:55:00 2009 -- Linux talos try qm-pubuntu-try01 Thu Mar 26 06:42:30 2009 -- Try linux hg unittest builder Thu Mar 26 05:22:50 2009 -- Try server linux hg builder Thu Mar 26 06:43:32 2009 -- Try win32 hg unittest builder Thu Mar 26 04:17:41 2009 -- Try server win32 hg builder Thu Mar 26 04:36:50 2009 -- Try server win32 hg builder Thu Mar 26 06:44:33 2009 -- Try server win32 hg builder Thu Mar 26 06:57:29 2009 -- Try server linux hg builder Thu Mar 26 05:59:27 2009 -- Try server linux hg builder Wed Mar 25 23:07:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01 Thu Mar 26 03:18:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01 Thu Mar 26 05:26:09 2009 -- Try linux hg unittest builder Thu Mar 26 07:22:19 2009 -- Try linux hg unittest builder Wed Mar 25 23:07:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02 Wed Mar 25 13:55:00 2009 -- Linux talos try qm-pubuntu-try02 Wed Mar 25 13:55:00 2009 -- Linux talos try qm-pubuntu-try01 Thu Mar 26 03:18:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02 Wed Mar 25 14:20:00 2009 -- Linux talos try qm-pubuntu-try02 Wed Mar 25 14:20:00 2009 -- Linux talos try qm-pubuntu-try01 Wed Mar 25 19:25:00 2009 -- WINNT 5.1 talos try qm-pxp-try02 Wed Mar 25 19:25:00 2009 -- WINNT 6.0 talos try qm-pvista-try01 Wed Mar 25 19:25:00 2009 -- WINNT 6.0 talos try qm-pvista-try02 Wed Mar 25 19:25:00 2009 -- WINNT 5.1 talos try qm-pxp-try01 Wed Mar 25 14:52:00 2009 -- WINNT 6.0 talos try qm-pvista-try01 Wed Mar 25 14:52:00 2009 -- WINNT 5.1 talos try qm-pxp-try02 Wed Mar 25 14:52:00 2009 -- WINNT 6.0 talos try qm-pvista-try02 Wed Mar 25 14:52:00 2009 -- WINNT 5.1 talos try qm-pxp-try01 Thu Mar 26 07:50:29 2009 -- Try server linux hg builder Thu Mar 26 06:57:29 2009 -- Try server linux hg builder Thu Mar 26 07:57:44 2009 -- Try server mac hg builder Thu Mar 26 06:32:00 2009 -- Try macosx hg unittest builder Thu Mar 26 08:03:47 2009 -- Try macosx hg unittest builder Thu Mar 26 06:37:00 2009 -- Try macosx hg unittest builder Thu Mar 26 03:18:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01 Thu Mar 26 04:51:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01 Thu Mar 26 06:35:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try01 Thu Mar 26 06:35:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try02 Thu Mar 26 06:37:01 2009 -- Try server mac hg builder Wed Mar 25 14:20:00 2009 -- Linux talos try qm-pubuntu-try02 Wed Mar 25 14:20:00 2009 -- Linux talos try qm-pubuntu-try01 Wed Mar 25 15:59:00 2009 -- Linux talos try qm-pubuntu-try02 Wed Mar 25 15:59:00 2009 -- Linux talos try qm-pubuntu-try01 Thu Mar 26 06:32:00 2009 -- Try server mac hg builder Thu Mar 26 03:18:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02 Thu Mar 26 04:51:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02 Thu Mar 26 08:18:03 2009 -- Try linux hg unittest builder Thu Mar 26 05:20:07 2009 -- Try linux hg unittest builder Thu Mar 26 04:51:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01 Thu Mar 26 06:35:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try02 Thu Mar 26 06:35:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01 Thu Mar 26 06:35:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try01 Thu Mar 26 06:30:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try02 Thu Mar 26 06:30:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try01 Thu Mar 26 08:26:42 2009 -- Try macosx hg unittest builder Thu Mar 26 04:51:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02 Thu Mar 26 08:26:41 2009 -- Try server mac hg builder Thu Mar 26 06:35:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02 Thu Mar 26 06:35:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01 Thu Mar 26 06:30:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01 Thu Mar 26 06:35:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02 Thu Mar 26 06:30:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02 Thu Mar 26 08:45:36 2009 -- Try server linux hg builder Thu Mar 26 07:50:29 2009 -- Try server linux hg builder Thu Mar 26 08:56:27 2009 -- Try win32 hg unittest builder Thu Mar 26 06:17:37 2009 -- Try win32 hg unittest builder Wed Mar 25 14:52:00 2009 -- WINNT 5.1 talos try qm-pxp-try02 Wed Mar 25 15:59:00 2009 -- Linux talos try qm-pubuntu-try01 Wed Mar 25 15:59:00 2009 -- Linux talos try qm-pubuntu-try02 Wed Mar 25 14:52:00 2009 -- WINNT 6.0 talos try qm-pvista-try01 Wed Mar 25 14:52:00 2009 -- WINNT 6.0 talos try qm-pvista-try02 Wed Mar 25 16:26:00 2009 -- Linux talos try qm-pubuntu-try01 Wed Mar 25 16:26:00 2009 -- WINNT 6.0 talos try qm-pvista-try01 Wed Mar 25 16:26:00 2009 -- WINNT 5.1 talos try qm-pxp-try02 Wed Mar 25 14:52:00 2009 -- WINNT 5.1 talos try qm-pxp-try01 Wed Mar 25 16:26:00 2009 -- Linux talos try qm-pubuntu-try02 Thu Mar 26 09:01:41 2009 -- Try server win32 hg builder Wed Mar 25 16:26:00 2009 -- WINNT 6.0 talos try qm-pvista-try02 Thu Mar 26 06:40:45 2009 -- Try server win32 hg builder Wed Mar 25 16:26:00 2009 -- WINNT 5.1 talos try qm-pxp-try01 Thu Mar 26 06:30:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try01 Thu Mar 26 07:22:19 2009 -- Try linux hg unittest builder Thu Mar 26 06:30:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try02 Thu Mar 26 09:11:42 2009 -- Try linux hg unittest builder Thu Mar 26 09:13:33 2009 -- Try win32 hg unittest builder Thu Mar 26 06:44:33 2009 -- Try server win32 hg builder Thu Mar 26 06:30:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01 Thu Mar 26 09:25:45 2009 -- Try server linux hg builder Thu Mar 26 06:42:30 2009 -- Try linux hg unittest builder Thu Mar 26 06:30:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02 Thu Mar 26 09:34:34 2009 -- Try server mac hg builder Thu Mar 26 07:57:44 2009 -- Try server mac hg builder Thu Mar 26 08:45:36 2009 -- Try server linux hg builder Thu Mar 26 09:38:36 2009 -- Try linux hg unittest builder Wed Mar 25 16:26:00 2009 -- Linux talos try qm-pubuntu-try01 Wed Mar 25 16:26:00 2009 -- Linux talos try qm-pubuntu-try02 Wed Mar 25 19:25:00 2009 -- Linux talos try qm-pubuntu-try01 Wed Mar 25 19:25:00 2009 -- Linux talos try qm-pubuntu-try02 Thu Mar 26 08:26:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try01 Thu Mar 26 08:26:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try01 Thu Mar 26 08:26:00 2009 -- MacOSX Darwin 8.8.1 talos try qm-ptiger-try02 Thu Mar 26 08:26:00 2009 -- MacOSX Darwin 9.2.2 talos try qm-pleopard-try02 Thu Mar 26 10:08:34 2009 -- Try macosx hg unittest builder Thu Mar 26 08:26:41 2009 -- Try server mac hg builder Thu Mar 26 10:11:01 2009 -- Try server mac hg builder Thu Mar 26 08:03:47 2009 -- Try macosx hg unittest builder Thu Mar 26 10:13:29 2009 -- Try server win32 hg builder Thu Mar 26 06:43:32 2009 -- Try win32 hg unittest builder Wed Mar 25 16:26:00 2009 -- WINNT 5.1 talos try qm-pxp-try02 Wed Mar 25 16:26:00 2009 -- WINNT 6.0 talos try qm-pvista-try01 Thu Mar 26 10:11:01 2009 -- Try server mac hg builder Thu Mar 26 10:16:50 2009 -- Try macosx hg unittest builder Wed Mar 25 23:07:00 2009 -- WINNT 5.1 talos try qm-pxp-try02 Wed Mar 25 16:26:00 2009 -- WINNT 6.0 talos try qm-pvista-try02 Wed Mar 25 23:07:00 2009 -- WINNT 6.0 talos try qm-pvista-try01 Wed Mar 25 16:26:00 2009 -- WINNT 5.1 talos try qm-pxp-try01 Wed Mar 25 23:07:00 2009 -- WINNT 6.0 talos try qm-pvista-try02

It looks like the problem doesn't occur in the last 200 build mails. 20 consecutive 'greater than 14 hr difference' builds is what causes tbox to stop reading build.dat. However, you clearly have a problem with the clocks on your servers. They shouldn't be jumping around like that. I'd expect them to consistently increase. (That's what they do on my local trees anyway.)

Is the time shown the start time of the build ?

Developers rely on the tryserver now. It's instrumental in keeping the tree healthy, so requesting that this be a P1 blocker.

updating summary to reflect the current state of the waterfall page.

(In reply to comment #8) > Developers rely on the tryserver now. It's instrumental in keeping the tree > healthy, so requesting that this be a P1 blocker. The Firefox 3.0.8 release is taking priority over this, so not sure when I'll be able to get to it. I'll get it done sometime later today, though (best I can do is remake the tree).

I ran the clean script again with only 3 days of logs kept, but there's obviously issues with the build machines, so passing this to RE to fix the time on all the machines.

guh, I ran the clean script last week as root rather than tinderbox, which caused tinderbox to not be able to update build.dat due to permission issues (plus, there was a stale semaphore lock file). While I don't know why this problem started, I at least know why it hasn't resolved itself yet, and I've fixed that.

Looks like the win32 talos machines have disappeared again...

I just fixed bug 485015. Are you still seeing this?