In bug 1049227 we had some masters where buildbot had been running for 3 months, and the memory consumption had gradually increased to the point that buildbot was really slow and using almost all the RAM. Please add a process age check to nagios that alerts after one month. Looks like /usr/lib64/nagios/plugins/check_procs -m ELAPSED -w 2678400 -C buildbot would do the trick. Puppet patch coming up.

Warn after 1 month. OK with you coop ?

Comment on attachment 8476275 [details] [diff] [review] [puppet] Add check on masters Landed: https://hg.mozilla.org/build/puppet/rev/14d2f770dc65 https://hg.mozilla.org/build/puppet/rev/2acea504ac23 Server ops, please add a check that uses command check_buildbot_age on all members of the buildbot master host group (sp?). Doesn't need any parameters passed in.

nthomas, Are these the hostgroups in question? 'dev-buildbot-masters', 'scl3-production-buildbot-masters', 'use1-production-buildbot-masters', 'usw2-production-buildbot-masters'

(In reply to dgarvey :dgarvey from comment #3) > 'scl3-production-buildbot-masters', > 'use1-production-buildbot-masters', > 'usw2-production-buildbot-masters' These ones please.

I have access to the hosts this time;) [dgarvey@buildbot-master81.srv.releng.scl3.mozilla.com ~]$ cat /etc/nagios/nrpe.d/buildbot.cfg # This Source Code Form is subject to the terms of the Mozilla Public # License, v. 2.0. If a copy of the MPL was not distributed with this # file, You can obtain one at http://mozilla.org/MPL/2.0/. command[check_buildbot]=/usr/lib64/nagios/plugins/check_procs -c 2:2 -C buildbot command[check_buildbot_age]=/usr/lib64/nagios/plugins/check_procs -m ELAPSED -w 2678400 -C buildbot [dgarvey@buildbot-master81.srv.releng.scl3.mozilla.com ~]$ /usr/lib64/nagios/plugins/check_procs -m ELAPSED -w 2678400 -C buildbot ELAPSED WARNING: 2 warn out of 2 processes with command name 'buildbot' [dgarvey@buildbot-master81.srv.releng.scl3.mozilla.com ~]$ echo $? 1 [dgarvey@buildbot-master81.srv.releng.scl3.mozilla.com ~]$ ps -efww | grep nrpe dgarvey 20232 16987 0 22:06 pts/0 00:00:00 grep nrpe nagios 26450 1 0 14:33 ? 00:00:01 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d [dgarvey@buildbot-master81.srv.releng.scl3.mozilla.com ~]$ cat /etc/nagios/nrpe.cfg # This Source Code Form is subject to the terms of the Mozilla Public # License, v. 2.0. If a copy of the MPL was not distributed with this # file, You can obtain one at http://mozilla.org/MPL/2.0/. #### This file under configuration management control #### DO NOT EDIT MANUALLY pid_file=/var/run/nrpe.pid server_port=5666 nrpe_user=nagios nrpe_group=nagios allowed_hosts=127.0.0.1,10.26.75.30 dont_blame_nrpe=1 include_dir=/etc/nagios/nrpe.d command_timeout=60 # basic, common checks command[check_load]=/usr/lib64/nagios/plugins/check_load -w $ARG1$ -c $ARG2$ command[check_users]=/usr/lib64/nagios/plugins/check_users -w $ARG1$ -c $ARG2$ command[check_disk]=/usr/lib64/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$ command[check_inodes]=/usr/lib64/nagios/plugins/check_disk -W $ARG1$ -C $ARG2$ -p $ARG3$ command[check_procs]=/usr/lib64/nagios/plugins/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$ command[check_file_age]=/usr/lib64/nagios/plugins/check_file_age $ARG1$ [dgarvey@buildbot-master81.srv.releng.scl3.mozilla.com ~]$ dig -x 10.26.75.30 ; <<>> DiG 9.7.3-P3-RedHat-9.7.3-8.P3.el6_2.2 <<>> -x 10.26.75.30 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59289 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 2 ;; QUESTION SECTION: ;30.75.26.10.in-addr.arpa. IN PTR ;; ANSWER SECTION: 30.75.26.10.in-addr.arpa. 3600 IN PTR nagios1.private.releng.scl3.mozilla.com. ;; AUTHORITY SECTION: 75.26.10.in-addr.arpa. 300 IN NS ns2.mozilla.org. 75.26.10.in-addr.arpa. 300 IN NS ns1.mozilla.org. ;; ADDITIONAL SECTION: ns1.mozilla.org. 60 IN A 63.245.215.5 ns2.mozilla.org. 60 IN A 63.245.218.7 ;; Query time: 13 msec ;; SERVER: 10.26.75.40#53(10.26.75.40) ;; WHEN: Thu Aug 21 22:07:21 2014 ;; MSG SIZE rcvd: 174 [dgarvey@buildbot-master81.srv.releng.scl3.mozilla.com ~]$ dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/modules/nagios$ svn commit -m "updated for bug 1056348" X11 forwarding request failed on channel 0 Sending manifests/mozilla/checkcommands.pp Sending manifests/releng/services.pp Transmitting file data .. Committed revision 92348. dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/modules/nagios$

Getting alerts in #buildduty channel of IRC. Is this related? nagios-releng 08:41:58 Thu 23:41:58 PDT [4565] buildbot-master102.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:41:59 Thu 23:41:58 PDT [4566] buildbot-master100.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:41:59 Thu 23:41:58 PDT [4567] buildbot-master106.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:41:59 Thu 23:41:58 PDT [4568] buildbot-master104.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:42:02 Thu 23:41:58 PDT [4569] buildbot-master108.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:42:05 Thu 23:41:58 PDT [4570] buildbot-master112.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:42:08 Thu 23:41:58 PDT [4571] buildbot-master05.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:42:11 Thu 23:41:59 PDT [4572] buildbot-master52.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:42:14 Thu 23:41:59 PDT [4573] buildbot-master67.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:42:17 Thu 23:41:59 PDT [4574] buildbot-master01.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:42:20 Thu 23:41:59 PDT [4575] buildbot-master03.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:42:23 Thu 23:41:59 PDT [4576] buildbot-master84.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:42:26 Thu 23:41:59 PDT [4577] buildbot-master82.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:42:29 Thu 23:41:59 PDT [4578] buildbot-master79.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:42:32 Thu 23:41:59 PDT [4579] buildbot-master73.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:42:35 Thu 23:41:59 PDT [4580] buildbot-master75.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:42:38 Thu 23:41:59 PDT [4581] buildbot-master77.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:42:41 Thu 23:41:59 PDT [4582] buildbot-master71.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:42:44 Thu 23:41:59 PDT [4583] buildbot-master88.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:42:47 Thu 23:41:59 PDT [4584] buildbot-master99.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:42:50 Thu 23:42:00 PDT [4585] buildbot-master86.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:42:53 Thu 23:42:00 PDT [4586] buildbot-master91.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:44:49 Thu 23:44:48 PDT [4587] cruncher.srv.releng.scl3.mozilla.com:Pending builds is WARNING: WARNING Pending Builds: 4417 (http://m.mozilla.org/Pending+builds) 08:46:09 Thu 23:46:09 PDT [4588] buildbot-master101.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:46:09 Thu 23:46:09 PDT [4589] buildbot-master103.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:46:09 Thu 23:46:09 PDT [4590] buildbot-master06.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:46:09 Thu 23:46:09 PDT [4591] buildbot-master02.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:46:12 Thu 23:46:09 PDT [4592] buildbot-master04.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:46:15 Thu 23:46:09 PDT [4593] buildbot-master105.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:46:18 Thu 23:46:09 PDT [4594] buildbot-master107.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:46:21 Thu 23:46:09 PDT [4595] buildbot-master117.bb.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:46:24 Thu 23:46:19 PDT [4596] buildbot-master83.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:46:27 Thu 23:46:19 PDT [4597] buildbot-master85.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:46:30 Thu 23:46:19 PDT [4598] buildbot-master89.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:46:33 Thu 23:46:19 PDT [4599] buildbot-master87.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:46:36 Thu 23:46:19 PDT [4600] buildbot-master81.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 2 warn out of 2 processes with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:46:39 Thu 23:46:19 PDT [4601] buildbot-master72.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:46:42 Thu 23:46:19 PDT [4602] buildbot-master53.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:46:45 Thu 23:46:19 PDT [4603] buildbot-master68.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:46:48 Thu 23:46:19 PDT [4604] buildbot-master74.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:46:51 Thu 23:46:19 PDT [4605] buildbot-master78.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:46:54 Thu 23:46:20 PDT [4606] buildbot-master51.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:46:57 Thu 23:46:20 PDT [4607] buildbot-master70.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:47:00 Thu 23:46:20 PDT [4608] buildbot-master94.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 08:47:03 Thu 23:46:20 PDT [4609] buildbot-master76.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) nagios-releng 09:11:23 Fri 00:11:23 PDT [4610] cruncher.srv.releng.scl3.mozilla.com:Pending builds is WARNING: WARNING Pending Builds: 4318 (http://m.mozilla.org/Pending+builds) 09:15:04 Fri 00:15:03 PDT [4611] buildbot-master68.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:15:04 Fri 00:15:03 PDT [4612] buildbot-master67.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:15:04 Fri 00:15:03 PDT [4613] buildbot-master70.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:15:04 Fri 00:15:03 PDT [4614] buildbot-master72.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:15:14 Fri 00:15:13 PDT [4615] buildbot-master81.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 2 warn out of 2 processes with command name buildbot (http://m.mozilla.org/buildbot+masters+age) nigelb|sheriffduty 09:17:38 pmoore: ^^ might want to look at tht :) nigelb|sheriffduty 09:17:40 *that nagios-releng 09:18:53 Fri 00:18:53 PDT [4617] buildbot-master01.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:18:53 Fri 00:18:53 PDT [4618] buildbot-master02.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:18:53 Fri 00:18:53 PDT [4619] buildbot-master04.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:18:54 Fri 00:18:54 PDT [4620] buildbot-master03.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:18:56 Fri 00:18:54 PDT [4621] buildbot-master05.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:18:59 Fri 00:18:54 PDT [4622] buildbot-master06.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:19:03 Fri 00:19:03 PDT [4623] buildbot-master100.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:19:05 Fri 00:19:03 PDT [4624] buildbot-master101.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:19:08 Fri 00:19:08 PDT [4625] buildbot-master102.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:19:11 Fri 00:19:09 PDT [4626] buildbot-master103.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:19:14 Fri 00:19:09 PDT [4627] buildbot-master104.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:19:17 Fri 00:19:09 PDT [4628] buildbot-master105.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:19:20 Fri 00:19:09 PDT [4629] buildbot-master107.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:19:23 Fri 00:19:09 PDT [4630] buildbot-master106.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:19:26 Fri 00:19:09 PDT [4631] buildbot-master108.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:19:29 Fri 00:19:13 PDT [4632] buildbot-master112.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:19:32 Fri 00:19:14 PDT [4633] buildbot-master117.bb.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:19:35 Fri 00:19:14 PDT [4634] buildbot-master51.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:19:38 Fri 00:19:14 PDT [4635] buildbot-master52.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:19:41 Fri 00:19:14 PDT [4636] buildbot-master53.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:19:44 Fri 00:19:23 PDT [4637] buildbot-master71.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:19:47 Fri 00:19:24 PDT [4638] buildbot-master73.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:19:50 Fri 00:19:24 PDT [4639] buildbot-master74.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:19:53 Fri 00:19:24 PDT [4640] buildbot-master75.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:19:56 Fri 00:19:24 PDT [4641] buildbot-master76.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:19:59 Fri 00:19:33 PDT [4642] buildbot-master78.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:20:02 Fri 00:19:34 PDT [4643] buildbot-master79.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:20:05 Fri 00:19:34 PDT [4644] buildbot-master77.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:20:08 Fri 00:19:34 PDT [4645] buildbot-master82.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:20:11 Fri 00:19:34 PDT [4646] buildbot-master83.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:20:14 Fri 00:19:34 PDT [4647] buildbot-master84.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:20:17 Fri 00:19:34 PDT [4648] buildbot-master85.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:20:20 Fri 00:19:34 PDT [4649] buildbot-master86.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:20:23 Fri 00:19:34 PDT [4650] buildbot-master87.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:20:26 Fri 00:19:34 PDT [4651] buildbot-master88.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:20:29 Fri 00:19:44 PDT [4652] buildbot-master89.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:20:32 Fri 00:19:44 PDT [4653] buildbot-master91.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:20:35 Fri 00:19:44 PDT [4654] buildbot-master99.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age) 09:20:38 Fri 00:19:44 PDT [4655] buildbot-master94.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)

Yup, this is a new regime of catching masters before they head off into the weeds by running for long times. I'll downtime the checks until my Monday and do a rolling graceful restart then.

I think this is fixed nthomas? closing.