The comm-central tinderboxes are trying to checkout from cvs-mirror and are failing, I can't check out locally either. Both cvs-mirror.mozilla.org and cvs.mozilla.org are failing, afaik there is no planned downtime for these. $ cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot -q checkout -P -r LDAPCSDK_6_0_6D_MOZILLA_RTM -d c-sdk mozilla/directory/c-sdk cvs [checkout aborted]: unrecognized auth response from cvs-mirror.mozilla.org: execle: cvs: No such file or directory $ cvs -d :pserver:anonymous@cvs.mozilla.org:/cvsroot -q checkout -P -r LDAPCSDK_6_0_6D_MOZILLA_RTM -d c-sdk mozilla/directory/c-sdk cvs [checkout aborted]: connect to cvs.mozilla.org(63.245.208.155):2401 failed: Operation timed out

Looking into this.

Okay, I'm unsure of what the issue is at this point. We rebooted dm-cvs01 and 02 yesterday and they came back fine. xinetd is running and I show the cvspserver running on 2401, but connections from the outside don't seem to make it to the box.

Talos is failing as well, and that's a really blocking issue for any push on mozilla central too. Please update us on situation.

So jailkit is to blame for this. 06:41:14 < dmoore> fox2mike: hmm... that directory got touched at 4:00 this morning 06:41:25 * fox2mike looks for crons 06:41:40 < dmoore> along with lib 06:42:00 < fox2mike> [root@dm-cvs02 bin]# cat /etc/cron.daily/jk_update 06:42:00 < fox2mike> #!/bin/bash 06:42:01 < fox2mike> /usr/sbin/jk_update -c /cvs/jail 06:42:23 < fox2mike> and from crontab 06:42:23 < fox2mike> 02 4 * * * root run-parts /etc/cron.daily 06:43:24 < fox2mike> jailkit 06:43:29 < dmoore> I think we're getting warmer 06:43:34 < fox2mike> [root@dm-cvs02 bin]# rpm -qf /usr/sbin/jk_update 06:43:35 < fox2mike> jailkit-2.11-1.el5.rf 06:43:40 < fox2mike> I bet my ass off 06:43:47 < fox2mike> that jailkit was updated yesterday 06:44:11 < fox2mike> okay dmoore who's the daddy now? :p 06:44:12 < fox2mike> [root@dm-cvs02 bin]# grep jailkit /var/log/yum.log 06:44:13 < fox2mike> Aug 11 07:58:10 Updated: jailkit-2.11-1.el5.rf.i386 06:44:33 < dmoore> [root@dm-cvs02 log]# jk_update -c /cvs/jail 06:44:33 < dmoore> removing outdated file /cvs/jail/usr/bin/cvs 06:44:34 < dmoore> lol So that's the cause. jailkit was updated to a new version and that basically removed the cvs binary, which caused cvs-mirror to hose itself. Working on fixing it.

Dave's our jailkit man and is taking over to fix this.

Should be fixed now. dmoore wasn't showing you the entire output from jk_update. It was removing the cvs binary because it was old, and then was supposed to put back the new one, but that didn't happen because the script crashed before it got that far. We picked up a new version of jailkit along with the rest of the security updates the other night. The new version of jailkit doesn't like symlinks, and the path to the jail that was in the config was a symlink. Changing the config to use the real path to the jail fixed most of the problem. The other change was that the new version of ld has an /etc/ld.so.conf.d/ directory with additional config to load in it, and jailkit was only configured to copy /etc/ld.so.conf itself into the jail. Adding the /etc/ld.so.conf.d/ directory to the jail configuration resolved the remaining issues.