Affecting at least mozilla-central and try.

Looks like talos-r3-fed64-* started backing between 4 and 5pm Pacific today.

Fallout from http://hg.mozilla.org/build/puppet-manifests/rev/829ee80177dc#l5.1 Feb 28 16:51:37 talos-r3-fed64-003 puppetd[1804]: Starting catalog run Feb 28 16:51:41 talos-r3-fed64-003 puppetd[1804]: (//Node[talos-r3-fed64-003.build.mozilla.org]/talosslave/talos_fedora/Package[python-devel]/ensure) change from absent to latest failed: Could not update: at /etc/puppet/manifests/os/talos_fedora.pp:27

So those machines just sit there polling puppet every few minutes. AFAICT, fedora12-x86_64/test/RPMs/python-devel-2.6.2-2.fc12.x86_64.rpm wasn't copied from staging/ to production/: [root@production-puppet N]# find /N -name '*python-devel*' /N/staging/fedora12-x86_64/test/RPMs/python-devel-2.6.2-2.fc12.x86_64.rpm [root@production-puppet N]# [root@scl-production-puppet production]# find /builds/production/ -name '*python-devel*' [root@scl-production-puppet production]#

I copied python-devel-2.6.2-2.fc12.x86_64.rpm from staging to production at mpt-p-p, and from there to mv-p-p and scl-p-p. My test slave picked that up and proceeded to start buildbot and collect a job.

Pending job count is collapsing as we clear the backlog. For m-c and tracemonkey and m-c only the most recent test request for each builder (tinderbox column) will get done. If there are regressions since 16:30 today we can trigger jobs to bisect the changes, where we = rail. ;-)