Needs a reboot.

grey screen | rebooted

grey screen | rebooted

Please reboot.

grey screen | rebooted

Up and running.

[56] talos-r3-fed64-030.build.scl1:PING is CRITICAL: PING CRITICAL - Packet loss = 100%

Back in production.

Machine seems to be unavailable on the network today.

https://tbpl.mozilla.org/php/getParsedLog.php?id=14238884&tree=Mozilla-Aurora Looks like maybe the system clock is set to the wrong date.

Disabled in slavealloc while I take a quick look.

I fixed the date up with ntpdate (it thought it was 2001), but there's may be a puppet issue to follow up on.

rebooted it and it re-puppetized correctly.

Reboot needed.

[18:12:13] nagios-releng Wed 15:12:36 PST [437] talos-r3-fed64-030.build.scl1.mozilla.com is DOWN :PING CRITICAL - Packet loss = 100%

Needs a reboot.

Back in action.

Just like comment 10, the clock is 13 years behind.

disabled in slavealloc

It seems that we still have clock issues and puppet issues. I recreated the certs and cleaned around but I get this error. [cltbld@talos-r3-fed64-030 ~]$ date Wed Jan 3 18:14:25 PST 2001 [cltbld@talos-r3-fed64-030 ~]$ su - Password: [root@talos-r3-fed64-030 ~]# puppetd --test --server scl-production-puppet.build.scl1.mozilla.com notice: Lock file /var/lib/puppet/state/puppetdlock exists; skipping catalog run [root@talos-r3-fed64-030 ~]# ls -l /var/lib/puppet/state/puppetdlock -rw-r--r-- 1 root root 0 2012-12-17 10:27 /var/lib/puppet/state/puppetdlock [root@talos-r3-fed64-030 ~]# vi /var/lib/puppet/state/puppetdlock [root@talos-r3-fed64-030 ~]# rm /var/lib/puppet/state/puppetdlock rm: remove regular empty file `/var/lib/puppet/state/puppetdlock'? y [root@talos-r3-fed64-030 ~]# puppetd --test --server scl-production-puppet.build.scl1.mozilla.com warning: Certificate validation failed; consider using the certname configuration option err: Could not retrieve catalog: Certificates were not trusted: certificate verify failed warning: Not using cache on failed catalog [root@talos-r3-fed64-030 ~]# puppetd --test --server scl-production-puppet.build.scl1.mozilla.com warning: Certificate validation failed; consider using the certname configuration option err: Could not retrieve catalog: Certificates were not trusted: certificate verify failed warning: Not using cache on failed catalog You have new mail in /var/spool/mail/root [root@talos-r3-fed64-030 ~]# rm -rf /var/lib/puppet/ssl/certs/* [root@talos-r3-fed64-030 ~]# puppetd --test --server scl-production-puppet.build.scl1.mozilla.com warning: peer certificate won't be verified in this SSL session notice: Did not receive certificate notice: Set to run 'one time'; exiting with no certificate [root@talos-r3-fed64-030 ~]# puppetd --test --server scl-production-puppet.build.scl1.mozilla.com warning: peer certificate won't be verified in this SSL session notice: Got signed certificate warning: Certificate validation failed; consider using the certname configuration option err: Could not retrieve catalog: Certificates were not trusted: certificate verify failed warning: Not using cache on failed catalog

Back in production.

And I should note that I found no clock issues on this machine even after multiple reboots.

needs a reboot

Back in production.

01:11 < nagios-releng> Tue 01:11:28 PST [494] talos-r3-fed64-030.build.scl1.mozilla.com is DOWN :PING CRITICAL - Packet loss = 100%