Delivered-To: dmitchell@mozilla.com Received: by 10.79.29.76 with SMTP id d73csp248786ivd; Wed, 19 Aug 2015 04:10:03 -0700 (PDT) X-Received: by 10.170.187.134 with SMTP id d128mr12817729yke.103.1439982603189; Wed, 19 Aug 2015 04:10:03 -0700 (PDT) Return-Path: <releng-puppet-mail+bncBAABBCWI2GXAKGQEI64GNGQ@mozilla.com> Received: from mail-yk0-f198.google.com (mail-yk0-f198.google.com. [209.85.160.198]) by mx.google.com with ESMTPS id u206si153876yke.75.2015.08.19.04.10.02 for <dmitchell@mozilla.com> (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Aug 2015 04:10:03 -0700 (PDT) Received-SPF: pass (google.com: domain of releng-puppet-mail+bncBAABBCWI2GXAKGQEI64GNGQ@mozilla.com designates 209.85.160.198 as permitted sender) client-ip=209.85.160.198; Authentication-Results: mx.google.com; spf=pass (google.com: domain of releng-puppet-mail+bncBAABBCWI2GXAKGQEI64GNGQ@mozilla.com designates 209.85.160.198 as permitted sender) smtp.mailfrom=releng-puppet-mail+bncBAABBCWI2GXAKGQEI64GNGQ@mozilla.com Received: by ykbj6 with SMTP id j6sf1425327ykb.3 for <dmitchell@mozilla.com>; Wed, 19 Aug 2015 04:10:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:delivered-to:from:to:subject :content-type:auto-submitted:message-id:date:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=vPwTr3mahENbBhriiyxS3uSHFB5y9tGhhhYYP7WamDw=; b=JiGagAu2XpdHc7F2OY9YEJM5FaffXCG/KFelffDOkdZZA20390znhbYCTOJzavJF3t a2rGkzWuIW6uaGzf/Dq5kgd1eSEjkMURAVebYkPEOmv/OQ2A+597tZJ2aNLgMawgxEp8 WZSUzoe3gNhdlR/CxKMSQSs4NADgFNhohRMIgE007Pe4slkzf+CLZb8Eep+1XkbiSdgB 8czdrlLUmb0xppI2kuYXDXmsj6asOr4oycrePg2zv+2fht7vR85MG6qh3paMQoxQgT7n oUKFbCnPw54lOyXurxt3mVwzNv7ssAY8D4M/WJudgQZW0DE/cckmUTtZKSWyX8AVpeC7 Rbeg== X-Gm-Message-State: ALoCoQncV0SCqmuFKZvBAkLRCNfU9ObzizRGY5UPXtLjGKBuVYJJlTppYbsZseOsG1NDHlDLS0Kz X-Received: by 10.129.85.76 with SMTP id j73mr9781869ywb.54.1439982602911; Wed, 19 Aug 2015 04:10:02 -0700 (PDT) MIME-Version: 1.0 X-BeenThere: releng-puppet-mail@mozilla.com Received: by 10.140.98.68 with SMTP id n62ls49527qge.10.gmail; Wed, 19 Aug 2015 04:10:02 -0700 (PDT) X-Received: by 10.140.106.5 with SMTP id d5mr1684439qgf.22.1439982602628; Wed, 19 Aug 2015 04:10:02 -0700 (PDT) Received: from smtp.mozilla.org (mx1.scl3.mozilla.com. [63.245.214.155]) by mx.google.com with ESMTPS id e4si512477qka.6.2015.08.19.04.10.02 for <releng-puppet-mail@mozilla.com> (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Aug 2015 04:10:02 -0700 (PDT) Received-SPF: pass (google.com: domain of root@releng-puppet2.srv.releng.scl3.mozilla.com designates 63.245.214.155 as permitted sender) client-ip=63.245.214.155; Received: from localhost (localhost6.localdomain [127.0.0.1]) by mx1.mail.scl3.mozilla.com (Postfix) with ESMTP id E18C2C590A for <releng-puppet-mail@mozilla.com>; Wed, 19 Aug 2015 11:10:01 +0000 (UTC) X-Virus-Scanned: amavisd-new at mozilla.org Received: from smtp.mozilla.org ([127.0.0.1]) by localhost (mx1.mail.scl3.mozilla.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lm86sS_Qj1zH for <releng-puppet-mail@mozilla.com>; Wed, 19 Aug 2015 11:10:01 +0000 (UTC) Received: from releng-puppet2.srv.releng.scl3.mozilla.com (releng-puppet2.srv.releng.scl3.mozilla.com [10.26.48.50]) by mx1.mail.scl3.mozilla.com (Postfix) with ESMTP id CE5CBBFE06 for <releng-puppet-mail@mozilla.com>; Wed, 19 Aug 2015 11:10:01 +0000 (UTC) Received: by releng-puppet2.srv.releng.scl3.mozilla.com (Postfix) id BF5BCCAF6D; Wed, 19 Aug 2015 04:10:01 -0700 (PDT) Delivered-To: root@releng-puppet2.srv.releng.scl3.mozilla.com Received: by releng-puppet2.srv.releng.scl3.mozilla.com (Postfix, from userid 0) id 9176ECAE3A; Wed, 19 Aug 2015 04:10:01 -0700 (PDT) From: root@releng-puppet2.srv.releng.scl3.mozilla.com (Cron Daemon) To: root@releng-puppet2.srv.releng.scl3.mozilla.com Subject: Cron <root@releng-puppet2> /var/lib/puppetmaster/ssl/scripts/do_requested_revocations.sh Content-Type: text/plain; charset=UTF-8 Auto-Submitted: auto-generated X-Cron-Env: <LANG=en_US.UTF-8> X-Cron-Env: <SHELL=/bin/sh> X-Cron-Env: <HOME=/root> X-Cron-Env: <PATH=/usr/bin:/bin> X-Cron-Env: <LOGNAME=root> X-Cron-Env: <USER=root> Message-Id: <20150819111001.9176ECAE3A@releng-puppet2.srv.releng.scl3.mozilla.com> Date: Wed, 19 Aug 2015 04:10:01 -0700 (PDT) X-Original-Sender: root@releng-puppet2.srv.releng.scl3.mozilla.com X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of root@releng-puppet2.srv.releng.scl3.mozilla.com designates 63.245.214.155 as permitted sender) smtp.mailfrom=root@releng-puppet2.srv.releng.scl3.mozilla.com Precedence: list Mailing-list: list releng-puppet-mail@mozilla.com; contact releng-puppet-mail+owners@mozilla.com List-ID: <releng-puppet-mail.mozilla.com> X-Spam-Checked-In-Group: releng-puppet-mail@mozilla.com X-Google-Group-Id: 532148886376 List-Post: <http://groups.google.com/a/mozilla.com/group/releng-puppet-mail/post>, <mailto:releng-puppet-mail@mozilla.com> List-Help: <http://support.google.com/a/mozilla.com/bin/topic.py?topic=25838>, <mailto:releng-puppet-mail+help@mozilla.com> List-Archive: <http://groups.google.com/a/mozilla.com/group/releng-puppet-mail/> List-Subscribe: <http://groups.google.com/a/mozilla.com/group/releng-puppet-mail/subscribe>, <mailto:releng-puppet-mail+subscribe@mozilla.com> List-Unsubscribe: <mailto:googlegroups-manage+532148886376+unsubscribe@googlegroups.com>, <http://groups.google.com/a/mozilla.com/group/releng-puppet-mail/subscribe> ==== revocation of dev-linux64-ec2-nthomas.dev.releng.use1.mozilla.com-for-releng-puppet1.srv.releng.use1.mozilla.com.crt failed: Using configuration from /var/lib/puppetmaster/ssl/ca/openssl.conf wrong number of fields on line 11109 (looking for field 6, got 1, '' left) -- You received this message because you are subscribed to the Google Groups "Releng Puppet Mail" group. To unsubscribe from this group and stop receiving emails from it, send an email to releng-puppet-mail+unsubscribe@mozilla.com.

11109 is the last line; the end of the file looks like R 200816180334Z 150818192611Z 2B5E unknown /CN=b-2008-ix-0175.wintry.releng.scl3.mozilla.com R 200816192611Z 150818213538Z 2B5F unknown /CN=b-2008-ix-0175.wintry.releng.scl3.mozilla.com V 200816213538Z 2B60 unknown /CN=b-2008-ix-0175.wintry.releng.scl3.mozilla.com V 200817083209Z 2B61 unknown /CN=try-linux64-ec2-golden.try.releng.use1.mozilla.com V 200817083723Z 2B62 unknown /CN=bld-linux64-ec2-golden.build.releng.use1.mozilla.com V 200817083837Z 2B63 unknown /CN=y-2008-ec2-golden.try.releng.use1.mozilla.com V 200817084239Z 2B64 unknown /CN=b-2008-ec2-golden.build.releng.use1.mozilla.com 0x2b65 2015-08-18T10:19:00GMT 2020-08-17T10:19:00GMT /CN=Puppet CA: releng-puppet2.srv.releng.scl3.mozilla.com A few odd things there: - wrong format (newer openssl??) - the CN is for the CA certificate itself

These shouldn't be here: -rw-r--r-- 1 puppet puppet 999 Aug 19 03:19 ca_crl.pem -rw-r--r-- 1 puppet puppet 2069 Aug 19 03:19 ca_crt.pem -rw-r--r-- 1 puppet puppet 3243 Aug 19 03:19 ca_key.pem and look at the dates.. [root@releng-puppet2.srv.releng.scl3.mozilla.com ca]# openssl x509 -in ca_crt.pem -text Certificate: Data: Version: 3 (0x2) Serial Number: 11109 (0x2b65) Signature Algorithm: sha256WithRSAEncryption Issuer: CN=Puppet CA: releng-puppet2.srv.releng.scl3.mozilla.com Validity Not Before: Aug 18 10:19:00 2015 GMT Not After : Aug 17 10:19:00 2020 GMT Subject: CN=Puppet CA: releng-puppet2.srv.releng.scl3.mozilla.com that is the bogus certificate! alin.sel pts/2 10-22-248-26.vpn Wed Aug 19 00:25 - 03:20 (02:54) I'm in touch with aselagea to see what happened.

Ah, "Puppet CA: .." is the name of the cert that puppet itself generates, and in this case it's a self-signed certificate. The puppet server has 'ca = false' in puppet.conf, so it shouldn't be generating CA keys: [master] # default to looking in the production/ subdirectory of /etc/puppet ssldir = /var/lib/puppetmaster/ssl ca = false but its directory is set correctly. It's possible that running 'puppet cert <something>' would cause this. I don't see that run by root, but it was run by aselagea. Interestingly, that "succeeeds": dmitchell@releng-puppet1 ~ $ puppet cert list Notice: Signed certificate request for ca but creates the cert under my homedir: 1577740 4 -rw-rw-r-- 1 dmitchell dmitchell 2069 Aug 20 10:12 .puppet/ssl/ca/ca_crt.pem so I can't quite see *how* it was done, since it's not in the command-line history, but I assume that at some point 'puppet cert <something>' got run as root or as puppet, generated the certs, and corrupted the inventory (with the Ruby OpenSSL format, instead of whatever the openssl command line uses). Given that, I've cleaned it up by removing the inventory line and deleting the cert, key, and crl.