Closed
Bug 1196723
Opened 9 years ago
Closed 9 years ago
revocations failing due to invalid inventory
Categories
(Infrastructure & Operations :: RelOps: Puppet, task)
Infrastructure & Operations
RelOps: Puppet
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: dustin, Assigned: dustin)
References
Details
Delivered-To: dmitchell@mozilla.com
Received: by 10.79.29.76 with SMTP id d73csp248786ivd;
Wed, 19 Aug 2015 04:10:03 -0700 (PDT)
X-Received: by 10.170.187.134 with SMTP id d128mr12817729yke.103.1439982603189;
Wed, 19 Aug 2015 04:10:03 -0700 (PDT)
Return-Path: <releng-puppet-mail+bncBAABBCWI2GXAKGQEI64GNGQ@mozilla.com>
Received: from mail-yk0-f198.google.com (mail-yk0-f198.google.com. [209.85.160.198])
by mx.google.com with ESMTPS id u206si153876yke.75.2015.08.19.04.10.02
for <dmitchell@mozilla.com>
(version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Wed, 19 Aug 2015 04:10:03 -0700 (PDT)
Received-SPF: pass (google.com: domain of releng-puppet-mail+bncBAABBCWI2GXAKGQEI64GNGQ@mozilla.com designates 209.85.160.198 as permitted sender) client-ip=209.85.160.198;
Authentication-Results: mx.google.com;
spf=pass (google.com: domain of releng-puppet-mail+bncBAABBCWI2GXAKGQEI64GNGQ@mozilla.com designates 209.85.160.198 as permitted sender) smtp.mailfrom=releng-puppet-mail+bncBAABBCWI2GXAKGQEI64GNGQ@mozilla.com
Received: by ykbj6 with SMTP id j6sf1425327ykb.3
for <dmitchell@mozilla.com>; Wed, 19 Aug 2015 04:10:02 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20130820;
h=x-gm-message-state:mime-version:delivered-to:from:to:subject
:content-type:auto-submitted:message-id:date:x-original-sender
:x-original-authentication-results:precedence:mailing-list:list-id
:x-spam-checked-in-group:list-post:list-help:list-archive
:list-subscribe:list-unsubscribe;
bh=vPwTr3mahENbBhriiyxS3uSHFB5y9tGhhhYYP7WamDw=;
b=JiGagAu2XpdHc7F2OY9YEJM5FaffXCG/KFelffDOkdZZA20390znhbYCTOJzavJF3t
a2rGkzWuIW6uaGzf/Dq5kgd1eSEjkMURAVebYkPEOmv/OQ2A+597tZJ2aNLgMawgxEp8
WZSUzoe3gNhdlR/CxKMSQSs4NADgFNhohRMIgE007Pe4slkzf+CLZb8Eep+1XkbiSdgB
8czdrlLUmb0xppI2kuYXDXmsj6asOr4oycrePg2zv+2fht7vR85MG6qh3paMQoxQgT7n
oUKFbCnPw54lOyXurxt3mVwzNv7ssAY8D4M/WJudgQZW0DE/cckmUTtZKSWyX8AVpeC7
Rbeg==
X-Gm-Message-State: ALoCoQncV0SCqmuFKZvBAkLRCNfU9ObzizRGY5UPXtLjGKBuVYJJlTppYbsZseOsG1NDHlDLS0Kz
X-Received: by 10.129.85.76 with SMTP id j73mr9781869ywb.54.1439982602911;
Wed, 19 Aug 2015 04:10:02 -0700 (PDT)
MIME-Version: 1.0
X-BeenThere: releng-puppet-mail@mozilla.com
Received: by 10.140.98.68 with SMTP id n62ls49527qge.10.gmail; Wed, 19 Aug
2015 04:10:02 -0700 (PDT)
X-Received: by 10.140.106.5 with SMTP id d5mr1684439qgf.22.1439982602628;
Wed, 19 Aug 2015 04:10:02 -0700 (PDT)
Received: from smtp.mozilla.org (mx1.scl3.mozilla.com. [63.245.214.155])
by mx.google.com with ESMTPS id e4si512477qka.6.2015.08.19.04.10.02
for <releng-puppet-mail@mozilla.com>
(version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Wed, 19 Aug 2015 04:10:02 -0700 (PDT)
Received-SPF: pass (google.com: domain of root@releng-puppet2.srv.releng.scl3.mozilla.com designates 63.245.214.155 as permitted sender) client-ip=63.245.214.155;
Received: from localhost (localhost6.localdomain [127.0.0.1])
by mx1.mail.scl3.mozilla.com (Postfix) with ESMTP id E18C2C590A
for <releng-puppet-mail@mozilla.com>; Wed, 19 Aug 2015 11:10:01 +0000 (UTC)
X-Virus-Scanned: amavisd-new at mozilla.org
Received: from smtp.mozilla.org ([127.0.0.1])
by localhost (mx1.mail.scl3.mozilla.com [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id lm86sS_Qj1zH for <releng-puppet-mail@mozilla.com>;
Wed, 19 Aug 2015 11:10:01 +0000 (UTC)
Received: from releng-puppet2.srv.releng.scl3.mozilla.com (releng-puppet2.srv.releng.scl3.mozilla.com [10.26.48.50])
by mx1.mail.scl3.mozilla.com (Postfix) with ESMTP id CE5CBBFE06
for <releng-puppet-mail@mozilla.com>; Wed, 19 Aug 2015 11:10:01 +0000 (UTC)
Received: by releng-puppet2.srv.releng.scl3.mozilla.com (Postfix)
id BF5BCCAF6D; Wed, 19 Aug 2015 04:10:01 -0700 (PDT)
Delivered-To: root@releng-puppet2.srv.releng.scl3.mozilla.com
Received: by releng-puppet2.srv.releng.scl3.mozilla.com (Postfix, from userid 0)
id 9176ECAE3A; Wed, 19 Aug 2015 04:10:01 -0700 (PDT)
From: root@releng-puppet2.srv.releng.scl3.mozilla.com (Cron Daemon)
To: root@releng-puppet2.srv.releng.scl3.mozilla.com
Subject: Cron <root@releng-puppet2> /var/lib/puppetmaster/ssl/scripts/do_requested_revocations.sh
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20150819111001.9176ECAE3A@releng-puppet2.srv.releng.scl3.mozilla.com>
Date: Wed, 19 Aug 2015 04:10:01 -0700 (PDT)
X-Original-Sender: root@releng-puppet2.srv.releng.scl3.mozilla.com
X-Original-Authentication-Results: mx.google.com; spf=pass (google.com:
domain of root@releng-puppet2.srv.releng.scl3.mozilla.com designates
63.245.214.155 as permitted sender) smtp.mailfrom=root@releng-puppet2.srv.releng.scl3.mozilla.com
Precedence: list
Mailing-list: list releng-puppet-mail@mozilla.com; contact releng-puppet-mail+owners@mozilla.com
List-ID: <releng-puppet-mail.mozilla.com>
X-Spam-Checked-In-Group: releng-puppet-mail@mozilla.com
X-Google-Group-Id: 532148886376
List-Post: <http://groups.google.com/a/mozilla.com/group/releng-puppet-mail/post>,
<mailto:releng-puppet-mail@mozilla.com>
List-Help: <http://support.google.com/a/mozilla.com/bin/topic.py?topic=25838>, <mailto:releng-puppet-mail+help@mozilla.com>
List-Archive: <http://groups.google.com/a/mozilla.com/group/releng-puppet-mail/>
List-Subscribe: <http://groups.google.com/a/mozilla.com/group/releng-puppet-mail/subscribe>,
<mailto:releng-puppet-mail+subscribe@mozilla.com>
List-Unsubscribe: <mailto:googlegroups-manage+532148886376+unsubscribe@googlegroups.com>,
<http://groups.google.com/a/mozilla.com/group/releng-puppet-mail/subscribe>
==== revocation of dev-linux64-ec2-nthomas.dev.releng.use1.mozilla.com-for-releng-puppet1.srv.releng.use1.mozilla.com.crt failed:
Using configuration from /var/lib/puppetmaster/ssl/ca/openssl.conf
wrong number of fields on line 11109 (looking for field 6, got 1, '' left)
--
You received this message because you are subscribed to the Google Groups "Releng Puppet Mail" group.
To unsubscribe from this group and stop receiving emails from it, send an email to releng-puppet-mail+unsubscribe@mozilla.com.
Assignee | ||
Comment 1•9 years ago
|
||
11109 is the last line; the end of the file looks like
R 200816180334Z 150818192611Z 2B5E unknown /CN=b-2008-ix-0175.wintry.releng.scl3.mozilla.com
R 200816192611Z 150818213538Z 2B5F unknown /CN=b-2008-ix-0175.wintry.releng.scl3.mozilla.com
V 200816213538Z 2B60 unknown /CN=b-2008-ix-0175.wintry.releng.scl3.mozilla.com
V 200817083209Z 2B61 unknown /CN=try-linux64-ec2-golden.try.releng.use1.mozilla.com
V 200817083723Z 2B62 unknown /CN=bld-linux64-ec2-golden.build.releng.use1.mozilla.com
V 200817083837Z 2B63 unknown /CN=y-2008-ec2-golden.try.releng.use1.mozilla.com
V 200817084239Z 2B64 unknown /CN=b-2008-ec2-golden.build.releng.use1.mozilla.com
0x2b65 2015-08-18T10:19:00GMT 2020-08-17T10:19:00GMT /CN=Puppet CA: releng-puppet2.srv.releng.scl3.mozilla.com
A few odd things there:
- wrong format (newer openssl??)
- the CN is for the CA certificate itself
Assignee | ||
Comment 3•9 years ago
|
||
These shouldn't be here:
-rw-r--r-- 1 puppet puppet 999 Aug 19 03:19 ca_crl.pem
-rw-r--r-- 1 puppet puppet 2069 Aug 19 03:19 ca_crt.pem
-rw-r--r-- 1 puppet puppet 3243 Aug 19 03:19 ca_key.pem
and look at the dates..
[root@releng-puppet2.srv.releng.scl3.mozilla.com ca]# openssl x509 -in ca_crt.pem -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 11109 (0x2b65)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=Puppet CA: releng-puppet2.srv.releng.scl3.mozilla.com
Validity
Not Before: Aug 18 10:19:00 2015 GMT
Not After : Aug 17 10:19:00 2020 GMT
Subject: CN=Puppet CA: releng-puppet2.srv.releng.scl3.mozilla.com
that is the bogus certificate!
alin.sel pts/2 10-22-248-26.vpn Wed Aug 19 00:25 - 03:20 (02:54)
I'm in touch with aselagea to see what happened.
Group: infra
Assignee | ||
Comment 4•9 years ago
|
||
Ah, "Puppet CA: .." is the name of the cert that puppet itself generates, and in this case it's a self-signed certificate.
The puppet server has 'ca = false' in puppet.conf, so it shouldn't be generating CA keys:
[master]
# default to looking in the production/ subdirectory of /etc/puppet
ssldir = /var/lib/puppetmaster/ssl
ca = false
but its directory is set correctly. It's possible that running 'puppet cert <something>' would cause this. I don't see that run by root, but it was run by aselagea. Interestingly, that "succeeeds":
dmitchell@releng-puppet1 ~ $ puppet cert list
Notice: Signed certificate request for ca
but creates the cert under my homedir:
1577740 4 -rw-rw-r-- 1 dmitchell dmitchell 2069 Aug 20 10:12 .puppet/ssl/ca/ca_crt.pem
so I can't quite see *how* it was done, since it's not in the command-line history, but I assume that at some point 'puppet cert <something>' got run as root or as puppet, generated the certs, and corrupted the inventory (with the Ruby OpenSSL format, instead of whatever the openssl command line uses).
Given that, I've cleaned it up by removing the inventory line and deleting the cert, key, and crl.
Group: infra
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•