3.4.3 was released while we were working on upgrading to 3.4.2. New puppetizations are, in fact, installing 3.4.3 and then downgrading on the first puppet run.

3.4.2 isn't available for trusty tahr. It looks like it may depend on virt-what, which isn't in main/restricted. If puppet is making us install universe, I'll be sad.

Yup, I am sad: https://launchpad.net/ubuntu/trusty/+source/virt-what/1.13-1

I'm attempting to sync universe on the relabs master, and it's used 7GB already and is only on the A's. G (gnu this and that) and L (libthis and libthat) tend to be larger than most other letters. So this is easily 150GB, which is 100GB more than any of the masters have space for. One option is to just increase the disk sizes of masters across the board, and mirror universe. The other is to try to mirror "just enough" to install puppet. The risk to the latter plan is that we'll end up in the same situation for every package we try to install -- that will quickly get frustrating. So let's increase the size to start with. Hopefully I can get a good estimate of the total size soon.

It's more complicated -- I'll handle re-mirroring in bug 999661

Puppet won't be shipped for trusty until 3.6.0, so we'll need to wait until then to run it.

(which is due out on May 1)

Yay, it's out as of yesterday :)

3.6.0 is filtering through the puppetagain mirrors.

Warning: Sections other than main, master, agent, user are deprecated in puppet.conf. Please use the directory environments feature to specify environments. (See http://docs.puppetlabs.com/puppet/latest/reference/environments.html) (at /usr/lib/ruby/site_ruby/1.8/puppet/settings/config_file.rb:77:in `collect') Warning: Setting manifest is deprecated in puppet.conf. See http://links.puppetlabs.com/env-settings-deprecations (at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1067:in `each') Warning: Setting modulepath is deprecated in puppet.conf. See http://links.puppetlabs.com/env-settings-deprecations (at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1067:in `each') Warning: Setting config_version is deprecated in puppet.conf. See http://links.puppetlabs.com/env-settings-deprecations (at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1067:in `each') Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations (at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1071:in `each') Warning: Setting manifestdir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations (at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1071:in `each') but otherwise seems to work OK on CentOS. There's a one-time error when a 3.4.2 agent contacts a 3.6.0 master: Error: Could not back up /var/lib/puppet/concat/_root_.ssh_authorized_keys/fragments/10__root::: Unsupported checksum type "md5" Error: Could not back up /var/lib/puppet/concat/_root_.ssh_authorized_keys/fragments/10__root::: Unsupported checksum type "md5" Error: /Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat::Fragment[/root::]/File[/var/lib/puppet/concat/_root_.ssh_authorized_keys/fragments/10__root::]/content: change from {md5}f009409a48ab4bd9e968a28fd61aef1c to {md5}48db268782c9e2b34206142b990ee6cd failed: Could not back up /var/lib/puppet/concat/_root_.ssh_authorized_keys/fragments/10__root::: Unsupported checksum type "md5" but that went away on the next run. It also loads all plugins twice on an environment run on the master: Info: Loading facts in /etc/puppet/environments/dmitchell/env/modules/shared/lib/facter/env.rb Info: Loading facts in /etc/puppet/environments/dmitchell/env/modules/shared/lib/facter/winrootlp.rb Info: Loading facts in /etc/puppet/environments/dmitchell/env/modules/slave_secrets/lib/facter/existing_slave_trustlevel.rb Info: Loading facts in /etc/puppet/environments/dmitchell/env/modules/needs_reboot/lib/facter/needs_reboot.rb Info: Loading facts in /etc/puppet/environments/dmitchell/env/modules/stdlib/lib/facter/pe_version.rb Info: Loading facts in /etc/puppet/environments/dmitchell/env/modules/stdlib/lib/facter/puppet_vardir.rb Info: Loading facts in /etc/puppet/environments/dmitchell/env/modules/stdlib/lib/facter/root_home.rb Info: Loading facts in /etc/puppet/environments/dmitchell/env/modules/stdlib/lib/facter/facter_dot_d.rb Info: Loading facts in /etc/puppet/environments/dmitchell/env/modules/firewall/lib/facter/iptables_version.rb Info: Loading facts in /etc/puppet/environments/dmitchell/env/modules/firewall/lib/facter/iptables_persistent_version.rb Info: Loading facts in /etc/puppet/environments/dmitchell/env/modules/firewall/lib/facter/ip6tables_version.rb Info: Loading facts in /etc/puppet/environments/dmitchell/env/modules/concat/lib/facter/concat_ruby_interpreter.rb Info: Loading facts in /etc/puppet/environments/dmitchell/env/modules/concat/lib/facter/concat_basedir.rb Info: Loading facts in /etc/puppet/environments/dmitchell/env/modules/vmwaretools/lib/facter/vmwaretools_version.rb Info: Loading facts in /etc/puppet/environments/dmitchell/env/modules/buildmaster/lib/facter/num_masters.rb Info: Loading facts in /var/lib/puppet/lib/facter/iptables_version.rb Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb Info: Loading facts in /var/lib/puppet/lib/facter/concat_ruby_interpreter.rb Info: Loading facts in /var/lib/puppet/lib/facter/env.rb Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb Info: Loading facts in /var/lib/puppet/lib/facter/iptables_persistent_version.rb Info: Loading facts in /var/lib/puppet/lib/facter/num_masters.rb Info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb Info: Loading facts in /var/lib/puppet/lib/facter/vmwaretools_version.rb Info: Loading facts in /var/lib/puppet/lib/facter/needs_reboot.rb Info: Loading facts in /var/lib/puppet/lib/facter/ip6tables_version.rb Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb Info: Loading facts in /var/lib/puppet/lib/facter/winrootlp.rb Info: Loading facts in /var/lib/puppet/lib/facter/existing_slave_trustlevel.rb that seems like a bug.

I did some work to try to transition to directory environments, and it turned out to be pretty complicated. I think it will be easier to transition once everything is running 3.6.0, and config environments do appear to still work properly in 3.6.0 (as promised). I'm testing Ubuntu now.

Ubuntu's OK, too. I'll test OS X on Monday. We should probably land those in phases anyway, even if they all seem to work.

OS X Lion seems fine too, but on Mountain Lion: [root@talos-mtnlion-r5-006.test.releng.scl3.mozilla.com ~]# puppet agent --test --environment=dmitchell --server=$PUPPET_SERVER /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require': no such file to load -- json (LoadError) from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `require' from /Library/Ruby/Site/puppet/module.rb:3 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `require' from /Library/Ruby/Site/puppet/parser/files.rb:1 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `require' from /Library/Ruby/Site/puppet/parser/templatewrapper.rb:1 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `require' from /Library/Ruby/Site/puppet/parser/scope.rb:6 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `require' from /Library/Ruby/Site/puppet/parser/methods.rb:2 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `require' from /Library/Ruby/Site/puppet/parser/ast/method_call.rb:2 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `require' from /Library/Ruby/Site/puppet/parser/ast.rb:115 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `require' from /Library/Ruby/Site/puppet/parser/parser.rb:11 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `require' from /Library/Ruby/Site/puppet/parser.rb:4 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `require' from /Library/Ruby/Site/puppet.rb:250 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `require' from /Library/Ruby/Site/puppet/util/command_line.rb:12 from /usr/bin/puppet:3:in `require' from /usr/bin/puppet:3

Works fine after gem install json_pure though. I'll see if I can find a way to package that up into a pkgdmg.

OK, I think I have this working - just reimaging the talos host to make sure.

Yay, success. I'll put up a three-part patch for CentOS, Ubuntu, and OS X tomorrow, and then land them one at a time.

We'll let that soak in, and upgrade the remaining two silos on Monday.

BTW, we've been seeing hundreds of failure reports a day after we upgraded Centos, looking like this one: Sun May 25 18:49:36 -0700 2014 Puppet (err): Could not retrieve catalog from remote server: Error 400 on SERVER: $concat_basedir not defined. Try running again with pluginsync=true on the [master] section of your node's '/etc/puppet/puppet.conf'. at /etc/puppet/production/modules/concat/manifests/setup.pp:30 on node tst-linux64-spot-504.test.releng.usw2.mozilla.com Sun May 25 18:49:36 -0700 2014 Puppet (err): Could not retrieve catalog; skipping run Since the reports also come from Ubuntu machines I think this is something puppet master related.

This seems to fix the issue with concat and not knowing checksum type "md5". I don't think that addresses the issue you're seeing, though.

I'm seeing reports like comment 21 from CentOS agents, too.

https://tickets.puppetlabs.com/browse/PUP-2665 for the increased error rate.

A closer look at a report like that in comment 21 shows, from foreman: notice /File[/var/lib/puppet/lib/facter]/ensure removed So puppet is *deleting* the plugin dirs. From access_log: 10.132.57.181 - - [27/May/2014:08:53:32 -0700] "GET /production/file_metadatas/plugins?ignore=.svn&ignore=CVS&ignore=.git&links=manage&recurse=true&checksum_type=md5 HTTP/1.1" 200 283 "-" "-" yet on the previous run: 10.132.57.181 - - [27/May/2014:08:29:09 -0700] "GET /production/file_metadatas/plugins?recurse=true&ignore=.svn&ignore=CVS&ignore=.git&links=manage&checksum_type=md5 HTTP/1.1" 200 46760 "-" "-" (note the wildly smaller size on the failed run)

Possibly related to https://tickets.puppetlabs.com/browse/PUP-2610 which suggests an upgrade to 3.6.1 would help.

It's running OK in relabs..

That's looking *much* better.

Comment on attachment 8426311 [details] [diff] [review] bug986599-ubuntu.patch The apt-get update of puppet itself kills puppet (by design, ugh, stupid debian packages), but it works OK on the next run.

Comment on attachment 8426312 [details] [diff] [review] bug986599-osx.patch Oops: the commit message says 3.6.0, but it's 3.6.1.

Hmmm: [root@talos-r4-snow-135.build.scl1.mozilla.com ~]# puppet --version /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require': no such file to load -- json (LoadError) I wonder if the snow box I tested that on had json installed for some other reason?

http://hg.mozilla.org/build/puppet/rev/1ea0abad379f (and I'll add a followup adjusting the comment) I've connected to all of the snow boxes to see if they were affected, and I'm in the process of remediating the 57 which were (by installing json_pure).

remediation complete

Hi Dustin, The json related issue is not solved on every boxes. On t-snow-r4-0017 and t-snow-r4-0002, for example, I still get: # puppet /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require': no such file to load -- json (LoadError) This may be related to Bug 1019523

As agreed with Dustin, I run the following script to install json_pure to the boxes which needed it: for s in t-snow-r4-0026 t-snow-r4-0011 t-snow-r4-0040 t-snow-r4-0033 t-snow-r4-0025 t-snow-r4-0006 t-snow-r4-0013 t-snow-r4-0030 t-snow-r4-0031 t-snow-r4-0004 t-snow-r4-0018 t-snow-r4-0037 t-snow-r4-0015 t-snow-r4-0027 t-snow-r4-0020 t-snow-r4-0034 t-snow-r4-0002 t-snow-r4-0012 t-snow-r4-0016 t-snow-r4-0036 t-snow-r4-0007 t-snow-r4-0035 t-snow-r4-0010; do ssh root@$s /usr/local/bin/wget http://repos/repos/DMGs/10.6/json_pure-1.8.1.dmg \&\& hdiutil mount json_pure-1.8.1.dmg \&\& installer -package /Volumes/Mozilla*/json* -target / \&\& hdiutil detach /Volumes/Mozilla*; done I then verified that puppet is no longer raising the json related LoadError: for s in t-snow-r4-0026 t-snow-r4-0011 t-snow-r4-0040 t-snow-r4-0033 t-snow-r4-0025 t-snow-r4-0006 t-snow-r4-0013 t-snow-r4-0030 t-snow-r4-0031 t-snow-r4-0004 t-snow-r4-0018 t-snow-r4-0037 t-snow-r4-0015 t-snow-r4-0027 t-snow-r4-0020 t-snow-r4-0034 t-snow-r4-0002 t-snow-r4-0012 t-snow-r4-0016 t-snow-r4-0036 t-snow-r4-0007 t-snow-r4-0035 t-snow-r4-0010; do echo $s && ssh root@$s puppet --version ; done

I missed those because my inventory search didn't find those hosts (because the names are different). Silly mistake on my part. Thanks, Simone, for fixing it!

Hi Dustin, I also had to fix talos-r4-snow-130 (same json Load Error).