Rules for flows from scl3 (outside of releng.scl3) to the releng AWS networks are currently governed *both* by rules on fw1.releng.scl3 and in AWS security groups. This is hard to reason about, and breaks the Mozilla practice of filtering traffic only at its destination or, more rarely, source. We should modify the security groups to limit the IP ranges appropriately for incoming traffic, and then remove the now-redundant "from-zone dc to-zone aws" policies. See bug 1044035.

Hi Dustin -- I will gladly work with you on this from the netops side. As you point out, on the firewalls, when packets are going to another security zone with the same data center or a remote data center, we put security policies on the destination side. (seems to me that we should treat VPC instances at AWS as remote data centers too, but that's just me...)

That's the idea here. I think things are as they are because of the weird history of the releng-in-AWS project, not for any technical reason.

A Pivotal Tracker story has been created for this Bug: https://www.pivotaltracker.com/story/show/80405588

Dustin J. Mitchell added a comment in Pivotal Tracker: Adjust AWS SG's test_buildmasters.py: # BUG: AWS flows don't limit http from AWS slaves to AWS masters test_buildmasters.py: # BUG: AWS flows don't limit http from AWS slaves to AWS masters test_buildmasters.py: # BUG: AWS is too permissive here test_buildmasters.py: # BUG 1078296: AWS masters don't have SSH access to scl3 individual masters Bug 1058225 - Restrict incoming flows in AWS SG's, remove scl3 -> aws rules from fw1.releng.scl3 test_puppetmasters.py:@skip("Flows from outside releng.scl3 to AWS are not represented properly; bug 1058225") test_puppetmasters.py:@skip("Flows from outside releng.scl3 to AWS are not represented properly; bug 1058225") test_buildmasters.py: # TODO: bug 1058225: denies on fw1.releng.scl3 for other_mozilla -> releng_aws are not visible to fwunit

Python script to print out security groups; the results for the current set of security groups in cloud-tools is Region: us-east-1 ID: sg-18a07677 Name: windows rules: allow 445/tcp from any allow 3389/tcp from any allow 5900/tcp from any rules_egress: allow None/any to any Region: us-east-1 ID: sg-31e8185e Name: buildbot-master rules: allow 9301/tcp from any allow 8001/tcp from any allow 9302/tcp from any allow 8201/tcp from any allow 5666/tcp from any allow 22/tcp from any allow 9101/tcp from any allow 8101/tcp from any allow 9001/tcp from any allow any/icmp from any allow 9201/tcp from any rules_egress: allow None/any to any Region: us-east-1 ID: sg-718b1214 Name: try rules: allow 5900/tcp from 10.22.248.0/22 10.22.75.6/31 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32 allow 22/tcp from 10.22.248.0/22 10.22.75.6/31 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32 allow any/icmp from 10.22.248.0/22 10.22.75.6/31 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32 rules_egress: allow None/any to any Region: us-east-1 ID: sg-e758e982 Name: build rules: allow 5900/tcp from 10.22.248.0/22 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32 10.22.75.6/31 allow 22/tcp from 10.22.248.0/22 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32 10.22.75.6/31 allow any/icmp from 10.22.248.0/22 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32 10.22.75.6/31 rules_egress: allow None/any to any Region: us-east-1 ID: sg-f0f1239f Name: tests rules: allow 5900/tcp from 10.26.48.17/32 10.26.48.16/32 10.26.48.25/32 10.26.48.43/32 10.22.252.0/22 10.22.248.0/22 10.26.48.23/32 10.22.75.6/31 allow 22/tcp from 10.26.48.17/32 10.22.252.0/22 10.26.48.43/32 10.26.48.16/32 10.22.248.0/22 10.26.48.25/32 10.26.48.23/32 10.22.75.6/31 allow any/icmp from 10.22.252.0/22 10.26.48.16/32 10.26.48.25/32 10.26.48.17/32 10.26.48.43/32 10.22.248.0/22 10.26.48.23/32 10.22.75.6/31 rules_egress: allow None/any to any Region: us-east-1 ID: sg-f3927c9c Name: default rules: allow 22/tcp from any allow any/icmp from any rules_egress: allow None/any to any Region: us-west-2 ID: sg-3aaa095f Name: try rules: allow 5900/tcp from 10.22.248.0/22 10.22.75.6/31 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32 allow 22/tcp from 10.22.248.0/22 10.22.75.6/31 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32 allow any/icmp from 10.22.248.0/22 10.22.75.6/31 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32 rules_egress: allow None/any to any Region: us-west-2 ID: sg-84beade6 Name: windows rules: allow 445/tcp from any allow 3389/tcp from any allow 5900/tcp from any rules_egress: allow None/any to any Region: us-west-2 ID: sg-8b9f7ce4 Name: tests rules: allow 5900/tcp from 10.22.248.0/22 10.22.252.0/22 10.26.48.17/32 10.26.48.16/32 10.26.48.43/32 10.26.48.25/32 10.26.48.23/32 10.22.75.6/31 allow 22/tcp from 10.22.248.0/22 10.22.252.0/22 10.26.48.17/32 10.26.48.16/32 10.26.48.43/32 10.26.48.25/32 10.26.48.23/32 10.22.75.6/31 allow any/icmp from 10.22.248.0/22 10.22.252.0/22 10.26.48.17/32 10.26.48.16/32 10.26.48.43/32 10.26.48.25/32 10.26.48.23/32 10.22.75.6/31 rules_egress: allow None/any to any Region: us-west-2 ID: sg-932e33ff Name: buildbot-master rules: allow 9301/tcp from any allow 8001/tcp from any allow 9302/tcp from any allow 8201/tcp from any allow 5666/tcp from any allow 22/tcp from any allow 9101/tcp from any allow 8101/tcp from any allow 9001/tcp from any allow any/icmp from any allow 9201/tcp from any rules_egress: allow None/any to any Region: us-west-2 ID: sg-d5617cb9 Name: default rules: allow 22/tcp from any allow any/icmp from any rules_egress: allow None/any to any Region: us-west-2 ID: sg-f5ca0690 Name: build rules: allow 5900/tcp from 10.22.248.0/22 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32 10.22.75.6/31 allow 22/tcp from 10.22.248.0/22 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32 10.22.75.6/31 allow any/icmp from 10.22.248.0/22 10.22.252.0/22 10.26.48.25/32 10.26.48.17/32 10.26.48.16/32 10.26.48.23/32 10.26.48.43/32 10.22.75.6/31 rules_egress: allow None/any to any Rail, what do you think is the best way to modify these? Alter the SG's in place? Cretae new SG's and update cloud-tools? When would either option take effect?

..or should we invent a way to program these via cloud-tools?

If you update/add rules in http://hg.mozilla.org/build/cloud-tools/file/5d32265cd9f1/configs/securitygroups.yml that should work too. The script using that config is idempotent and will replace the rules in place. Will this help?

Hah, well, I feel dumb. Per irc, the applies-to in securitygroups.yml can be ignored, and the fact that the config/<hotstype> files refer to sg's by id isn't important here (although it would confuse things if we had to wipe our config and start over). So, I'll make a patch.

I'm writing rules for 19 - cruncher.build.mozilla.org 20 - slaveapi1.srv.releng.scl3.mozilla.com 21 - slaveapi-dev1.srv.releng.scl3.mozilla.com 22 - 10.22.75.6/31 23 - 10.22.252.0/22 24 - 10.22.248.0/22 25 - dev-master1.build.mozilla.org 26 - aws-manager1.srv.releng.scl3.mozilla.com and it's worth noting from that set that the dev-master1 flow actually doesn't work (it's blocked by the SRX). The 10.22 IPs are admin1a/b, scl3-vpn, and the cloud VLANs, respectively. The latter is gone now.

Here are the rules to replicate: https://infra.etherpad.mozilla.org/1067178 (sorry for those without access)

Bug 1058225: attempt to (mostly) replicate the current AWS config This replaces the specific source IPs for ping with 0.0.0.0/0, which is what we do for the rest of Mozilla (ever notice you can ping anything?). It also removes the openstack network, gets the netmask right for the VPN, and adds some comments to identify subnets. This is based on clicking around the AWS UI. If this is OK, how should I go about landing it safely? Once it's landed, I'll submit a few patches further limiting access based on what's currently configured in the SRXes, until I'm confident that everything's represented in the security groups. Then we can remove the rules from the SRXes.

ugh, aws_manage_securitygroups.py is really buggy. Although it asks to confirm each change, it then makes different changes: tests - Delete rule ('inbound', u'icmp', u'-1', u'-1') to set([u'10.22.248.0/22', u'10.22.75.6/31', u'10.22.252.0/22', u'10.26.48.25/32', u'10.26.48.17/32', u'10.26.48.16/32', u'10.26.48.23/32', u'10.26.48.43/32']) (y/N) y 2014-11-07 12:03:36,376 - tests - removing rule for ('inbound', u'icmp', u'-1', u'-1') to set(['10.22.240.0/20']) So, that sucks.

I edited up the bits it modified by hand, so we should be good for now. There are still a bunch of security groups to go, though.

I found port 8201 used in one region and not the others.

Comment on attachment 8524080 [details] [diff] [review] add-8201.patch r+ from nthomas in rc

This should help as things get longer :)

It turns out includes within includes are useful..

This one needs some serious squinting at! I based this on a read of all flows from scl3 to the VPC (in the SRX configuration), intersected with existing allowed flows in scl3, plus my understanding of what intra-AWS flows should look like. Specific notes: - admin-access contains more hosts than had access before, and allows any port, not just ssh - observium and infra-puppetize are new, but don't hurt - the puppet security group has been removed (bug 1022368) - incoming access on port 5900 doesn't work now, so it has been removed - blobber access is now limited to slave VLANs, and only HTTPS was permitted by the Junipers so only HTTPS is listed here - incoming access to buildbot masters was refactored to include both traffic over the VPC and internal traffic from buildslaves If it helps, here's the output of the script (with everything answered 'n'): > 2014-11-17 19:28:26,725 - Working in regions set(['us-west-2', 'us-east-1', 'us-west-1']) > 2014-11-17 19:28:26,725 - Loading groups for us-west-2 > 2014-11-17 19:28:27,351 - Loading groups for us-east-1 > 2014-11-17 19:28:27,710 - Loading groups for us-west-1 > 2014-11-17 19:28:28,252 - Working in us-west-2 > 2014-11-17 19:28:28,252 - Looking for sg tests > 2014-11-17 19:28:28,252 - Found SecurityGroup:tests > tests - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N) > tests - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N) > tests - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N) > tests - Delete rule ('inbound', u'tcp', u'5900', u'5900') to set([u'10.22.75.6/31', u'10.26.48.25/32', u'10.26.48.17/32', u'10.26.48.16/32', u'10.26.48.23/32', u'10.22.240.0/20', u'10.26.48.43/32']) (y/N) > tests - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'10.22.75.6/31', u'10.22.240.0/20']) (y/N) > 2014-11-17 19:28:30,763 - Working in us-east-1 > 2014-11-17 19:28:30,764 - Looking for sg tests > 2014-11-17 19:28:30,764 - Found SecurityGroup:tests > tests - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N) > tests - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N) > tests - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N) > tests - Delete rule ('inbound', u'tcp', u'5900', u'5900') to set([u'10.22.75.6/31', u'10.26.48.25/32', u'10.26.48.17/32', u'10.26.48.16/32', u'10.26.48.23/32', u'10.22.240.0/20', u'10.26.48.43/32']) (y/N) > tests - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'10.22.75.6/31', u'10.22.240.0/20']) (y/N) > 2014-11-17 19:28:33,644 - Working in us-west-1 > 2014-11-17 19:28:33,644 - Looking for sg tests > 2014-11-17 19:28:33,644 - Found SecurityGroup:tests > tests - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N) > tests - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N) > tests - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N) > tests - Delete rule ('inbound', u'tcp', u'5900', u'5900') to set([u'10.22.75.6/31', u'10.26.48.25/32', u'10.26.48.17/32', u'10.26.48.16/32', u'10.26.48.23/32', u'10.22.240.0/20', u'10.26.48.43/32']) (y/N) > tests - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'10.22.75.6/31', u'10.22.240.0/20']) (y/N) > 2014-11-17 19:28:33,962 - Working in us-west-2 > 2014-11-17 19:28:33,963 - Looking for sg buildbot-master > 2014-11-17 19:28:33,963 - Found SecurityGroup:buildbot-master > buildbot-master - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.26.48.24/32', '10.26.48.26/32', '10.26.48.30/32', '10.132.49.154/32', '10.26.48.19/32', '10.26.48.25/32', '10.134.49.77/32', '10.26.48.59/32', '10.26.48.36/32', '10 > .26.48.52/32', '10.132.49.29/32', '10.134.49.144/32', '10.26.48.57/32', '10.132.49.158/32', '10.132.50.54/32', '10.134.68.0/24', '10.26.68.0/24', '10.134.48.7/32', '10.26.48.51/32', '10.132.48.17/32', '10.134.49.94/32', '10.132.48.136/32', '10.132.50.189/32', '10.26.48.40/32', '10.26.48.38/32', '10.26.48.55/32', '10.132.50.240/32', '10.134.48.122/32', '10.132.49.181/32', '10.132.49.94/32', '10.26.48.56/32', '10.26.48.54/32', '10.26.48.20/32', '10.134.48.86/32', '10.26.48.37/32', '10.26.48.28/32', '10.132.48.18/32', '10.26.48.23/32', '10.132.48.16/32', '10.132.68.0/24', '10.132.49.117/32', '10.134.48.8/32', '10.134.49.70/32', '10.134.48.9/32', '10.134.48.31/32', '10.26.48.17/32', '10.26.48.39/32', '10.26.48.27/32', '10.134.48.10/32', '10.134.48.234/32', '10.26.48.16/32', '10.26.48.43/32', '10.22.75.36/32', '10.26.48.53/32', '10.134.48.40/32', '10.134.48.106/32', '10.26.48.32/32', '10.132.50.247/32', '10.132.50.89/32', '10.134.49.111/32'])? (y/N) > buildbot-master - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N) > buildbot-master - Add rule for ('inbound', 'tcp', '8000', '8999') to set(['10.26.48.24/32', '10.26.48.26/32', '10.26.48.30/32', '10.132.49.154/32', '10.26.48.19/32', '10.134.49.77/32', '10.26.48.59/32', '10.26.48.36/32', '10.26.48.52/32', '10.132.49.29/32', '10.134.49.144/32', '10.26.48.57/32', '10.132.49.158/32', '10.132.50.54/32', '10.134.68.0/24', '10.26.68.0/24', '10.134.48.7/32', '10.26.48.51/32', '10.132.48.17/32', '10.134.49.94/32', '10.132.48.136/32', '10.132.50.189/32', '10.26.48.40/32', '10.26.48.55/32', '10.132.50.240/32', '10.134.48.122/32', '10.132.49.181/32', '10.26.48.56/32', '10.26.48.54/32', '10.26.48.20/32', '10.134.48.86/32', '10.26.48.37/32', '10.26.48.28/32', '10.132.48.18/32', '10.26.48.23/32', '10.132.48.16/32', '10.132.68.0/24', '10.132.49.117/32', '10.134.48.8/32', '10.134.49.70/32', '10.134.48.9/32', '10.134.48.31/32', '10.26.48.39/32', '10.26.48.27/32', '10.134.48.10/32', '10.134.48.234/32', '10.26.48.38/32', '10.26.48.43/32', '10.132.49.94/32', '10.26.48.53/32', '10.134.48.40/32', '10.134.48.106/32', '10.26.48.32/32', '10.132.50.247/32', '10.132.50.89/32', '10.134.49.111/32'])? (y/N) > buildbot-master - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N) > buildbot-master - Add rule for ('inbound', 'tcp', '9000', '9999') to set(['10.26.36.0/22', '10.26.48.24/32', '10.26.48.53/32', '10.26.48.26/32', '10.134.68.0/24', '10.132.49.154/32', '10.26.48.19/32', '10.134.49.77/32', '10.134.156.0/22', '10.132.56.0/22', '10.26.48.59/32', '10.26.48.36/32', '10.26.48.52/32', '10.132.49.29/32', '10.132.64.0/22', '10.134.49.144/32', '10.26.48.57/32', '10.132.49.158/32', '10.132.50.54/32', '10.26.48.30/32', '10.26.68.0/24', '10.134.48.7/32', '10.26.48.51/32', '10.132.48.17/32', '10.134.49.94/32', '10.132.48.136/32', '10.132.50.189/32', '10.26.48.40/32', '10.26.48.39/32', '10.134.48.106/32', '10.26.48.55/32', '10.132.50.240/32', '10.134.48.122/32', '10.132.49.181/32', '10.26.128.0/17', '10.26.48.56/32', '10.26.48.54/32', '10.134.52.0/22', '10.26.48.20/32', '10.134.48.86/32', '10.26.48.37/32', '10.26.48.28/32', '10.26.88.0/21', '10.132.48.18/32', '10.132.48.16/32', '10.132.68.0/24', '10.132.49.117/32', '10.134.48.8/32', '10.26.56.0/22', '10.134.49.70/32', '10.134.48.9/32', '10.132.52.0/22', '10.134.48.31/32', '10.134.64.0/22', '10.26.44.0/22', '10.26.52.0/22', '10.132.156.0/22', '10.26.48.27/32', '10.134.48.10/32', '10.134.48.234/32', '10.26.48.38/32', '10.26.48.43/32', '10.132.49.94/32', '10.134.56.0/22', '10.134.48.40/32', '10.26.40.0/22', '10.26.48.32/32', '10.132.50.247/32', '10.132.50.89/32', '10.134.49.111/32'])? (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'9301', u'9301') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'8202', u'8202') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'5666', u'5666') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'9001', u'9001') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'8201', u'8201') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'9302', u'9302') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'8001', u'8001') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'8101', u'8101') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'9101', u'9101') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'9201', u'9201') to set([u'0.0.0.0/0']) (y/N) > 2014-11-17 19:28:33,963 - No interface filters to apply, skipping. > 2014-11-17 19:28:33,963 - No interface filters to apply, skipping. > 2014-11-17 19:28:33,963 - Working in us-east-1 > 2014-11-17 19:28:33,964 - Looking for sg buildbot-master > 2014-11-17 19:28:33,964 - Found SecurityGroup:buildbot-master > buildbot-master - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.26.48.24/32', '10.26.48.26/32', '10.26.48.30/32', '10.132.49.154/32', '10.26.48.19/32', '10.26.48.25/32', '10.134.49.77/32', '10.26.48.59/32', '10.26.48.36/32', '10.26.48.52/32', '10.132.49.29/32', '10.134.49.144/32', '10.26.48.57/32', '10.132.49.158/32', '10.132.50.54/32', '10.134.68.0/24', '10.26.68.0/24', '10.134.48.7/32', '10.26.48.51/32', '10.132.48.17/32', '10.134.49.94/32', '10.132.48.136/32', '10.132.50.189/32', '10.26.48.40/32', '10.26.48.38/32', '10.26.48.55/32', '10.132.50.240/32', '10.134.48.122/32', '10.132.49.181/32', '10.132.49.94/32', '10.26.48.56/32', '10.26.48.54/32', '10.26.48.20/32', '10.134.48.86/32', '10.26.48.37/32', '10.26.48.28/32', '10.132.48.18/32', '10.26.48.23/32', '10.132.48.16/32', '10.132.68.0/24', '10.132.49.117/32', '10.134.48.8/32', '10.134.49.70/32', '10.134.48.9/32', '10.134.48.31/32', '10.26.48.17/32', '10.26.48.39/32', '10.26.48.27/32', '10.134.48.10/32', '10.134.48.234/32', '10.26.48.16/32', '10.26.48.43/32', '10.22.75.36/32', '10.26.48.53/32', '10.134.48.40/32', '10.134.48.106/32', '10.26.48.32/32', '10.132.50.247/32', '10.132.50.89/32', '10.134.49.111/32'])? (y/N) > buildbot-master - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N) > buildbot-master - Add rule for ('inbound', 'tcp', '8000', '8999') to set(['10.26.48.24/32', '10.26.48.26/32', '10.26.48.30/32', '10.132.49.154/32', '10.26.48.19/32', '10.134.49.77/32', '10.26.48.59/32', '10.26.48.36/32', '10.26.48.52/32', '10.132.49.29/32', '10.134.49.144/32', '10.26.48.57/32', '10.132.49.158/32', '10.132.50.54/32', '10.134.68.0/24', '10.26.68.0/24', '10.134.48.7/32', '10.26.48.51/32', '10.132.48.17/32', '10.134.49.94/32', '10.132.48.136/32', '10.132.50.189/32', '10.26.48.40/32', '10.26.48.55/32', '10.132.50.240/32', '10.134.48.122/32', '10.132.49.181/32', '10.26.48.56/32', '10.26.48.54/32', '10.26.48.20/32', '10.134.48.86/32', '10.26.48.37/32', '10.26.48.28/32', '10.132.48.18/32', '10.26.48.23/32', '10.132.48.16/32', '10.132.68.0/24', '10.132.49.117/32', '10.134.48.8/32', '10.134.49.70/32', '10.134.48.9/32', '10.134.48.31/32', '10.26.48.39/32', '10.26.48.27/32', '10.134.48.10/32', '10.134.48.234/32', '10.26.48.38/32', '10.26.48.43/32', '10.132.49.94/32', '10.26.48.53/32', '10.134.48.40/32', '10.134.48.106/32', '10.26.48.32/32', '10.132.50.247/32', '10.132.50.89/32', '10.134.49.111/32'])? (y/N) > buildbot-master - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N) > buildbot-master - Add rule for ('inbound', 'tcp', '9000', '9999') to set(['10.26.36.0/22', '10.26.48.24/32', '10.26.48.53/32', '10.26.48.26/32', '10.134.68.0/24', '10.132.49.154/32', '10.26.48.19/32', '10.134.49.77/32', '10.134.156.0/22', '10.132.56.0/22', '10.26.48.59/32', '10.26.48.36/32', '10.26.48.52/32', '10.132.49.29/32', '10.132.64.0/22', '10.134.49.144/32', '10.26.48.57/32', '10.132.49.158/32', '10.132.50.54/32', '10.26.48.30/32', '10.26.68.0/24', '10.134.48.7/32', '10.26.48.51/32', '10.132.48.17/32', '10.134.49.94/32', '10.132.48.136/32', '10.132.50.189/32', '10.26.48.40/32', '10.26.48.39/32', '10.134.48.106/32', '10.26.48.55/32', '10.132.50.240/32', '10.134.48.122/32', '10.132.49.181/32', '10.26.128.0/17', '10.26.48.56/32', '10.26.48.54/32', '10.134.52.0/22', '10.26.48.20/32', '10.134.48.86/32', '10.26.48.37/32', '10.26.48.28/32', '10.26.88.0/21', '10.132.48.18/32', '10.132.48.16/32', '10.132.68.0/24', '10.132.49.117/32', '10.134.48.8/32', '10.26.56.0/22', '10.134.49.70/32', '10.134.48.9/32', '10.132.52.0/22', '10.134.48.31/32', '10.134.64.0/22', '10.26.44.0/22', '10.26.52.0/22', '10.132.156.0/22', '10.26.48.27/32', '10.134.48.10/32', '10.134.48.234/32', '10.26.48.38/32', '10.26.48.43/32', '10.132.49.94/32', '10.134.56.0/22', '10.134.48.40/32', '10.26.40.0/22', '10.26.48.32/32', '10.132.50.247/32', '10.132.50.89/32', '10.134.49.111/32'])? (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'9301', u'9301') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'8202', u'8202') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'5666', u'5666') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'9001', u'9001') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'8201', u'8201') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'9302', u'9302') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'8001', u'8001') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'8101', u'8101') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'9101', u'9101') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'9201', u'9201') to set([u'0.0.0.0/0']) (y/N) > 2014-11-17 19:28:33,964 - No interface filters to apply, skipping. > 2014-11-17 19:28:33,964 - No interface filters to apply, skipping. > 2014-11-17 19:28:33,964 - Working in us-west-1 > 2014-11-17 19:28:33,964 - Looking for sg buildbot-master > 2014-11-17 19:28:33,964 - Found SecurityGroup:buildbot-master > buildbot-master - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.26.48.24/32', '10.26.48.26/32', '10.26.48.30/32', '10.132.49.154/32', '10.26.48.19/32', '10.26.48.25/32', '10.134.49.77/32', '10.26.48.59/32', '10.26.48.36/32', '10.26.48.52/32', '10.132.49.29/32', '10.134.49.144/32', '10.26.48.57/32', '10.132.49.158/32', '10.132.50.54/32', '10.134.68.0/24', '10.26.68.0/24', '10.134.48.7/32', '10.26.48.51/32', '10.132.48.17/32', '10.134.49.94/32', '10.132.48.136/32', '10.132.50.189/32', '10.26.48.40/32', '10.26.48.38/32', '10.26.48.55/32', '10.132.50.240/32', '10.134.48.122/32', '10.132.49.181/32', '10.132.49.94/32', '10.26.48.56/32', '10.26.48.54/32', '10.26.48.20/32', '10.134.48.86/32', '10.26.48.37/32', '10.26.48.28/32', '10.132.48.18/32', '10.26.48.23/32', '10.132.48.16/32', '10.132.68.0/24', '10.132.49.117/32', '10.134.48.8/32', '10.134.49.70/32', '10.134.48.9/32', '10.134.48.31/32', '10.26.48.17/32', '10.26.48.39/32', '10.26.48.27/32', '10.134.48.10/32', '10.134.48.234/32', '10.26.48.16/32', '10.26.48.43/32', '10.22.75.36/32', '10.26.48.53/32', '10.134.48.40/32', '10.134.48.106/32', '10.26.48.32/32', '10.132.50.247/32', '10.132.50.89/32', '10.134.49.111/32'])? (y/N) > buildbot-master - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N) > buildbot-master - Add rule for ('inbound', 'tcp', '8000', '8999') to set(['10.26.48.24/32', '10.26.48.26/32', '10.26.48.30/32', '10.132.49.154/32', '10.26.48.19/32', '10.134.49.77/32', '10.26.48.59/32', '10.26.48.36/32', '10.26.48.52/32', '10.132.49.29/32', '10.134.49.144/32', '10.26.48.57/32', '10.132.49.158/32', '10.132.50.54/32', '10.134.68.0/24', '10.26.68.0/24', '10.134.48.7/32', '10.26.48.51/32', '10.132.48.17/32', '10.134.49.94/32', '10.132.48.136/32', '10.132.50.189/32', '10.26.48.40/32', '10.26.48.55/32', '10.132.50.240/32', '10.134.48.122/32', '10.132.49.181/32', '10.26.48.56/32', '10.26.48.54/32', '10.26.48.20/32', '10.134.48.86/32', '10.26.48.37/32', '10.26.48.28/32', '10.132.48.18/32', '10.26.48.23/32', '10.132.48.16/32', '10.132.68.0/24', '10.132.49.117/32', '10.134.48.8/32', '10.134.49.70/32', '10.134.48.9/32', '10.134.48.31/32', '10.26.48.39/32', '10.26.48.27/32', '10.134.48.10/32', '10.134.48.234/32', '10.26.48.38/32', '10.26.48.43/32', '10.132.49.94/32', '10.26.48.53/32', '10.134.48.40/32', '10.134.48.106/32', '10.26.48.32/32', '10.132.50.247/32', '10.132.50.89/32', '10.134.49.111/32'])? (y/N) > buildbot-master - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N) > buildbot-master - Add rule for ('inbound', 'tcp', '9000', '9999') to set(['10.26.36.0/22', '10.26.48.24/32', '10.26.48.53/32', '10.26.48.26/32', '10.134.68.0/24', '10.132.49.154/32', '10.26.48.19/32', '10.134.49.77/32', '10.134.156.0/22', '10.132.56.0/22', '10.26.48.59/32', '10.26.48.36/32', '10.26.48.52/32', '10.132.49.29/32', '10.132.64.0/22', '10.134.49.144/32', '10.26.48.57/32', '10.132.49.158/32', '10.132.50.54/32', '10.26.48.30/32', '10.26.68.0/24', '10.134.48.7/32', '10.26.48.51/32', '10.132.48.17/32', '10.134.49.94/32', '10.132.48.136/32', '10.132.50.189/32', '10.26.48.40/32', '10.26.48.39/32', '10.134.48.106/32', '10.26.48.55/32', '10.132.50.240/32', '10.134.48.122/32', '10.132.49.181/32', '10.26.128.0/17', '10.26.48.56/32', '10.26.48.54/32', '10.134.52.0/22', '10.26.48.20/32', '10.134.48.86/32', '10.26.48.37/32', '10.26.48.28/32', '10.26.88.0/21', '10.132.48.18/32', '10.132.48.16/32', '10.132.68.0/24', '10.132.49.117/32', '10.134.48.8/32', '10.26.56.0/22', '10.134.49.70/32', '10.134.48.9/32', '10.132.52.0/22', '10.134.48.31/32', '10.134.64.0/22', '10.26.44.0/22', '10.26.52.0/22', '10.132.156.0/22', '10.26.48.27/32', '10.134.48.10/32', '10.134.48.234/32', '10.26.48.38/32', '10.26.48.43/32', '10.132.49.94/32', '10.134.56.0/22', '10.134.48.40/32', '10.26.40.0/22', '10.26.48.32/32', '10.132.50.247/32', '10.132.50.89/32', '10.134.49.111/32'])? (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'9301', u'9301') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'8202', u'8202') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'5666', u'5666') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'9001', u'9001') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'8201', u'8201') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'9302', u'9302') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'8001', u'8001') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'8101', u'8101') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'9101', u'9101') to set([u'0.0.0.0/0']) (y/N) > buildbot-master - Delete rule ('inbound', u'tcp', u'9201', u'9201') to set([u'0.0.0.0/0']) (y/N) > 2014-11-17 19:28:33,967 - No interface filters to apply, skipping. > 2014-11-17 19:28:33,967 - No interface filters to apply, skipping. > 2014-11-17 19:28:33,967 - Working in us-west-2 > 2014-11-17 19:28:33,967 - Looking for sg nagios > 2014-11-17 19:28:33,967 - Found SecurityGroup:nagios > nagios - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N) > nagios - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N) > nagios - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N) > nagios - Delete rule ('inbound', u'tcp', u'443', u'443') to set([u'0.0.0.0/0']) (y/N) > nagios - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'0.0.0.0/0']) (y/N) > nagios - Delete rule ('inbound', u'tcp', u'80', u'80') to set([u'0.0.0.0/0']) (y/N) > 2014-11-17 19:28:33,967 - No interface filters to apply, skipping. > 2014-11-17 19:28:33,967 - No interface filters to apply, skipping. > 2014-11-17 19:28:33,967 - Working in us-east-1 > 2014-11-17 19:28:33,967 - Looking for sg nagios > 2014-11-17 19:28:33,967 - Found SecurityGroup:nagios > nagios - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N) > nagios - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N) > nagios - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N) > nagios - Delete rule ('inbound', u'tcp', u'443', u'443') to set([u'0.0.0.0/0']) (y/N) > nagios - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'0.0.0.0/0']) (y/N) > nagios - Delete rule ('inbound', u'tcp', u'80', u'80') to set([u'0.0.0.0/0']) (y/N) > 2014-11-17 19:28:33,968 - No interface filters to apply, skipping. > 2014-11-17 19:28:33,968 - No interface filters to apply, skipping. > 2014-11-17 19:28:33,968 - Working in us-west-1 > 2014-11-17 19:28:33,968 - Looking for sg nagios > 2014-11-17 19:28:33,968 - Found SecurityGroup:nagios > nagios - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N) > nagios - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N) > nagios - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N) > nagios - Delete rule ('inbound', u'tcp', u'443', u'443') to set([u'0.0.0.0/0']) (y/N) > nagios - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'0.0.0.0/0']) (y/N) > nagios - Delete rule ('inbound', u'tcp', u'80', u'80') to set([u'0.0.0.0/0']) (y/N) > 2014-11-17 19:28:33,968 - No interface filters to apply, skipping. > 2014-11-17 19:28:33,968 - No interface filters to apply, skipping. > 2014-11-17 19:28:33,968 - Working in us-west-2 > 2014-11-17 19:28:33,968 - Looking for sg try > 2014-11-17 19:28:33,968 - Found SecurityGroup:try > try - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N) > try - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N) > try - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N) > try - Delete rule ('inbound', u'tcp', u'5900', u'5900') to set([u'10.22.75.6/31', u'10.26.48.25/32', u'10.26.48.17/32', u'10.26.48.16/32', u'10.26.48.23/32', u'10.22.240.0/20', u'10.26.48.43/32']) (y/N) > try - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'10.22.75.6/31', u'10.22.240.0/20']) (y/N) > 2014-11-17 19:28:34,374 - Working in us-east-1 > 2014-11-17 19:28:34,374 - Looking for sg try > 2014-11-17 19:28:34,375 - Found SecurityGroup:try > try - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N) > try - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N) > try - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N) > try - Delete rule ('inbound', u'tcp', u'5900', u'5900') to set([u'10.22.75.6/31', u'10.26.48.25/32', u'10.26.48.17/32', u'10.26.48.16/32', u'10.26.48.23/32', u'10.22.240.0/20', u'10.26.48.43/32']) (y/N) > try - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'10.22.75.6/31', u'10.22.240.0/20']) (y/N) > 2014-11-17 19:28:35,024 - Working in us-west-1 > 2014-11-17 19:28:35,024 - Looking for sg try > 2014-11-17 19:28:35,025 - Found SecurityGroup:try > try - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N) > try - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N) > try - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N) > try - Delete rule ('inbound', u'tcp', u'5900', u'5900') to set([u'10.22.75.6/31', u'10.26.48.25/32', u'10.26.48.17/32', u'10.26.48.16/32', u'10.26.48.23/32', u'10.22.240.0/20', u'10.26.48.43/32']) (y/N) > try - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'10.22.75.6/31', u'10.22.240.0/20']) (y/N) > 2014-11-17 19:28:35,315 - Working in us-west-2 > 2014-11-17 19:28:35,315 - Looking for sg build > 2014-11-17 19:28:35,315 - Found SecurityGroup:build > build - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N) > build - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N) > build - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N) > build - Delete rule ('inbound', u'tcp', u'5900', u'5900') to set([u'10.22.75.6/31', u'10.26.48.25/32', u'10.26.48.17/32', u'10.26.48.16/32', u'10.26.48.23/32', u'10.22.240.0/20', u'10.26.48.43/32']) (y/N) > build - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'10.22.75.6/31', u'10.22.240.0/20']) (y/N) > 2014-11-17 19:28:35,871 - Working in us-east-1 > 2014-11-17 19:28:35,871 - Looking for sg build > 2014-11-17 19:28:35,871 - Found SecurityGroup:build > build - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N) > build - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N) > build - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N) > build - Delete rule ('inbound', u'tcp', u'5900', u'5900') to set([u'10.22.75.6/31', u'10.26.48.25/32', u'10.26.48.17/32', u'10.26.48.16/32', u'10.26.48.23/32', u'10.22.240.0/20', u'10.26.48.43/32']) (y/N) > build - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'10.22.75.6/31', u'10.22.240.0/20']) (y/N) > 2014-11-17 19:28:36,630 - Working in us-west-1 > 2014-11-17 19:28:36,630 - Looking for sg build > 2014-11-17 19:28:36,630 - Found SecurityGroup:build > build - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N) > build - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N) > build - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N) > build - Delete rule ('inbound', u'tcp', u'5900', u'5900') to set([u'10.22.75.6/31', u'10.26.48.25/32', u'10.26.48.17/32', u'10.26.48.16/32', u'10.26.48.23/32', u'10.22.240.0/20', u'10.26.48.43/32']) (y/N) > build - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'10.22.75.6/31', u'10.22.240.0/20']) (y/N) > 2014-11-17 19:28:36,861 - Working in us-west-2 > 2014-11-17 19:28:36,861 - Looking for sg blobber > 2014-11-17 19:28:36,862 - Found SecurityGroup:blobber > blobber - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N) > blobber - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N) > blobber - Add rule for ('inbound', 'tcp', '443', '443') to set(['10.26.36.0/22', '10.132.64.0/22', '10.26.128.0/17', '10.134.64.0/22', '10.26.52.0/22', '10.134.52.0/22', '10.26.44.0/22', '10.132.52.0/22', '10.134.156.0/22', '10.132.156.0/22', '10.26.88.0/21', '10.134.56.0/22', '10.132.56.0/22', '10.26.56.0/22', '10.26.40.0/22'])? (y/N) > blobber - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N) > blobber - Delete rule ('inbound', u'tcp', u'443', u'443') to set([u'10.0.0.0/8']) (y/N) > blobber - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'0.0.0.0/0']) (y/N) > blobber - Delete rule ('inbound', u'tcp', u'80', u'80') to set([u'10.0.0.0/8']) (y/N) > 2014-11-17 19:28:36,862 - No interface filters to apply, skipping. > 2014-11-17 19:28:36,862 - No interface filters to apply, skipping. > 2014-11-17 19:28:36,862 - Working in us-east-1 > 2014-11-17 19:28:36,862 - Looking for sg blobber > 2014-11-17 19:28:36,862 - Found SecurityGroup:blobber > blobber - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N) > blobber - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N) > blobber - Add rule for ('inbound', 'tcp', '443', '443') to set(['10.26.36.0/22', '10.132.64.0/22', '10.26.128.0/17', '10.134.64.0/22', '10.26.52.0/22', '10.134.52.0/22', '10.26.44.0/22', '10.132.52.0/22', '10.134.156.0/22', '10.132.156.0/22', '10.26.88.0/21', '10.134.56.0/22', '10.132.56.0/22', '10.26.56.0/22', '10.26.40.0/22'])? (y/N) > blobber - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N) > blobber - Delete rule ('inbound', u'tcp', u'443', u'443') to set([u'10.0.0.0/8']) (y/N) > blobber - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'0.0.0.0/0']) (y/N) > blobber - Delete rule ('inbound', u'tcp', u'80', u'80') to set([u'10.0.0.0/8']) (y/N) > 2014-11-17 19:28:36,863 - No interface filters to apply, skipping. > 2014-11-17 19:28:36,863 - No interface filters to apply, skipping. > 2014-11-17 19:28:36,863 - Working in us-west-1 > 2014-11-17 19:28:36,863 - Looking for sg blobber > 2014-11-17 19:28:36,863 - Found SecurityGroup:blobber > blobber - Add rule for ('inbound', 'tcp', '22', '22') to set(['10.22.75.36/32'])? (y/N) > blobber - Add rule for ('inbound', '-1', None, None) to set(['10.22.72.155/32', '10.22.72.158/32', '10.22.75.6/31', '10.22.75.5/32', '10.22.20.0/25', '10.22.8.128/32', '10.22.72.136/32', '10.22.72.159/32', '10.26.74.22/32', '10.22.240.0/20', '10.26.75.30/32'])? (y/N) > blobber - Add rule for ('inbound', 'tcp', '443', '443') to set(['10.26.36.0/22', '10.132.64.0/22', '10.26.128.0/17', '10.134.64.0/22', '10.26.52.0/22', '10.134.52.0/22', '10.26.44.0/22', '10.132.52.0/22', '10.134.156.0/22', '10.132.156.0/22', '10.26.88.0/21', '10.134.56.0/22', '10.132.56.0/22', '10.26.56.0/22', '10.26.40.0/22'])? (y/N) > blobber - Add rule for ('inbound', 'udp', '161', '161') to set(['10.22.75.137/32'])? (y/N) > blobber - Delete rule ('inbound', u'tcp', u'443', u'443') to set([u'10.0.0.0/8']) (y/N) > blobber - Delete rule ('inbound', u'tcp', u'22', u'22') to set([u'0.0.0.0/0']) (y/N) > blobber - Delete rule ('inbound', u'tcp', u'80', u'80') to set([u'10.0.0.0/8']) (y/N) > 2014-11-17 19:28:36,863 - No interface filters to apply, skipping. > 2014-11-17 19:28:36,863 - No interface filters to apply, skipping.

So midway through applying that, I ran into RulesPerSecurityGroupLimitExceeded: The maximum number of rules per security group has been reached I had been hoping that "Rules" in http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_Limits.html referred to, you know, rules. It doesn't -- it refers to the total number of *grants* in the security group, across all rules. I'd like to do two things: 1. Raise "Rules per security group" to 125 (reducing "Security groups per network interface" to 2), which needs the account rep's interaction; and 2. Try to combine some rules into aggregates; slaveapi{-dev,}1 is the first one I see.

Can you connect me with the account rep for #1?

I filed case 1271429571 with Amazon to raise the limits, but also managed to hack things to get under 50 temporarily (although the result is substantially too open)

Comment on attachment 8524693 [details] [diff] [review] bug1058225-sg-size.patch thanks!

The part where all of srv.releng.* gets unfettered access to buildmasters makes me deeply sad, so let's hope AWS comes through soon!

attachment 8524726 [details] [diff] [review] is applied for usw1/usw2, but *not* use1. Nothing has exploded yet, but I'll give it a while longer before applying for use1. So at worst we're on a single region until we can revert the SG's for usw2.

Comment on attachment 8524693 [details] [diff] [review] bug1058225-sg-size.patch https://hg.mozilla.org/build/cloud-tools/rev/5153c591f335

use1 is in place now too. Now to un-restrict flows on the SRX.

We got the limit raised to 125, so we can put some more accurate flows into the SG.

Comment on attachment 8526943 [details] [diff] [review] bug1058225-expand.patch Landed in the repo, and deployed to usw2 only - I'll do use1 once everything seems clear. https://hg.mozilla.org/build/cloud-tools/rev/202bb8eea212

remainder is now landed.