+++ This bug was initially created as a clone of Bug #1372172 +++ Fom bug 1372172 comment 24: Trees are closing again because Windows builds stopped running after some time. Do we hit an AWS query limit? We had increased the max capacities, hit a rate limit, decreased it but are still at 2x the default. https://tools.taskcluster.net/aws-provisioner/gecko-3-b-win2012/ currently lists 256 pending jobs for gecko-3-b-win2012. Pending build jobs whose platforms start with 'win' grouped by state: https://sql.telemetry.mozilla.org/queries/51150 pending 394 running 5 The 'running' number is falling. Pending build jobs whose platforms start with 'win', looked at the column 'state': https://sql.telemetry.mozilla.org/queries/51149

turned out the problem was the same as this morning. the ssh key associated with windows instances changed its naming convention to no longer include the workerType name in the ssh key name. occ required patching in two places to no longer obtain the worker type from the ssh key name but instead from userdata on provisioned instances. this mornings patch was: https://github.com/mozilla-releng/OpenCloudConfig/commit/d7d2df5a174087bad52e7d3636ae92e043f999f0 and this afternoons patch is: https://github.com/mozilla-releng/OpenCloudConfig/commit/137c8c1b0e4b3927f15cf38ee4f9771894818221 we will need to manually terminate windows instances close to the time that these jobs are nearing completion: https://tools.taskcluster.net/groups/cBf4vjVTTrmTaZHrAXQhJA in order to allow the new amis to come into use.