According to sanity check email these have been up for ~200 hours. 'aws_manage_instances.py status' says they are both running. Slave Health says they are 'stopped' AWS web console (https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Instances:) can't find them when I search. slavealloc says they are attached to bm70 for 'Date branch purposes' I can't ssh or ping them. I am creating this bug because, as far as I can tell, we are getting conflicting results from various sources. This could be a bug or it could be me failing to understand how things work. Either way, I don't want to 'aws_manage_instances.py stop tst-w64-ec2-001' until I know more.

see https://bugzilla.mozilla.org/show_bug.cgi?id=968413#c0 for similar situation regarding slave health vs aws_manage_instances discrepancy

I think these should be marked as loaned to jhopkins?

in our sanity email I see ones with 'dev' and ones without in hostname: == Loaned == dev-tst-w64-ec2-002 (i-873c0c8e, us-west-2): Loaned to jhopkins@mozilla.com in an unknown bug, stopped dev-tst-w64-ec2-001 (i-29566620, us-west-2): Loaned to jhopkins@mozilla.com in an unknown bug, stopped dev-tst-w64-ec2-000 (i-b21a4c86, us-west-2): Loaned to jhopkins@mozilla.com in an unknown bug, stopped ==== Long running instances ==== 0 tst-w64-ec2-002 (i-9a4e93ac, us-west-2): up for 208 hours 1 tst-w64-ec2-001 (i-be06698a, us-west-2): up for 208 hours The ones that specifically say jhopkins in loaned section, are apparently stopped. I am not sure if they are related but something seems fishy here. Either we have duplicates and aws_manage_instances.py is confused or we have a some instances that are running/hiding in a dark cloud ;)

I've terminated the dev-tst-w64-ec2-* instances listed above. I left the tst-w64-ec2-* instances running. I see that moz-loaned-to on each of those instances is set to jhopkins@mozilla.com which is fine.