autophone-2 experienced a set of disconnections and reboots after the new hub was installed and after attempting to reboot at 4:03PM has failed to come back up. Van, is there any way you can take a look at it before Monday? uber, if Van can't make it will you be in the office on Friday? I'd rather not leave it in what ever state it is in over the weekend. If we can get it to boot back up, I would like to remove the hub and connect the devices back to the host directly. If we can't get it to boot back up, I would like to power it down until we can investigate whether we need to have it serviced.

Timeline of disconnections and reboots since new hubs added: Inventory https://docs.google.com/spreadsheets/d/1ppMtFUS6by42QCv_b_6EPQko30TaswAxrmFLb-FxHcI/edit?ts=58a61d0d#gid=0 14:22 autophone-2 process finished starting 14:41 autophone-2 nexus-6p-12 disconnected USB3HUB2.5 14:46 autophone-2 rebooted 14:54 autophone-2 nexus-4-8 failed to init USB3HUB2.2 14:59 autophone-2 nexus-4-9 failed to init USB3HUB2.3 15:04 autophone-2 nexus-5-1 failed to init USBCARD3.1 15:09 autophone-2 nexus-5-8 failed to init USB2.1 15:14 autophone-2 nexus-6-1 failed to init USBCARD3.3 15:19 autophone-2 nexus-6p-7 failed to init USB3HUB2.4 15:24 autophone-2 nexus-6p-12 failed to init USB3HUB2.5 15:29 autophone-2 nexus-6p-13 failed to init USBCARD3.2 15:34 autophone-2 nexus-9-1 failed to init USB3HUB2.1 15:34 autophone-2 process finished starting 15:44 autophone-2 nexus-5-3 disconnected USB2.4 16:03 autophone-2 rebooted autophone-2 offline 14:10 autophone-1 process finished starting 23:45 autophone-1 nexus-6p-11 disconnected USB3HUB2.2 23:49 autophone-1 rebooted 23:53 autophone-1 process finished starting 14:10 autophone-3 process finished starting 14:48 autophone-3 pixel-02 disconnected USB3HUB2.1 15:07 autophone-3 rebooted 15:12 autophone-3 process finished starting 18:49 autophone-3 pixel-02 disconnected USB3HUB2.1 19:08 autophone-3 rebooted 19:13 autophone-3 process finished starting 20:33 autophone-3 pixel-02 disconnected USB3HUB2.1 22:52 autophone-3 rebooted 22:56 autophone-3 process finished starting No further issues past 22:56

2017-02-17 02:55 autophone-3 pixel-02 disconnected USB3HUB2.1 03:14 autophone-3 rebooted 03:18 autophone-3 process finished starting

06:31 autophone-1 nexus-6p-11 disconnected USB3HUB2.2 06:35 autophone-1 rebooted 06:39 autophone-1 process finished starting

08:25 autophone-3 pixel-02 disconnected USB3HUB2.1 08:44 autophone-3 rebooted 08:48 autophone-3 process finished starting

10:00 autophone-3 pixel-02 disconnected USB3HUB2.1 10:19 autophone-3 rebooted 10:23 autophone-3 process finished starting

12:04 autophone-3 pixel-02 disconnected USB3HUB2.1 12:23 autophone-3 rebooted 12:27 autophone-3 process finished starting I removed pixel-02 from the manifests to prevent this from reoccurring until we can fix the issue. https://github.com/mozilla/autophone/commit/f36f7948d5580d7e1dbf85a69f598e2e68421e50

brought autophone2 back online and worked with :bc to move some phones around (especially off the hub connected to autophone2).

With Van's help autophone-2 was rebooted. It appeared to have hung. We've moved most devices off of the hubs until we have a handle on the problem with autophone-2. nexus-6p-7, nexus-6p-12 moved from autophone-2 to autophone-3 pixel-02 moved from autophone-3 to autophone-2 deployed https://github.com/mozilla/autophone/commit/4df2d5a533e31a9ecee5b0e50f6e50db0eeaf3eb and rebooted everyone. nexus-6p-4 on autophone-3 (twitter-remote on mozilla-inbound, autoland) is missing but hopefully we can get it to come back to us. We'll revisit the hubs when we have more time. Thanks Van!