kgrandon, reported disk space issues here: https://tools.taskcluster.net/task-inspector/#4Wy1Uh6XQ--DMWEHlf2eIw/1 For some reasons docker-worker is running out of disk space on /mnt. I suspect it has something to do with misconfiguration after jlal reconfigured all workerTypes when the workerTypes azure table was dropped. I ssh'ed in and dumped some data here: https://gist.github.com/jonasfj/581d8356410c35fb2d39 Potential issues: 1) The number of caches in /mnt/var/cache/docker-worker/gaia-misc-caches is alarming. Shouldn't there atmost be <capacity> instances of any named cache. 2) `docker ps` and `docker ps -a` doesn't show the same number of containers, implying that there is some halted containers that just sits around. They should always be deleted rather quickly afaik. 3) `docker ps` indicates that there is a lot of logserve containers running. Somehow it seems that they aren't killed. instance-id: i-ff57e4f1 Note, this seems to happening on multiple instances, not this single instance. See the other runs on task.

Note, it appears my previous "note" was wrong. This is not happening for all instances. I feared it was one of those everything is broken things, but I just looked wrong at the other runs which seems to have been running currently.

This has been resolved on multiple levels with reducing the amount of lingering containers, better garbage collection checks around volumes, and handling stalled containers.