To best understand this idea please open this link: https://tools.taskcluster.net/provisioners/releng-hardware/worker-types To better detect issues and problematic machines, after clicking on a specific worker type, I suggest adding an option in the "filter by:" drop down menu, right under "Quarantine", that would say "Failures". Allowing us to sort all the workers with a failed (red) task state first, followed by exceptions (yellow), running (blue) and completed (green) as last. This would allow us to better and faster identify the workers that are failing tests. I see 2 options for how the sorting priority on this should work. Option 1 is: MDC1 red > yellow > blue > green > MDC2 red > yellow > blue > green. Option 2 would be: MDC1 red > MDC2 red > MDC1 yellow > MDC2 yellow > and so on... Personally i would be more inclined towards the 2nd option.

A secondary idea is having an overall check for whenever a machine has the last ~10 jobs non-green, to alert the build duty / task cluster teams via e-mail so that we can look at what causes the issue and decide if quarantine is needed.