In bug 1012633 it took some time for us to realise that there was a backlog building up - since the normal metric of "are there more than N total queued jobs" missed the fact that most jobs were being picked up by a machine and run fine, those specific ones were not. In order to not miss this in future, we should add a Nagios alert (or fix/tweak the current rule if we already have something similar) that checks for pending (ie scheduled but not started) jobs in the scheduler DB that have a submit time > N hours ago. Up for suggestions as to what we set N to - and we may need to vary N depending on whether the job is scheduled on a main tree ({mozilla-central, mozilla-inbound, b2g-inbound, fx-team, mozilla-aurora, mozilla-beta, mozilla-release, mozilla-b2g*, ...}) or a lower priority tree ({try, cedar, ash, ...}).

Similar/dupe of bug 978956?

Yeah though that states it's just for tests; happy for you to dupe either way and/or morph

Given the total lack of activity in that bug, I don't really care at this point.

FWIW we have https://secure.pub.build.mozilla.org/builddata/reports/reportor/hourly/long_jobs/long_jobs.html or https://secure.pub.build.mozilla.org/builddata/reports/reportor/hourly/long_jobs/long_jobs.json in json form