One of the reasons bug 823989 was overlooked for some time, is that the permaorange tests were on Nightlies only. Currently tests run on Nightlies have "pgo" in their buildername, meaning they appear on the pgo row with everything else. Therefore a green pgo test result can incorrectly convince people looking at the tree, that the orange was just intermittent, when in fact it was permaorange, but just on Nightly. If we s/pgo/nightly/ for the buildernames, I can then adjust TBPL to create a new nightly row, giving us separate rows for {opt,debug,pgo,nightly}.

I suspect that giving them different names would cause them to be treated differently for self-serve and coalescing - if so, good. I got lucky, and when aurora coalesced a test against a dep build and a test against a nightly on one push into a single test against a dep build on the push above it, self-serve's retrigger for the only thing it listed as a result did happen to retrigger the test against the nightly, so I didn't have to re-run nightly builds and tests on every platform to get a single test suite on a single platform.

We need this fairly soon to ensure we don't have another instance like bug 823989 on the next uplift.

let's make this soft-block the next migration bug. hopefully we can get to it by then....

This never made it for today's merge day.

Would a better change be to make dep pgo builds the same, code-wise, as a nightly?

We have nightly-only bustage again on mozilla-central - which wasn't obvious initially due to the identical builder names :-(

so one thing I think we could do in the meanwhile is prevent tests for nightly builds from being coalesced with tests from dep builds.

Where would one start with trying to implement this? :-)

Bug 994686 was overlooked for a day or so because of this.

Doing this will nearly double the number of builders we have, which isn't really a good option. We already set the "nightly_build" property for these builds/tests. If we disabled coalescing between nightly and non-nightly tests, would that be sufficient to detangle nightly-specific bustage?

(In reply to Chris AtLee [:catlee] from comment #10) > Doing this will nearly double the number of builders we have, which isn't > really a good option. It makes me sad that buildbot's architecture makes this problematic :-( That said, I don't think we'll be doubling - we currently have {opt, debug, pgo} and we're adding one more. Also, the buildbot builders are specific to a tree, right? So only the trees with nightlies enabled should need those buildernames? > We already set the "nightly_build" property for these builds/tests. If we > disabled coalescing between nightly and non-nightly tests, would that be > sufficient to detangle nightly-specific bustage? The problem is that the only way things like TBPL (and even treeherder, given the bandwidth issues they've experienced with bugs filed to extract the required info from buildbot) have to differentiate between a whole bunch of jobs is the buildername. So whilst coalescing is indeed a problem too, it's as much the ability to separate jobs out in the UI so humans don't miss them, that's the reason for this bug being filed. Totally understand we're between a rock and a hard place - and ultimately the best long term fix of all is to move all data mangling into buildbot/mozharness/wherever that's upstream of builds-4hr/pulse, so downstream consumers don't have to maintain mappings and extract a whole bunch of only vaguely related metadata from a single buildername string (eg: https://github.com/mozilla/treeherder-service/blob/master/treeherder/etl/buildbot.py).

I'm disabling coalescing for jobs with the 'nightly_build' property set in bug 1013493.