When merging to m-c, it's standard practice for our team to wait until all builds have started before merging the next tree over (to avoid long retrigger delays in the even of merge-induced bustage). With 10+min lags starting AWS-based builds being the new norm (and expectation), this makes for lots of idle time waiting for builds to start, hurting productivity. If we were guaranteed to never coalesce these builds, it would speed up the process. Also, the same AWS-induced lag often lengthens tree closures on backouts while we wait for builds to start on bustage backouts (i.e. if Android is busted, we'd like to for sure get an Android build on the backout and will keep the tree closed until it starts).

Do you have any suggestions for how to identify merges and backouts, or are you just looking for a keyword you can use?

We use pretty standard terminology in our commit messages, so they should be pretty easy to find. See lines 22-24 below for example: https://mxr.mozilla.org/webtools-central/source/tbpl/mcmerge/js/Config.js#22

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #2) > We use pretty standard terminology in our commit messages, so they should be > pretty easy to find. See lines 22-24 below for example: > https://mxr.mozilla.org/webtools-central/source/tbpl/mcmerge/js/Config.js#22 Those are pretty generic....f we look at more than just the tip commit that's going to be problematic on branches that don't need those semantics. Eg, most project branches.

(In reply to Ben Hearsum [:bhearsum] from comment #3) > Those are pretty generic....f we look at more than just the tip commit > that's going to be problematic on branches that don't need those semantics. > Eg, most project branches. We only need to look at the first line of the tip commit.

(In reply to Ed Morley [:edmorley UTC+0] from comment #4) > (In reply to Ben Hearsum [:bhearsum] from comment #3) > > Those are pretty generic....f we look at more than just the tip commit > > that's going to be problematic on branches that don't need those semantics. > > Eg, most project branches. > > We only need to look at the first line of the tip commit. Yeah, I'm not sure what the Buildbot code does though - it may look at all of them. (If CLOBBER works in a merged commit, that's almost certainly true.)

The CLOBBER behaviour is triggered by a change to a checked in file (hence working across merges), whereas we're talking about commit messages for this bug; I'm confused?

*All* coalescing of builds was supposed to be disabled in bug 1013493 (note that the hg poll interval is 60s, so we can't get any less than that... but IMO that's as good as coalescing disabled) - so we must have regressed bug 1013493. Chris, don't suppose you could take a look?

(In reply to Ed Morley [:edmorley UTC+0] from comment #7) > *All* coalescing of builds was supposed to be disabled in bug 1013493 (note > that the hg poll interval is 60s, so we can't get any less than that... but > IMO that's as good as coalescing disabled) - so we must have regressed bug > 1013493. > > Chris, don't suppose you could take a look?

Regressed bug 684436 that is. Sigh, not with it this afternoon lol...

So, slight clarification here, bug 684436 was about disabling the 'tree stable timer'. i.e. stop waiting 3 minutes after seeing a change in hg before triggering jobs. What's going on here (according to comment #0) is that you want to avoid coalescing between pushes from e.g. m-i to m-c, and a subsequent push from fx-team to m-c. Because of the lag we see sometimes when starting up builds, you currently need to wait to see builds running on the initial push of m-i to m-c before pushing fx-team to m-c. If you pushed them at least a minute apart, you would see 2 sets of jobs scheduled, but possibly would see coalescing if there weren't machines available immediately to take the 1st set before you pushed the second set. Is that a correct summary? If so, I believe the solution is to adjust the buildbot merge functions to disallow coalescing between pushes that look like merge commits.

No, I'd like to disable coalescing (aside from the unavoidable 60s hg poller interval) between all builds on all pushes to trunk trees - which aiui we were closer to having in the past, but seems to have regressed since.

We haven't changed anything that would have changed behaviour here lately. Regardless, I think we can do this pretty easily if you want to disable all coalescing of builds on trunk trees. These are mozilla-central, mozilla-aurora, mozilla-beta, mozilla-esr*? What else?

(In reply to Chris AtLee [:catlee] from comment #12) > Regardless, I think we can do this pretty easily if you want to disable all > coalescing of builds on trunk trees. These are mozilla-central, > mozilla-aurora, mozilla-beta, mozilla-esr*? What else? mozilla-central & all trees that are sheriff-managed + release, I think? ie: mozilla-central mozilla-inbound b2g-inbound fx-team + mozilla-aurora mozilla-beta mozilla-release mozilla-esr24 mozilla-b2g30_v1_4 mozilla-b2g28_v1_3 mozilla-b2g28_v1_3t Thanks :-)

This adds the following builders to buildbotcustom.misc.nomergeBuilders: --- old.txt 2014-11-27 17:08:20.000000000 -0500 +++ new.txt 2014-11-27 17:15:42.000000000 -0500 +Android 2.3 Armv6 mozilla-esr31 build +Android 2.3 Debug b2g-inbound build +Android 2.3 Debug fx-team build +Android 2.3 Debug mozilla-aurora build +Android 2.3 Debug mozilla-beta build +Android 2.3 Debug mozilla-central build +Android 2.3 Debug mozilla-inbound build +Android 2.3 Debug mozilla-release build +Android 2.3 b2g-inbound build +Android 2.3 fx-team build +Android 2.3 mozilla-aurora build +Android 2.3 mozilla-beta build +Android 2.3 mozilla-central build +Android 2.3 mozilla-inbound build +Android 2.3 mozilla-release build +Android 4.2 x86 b2g-inbound build +Android 4.2 x86 fx-team build +Android 4.2 x86 mozilla-aurora build +Android 4.2 x86 mozilla-beta build +Android 4.2 x86 mozilla-central build +Android 4.2 x86 mozilla-inbound build +Android 4.2 x86 mozilla-release build +Linux b2g-inbound build +Linux b2g-inbound leak test build +Linux fx-team build +Linux fx-team leak test build +Linux mozilla-aurora build +Linux mozilla-aurora leak test build +Linux mozilla-b2g30_v1_4 build +Linux mozilla-b2g30_v1_4 leak test build +Linux mozilla-b2g32_v2_0 build +Linux mozilla-b2g32_v2_0 leak test build +Linux mozilla-b2g34_v2_1 build +Linux mozilla-b2g34_v2_1 leak test build +Linux mozilla-beta build +Linux mozilla-beta leak test build +Linux mozilla-central build +Linux mozilla-central leak test build +Linux mozilla-esr31 build +Linux mozilla-esr31 leak test build +Linux mozilla-inbound build +Linux mozilla-inbound leak test build +Linux mozilla-release build +Linux mozilla-release leak test build +Linux x86-64 b2g-inbound asan build +Linux x86-64 b2g-inbound build +Linux x86-64 b2g-inbound debug asan build +Linux x86-64 b2g-inbound debug static analysis build +Linux x86-64 b2g-inbound leak test build +Linux x86-64 b2g-inbound valgrind +Linux x86-64 fx-team asan build +Linux x86-64 fx-team build +Linux x86-64 fx-team debug asan build +Linux x86-64 fx-team debug static analysis build +Linux x86-64 fx-team leak test build +Linux x86-64 fx-team valgrind +Linux x86-64 mozilla-aurora asan build +Linux x86-64 mozilla-aurora build +Linux x86-64 mozilla-aurora debug asan build +Linux x86-64 mozilla-aurora leak test build +Linux x86-64 mozilla-b2g30_v1_4 build +Linux x86-64 mozilla-b2g30_v1_4 leak test build +Linux x86-64 mozilla-b2g32_v2_0 build +Linux x86-64 mozilla-b2g32_v2_0 leak test build +Linux x86-64 mozilla-b2g34_v2_1 build +Linux x86-64 mozilla-b2g34_v2_1 leak test build +Linux x86-64 mozilla-beta asan build +Linux x86-64 mozilla-beta build +Linux x86-64 mozilla-beta debug asan build +Linux x86-64 mozilla-beta leak test build +Linux x86-64 mozilla-central asan build +Linux x86-64 mozilla-central build +Linux x86-64 mozilla-central debug asan build +Linux x86-64 mozilla-central debug static analysis build +Linux x86-64 mozilla-central leak test build +Linux x86-64 mozilla-central valgrind +Linux x86-64 mozilla-esr31 build +Linux x86-64 mozilla-esr31 leak test build +Linux x86-64 mozilla-inbound asan build +Linux x86-64 mozilla-inbound build +Linux x86-64 mozilla-inbound debug asan build +Linux x86-64 mozilla-inbound debug static analysis build +Linux x86-64 mozilla-inbound leak test build +Linux x86-64 mozilla-inbound valgrind +Linux x86-64 mozilla-release asan build +Linux x86-64 mozilla-release build +Linux x86-64 mozilla-release debug asan build +Linux x86-64 mozilla-release leak test build +OS X 10.7 64-bit b2g-inbound leak test build +OS X 10.7 64-bit fx-team leak test build +OS X 10.7 64-bit mozilla-aurora leak test build +OS X 10.7 64-bit mozilla-b2g30_v1_4 leak test build +OS X 10.7 64-bit mozilla-b2g32_v2_0 leak test build +OS X 10.7 64-bit mozilla-b2g34_v2_1 leak test build +OS X 10.7 64-bit mozilla-beta leak test build +OS X 10.7 64-bit mozilla-central leak test build +OS X 10.7 64-bit mozilla-esr31 leak test build +OS X 10.7 64-bit mozilla-inbound leak test build +OS X 10.7 64-bit mozilla-release leak test build +OS X 10.7 b2g-inbound build +OS X 10.7 fx-team build +OS X 10.7 mozilla-aurora build +OS X 10.7 mozilla-b2g30_v1_4 build +OS X 10.7 mozilla-b2g32_v2_0 build +OS X 10.7 mozilla-b2g34_v2_1 build +OS X 10.7 mozilla-beta build +OS X 10.7 mozilla-central build +OS X 10.7 mozilla-esr31 build +OS X 10.7 mozilla-inbound build +OS X 10.7 mozilla-release build +WINNT 5.2 b2g-inbound build +WINNT 5.2 b2g-inbound leak test build +WINNT 5.2 fx-team build +WINNT 5.2 fx-team leak test build +WINNT 5.2 mozilla-aurora build +WINNT 5.2 mozilla-aurora leak test build +WINNT 5.2 mozilla-b2g30_v1_4 build +WINNT 5.2 mozilla-b2g30_v1_4 leak test build +WINNT 5.2 mozilla-b2g32_v2_0 build +WINNT 5.2 mozilla-b2g32_v2_0 leak test build +WINNT 5.2 mozilla-b2g34_v2_1 build +WINNT 5.2 mozilla-b2g34_v2_1 leak test build +WINNT 5.2 mozilla-beta build +WINNT 5.2 mozilla-beta leak test build +WINNT 5.2 mozilla-central build +WINNT 5.2 mozilla-central leak test build +WINNT 5.2 mozilla-esr31 build +WINNT 5.2 mozilla-esr31 leak test build +WINNT 5.2 mozilla-inbound build +WINNT 5.2 mozilla-inbound leak test build +WINNT 5.2 mozilla-release build +WINNT 5.2 mozilla-release leak test build +WINNT 6.1 x86-64 b2g-inbound build +WINNT 6.1 x86-64 b2g-inbound leak test build +WINNT 6.1 x86-64 fx-team build +WINNT 6.1 x86-64 fx-team leak test build +WINNT 6.1 x86-64 mozilla-central build +WINNT 6.1 x86-64 mozilla-central leak test build +WINNT 6.1 x86-64 mozilla-inbound build +WINNT 6.1 x86-64 mozilla-inbound leak test build +linux64-br-haz_b2g-inbound_dep +linux64-br-haz_fx-team_dep +linux64-br-haz_mozilla-aurora_dep +linux64-br-haz_mozilla-beta_dep +linux64-br-haz_mozilla-central_dep +linux64-br-haz_mozilla-inbound_dep +linux64-br-haz_mozilla-release_dep

In production: https://hg.mozilla.org/build/buildbot-configs/rev/9d05b7845146

In production: https://hg.mozilla.org/build/buildbotcustom/rev/4a4350574867