They've already been scheduled on cedar. It looks like crashtests, reftests and xpcshell tests are already green and can be immediately scheduled on the new build for all branches (and subsequently removed from the ics_armv7a_gecko build). Mochitests have some issues I'm currently investigating, and Mn tests have some intermittent failures. Aki, would you prefer I filed separate bugs per test suite or just keep track of everything here?

I'm ok tracking things here. I would like to see the results from https://tbpl.mozilla.org/?tree=Cedar&rev=1be607d22e17 so we can decide what to do with debug this first pass.

This is complex enough that I need to do a builder list dump & compare. Also, our mozilla-tests/b2g_config.py has gotten seriously ugly. Hopefully as we get rid of the old emulator builds it'll get less so.

This is the diff in builders. This may be more far-reaching than we'd like, or it may normalize things in a way we wanted in the first place. I'm not sure I know.

Comment on attachment 766116 [details] [diff] [review] (configs) full emulator test rollout, part 1 Also see the builder diff -- that's what I used to do a short eyeball verification.

Looking at Cedar, the right builder diff would only be adding emulator_vm crashtest-1 and -2 and -3, emulator_vm xpcshell, and emulator (without the vm) reftest-1 through -10. We can't run reftests on VMs yet, but since we can run crashtests and xpcshell on VMs, we don't have any need to run those on hardware.

Comment on attachment 766171 [details] [diff] [review] builder list diff Review of attachment 766171 [details] [diff] [review]: ----------------------------------------------------------------- What philor said. I think we only want to run each suite on either a VM or physical hardware, but not both. I believe R is the only suite not capable of running on the VM's at the moment so: C, X on b2g_emulator_vm R on b2g_emulator ::: clean_builder @@ -3755,4 @@ > -b2g_emulator cedar opt test crashtest-1 ScriptFactory > -b2g_emulator cedar opt test crashtest-2 ScriptFactory > -b2g_emulator cedar opt test crashtest-3 ScriptFactory > -b2g_emulator cedar opt test marionette-webapi ScriptFactory Hard to tell from the diff, but is this removing the marionette-webapi on b2g_emulator from cedar? Or will it still be running on the VM? We still need to fix some test failures for this suite on cedar.

Comment on attachment 766116 [details] [diff] [review] (configs) full emulator test rollout, part 1 Review of attachment 766116 [details] [diff] [review]: ----------------------------------------------------------------- > +b2g_emulator_vm birch opt test crashtest-1 ScriptFactory This doesn't work yet. See bug 818968. ::: mozilla-tests/b2g_config.py @@ +279,5 @@ > 'unittest-env': {'DISPLAY': ':0'}, > 'enable_opt_unittests': True, > 'enable_debug_unittests': False, > 'fedora-b2g': { > + 'opt_unittest_suites': MOCHITEST + MARIONETTE, this change would affect all branches, including b2g18*, on which we don't use Ubuntu yet (trains...) I'm not sure if we want to switch to Ubuntu with this patch.

Ok. I was trying to do the right thing and not make our configs fuglier, but it looks like our requirements won't allow for that. So: can someone tell me specifically what I should turn off and what I should turn on, or should I try to figure it out from the above multiple comments and most likely get it wrong again?

So the latest Cedar push made all of the crashtests and reftests (which were previously green) orange. I don't know why, but I think you should hold off on this bug for a little while. Sorry, it is super confusing with the two emulator builds + vm/hw + opt/debug. I just wanted to switch the suites over incrementally so we could avoid bustage like what just happened. But looks like we'll need to spend some more time on them now on Cedar.

Probably due to https://bugzilla.mozilla.org/show_bug.cgi?id=885802#c5 .

(In reply to Rail Aliiev [:rail] from comment #7) > > +b2g_emulator_vm birch opt test crashtest-1 ScriptFactory > > This doesn't work yet. See bug 818968. It looks like crashtests work, it's just reftests that don't. So... Suites to enable: b2g_emulator_vm opt - C, X b2g_emulator opt - R Suites to disable: ics_armv7a_gecko_vm opt - C, X ics_armv7a_gecko opt - R (also C, X on cedar) I haven't seen b2g_emulator results for debug yet, so let's leave that as is.

...and now cedar reftests/crashtests are busted for both the old and new emulator builds

Do we know what happened? It's odd to me that old emulator tests are busted, but m-c isn't.

No, I'm not sure. It looks like bug 882888 except not intermittent. I'll see if I can find some time to look at it tomorrow.

I think I fixed the OnReftestLoad problem, so when bug 882888 lands we can go ahead with this. The mochitests got fixed with bug 885454 and are also now all green. To clarify: Step 1 - enable suites: B2G Emu opt - R, Mn B2G Emu (VM) opt - M, C, X Step 2 - disable suites: B2G Arm opt - All tests and builds! Notes: * Ideally we want to run Mn tests on the (VM) pool, but there are still some intermittent oranges there. If we land on the hardware pool first we'd be able to nuke the ics_armv7a_gecko builds wholesale. If you'd prefer to have them running on the (VM) pool right away we could disable the tests or look into the failures first. * We might want to do step 1 before step 2 just to be safe, though I'll leave that up to you.

Also just to clarify, this is only for m-c based branches. The b2g18 branches should remain as is.

Thanks Andrew!

I'm guessing the cedar bit is just a re-ordering.

Comment on attachment 768668 [details] [diff] [review] as requested https://hg.mozilla.org/build/buildbot-configs/rev/82f84bdf4492

Reconfiged.

Any reason birch was left out on this? As the main integration branch for b2g, I would think running these builds and tests there would be of high importance.

Weird, I didn't special case birch at all.

I see them running now. Sorry for the false alarm.

This has been running for awhile now without problem. I'll file a new bug for turning off the old builds.

I see this in buildbot-configs: 1092 # emulator hacks. See bug 885456 1093 # MERGE DAY This will someday ride trains... 1094 for branch in BRANCHES.keys(): 1095 if branch in ('mozilla-aurora', 'mozilla-beta', 'mozilla-release', 1096 'mozilla-esr17', 'mozilla-b2g18_v1_0_0', 1097 'mozilla-b2g18_v1_0_1'): 1098 if 'emulator' in BRANCHES[branch]['platforms']: 1099 del BRANCHES[branch]['platforms']['emulator'] When is the code suppose to be riding the trains?

(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #25) > I see this in buildbot-configs: > 1092 # emulator hacks. See bug 885456 > 1093 # MERGE DAY This will someday ride trains... > 1094 for branch in BRANCHES.keys(): > 1095 if branch in ('mozilla-aurora', 'mozilla-beta', 'mozilla-release', > 1096 'mozilla-esr17', 'mozilla-b2g18_v1_0_0', > 1097 'mozilla-b2g18_v1_0_1'): > 1098 if 'emulator' in BRANCHES[branch]['platforms']: > 1099 del BRANCHES[branch]['platforms']['emulator'] > > When is the code suppose to be riding the trains? Probably now, now that b2g26 is riding the trains.