Currently we have both the marionette unit tests and the web api tests run on the same job on B2G. Unfortunately WebAPI tests have a high intermittent count (either from b2g or gecko crashes) and are on the verge of being hidden on TBPL. See https://tbpl.mozilla.org/?tree=Try&rev=8fa5dc94a2d2 as an example of how intermittent they are. Losing the Mn tests from TBPL has the potential to allow marionette regressions on to b2g which has a bigger impact than potentially just letting through WebAPI regressions. we should split the suites or have it as a "chunk" in Mn

Thank you for filing this after our meeting Monday - splitting these two into separate jobs will: * Mean any future hiding of webapi tests will not affect the visibility of marionette sanity tests * Make it easier for sheriffs and other devs to know where to file bugs against when issues occur (as well as reduce the tainting of Marionette as a harness by flaky webapi tests; which would inevitably be followed by the boy-who-called-wolf effect, similar to what happened with Android infra issues a few years ago).

Comment on attachment 8448253 [details] [diff] [review] Part 1 - Update buildconfig to split Mn and Mnw Review of attachment 8448253 [details] [diff] [review]: ----------------------------------------------------------------- Looks good so far; you'll also have to add a 'marionette' section to the emulator's 'suite_config', here: http://hg.mozilla.org/build/buildbot-configs/file/680a941f66fe/mozilla-tests/b2g_config.py#l1095

now with 100% more manifest file

Comment on attachment 8488081 [details] [diff] [review] Create a manifest just for marionette web-api tests Review of attachment 8488081 [details] [diff] [review]: ----------------------------------------------------------------- lgtm

Comment on attachment 8488065 [details] [diff] [review] Update buildconfig to split Mn and Mnw Review of attachment 8488065 [details] [diff] [review]: ----------------------------------------------------------------- Looks good so far. A couple of other changes you'll need: Everywhere you see 'marionette-webapi', like http://hg.mozilla.org/build/buildbot-configs/file/4092544e664c/mozilla-tests/b2g_config.py#l1124, you'll need to add a --test-manifest=webapi-tests.ini argument. Everywhere you see 'marionette-webapi', you'll need to add a new block for 'marionette' (the one you just created) and pass it the same --cfg arg as 'marionette-webapi' currently uses, but no need to include a --test-manifest argument.

Updated as suggested, if you think it is ready for review please defer to the correct person.

Comment on attachment 8489916 [details] [diff] [review] Update buildconfig to split Mn and Mnw Review of attachment 8489916 [details] [diff] [review]: ----------------------------------------------------------------- Almost ready for review! ::: mozilla-tests/b2g_config.py @@ +1376,4 @@ > 'marionette-webapi': { > 'extra_args': [ > "--cfg", "marionette/automation_emulator_config.py", > + "--manifest", "webapi-tests.ini" Should be --test-manifest, see http://hg.mozilla.org/build/mozharness/file/d75b03fc4d56/scripts/marionette.py#l99 @@ +1883,3 @@ > 'marionette-webapi': { > 'extra_args': [ > "--cfg", "marionette/automation_emulator_config.py", Need to add the --test-manifest arg here. @@ +1916,3 @@ > 'marionette-webapi': { > 'extra_args': [ > "--cfg", "marionette/automation_emulator_config.py", Need to add the --test-manifest arg here.

Changes made as suggested, if there isnt anything else you can suggest can you recommend a reviewer?

Comment on attachment 8491150 [details] [diff] [review] Update buildconfig to split Mn and Mnw Review of attachment 8491150 [details] [diff] [review]: ----------------------------------------------------------------- lgtm :) just a sanity check, IIUC the suite we are adding, 'marionette', looks to be the exact same as 'marionette-webapi' used to be prior to this patch. I can see how we add a manifest to marionette-webapi which I assume tells that suite to run webapi component only. but how does the new suite/job we are adding run only marionette and not the webapi part? Does that need a manifest too or am I grepping things wrong?

(In reply to Jordan Lund (:jlund) from comment #16) > Comment on attachment 8491150 [details] [diff] [review] > Update buildconfig to split Mn and Mnw > > Review of attachment 8491150 [details] [diff] [review]: > ----------------------------------------------------------------- > > lgtm :) > > just a sanity check, IIUC the suite we are adding, 'marionette', looks to be > the exact same as 'marionette-webapi' used to be prior to this patch. I can > see how we add a manifest to marionette-webapi which I assume tells that > suite to run webapi component only. > > but how does the new suite/job we are adding run only marionette and not the > webapi part? Does that need a manifest too or am I grepping things wrong? Currently there is a new manifest in central for only webapi but because this change isnt being rolled out to all twigs/branches I want to get my build config change rolled out and then on cedar remove the webapi tests from the main manifest and just triple check (mainly to keep the sheriffs from locking me up and throwing away the key) that I don't break anything when i try do this on Inbound. There shouldnt be any weird dependencies but want to check that before the major rollout.

what's the process for getting this landed and rolled out? Do I need to set any flags?

(In reply to David Burns :automatedtester from comment #18) > what's the process for getting this landed and rolled out? Do I need to set > any flags? land on default at hg.m.o/build/buildbot-configs whenever (it's a no-op) that will run through some tests and show up in #releng whether it passed or failed. releng will merge it to production and roll it out across our integration system ~3 times a week. this bug will be updated whenever that happens.

In prod with reconfig on 2014-09-22 08:20 PT

This broke Mnw on Aurora (and probably the other B2G release branches as well): https://tbpl.mozilla.org/php/getParsedLog.php?id=48588232&tree=Mozilla-Aurora Also, is it expected that Mn appears to be running the full webapi suite as well? Based on the test counts, it appears that way. Mnw before: marionette: 533/0/42 Mn after: marionette: 535/0/42 Mnw after: marionette: 203/0/9

I've reverted the buildbot-configs patch. http://hg.mozilla.org/build/buildbot-configs/rev/08c6e52d150c I wasn't comfortable backporting the in-tree manifest patch to the various release branches due to not being sure how comparable the list of tests is from branch to branch. I'd rather not try to sort that out while the tree is actively burning. But to be clear, we will need in-tree manifests backported to Aurora, b2g32, b2g30, and b2g28_v1_3t before this goes live in production again. They can land a=test-only.

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #22) > Also, is it expected that Mn appears to be running the full webapi suite as > well? Based on the test counts, it appears that way. > Mnw before: marionette: 533/0/42 > Mn after: marionette: 535/0/42 > Mnw after: marionette: 203/0/9 OK, I guess this part was expected based on the earlier comments in the bug.

Please land the manifest in Aurora, b2g32, b2g30, and b2g28_v1_3t (see comment 23)

https://hg.mozilla.org/releases/mozilla-aurora/rev/205e1704dcb6 https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/f18695cae418 https://hg.mozilla.org/releases/mozilla-b2g30_v1_4/rev/0b494a495ca3 v1.3t is near death, so I'm not going to bother landing this there.

So apparently the manifest cleanups I snuck in with these patches made Mnw perma-fail (see bug 1067363). Fantastic. Reverted that part: https://hg.mozilla.org/mozilla-central/rev/ad2e5aff3a3a https://hg.mozilla.org/releases/mozilla-aurora/rev/973e99d258a8 https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/75521744e0e8 https://hg.mozilla.org/releases/mozilla-b2g30_v1_4/rev/1dad764fbd52

Comment on attachment 8500039 [details] [diff] [review] remove WebApi tests from Marionette unit tests manifests: Review of attachment 8500039 [details] [diff] [review]: ----------------------------------------------------------------- lgtm

https://hg.mozilla.org/releases/mozilla-aurora/rev/b9d04c58580a https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/4c0eb73dfac7 https://hg.mozilla.org/releases/mozilla-b2g30_v1_4/rev/aae7c12029c2

https://hg.mozilla.org/releases/mozilla-b2g28_v1_3/rev/3a208c39bcb9 https://hg.mozilla.org/releases/mozilla-b2g28_v1_3t/rev/3a208c39bcb9 Don't ask.