We're switching the jetpack test harness to use the mochitest python code to run Firefox, this will give us many benefits like screenshot on fail, crash stacks etc. I have landed the harness code on Gum and now I want to get the new test jobs created and running there. The tests consist of two different jobs. jetpack-addon and jetpack-package. Both are run in exactly the same way as browser-chrome are, through runtests.py so hopefully this should be some pretty straightforward copy and pasting to get going. I have some blind copy-paste patches that might be a good start to attach in a moment. Once this all works on gum the plan would be to land the code on m-c and turn off the older JP tests there and turn on the new ones instead so this shouldn't use any more resources than at present.

Does some additional checks for parsing the jetpack test output in the same way as browser-chrome

Not really sure this is right and I haven't attempted to add it to gum's config as it wasn't obvious how to do that.

I'm not really sure who is right to look at these patches or help me get this working, catlee can you help me find someone?

Jordan, can you have a look?

sure np! I'm just back from PTO so my queue was abnormally large today. I'll have a peak tomorrow.

Comment on attachment 8476783 [details] [diff] [review] buildbotcustom changes Review of attachment 8476783 [details] [diff] [review]: ----------------------------------------------------------------- I think you've got a large chunk of it right but IIUC, we may need to tweak it a bit. first, let's see if we are on the same page: 1) we currently run jetpack via UnittestPackagedBuildFactory[1]? 2) we want to stop doing that and use mochitest to run jetpack ? if so, I think this patch is not needed. Mochitest suites are now run through mozharness. Mozharness scripts are called via ScriptFactory[2] not UnittestPackagedBuildFactory. hopefully all we need here is to add jetpack mochitest builders (your buildbot-config patch) and then support those builders in mozharness (your mozharness patch) I'll look at those patches now. [1] http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#4892 [2] http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#6049

Comment on attachment 8476782 [details] [diff] [review] mozharness changes Review of attachment 8476782 [details] [diff] [review]: ----------------------------------------------------------------- ::: configs/unittests/linux_unittest.py @@ +72,5 @@ > "mochitest-devtools-chrome-2": ["--browser-chrome", "--subsuite=devtools", "--chunk-by-dir=5", "--total-chunks=3", "--this-chunk=2"], > "mochitest-devtools-chrome-3": ["--browser-chrome", "--subsuite=devtools", "--chunk-by-dir=5", "--total-chunks=3", "--this-chunk=3"], > "mochitest-devtools-chrome-chunked": ["--browser-chrome", "--subsuite=devtools", "--chunk-by-dir=5"], > + "jetpack-package": ["--jetpack-package"], > + "jetpack-addon": ["--jetpack-addon"], to clarify, do we want to: 1) run one job and pass --jetpack-package and --jetpack-addon to runtests.py in a single call 2) run one job and call runtests.py twice, once with --jetpack-package and once with --jetpack-addon 3) run two jobs and call runtests.py only once for each job. passing --jetpack-package for one job and --jetpack-addon for the second up till now we have pretty much created one job for each suite (mochitest chunk 1, xpcshell, etc). Mozharness can do more than one suite(single runtests.py call) per script run(job) but if we want to do that, we might need to play with some dials: timeout limits, which suite goes first, etc.

Comment on attachment 8476784 [details] [diff] [review] buildbot-configs changes Review of attachment 8476784 [details] [diff] [review]: ----------------------------------------------------------------- I think this puts the pieces into place but it will not be run anywhere. i.e MOCHITEST_JP is never used you are going to want to specify which branch(es) you want these suites to run on. Right now we currently use Cedar to test out new or failing jobs. We should probably limit this out to there until we green them up. For that you will need something like: http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/config.py#1912 does this make sense? ::: mozilla-tests/config.py @@ +386,5 @@ > +MOCHITEST_JP = [ > + ('mochitest-jetpack', { > + 'use_mozharness': True, > + 'script_path': 'scripts/desktop_unittest.py', > + 'extra_args': ['--mochitest-suite', 'jetpack-package,jetpack-addon'], this will need to change depending on your answer to the question I ask in your buildbot-config patch.

(In reply to Jordan Lund (:jlund) from comment #8) > Comment on attachment 8476782 [details] [diff] [review] > mozharness changes > > Review of attachment 8476782 [details] [diff] [review]: > ----------------------------------------------------------------- > > ::: configs/unittests/linux_unittest.py > @@ +72,5 @@ > > "mochitest-devtools-chrome-2": ["--browser-chrome", "--subsuite=devtools", "--chunk-by-dir=5", "--total-chunks=3", "--this-chunk=2"], > > "mochitest-devtools-chrome-3": ["--browser-chrome", "--subsuite=devtools", "--chunk-by-dir=5", "--total-chunks=3", "--this-chunk=3"], > > "mochitest-devtools-chrome-chunked": ["--browser-chrome", "--subsuite=devtools", "--chunk-by-dir=5"], > > + "jetpack-package": ["--jetpack-package"], > > + "jetpack-addon": ["--jetpack-addon"], > > to clarify, do we want to: > > 1) run one job and pass --jetpack-package and --jetpack-addon to runtests.py > in a single call > > 2) run one job and call runtests.py twice, once with --jetpack-package and > once with --jetpack-addon > > 3) run two jobs and call runtests.py only once for each job. passing > --jetpack-package for one job and --jetpack-addon for the second > > up till now we have pretty much created one job for each suite (mochitest > chunk 1, xpcshell, etc). Mozharness can do more than one suite(single > runtests.py call) per script run(job) but if we want to do that, we might > need to play with some dials: timeout limits, which suite goes first, etc. I don't know that this matters much. Right now I think JP tests run as (2). These two tests tend to test the same code so it makes sense to just have them in one box on tbpl. If there is a standard way that makes most sense then let's just do that. (In reply to Jordan Lund (:jlund) from comment #9) > Comment on attachment 8476784 [details] [diff] [review] > buildbot-configs changes > > Review of attachment 8476784 [details] [diff] [review]: > ----------------------------------------------------------------- > > I think this puts the pieces into place but it will not be run anywhere. i.e > MOCHITEST_JP is never used > > you are going to want to specify which branch(es) you want these suites to > run on. Right now we currently use Cedar to test out new or failing jobs. We > should probably limit this out to there until we green them up. For that you > will need something like: > http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/config. > py#1912 > > does this make sense? > > ::: mozilla-tests/config.py > @@ +386,5 @@ > > +MOCHITEST_JP = [ > > + ('mochitest-jetpack', { > > + 'use_mozharness': True, > > + 'script_path': 'scripts/desktop_unittest.py', > > + 'extra_args': ['--mochitest-suite', 'jetpack-package,jetpack-addon'], > > this will need to change depending on your answer to the question I ask in > your buildbot-config patch. I didn't add this anywhere yet because I didn't really understand how to. The code to land and ship these harnesses in the test package is landed on Gum and I anticipated that there might need to be some tweaking both to the mozharness side and the m-c side to get it working. If it makes more sense to do that on cedar then that works. Once this all works these should be enabled like any other tests on m-c and related branches and then ride the trains down to release.

Any progress here?

(In reply to Dave Townsend [:mossop] from comment #11) > Any progress here? Oh sorry I thought you just needed feedback on your patch and I missed your follow-up comments. I am pretty busy trying to wrap up some goals so if you would like me to drive this I'll run it by my manager. Hopefully though there isn't much work. You might be very close. Might just a matter of taking this: 'extra_args': ['--mochitest-suite', 'jetpack-package,jetpack-addon'] And instead doing: 'extra_args': ['--mochitest-suite', 'jetpack-package', --mochitest-suite', 'jetpack-addon'] To accomplish number two mentioned above. And then to a enable on Cedar or gum. I suppose gum makes the most sense. Again in a enabling on a single branch is done via: http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/config.py#1912 Feel free to make those two changes and I'll review. Then we can iterate on gum :)

Ok here are those changes.

Comment on attachment 8476782 [details] [diff] [review] mozharness changes Sounds like this bit is ready to land?

Comment on attachment 8476782 [details] [diff] [review] mozharness changes Review of attachment 8476782 [details] [diff] [review]: ----------------------------------------------------------------- lgtm. this should be a good start. We may need to tweak mozharness more depending on how this does on gum

Comment on attachment 8486818 [details] [diff] [review] buildbot-configs changes Review of attachment 8486818 [details] [diff] [review]: ----------------------------------------------------------------- Awesome, this looks good to me. I think you meant to put gum instead of Cedar (see in-line). But I'm just thinking, I may be wrong about doing this on gum. cedar has the benefit of using mozharness 'default' branch instead of production. Production is used on most branches including gum. So if we need to make future changes, at least we can see the results on Cedar before merging possible broken patches to production. You mentioned you didn't mind where we test this so how about we leave your patch as is but fix the comment mentioning gum? r+ to land on default any time. ::: mozilla-tests/config.py @@ +1971,5 @@ > if slave_platform in BRANCHES['cedar']['platforms'][platform]: > BRANCHES['cedar']['platforms'][platform][slave_platform]['opt_unittest_suites'] += MOZBASE[:] > BRANCHES['cedar']['platforms'][platform][slave_platform]['debug_unittest_suites'] += MOZBASE[:] > > +# Enable webapprt-chrome tests on gum gum mentioned here @@ +1977,5 @@ > + for slave_platform in PLATFORMS[platform]['slave_platforms']: > + if slave_platform not in BRANCHES['gum']['platforms'][platform]: > + continue > + BRANCHES['cedar']['platforms'][platform][slave_platform]['opt_unittest_suites'] += MOCHITEST_JP[:] > + BRANCHES['cedar']['platforms'][platform][slave_platform]['debug_unittest_suites'] += MOCHITEST_JP[:] cedar is used for the above two lines

We will also need a tbpl patch so the job shows correctly. Without patch it might show up as some default like M instead of a custom one like MJP. This isn't such a big deal for Cedar but it will need to be done. I will ping Sheriffs to see if they have docs for this.

Landed: https://hg.mozilla.org/build/buildbot-configs/rev/4092544e664c https://hg.mozilla.org/build/mozharness/rev/4a63895bd8b2

In prod with reconfig on 2014-09-15 08:30 PT

All the appropriate things have landed now. How do I get cedar updated to the latest m-c so we can see how the tests are running?

(In reply to Dave Townsend [:mossop] from comment #20) > All the appropriate things have landed now. How do I get cedar updated to > the latest m-c so we can see how the tests are running? hg clone https://hg.mozilla.org/projects/cedar && cd cedar && hg pull https://hg.mozilla.org/mozilla-central && hg merge && hg commit -m "Merge mozilla-central into cedar" && hg push ssh://hg.mozilla.org/projects/cedar

(In reply to Ed Morley [:edmorley] from comment #21) > (In reply to Dave Townsend [:mossop] from comment #20) > > All the appropriate things have landed now. How do I get cedar updated to > > the latest m-c so we can see how the tests are running? > > hg clone https://hg.mozilla.org/projects/cedar && cd cedar && hg pull > https://hg.mozilla.org/mozilla-central && hg merge && hg commit -m "Merge > mozilla-central into cedar" && hg push ssh://hg.mozilla.org/projects/cedar Unfortunately there are merge conflicts in the web platform tests that it looks like are being experimented on on cedar right now so I'm not sure I know enough to resolve them.

(In reply to Dave Townsend [:mossop] from comment #22) > Unfortunately there are merge conflicts in the web platform tests that it > looks like are being experimented on on cedar right now so I'm not sure I > know enough to resolve them. Ah - James I don't suppose you could do an m-c import into Cedar for Dave? :-)

Something like https://hg.mozilla.org/projects/cedar/rev/649768be4000 ?

(In reply to James Graham [:jgraham] from comment #24) > Something like https://hg.mozilla.org/projects/cedar/rev/649768be4000 ? Wonderful, thanks