We need to run mochitests on fennec in the same fashion that we do for firefox. The issue with fennec is we need to run on a mobile device. Currently we support the Nokia 800/810 device, but we will support windows mobile devices in the near future. In addition to mobile devices, we can run on desktop versions as well if we find it necessary to supplement. Here are some of the issues with running on the mobile device (N8xx): 1) getting the device, physically locating it, managing it 2) connecting to the device (N810 has USB cable, N8xx has wifi) 3) setting up device with SSH access and other required tools 4) adding a memory card for additional storage, swap space 5) cross compiling a build of fennec in scratchbox 6) packaging up fennec (make package in $objdir/mobile) 7) packaging up test files (xulrunner/_profile, xulrunner/_tests) 8) copying packaged files to device (and not running out of room) 9) installing packages on device 10) running test drivers on device 11) uploading/copying results off device 12) dealing with 20+ hour runs for mochitests (maybe only run on nightly builds) 13) troubleshooting hangs, status, device issues remotely I have attached a run_tests_chunked.py file which is required to avoid a long list of OOM, Timeout, and hung test errors. This resolves the problem by running mochitests on each directory (not as a single test with many subdirs) and going a step further and splitting larger directories up into smaller chunks. Lastly, it times out after 60 minutes if you are hung and will terminate the test. I can provide some additional tools, scripts to aide in this process. My understanding is there is an existing framework developed to manage this process which might make my scripts not useful or incompatible.

Some helpful how-to pages on building and moving the tests over: * Building: https://wiki.mozilla.org/Mobile/Build/Fennec * Putting the tests on the device: https://wiki.mozilla.org/Mobile/Fennec_Mochitest And my scratchbox mozconfig file for building a test-enabled fennec: mk_add_options MOZ_BUILD_PROJECTS="xulrunner mobile" mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../mobile-dbg # clint additions ac_add_options --enable-debug ac_add_options --disable-optimize # ac_add_options --disable-dbus ac_add_options --disable-crashreporter ac_add_options --enable-tests ac_add_options --enable-mochitest # XULRunner options ac_add_app_options xulrunner --enable-application=xulrunner ac_add_app_options xulrunner --disable-javaxpcom # mobile options ac_add_app_options mobile --enable-application=mobile ac_add_app_options mobile --with-libxul-sdk=../xulrunner/dist

From talking with Stuart, the mochitest suite is just the first testsuite they are looking for. They actually want all the suites that are run in a typical UnitTest run for a desktop Firefox build: TUnit, reftest, crashtest, mochitest, chrome, browser Updating this bug to match this.

(In reply to comment #2) > From talking with Stuart, the mochitest suite is just the first testsuite they > are looking for. They actually want all the suites that are run in a typical > UnitTest run for a desktop Firefox build: > TUnit, reftest, crashtest, mochitest, chrome, browser I totally agree with this. However, from my understanding to date, the browser tests aren't going to work on Fennec because we've changed the underlying architecture. For example, there is no global gBrowser object in Fennec, and so I'd expect many of the tests to fail. Just trying to start it out using the command line: python runtests.py --appname=../../../../mobile/dist/bin/fennec --browser-chrome and I don't even get the browser-chrome test to come up, I just get a blank fennec window. So, I think the list we really should focus on is: TUnit, reftest, crashtest, mochitest, chrome I think that if we are going to get browser tests to work on Fennec, we're going to need to re-implement some of the tests and quite possibly refactor some of the test framework when it comes to browser-chrome.

We can not build on these n8x0 devices; instead we can only run tests on what was build previously elsewhere. Also, given how slow these devices are, we really need to be able to run separate unittest suites concurrently on different devices. Both of these requirements mean we need to be able to run unittests on pre-existing build. Adding dependency and moving to Future per discussions with Aki and Ted.

can we get an eta from ted as to when the unittest without build stuff is coming? would be nice to have releng tee'd up once this is ready.

I'm aiming to have it done in Q1, but that's kind of a SWAG. Joel is also proceeding pretty well without me with his own set of scripts. My work is a lot of taking what he's already doing and integrating it into the build system so that it's easy for everyone to use.

My original steps to this bug outline how what is necessary to get mochitests running on maemo. We already build in a scratchbox environment, package up the bits, and copy to the device. This process is the same for the tests. They come from the same place you build the fennec binary (the scratchbox environment). Right now each test suite can run on its own (mochitest, reftest, crashtest, chrome, xpcshell) independent of each other. One big problem is how to get mochitests to run in <1 day! That is not in scope for any bugs or discussed as a blocking issue here. As Ted mentioned above, he is taking pieces of the work I have done (as well as other TODO items) and rolling them into the build tree. The end result is we will still have to copy the test bits from the build tree to the device, unpack them and run them. This is the same process I originally outlined above. What is the ideal environment for the unittest to run on mobile?

My question in comment #7 is not too clear. My confusion stems from wanting to know what we need to do in order to get the unittests running on tinderbox/buildbot. It makes sense that we would only want to modify the tinderbox scripts once to accommodate fennec. The problem is we don't have a clear understanding of what we need to do as a final goal. Here are a list of issues we will need to overcome: * packaging up the tests (works great, ted's patch(s) makes it more uniform) * getting the tests to a device * running the tests reliably (using a script to chunk up the run) * reporting the results ** results are in chunked logs, not a single log ** results are on a remote device ** results have a long list of failures (http://people.mozilla.com/~jmaher/mochitest.htm) * need a method for removing tests which are not relevant on maemo (tests that require a cursor) * need a method for making the tests run faster (mochitests take 22+ hours now) such as running in parallel * dealing with communication and run time issues on other platforms. These are some of the big issues that I imagine we will need to figure out. So my question in comment #7 should be phrased more like "what framework is required for unittests to run in tinderbox?" I suspect the ideal environment is exactly what firefox has. In reality we need to figure out how we would like to see this work. If it requires a lot of patches to the test harnesses we need to outline those and get bugs filed. Since this is not well defined, I don't see us getting any unittests running in Q1 2009. So is it worth it to figure out an interim solution so we can get daily results for the 1.0 release?

Joel: if you can get your new steps to me and/or in this bug, I will attempt to automate them in buildbot.

Should be pretty similar to what catlee is doing in bug 383136.

after the rework of the test harnesses to support running on arbitrary builds, this has become a little easier. I have methods for running: Mochitest - https://wiki.mozilla.org/Mobile/Fennec_Mochitest Mochitest-chrome - https://wiki.mozilla.org/Mobile/Fennec_Chrome reftest/crashtest - https://wiki.mozilla.org/Mobile/Fennec_Reftests XPCShell - https://wiki.mozilla.org/Mobile/Fennec_Xpcshell step by step: 1) build in scratchbox 2) make package <- in $(objdir) 3) make package-test PYTHON=python2.5 <- in $(xr_objdir) 4) copy over .tar files to device storage card (/media/mmc1/release) 5) copy over http://hg.mozilla.org/qa/maemkit/ to /media/mmc1/maemkit 6) in the maemkit dir, run: "python maemkit-chunked.py --testtype=mochitest" 7) wait ~10 hours (non debug build) and view /media/mmc1/maemkit/log_mochitest.txt 8) repeat for --testtype=[chrome|reftest|crashtest|xpcshell] If there are issues, most likely it is related to the maemkit.cfg file which has the paths and config options. Those are the basic steps. If you get this far, lets sort out the way to split this across multiple devices (this works in maemkit.cfg with the total-clients and client-number options).

Package/upload maemo tests. Patch 1 of 2.

|make package-tests| breaks without these set =)

Comment on attachment 376148 [details] [diff] [review] add 'make package-tests' and upload xulrunner*.tar.bz2 Seems fine...out of curiosity, why do you need PYTHON=python2.5?

Comment on attachment 376151 [details] [diff] [review] update maemo mozconfig to enable tests afaict --enable-mochitest is implied with --enable-tests...is that incorrect for mobile?

re: both, I'm just following jmaher's directions =) I'm guessing python2.5 because the default python in scratchbox is python2.3. I can try without one or both on staging if you want.

we need PYTHON=python2.5 for the 'make package-tests' step where we package up the xpcshell tests. When ted rewrote the script to work on arbitrary builds in bug 421611, he made a requirement for python2.5. The scratchbox environment (cross compile to maemo) has python 2.3? which would not copy the xpcshell tests to the test package.

(In reply to comment #17) > we need PYTHON=python2.5 for the 'make package-tests' step where we package up > the xpcshell tests. When ted rewrote the script to work on arbitrary builds in > bug 421611, he made a requirement for python2.5. The scratchbox environment > (cross compile to maemo) has python 2.3? which would not copy the xpcshell > tests to the test package. OK, thanks for the explanation.

(In reply to comment #15) > (From update of attachment 376151 [details] [diff] [review]) > afaict --enable-mochitest is implied with --enable-tests...is that incorrect > for mobile? Ted just told me that --enable-mochitest doesn't exist anymore. Please remove it - --enable-tests should do the job here.

The PYTHON bit shouldn't be required anymore, I landed a patch that explicitly looks for the `python2.5` binary before `python`: http://mxr.mozilla.org/mozilla-central/source/configure.in#814 (this hasn't landed on 1.9.1, however)

(In reply to comment #20) > The PYTHON bit shouldn't be required anymore, I landed a patch that explicitly > looks for the `python2.5` binary before `python`: > http://mxr.mozilla.org/mozilla-central/source/configure.in#814 > (this hasn't landed on 1.9.1, however) ...umm... when can this land on 191? We're working on moving mobile builds to using 1.9.1 as we get closer to release date, so will need this there also, yes?

Didn't really seem necessary, but if you want it feel free to request approval on the patch in bug 427750.

As for PYTHON=python2.5, I think I'll leave it there in the meantime, so it works on mozilla-1.9.1 once the build goes green =) Unless the 1.9.1 review/landing process is relatively quick.

Comment on attachment 376440 [details] [diff] [review] remove --enable-mochitest rev 1136:a76c6dfdacbd Should get picked up automatically.

Comment on attachment 376148 [details] [diff] [review] add 'make package-tests' and upload xulrunner*.tar.bz2 rev 273:c90e65ea30d9 p-m reconfig'ed, kicked off test build.

Current status: o Running with jmaher's patches from bug 495164. o Mochitest keeps crashing on production, so I'm splitting it into mochitest1 and mochitest2. o Hoping jmaher's fixes stabilize the number of tests; I've created new unittest parsers to deal with additional knownFailCount since the functions in buildbotcustom.steps.unittest aren't easily overridable/customizable. o mochi, chrome, and xpcshell look like they'll be ready soon. crashtest and reftest are still 100% T-FAIL.

(crashtest and reftest tracked in bug 471579 )

These are running and reporting to the Mobile tinderbox tree. We should continue to adjust the threshold between orange and green, and bring the known mobile fail count to 0, but as far as running and reporting we're done.