Using the patch in attachment 367205 [details] [diff] [review] from bug 421611, I packaged a build and all its tests, then unpacked the tests on the same machine and ran them like so: cat xpcshell/tests/all-test-dirs.list | sed "s|^|./xpcshell/tests/|" | xargs python xpcshell/runxpcshelltests.py --keep-going --xre-path=./firefox/ ./bin/xpcshell A lot of tests are failing. I'm not sure if it's problems with the tests or files that I'm not packaging, or what. I'll attach the full test log and try to narrow it down from there.

Joel, are any of these failures you've seen in Fennec testing?

test_bug455213.js is failing because it isn't finding the plugins directory alongside the CurProcD. Presumably there is some new way of finding the binary directory of firefox?

"GreD" will get the Firefox dir in this case.

I have done this for Fennec and run the tests on the device. Here is a list of tests that fail on Fennec: http://people.mozilla.com/~jmaher/xpcshell.html there are 26 tests that fail. I need to update the list a bit with some adjusted bug numbers which should happen soon.

I changed the test packaging to put the test plugin in bin/plugins (next to xpcshell), and that fixes that particular test.

http://mavra.perilith.com/~luser/latest-teds-builds/ There's a mac build and test package. I don't think I can fix all these tests myself. python -u xpcshell/runxpcshelltests.py --xre-path=./firefox/ ./bin/xpcshell ./xpcshell/tests/test_update/unit/ will run that directory of tests.

Looking into this test: parser/xml/test/unit/test_parser.js It appears that the test is in fact working correctly, but the error handler is called with an empty string for an error message, which the test treats as no error. I think the stringbundle isn't being loaded properly.

What's happening appears to be that xpcshell isn't loading chrome from the GRE dir, only the app dir, which in this case is pretty useless. See http://mxr.mozilla.org/mozilla-central/source/toolkit/xre/nsXREDirProvider.cpp#709 and http://mxr.mozilla.org/mozilla-central/source/chrome/src/nsChromeRegistry.cpp#1252 There are a couple options we can try: * have xpcshell provide NS_CHROME_MANIFESTS_FILE_LIST * have the chrome registry use a better default (include gredir/chrome by default) * stick xpcshell in the appdir instead of a separate test dir, which we've been avoiding so far

(In reply to comment #8) > * stick xpcshell in the appdir instead of a separate test dir, which we've been > avoiding so far This seems to remove most of the test failures I was seeing. The downside is that I not only need to copy xpcshell, I need to copy test_necko.xpt and httpd.js to firefox/components, since the tests rely on those. It takes me down to 4 failures which are easily explained: test_punycodeURIs.js - this test expects to find the WriteArgument executable, which isn't currently packaged in the app or test package (we should package it with the tests) http://mxr.mozilla.org/mozilla-central/source/uriloader/exthandler/tests/unit/test_punycodeURIs.js#106 test_bug455213.js: This is a plugin unittest, and expects to find the test plugin somewhere (which isn't in the app dir, of course): http://mxr.mozilla.org/mozilla-central/source/modules/plugin/test/unit/test_bug455213.js#111 I'm not really sure how to fix this one, short of copying the test plugin to appdir/plugins. test_nsIProcess.js: Also expects to find a couple of test executables: http://mxr.mozilla.org/mozilla-central/source/xpcom/tests/unit/test_nsIProcess.js#61 Easy enough to make them get copied to the test dir. test_bug378839.js: The tests dir contains strres.properties, which the test expects to be able to load, but this is a test .properties file that isn't shipped with the app. I think we can instead put it in the test dir, and make the test register its own resource path to load the bundle from. (Or use a file URL if you can load bundles from those.) http://mxr.mozilla.org/mozilla-central/source/intl/strres/tests/unit/test_bug378839.js#35

Comment on attachment 367207 [details] log of full xpcshell test run from packaged build+tests bsmedberg's comment explains all the other failures, we'll have to fix the underlying problem, clearly.

(In reply to comment #9) > test_bug455213.js: This is a plugin unittest, and expects to find the test > plugin somewhere (which isn't in the app dir, of course): > http://mxr.mozilla.org/mozilla-central/source/modules/plugin/test/unit/test_bug455213.js#111 > I'm not really sure how to fix this one, short of copying the test plugin to > appdir/plugins. I fixed this in my existing patch before by putting it in bin/plugins, next to xpcshell. Copying it to firefox/plugins would fix the test, but it's kind of a hassle. :-/

Ok, If I copy: bin/xpcshell -> firefox/ bin/components/httpd.js -> firefox/components bin/components/test_necko.xpt -> firefox/components bin/plugins/libnptest.so -> firefox/plugins Then run: cat xpcshell/tests/all-test-dirs.list | sed "s|^|./xpcshell/tests/|" | xargs python -u xpcshell/runxpcshelltests.py --keep-going ./firefox/xpcshell All the tests pass for me now on Linux.

Morphing this bug to cover bsmedberg's suggestions from comment 8. The workaround is working, but it sucks.

I've verified that this works manually... the directory service changes were required because by golly the directory service was appending "components" to the GRE directory without cloning it!

Comment on attachment 382913 [details] [diff] [review] Load chrome from the GRE directory, rev. 1 Seems sensible, yeah. Would be nice to have a test for this, although testing it in the existing xpcshell harness would be pretty difficult. Could probably do a custom test in check::.