Device: Nexus 4 A non-debug build of B2G runs Mochitests consistently and correctly. A debug build does not - the mochitest iframe in the test-container displays "404 Not Found: /tests/content/base/test/csp/test_CSP.html was not found". To be clear: a given Mochitest runs correctly on a non-debug build. I add "B2G_DEBUG=1" to .userconfig, re-run ./build.sh, flash the device ,and re-run the tests, and I get a 404 for the Mochitest .html file. This makes debugging impossible. Not only can I not use GDB or Valgrind, a lot of PR Logging is only enabled for debug builds.

This should be disabled on B2G debug builds. This may be a side effect of the fact that we're now using manifests for B2G mochitests, and it looks like your test list wasn't regenerated correctly after changing configs. (Removing objdir-gecko/_tests before building is probably enough to get you going.) Cc'ing Ted in case he has some idea of what to do to fix this.

Try running "make -C objdir-gecko install-tests".. just a shot in the dark.

Note using mach should do this for you.. I can't remember if the mach target supports debug emulators or not though.

n.b., we're not yet using manifests for running the tests, only for copying them from the source directory to the object directory. In general, if you change your build configuration you should clobber and build from scratch. The build system doesn't always cope well with configuration changes.

I am using build.sh, not sure if that's using mach under the hood, and using runtestsb2g.py to run the tests (as described here [0]). I have manually clobbered (rm -rf objdir-gecko out) before re-building with the debug flag set. It does not make a difference, I still get the same 404 error. [0] https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/Mochitests

(In reply to Andrew Halberstadt [:ahal] from comment #3) > Note using mach should do this for you.. I can't remember if the mach target > supports debug emulators or not though. "mach mochitest-remote" is meant for running mochitests on the emulator, but I am interested in running tests on the device. I don't understand why whether the build is debug or not should make a difference for running tests.

Can you paste the command line you are using? Half-baked theory: possibly what's happening is that you are trying to run a directory of tests using --test-path, but this isn't first filtering tests based on the mochitest.ini in said directory. The test_CSP.html not found would make sense because it wouldn't get copied over by the build system as test filtering currently happens at build time (should change soon).

That would also explain why it worked on non-debug since the test isn't skipped there.

(In reply to Andrew Halberstadt [:ahal] from comment #7) > Can you paste the command line you are using? python runtestsb2g.py --b2gpath ~/B2G --xre-path ~/mozilla-central/obj-x86_64-unknown-linux-gnu/dist/bin --console-level INFO --httpd-path ./ --test-path content/base/test/csp/test_CSP.html > Half-baked theory: possibly what's happening is that you are trying to run a > directory of tests using --test-path, but this isn't first filtering tests > based on the mochitest.ini in said directory. The test_CSP.html not found > would make sense because it wouldn't get copied over by the build system as > test filtering currently happens at build time (should change soon). Sounds plausible. I am using --test-path. I'm not sure I understand what you're saying about test filtering though.

Ah that explains it. This test is disabled for debug builds, which means that the test is never copied to where it would have to be in order to work. If you want to test it, you'll need to enable it at http://mxr.mozilla.org/mozilla-central/source/content/base/test/csp/mochitest.ini#129 and rebuild (hopefully no clobber needed, but I'm not 100% sure).

Thanks jgriffin, that was the cause of the problem. I had no idea the manifest format was changed recently. I'm concerned by the commit that introduces the change [0]. Its commit message just says "update to new manifest format", but it also disables a number of CSP tests on various platforms. That kind of change does not seem appropriate to me, and I'm concerned that I wasn't made aware of it when it happened. If there really are issues with observers on B2G desktop, that is a regression of bug 945268. This commit *hid that regression*. It generally indicates that there are breaking differences in the multiprocess handling between B2G desktop, the emulator, device builds, and e10s, because the CSP tests are still passing on TBPL in the emulator. [0] http://hg.mozilla.org/mozilla-central/diff/72ffff0fdad2/content/base/test/csp/mochitest.ini

That commit didn't change any behavior, it simply migrated disabled tests from b2g-debug.json (which we formerly used to manage test disabling on B2G) to the new .ini file. See e.g., for an older version of b2g-desktop.json: http://hg.mozilla.org/mozilla-central/annotate/d4b5b3c9f947/testing/mochitest/b2g-desktop.json It's likely this test never worked correctly on debug emulators or B2G desktop builds, and so was disabled as part of getting a green set of tests running in TBPL. We didn't file bugs for these, per agreement with engineering management, due to the volume of bugs and the desire to get something on TBPL quickly. But, if you'd like to investigate and help re-enable the CSP tests, I'm sure we'd all appreciate that!

(In reply to Jonathan Griffin (:jgriffin) from comment #12) > That commit didn't change any behavior, it simply migrated disabled tests > from b2g-debug.json (which we formerly used to manage test disabling on B2G) > to the new .ini file. See e.g., for an older version of b2g-desktop.json: Ah, I see. The patch for bug 945268 only updated b2g.json. I didn't realize there was also b2g-debug.json. > http://hg.mozilla.org/mozilla-central/annotate/d4b5b3c9f947/testing/ > mochitest/b2g-desktop.json > > It's likely this test never worked correctly on debug emulators or B2G > desktop builds, and so was disabled as part of getting a green set of tests > running in TBPL. We didn't file bugs for these, per agreement with > engineering management, due to the volume of bugs and the desire to get > something on TBPL quickly. But, if you'd like to investigate and help > re-enable the CSP tests, I'm sure we'd all appreciate that! I'll look into that. Looks like I didn't catch everything with bug 945268! Thanks for helping me resolve this.

(In reply to Jonathan Griffin (:jgriffin) from comment #10) > If you want to test it, you'll need to enable it at > http://mxr.mozilla.org/mozilla-central/source/content/base/test/csp/ > mochitest.ini#129 and rebuild (hopefully no clobber needed, but I'm not 100% > sure). For future reference you should be able to run "make -C objdir-gecko install-tests" when manifests are changed or tests are added without needing to re-build. This bug is a good example of why test filtering at build time is confusing :). Luckily I think the patch to filter tests from within the harness is coming soon. Once that happens you would see "1 tests skipped" instead of a 404.