To move all of the referenced testcases into the repository we need a better integration of the httpd server, which also allows us to specify proxy rules inside the proxy.pac. That would allow us to run tests against different domains. Looks like that we eventually have to fix bug 554197 first so that we can use an unmodified version of the server.

I'm proposing that we solve this only for the case where Mozmill is packaged along with the other unittests. When we package the unittests we will already have all the items we need to make httpd.js run, and using that shared code, we can allow Mozmill to easily and simply start httpd.js without bundling it or forking it. I think this solves the actual problem here (that we can't run some of our tests in a buildbot scenario because they hit outside sites). You could run this on your own machine by downloading the packaged tests and running mozmill using the same scripts that buildbot uses. If we can do that, then we can solve this more easily for 1.4.2. And for mozmill.next we can evaluate if we really want to go with httpd.js or if we want to use standard cross-unittest refactoring to start httpd.js more easily for all unittests (mozmill included). Does that sound ok? Proposal: in 1.4.2 timeframe, we only solve this for mozmill when it is bundled as a test package with the other unittests.

Clint, isn't the only thing we would need stored inside the bin/components folder of the packaged tests? There is the httpd.js and the test_necko.xpt. Both files are required to run httpd as a service. I would assume that it shouldn't be that hard to get it working with Mozmill. If I'm wrong, can you please explain what else (except some code modification) should be necessary to get it running as a XPCOM component? Also I would suggest to move the proxy.pac request into its own bug because it doesn't belong in here.

(In reply to comment #2) > Clint, isn't the only thing we would need stored inside the bin/components > folder of the packaged tests? There is the httpd.js and the test_necko.xpt. > Both files are required to run httpd as a service. I would assume that it > shouldn't be that hard to get it working with Mozmill. If I'm wrong, can you > please explain what else (except some code modification) should be necessary to > get it running as a XPCOM component? > > Also I would suggest to move the proxy.pac request into its own bug because it > doesn't belong in here. That's exactly what I'm saying. We run it only from packaged tests. We don't run it from non-packaged tests. i.e. if you just check out mozmill from PypI and install it, then you don't get the ability to run httpd.js automatically. In order to run httpd.js you would have to download a packaged build of tests from mozilla's ftp, unpack it and run the mozmill tests.

We should re-use the code in mochitest for this.

For reference, the proxy PAC is used in mochitest here: http://mxr.mozilla.org/mozilla-central/source/build/automation.py.in#364

I just started trying to figure out the httpd.js + mozmill bugs (as well as learning how mochitest does things). What is not debatable (thanks Henrik for helping me figure this out): - there is a patch to m-c httpd.js that suppresses output, introduces a getServer function, and makes this function the only exported symbol: http://github.com/mikeal/mozmill/blob/master/patches/httpd.patch - a script, sync_dependencies.py , will download the latest version of httpd.js from m-c and apply the patch: http://github.com/mikeal/mozmill/blob/master/scripts/sync_dependencies.py - the patched version of httpd.js then lives in the mozmill repository The details I have questions on: - Is it a good idea to have httpd.js live in mozmill at all (patched or otherwise)? Or, as Clint suggests, should this be available only to those that have the packaged tests? - [Note: existing tests in hg.m.o/qa/mozmill-tests make use of httpd.js, so removing it from mozmill master would involve changing or moving these tests. My personal recommendation is to make (at least) two strains of tests: those that run OOTB with mozmill and those that require httpd.js] - Why is it necessary to patch httpd.js at all? ABICT, getServer can live directly in frame.js (where it is used). Am I missing something there? Hiding the other symbols from export is a nicety, but IMHO is not worth the patch workflow. So this leaves the suppression of output. Is there no way that this can be done external to the module? (I'm no JS expert, so I'd really appreciate input on this). And, if it is impossible, is it worth patching just for output suppression? Or is a better idea to ticket this as an upstream fix to m-c httpd.js to optionally suppress output (or log it). I don't mean to raise *big ?s* at this point, but I do want to make sure we know where we're going. I'll continue investigating this and bug 554197 to get stuff out the door, but would like to aim as far towards the horizon as we can reasonably see right now. I'll continue to

(In reply to comment #6) > - Is it a good idea to have httpd.js live in mozmill at all (patched or > otherwise)? Or, as Clint suggests, should this be available only to those that > have the packaged tests? It's necessary. There is no way around. > - [Note: existing tests in hg.m.o/qa/mozmill-tests make use of httpd.js, > so removing it from mozmill master would involve changing or moving these > tests. My personal recommendation is to make (at least) two strains of tests: > those that run OOTB with mozmill and those that require httpd.js] No, that wouldn't make sense from our perspective when writing those tests. We shouldn't differentiate between those two types. > - Why is it necessary to patch httpd.js at all? ABICT, getServer can live > directly in frame.js (where it is used). Am I missing something there? Before we can start with this bug you should probably fix bug 554197 first. It will explain everything (I'm sure).

(In reply to comment #7) > (In reply to comment #6) > > - Is it a good idea to have httpd.js live in mozmill at all (patched or > > otherwise)? Or, as Clint suggests, should this be available only to those that > > have the packaged tests? > > It's necessary. There is no way around. Why is this? > > - [Note: existing tests in hg.m.o/qa/mozmill-tests make use of httpd.js, > > so removing it from mozmill master would involve changing or moving these > > tests. My personal recommendation is to make (at least) two strains of tests: > > those that run OOTB with mozmill and those that require httpd.js] > > No, that wouldn't make sense from our perspective when writing those tests. We > shouldn't differentiate between those two types. I guess I have a few big questions here, which I'll try to consolidate into one statement and one question. Statement: We should clarify what our goal is here and much such improvements that give a short term solution (being able to run the tests we want now) that leads towards a long term solution (a consolidation of our test frameworks where there is a solid maintainable codebase with one extensible way of doing things that avoids code replication; see https://wiki.mozilla.org/Auto-tools/Projects/Loderunner ) Question: What problem are we trying to solve that it is inappropriate to use the packaged tests? Okay, I lied, a second statement. To me, running the live tests (that is, tests that live on the web) seem to be a different issue than running tests with cached web-pages. The latter should, in general, mirror the former, but I don't see a reason they should live in the same place. Am I missing context here? > > - Why is it necessary to patch httpd.js at all? ABICT, getServer can live > > directly in frame.js (where it is used). Am I missing something there? > Before we can start with this bug you should probably fix bug 554197 first. It > will explain everything (I'm sure). I've commented on that bug (see https://bugzilla.mozilla.org/show_bug.cgi?id=554197#c6 ) and will try to keep most of the discussion there. To me, these issues are very similar though, or at least the overall problem is what I need to understand first.

This is a much larger issue than mozmill. It extends to all the test harnesses that need an http server. We need a better way to package httpd.js to make it into a real callable module. I'm not going to hold mozmill 2.0 on creation of that package. It's something that can and should be done in parallel.

Not working on this ATM

With Mozmill 2.1 we make use of mozhttpd, so httpds.js is no longer with us. It was never a high demand for us to get this integrated into Mozmill proper. Also Mozmill will reach its end of life soon. We are currently working on getting all the tests for Firefox ported to Marionette. For status updates please see bug 1080766.