Mozmill should be integrated into the build system when it's ready. That will enable us to have automated tests runs on tinderboxes. Until then tests have to be run manually by the user. The source code of Mozmill can be found here: http://code.google.com/p/mozmill/ Currently available tests which can be run by users: http://hg.mozilla.org/users/ctalbert_mozilla.com/gristmill-test/file/

This work is slated for MozMill RC1 and will use the jsbridge library for communication back to the Python end, and mozrunner for start and stop of mozilla applications. http://code.google.com/p/jsbridge http://code.google.com/p/mozrunner

(In reply to comment #1) > This work is slated for MozMill RC1 Now that the RC is out, where can the linkage with the build system be found (if it's completed)? TB should have this too, cc'ing gozer.

I documented the command line interface for mozmill http://code.google.com/p/mozmill/wiki/CommandLine This should all work OOTB with buildbot.

Mikeal is right -- this is very easy to setup with buildbot. The problem is that the current reference platforms [1] don't have the necessary python dependencies installed. I suppose getting these added to the reference platform wouldn't be too hard. My only concern would be burdening the build team with keeping the version of mozmill/jsbridge/mozrunner constantly updated. Maybe we could just have the two external dependencies (simplesettings and simplejson) on the reference platform and download mozmill, jsbride and mozrunner each time we run the tests? I'm not sure how feasible this would be... Or am I misjudging how often we would need to update these? Maybe we should just hold off until they are more stable? [1] https://wiki.mozilla.org/ReferencePlatforms

I think the only thing that would need to be added to the reference platform is setuptools. If you're provisioning a new mozmill slave you can just $ easy_install mozmill and all the dependencies will get installed. This way we can set dependency requirements in setup.py for each release of mozmill so that those deps are only updated when we decide to update to the newest version of mozmill.

To clarify for those not familiar with setuptools, you can set the dependency to a specific version, or even to a certain version or higher. So if mozmill 1.0 gets installed on a bunch of slaves and mozmill 1.1 gets released with a requirement on a version 0.9.4 of mozrunner we can set mozmill's setupt.py mozrunner requirement version to 0.9.4 or higher and when you $ easy_install -U mozmill, it will check your local mozrunner version and update to the newest.

This is untested. I still need to figure out how to get the correct path to add to PYTHONPATH in MozillaUpdateMozmill. This patch will also require easy_install to be added to all the unittest reference platforms.

I've two concerns about this patch. First is that the functions in mozbuild.py are very Firefox specific. Thunderbird want to be able to run mozmill tests, and I bet SeaMonkey do as well. Although we may need to end up with different options/more advanced options, how about allowing Mozilla*MozmillTest to take arguments for the different apps? If not, I think they should be renamed to Firefox*MozmillTest to reflect what they actually do. Second, whenever a new version of mozmill (and all its dependencies) are released, are we guaranteed it is backwards compatible with the previous version? From what I can tell of this patch, mozmill could be updated at any time to the latest version and if we don't have that guarantee, then it could break our build boxes and the cause be hard to find. Additionally, I'd also like to know when it updates (i.e. make it obvious on the tinderbox output/display), for example, if I use Mozmill for leak and bloat, then a change in mozmill version may slightly affect numbers. I want to know when these happen so again we haven't got random changes.

I'd like to second what Mark said, I'm listening in here to see if we can use this as a base for l10n testing, too. So being generic is a good idea. I wonder if we should make the slaves easy_install mozmill with a version number. Though I don't know if we can do the same thing for the extension part.

We had a conversation on IRC about what the update policy should be. I stressed that mozmill should not be updated to the newest available but instead should poll for an update from a private update server, which seems to be reflected in this patch. The --find-links=http://152.7.7.176/mozmill directive should look for updates to mozmill at this page rather than on PyPI. In my opinion we should setup a different page for each group that is running mozmill tests so that they can control their own mozmill update schedule for their tests.

I mentioned this to ispiked on irc, but wouldn't it make more sense to just have a static pre-built checked in copy of whatever we need to run mozmill in the mozilla-central tree? We could then update it as necessary like any other upstream software.

(In reply to comment #11) > I mentioned this to ispiked on irc, but wouldn't it make more sense to just > have a static pre-built checked in copy of whatever we need to run mozmill in > the mozilla-central tree? We could then update it as necessary like any other > upstream software. Pre-built as long as we can alter the install.rdf files when our versions change or some other way so that all apps can keep the versions right.

MozMill should be treated like an external dependency. Checking a copy in to the tree is just going to encourage people to make tweaks to the tool in the tree instead of in the main project. We have a way to auto-update the slaves and we have a way to control the updates so they only happen when we want them. What's the problem?

If the install.rdf excludes any in-development version of Firefox then we need to update it and push a .0.1 release of MozMill to PyPI. Anyone should be able to $ easy_install -U mozmill, on their local machine and test against the newest nightly, if not, it's a bug and it needs to be fixed in mozmill trunk and latest release branch.

(In reply to comment #13) > MozMill should be treated like an external dependency. > > Checking a copy in to the tree is just going to encourage people to make tweaks > to the tool in the tree instead of in the main project. I don't think this is clearly stated. One of the reasons we wanted Mozmill to be external is to enable the Mozmill team to completely break things from release to release. If we put this in the m-c trunk and point everyone at using that, then we lose the ability to do whole-scale quick turn around changes without breaking everyone and their brother in the meantime. Keep in mind that this tool while almost at 1.0 is still developing very quickly and is still under a good bit of churn.

(In reply to comment #13) > MozMill should be treated like an external dependency. > > Checking a copy in to the tree is just going to encourage people to make tweaks > to the tool in the tree instead of in the main project. We have other third-party libraries that we keep in the tree in source form but are strict about not taking local patches to. For example, we do not locally patch sqlite. It comes down to the policy the module owner puts in place. (In reply to comment #15) > I don't think this is clearly stated. One of the reasons we wanted Mozmill to > be external is to enable the Mozmill team to completely break things from > release to release. If we put this in the m-c trunk and point everyone at > using that, then we lose the ability to do whole-scale quick turn around > changes without breaking everyone and their brother in the meantime. Keep in > mind that this tool while almost at 1.0 is still developing very quickly and is > still under a good bit of churn. Right, I'm not saying you should keep your development sources in the tree or anything, just checking in an XPI or whatever for the test harness would be fine. It just feels like overkill to make the buildbot slaves pull from somewhere else to get stuff they need when we could just as easily commit a copy to testing/ and update it whenever we like. I'm not going to argue this heavily, I just don't see a good reason not to do it. Also, as a note on the actual attached patch here, MozillaBuild doesn't ship with easy_install.

I think the patch assumed that setuptools would be installed after the slave was provisioned or added to default build image. IMHO it should be there anyway. I'm not that opposed to sticking a tarball in the tree, I do think the easy_install method is cleaner but I'm not the one that has to maintain these boxes so I'll stay out of it :)

I'm advocating for setuptools to deploy code on the slaves, too, fwiw. I don't think that it's a major drawback that it's not available in MozillaBuild, I somehow smell that a new release might not be too far out to take the stability fixes of hg 1.1. Sorry, Ted. Still waiting for tortoisehg to release on 1.1 until I eagerly push for that, though. Note that easy_install isn't part of the refplatform on any of our other platforms either, though.

(In reply to comment #10) > The --find-links=http://152.7.7.176/mozmill directive should look for updates > to mozmill at this page rather than on PyPI. Why not just allow a version to be specified to easy_install? Wouldn't that do the same job, but easier? (In reply to comment #14) > If the install.rdf excludes any in-development version of Firefox then we need > to update it and push a .0.1 release of MozMill to PyPI. I agree, but if we're automatically pulling in a new version of Mozmill (i.e. no way to update this in-tree) then version changes are going to have to be delay until MozMill is updated, else our test machines will break. That will probably be alright - if you guys are going to be happy to update the version number a week beforehand so the release process doesn't get delayed. The only thing would be that the tests being run would have to be running on the latest version of MozMill at each release time (which possibly isn't a bad thing).

There seem to be two main concerns here: 1) the patch isn't generic enough and 2) how we should download mozmill and friends. This patch is written specifically for the Firefox unittest tinderboxen. Apparently code sharing has taken a cut-and-paste approach rather than an |import| approach in these config. files. IMO, it would be fairly simple to copy over these functions and modify a few lines to get them working for Thunderbird and l10n tinderboxen. The MozillaUpdateMozmill and MozillaMozmillTest functions are actually fairly generic. One would just need to subclass them and specify new command attrs. As for the mozmill and friends dependency... We cannot burden build & release with having to update the ref. platform each time a new version of mozmill is released, so we need to download it each time we run the test. We also need to ensure two additional things: 1) we don't inundate PyPI's servers with download requests and 2) the ability to control what version of mozmill the tinderboxes use. Mikeal decided that the best way to solve this would be to setup our own easy_install update "mirror" where the latest version of mozmill and friends would live. Yes, this requires that someone uploads a new version of mozmill to the update site whenever it's determined that the current tests pass with it, but this is really the only way to ensure that tests don't break due to mozmill updates. I still need to get buildbot setup so I can test this and work out the remaining issues. Once I do this, I'll update the patch.

Also, let me just note that I'm not trying to be stop-energy here, I'm just noting how we have done other things, and also noting that it's a big pain for RelEng to update refplatforms. I think the mozmill work is awesome, and look forward to having it available for unit tests!

> Why not just allow a version to be specified to easy_install? Wouldn't that do > the same job, but easier? I'm not familiar enough with setuptools internals to know for certain, but I think easy_install will still make a request to PyPI when you specify a version that is already installed. We could check to make sure, but if it does we can't be making that many requests per day to PyPI. About the refplatform. IMHO setuptools should be added to the refplatform the next time the build team needs to update the ref images. This however shouldn't be a blocker, setuptools is pretty easy to install by hand on newly provisioned slaves. I strongly advise against trying to install mozmill without setuptools as it's got a pretty long list of dependencies that is more likely to get larger than smaller. I also think the last thing we should do is add mozmill and it's deps to the refplatform as the release cycle for mozmill will be more frequent than the build platform updates.

One last comment on the install.rdf updates. MozMill's install.rdf that is checked in and packaged up with the Python package should have as aggressive a version setting as all the products will allow. Ideally MozMill's install.rdf version compatibility should be ahead of the development version of the products if the products will tolerate it. When we package up a release we can bring down the setting by hand before uploading the package to AMO since it won't accept it.

Mirroring a setup-tools server sounds like a bunch of headache and more maintenance work than simply checking in a read-only version of source and dependencies into the tree. At the moment, there are four or five dependencies. That's not enough to justify our own setup-tools infrastructure as far as I know. I don't think Momzill should have many more dependencies than it currently has. Furthermore, the releases will slow down after 1.0 and we'll have more measured releases of Mozmill as functionality solidifies. This shouldn't be that much of a headache regardless of which path is taken. If we can't get infrastructure for our own setup-tools server and the setuptools client into the standard mozilla-build platform anytime soon, then we should opt for the other method. Now that I see that we are advocating for a readonly copy a la every other third party lib we've got in our tree, then I'm thinking that is a pretty viable short term solution. As you note in comment 23, install.rdf is definitely a non-issue.

I think you are grossly overestimating the amount of infrastructure this takes. It is not a "setuptools mirror" or anything like that. When you run easy_install you give it an url. That url is literally a single static html page with a link in it, that's it. I'm sure we can host a static html page, we're f'ing Mozilla.

(In reply to comment #25) > I think you are grossly overestimating the amount of infrastructure this takes. > > It is not a "setuptools mirror" or anything like that. When you run > easy_install you give it an url. That url is literally a single static html > page with a link in it, that's it. Then I'm completely confused about what you're proposing. Assuming we don't overcome build inertia and we can't update the build ref platform until April of 09 (after 1.9.1 ship) what's the plan?

Forgive the double spam, let me explain... We all want this integrated sooner rather than later. I have serious doubts that we'll see any update to the build platforms before 1.9.1 ships in Feb/March. I think I understand the situation if we can update the build ref platform. In that case, we plug in setuptools, have it ping the easyinstall server which in turn pings PyPi and gets all our goodies. (correct if I'm wrong) If we can't update the ref platform...that's the part I really don't understand at all, and can't hazard a guess as to what the plan should be, and what it will take to get that plan implemented. Thanks for clarifying.

Nothing needs to be added to the ref platform. Eventually setuptools should be added but that is all. Mozmill update checks are happening in the buildbot steps. The debate is about how that update is done. Whether we stick a tarball in the tree or update a page with a link in it. I've stated my preference but at the end of the day I won't be maintaining these boxes. In the interest of moving forward I think whoever is setting these up should just make the call and move ahead.

When new slaves are setup in the buildbot system there are some manual steps after the ref image is installed. Setuptools can just be insalled as an additional step manually, so we aren't blocked by it not being part of the ref image. Since we will always need setuptools it should be added to the ref image next time they update it to reduce manual setup steps, but this is not a blocker.

Oh cool. Somehow I got it into my head that updating the ref platform was an absolute must and I got worried. I'll file a bug to get setuptools added to the next rev of the platform. I think that makes good sense to me since we have so much invested in all our python tools. > In the interest of moving forward I think whoever is setting these up should > just make the call and move ahead. I agree. Thanks for the clarification.

(In reply to comment #30) > Oh cool. Somehow I got it into my head that updating the ref platform was an > absolute must and I got worried. I'll file a bug to get setuptools added to > the next rev of the platform. I think that makes good sense to me since we have > so much invested in all our python tools. Filed bug 471244

Comment on attachment 353484 [details] [diff] [review] Add buildbot configs. for Mozmill, round 1 >Index: tools/buildbot-configs/testing/unittest/mozbuild.py >=================================================================== >+class MozillaUpdateMozmill(ShellCommandReportTimeout): Is there a reason why you need to subclass for this? I think it can be done as a plain ShellCommand in the master.cfg. >+ warnOnFailure = True >+ name = "update mozmill" >+ description = ["update mozmill"] >+ descriptionDone = ["update mozmill complete"] >+ command = ["easy_install", >+ "--install-dir=.", >+ "--find-links=http://152.7.7.176/mozmill", This link looks broken. More importantly, this needs to be hosted somewhere inside Mozilla. Pulling in random code from an outside address is a huge security risk. >+ def __init__(self, **kwargs): >+ env = {} >+ if 'env' in kwargs: >+ env = kwargs['env'].copy() >+ #XXX how to get current directory? >+ env['PYTHONPATH'] = os.getcwd() This doesn't do what you think it does. All of this code is run on the master. PYTHONPATH supports relative addresses, so you should be able to just do PYTHONPATH='.'. Note that Buildbot will automatically prepend this to the existing slave-side PYTHONPATH. Please don't return WARNINGS for this. It's best to reserve that state for actual test failures. If there's a problem with the harness or setup FAILURE is more appropriate. I really think this steps should just be done as: ShellCommand(name=..., description=..., command=[...], env={'PYTHONPATH': '.'}) >+ >+class MozillaMozmillTest(ShellCommandReportTimeout): >+ warnOnFailure = True >+ name = "mozmill tests" >+ description = ["mozmill tests"] >+ descriptionDone = ["mozmill tests complete"] >+ >+ def __init__(self, **kwargs): >+ env = {} >+ if 'env' in kwargs: >+ env = kwargs['env'].copy() >+ #XXX how to get current directory? >+ env['PYTHONPATH'] = os.getcwd() same thing here about PYTHONPATH. Just pass env in and set PYTHONPATH to '.'. >+class MozillaOSXMozmillTest(MozillaMozmillTest): >+ command = ["python", >+ "mozmill", >+ "--binary=../../objdir/dist/Minefield.app/Contents/MacOS/firefox", >+ "--default-profile=../../objdir/Minfield.app/Contents/MacOS/defaults/profile", >+ "--tests=tests"] >+ If this is for 3.0 this is fine. If it's for 3.1/3.2 you'll need to update this code to work with the brand_name parameter - like this: http://mxr.mozilla.org/mozilla/source/tools/buildbotcustom/unittest/steps.py#272, which is passed in UnittestBuildFactory. If you're only looking for 3.0 support please replace the MozillaUpdateMozmill class with a plain ShellCommand in the master.cfg and fixup the PYTHONPATH variable in the MozillaMozmillTest class. If you're looking for 3.1/3.2 please do that plus move all of this code to the right files. (buildbotcustom/unittest/steps.py for steps, buildbotcustom/process/factory.py for the "master.cfg" code). In either case, the update server needs to be hosted somewhere more secure.

When reading the bug title here, I kind of expected that the mozilla build system is modified so that for a third party app I could do something like: cd $OBJ/calendar/test/mozmill make check Which would obviosly mean we need some changes to rules.mk or such, that allow a Makefile.in to specify MOZMILL_TESTS = <directory or file with test(s)> which would in turn: * Find out the binary path of the current project * Allow specifying the default profile, either through environment variables or maybe a second variable MOZMILL_CONFIG = <path to simplesettings config file> * Possibly allow setting other commandline parameters, i.e MOZMILL_OPTIONS = ... Looking at the code rather makes me assume that the buildbots are taking over the logic of finding out where the test directory is (which is hardcoded, as noted above). Any chance this could change? Also, I'd appreciate if this bug could move forward, having automated tests in calendar would be a big step forward and we need this rather sooner than later.

Just a note on the previous comment, this can't be run from "make check" as that target is not allowed to run anything that requires a display. We could probably use automation.py and build a runtests.py like equivalent to run Mozmill tests.

Adam G., what's your status here? Do you have another patch in the works?

Clint, I quit working on this because I was unable to replicate a buildbot environment on my own machine. It's unlikely that I'll be able to do this anytime soon. Someone who has access to a buildbot environment could probably update my patch according to bhearsum's comments and get thigns working quite easily, however.

Thanks for the update Adam. I've got a buildbot environment for linux, so I guess I'm the guy to pick up where you left off. Thanks for getting the effort started!

Hey Clint, any updates on this?

This doesn't stop us from creating tests for now. So marking as P2. Clint, please reconsider the priority based on the time your team has to get this into the build system.

I'm going to mark this as a P1. It's not blocking the creation of new tests but it is blocking some forward momentum on the general usefulness of the tests. I'd also like this to be P1 so that any fixes that might need to be made in mozmill itself to enable this are treated as blockers.

This is a first step to port iSpiked's patch over to the current buildbot infrastructure. I've tried to copy the existing setup as closely as possible. I haven't tested it yet, but I'd like to get a 10,000 foot glance on the approach as this is the first buildbot patch I've successfully written, and I want to know early on if I'm doing something dumb. = Things remaining to address = * Updating Mozmill * Pulling in the Mozmill Tests from http://hg.mozilla.org/qa/mozmill-tests/ For updating Mozmill, I'd love to get setuptools installed, but I'm not sure if that is realistic to do on the 1.9.1 infrastructure. If not, then I'm leaning toward checking in a tar.bz of mozmill and unzipping it as part of my "update" step. When getting the tests, I'd like to do two things * 1. pull them down during build so that they end up in the objdir under the _tests subdirectory (this way community people running with --enable-tests will get them * 2. when make test-package is run, package the tests with the other tests. --> But in both cases, put the tests in a mozmill/tests directory that is relative to our cwd in buildbot. That seems to be how the other package tests run, and I'm basing my approach off of those. I hope to only need one buildstep for Mozmill tests - whether running in package mode or not. Thanks for the feedback!

(In reply to comment #41) > = Things remaining to address = > * Updating Mozmill > * Pulling in the Mozmill Tests from http://hg.mozilla.org/qa/mozmill-tests/ This is a surprise, I thought the intention (explicitly) was that those tests in that repository were not going to be run automated. + self.command = ['python', 'reftest/runreftest.py', + WithProperties('--binary=%(exepath)s'), + '--default-profile=bin/defaults/profile', + '--test=mozmill/tests'] So mozmill can be run via runreftest.py now? Can we not get a makefile target to do this, so that those with mozmill installed can just do "make mozmill"? Note that Thunderbird has been building up a mozmill python script here: http://mxr.mozilla.org/comm-central/source/mail/test/mozmill/runtest.py Its not generic enough yet (mainly specific to TB and doesn't accept arguments for the run directory etc which make package-tests would need to use), but its slowly progressing.

(In reply to comment #41) > * Pulling in the Mozmill Tests from http://hg.mozilla.org/qa/mozmill-tests/ When we wanna do this we should first find a way how to exclude tests from running. For existing tests in m-c and 1.9.1 we can specify those in the list file. We don't have such a file for Mozmill right now. We could only rename a folder so it doesn't start with test. But that will affect all tests under this folder which is probably not the way we wanna go.

Comment on attachment 383605 [details] [diff] [review] Updated WIP patch for 1.9.1/1.9.2 build/test infrastructure (In reply to comment #41) > Created an attachment (id=383605) [details] > = Things remaining to address = > * Updating Mozmill > * Pulling in the Mozmill Tests from http://hg.mozilla.org/qa/mozmill-tests/ > > For updating Mozmill, I'd love to get setuptools installed, but I'm not sure if > that is realistic to do on the 1.9.1 infrastructure. If not, then I'm leaning > toward checking in a tar.bz of mozmill and unzipping it as part of my "update" > step. I'm still against having Mozmill installed on these machines in any global way. We pull all other test harnesses (Talos, runtests.py, et. al) on each build, let's do the same here. Checking in a bz2 would be OK with me, but pulling from source is arguably better. > > When getting the tests, I'd like to do two things > * 1. pull them down during build so that they end up in the objdir under the > _tests subdirectory (this way community people running with --enable-tests will > get them I'm not following how us putting the tests in _tests helps anyone doing third party builds, can you expand on that? > * 2. when make test-package is run, package the tests with the other tests. > --> But in both cases, put the tests in a mozmill/tests directory that is > relative to our cwd in buildbot. That seems to be how the other package tests > run, and I'm basing my approach off of those. I hope to only need one > buildstep for Mozmill tests - whether running in package mode or not. > Yeah, I think that makes sense. Not doing a full code review, but here's what sticks out immediately: >+class MozillaMozmillTests(ShellCommandReportTimeout): >+ warnOnFailure = True >+ warnOnWarnings = True >+ >+ def __init__(self, **kwargs): >+ env = {} >+ if 'env' in kwargs: >+ env = kwargs['env'].copy() Don't use kwarg manipulation, please. It's caused us troubles related to 'buildbot reconfig' in the past. Here's how we do it: http://hg.mozilla.org/build/buildbotcustom/file/tip/steps/unittest.py#l377. I think this is promising, thank you for doing it in the packaged unittest style! You'll probably want to set r?catlee and maybe someone else when you need a full review.

You can't run mozmill if it's not installed. Mozmill isn't a script like the other test tools that are packaged up with the build. Mozmill is a reusuable Python module, the command line interfaces (yes, there is more than one command line interface) are built using setuptools/buildout directives. It *must* be installed *somewhere*. That said, if we don't want it installed globally we can just use virtualenv. http://pypi.python.org/pypi/virtualenv

(In reply to comment #44) > (From update of attachment 383605 [details] [diff] [review]) > (In reply to comment #41) > > Created an attachment (id=383605) [details] [details] > > = Things remaining to address = > > * Updating Mozmill > > * Pulling in the Mozmill Tests from http://hg.mozilla.org/qa/mozmill-tests/ > > > > For updating Mozmill, I'd love to get setuptools installed, but I'm not sure if > > that is realistic to do on the 1.9.1 infrastructure. If not, then I'm leaning > > toward checking in a tar.bz of mozmill and unzipping it as part of my "update" > > step. > > I'm still against having Mozmill installed on these machines in any global way. > We pull all other test harnesses (Talos, runtests.py, et. al) on each build, > let's do the same here. Checking in a bz2 would be OK with me, but pulling from > source is arguably better. > Ideally, I'd love to do something like this: 1. checkout some target of the mozmill code 2. run python mozmill/<mozmill component>/setup.py install on the requisite mozmill pieces 3. run tests 4. run python mozmill/<mozmill component>/setup.py uninstall on the requisite mozmill pieces. But in my limited experience with python, all the frustration has come from trying to do point 4 (for various unrelated reasons). Does "uninstall" support actually exist in python? Ben, if we could get a solution for point 4, would that approach work? > > I'm not following how us putting the tests in _tests helps anyone doing third > party builds, can you expand on that? > My thought here is that people will want to run mozmill tests using a similar setup to the way that they run mochitest/xpcshell tests. Currently that isn't possible. If you want to run mozmill tests today you must: 1. checkout the tests from hg.mozilla.org/qa/mozmill-testss 2. checkout mozmill source (or download from amo if running in browser) 3. run tests If I edit the makefiles to package up the tests into the "standard" objdir/_tests directory then I can at least help solve one of those steps for people interested in running mozmill tests. I could even supply a simple runmozmilltests.py that would run them for people in the same way that they now run everything else. That's my thought. For this patch, we just need to put the tests someplace when we pull them down from the qa hg repo, so putting them in _tests seems like the right approach. Later on, I can finish out the rest of the automation work. > > >+class MozillaMozmillTests(ShellCommandReportTimeout): > >+ warnOnFailure = True > >+ warnOnWarnings = True > >+ > >+ def __init__(self, **kwargs): > >+ env = {} > >+ if 'env' in kwargs: > >+ env = kwargs['env'].copy() > > Don't use kwarg manipulation, please. It's caused us troubles related to > 'buildbot reconfig' in the past. Here's how we do it: > http://hg.mozilla.org/build/buildbotcustom/file/tip/steps/unittest.py#l377. > Cool, thanks for the pointer. > > I think this is promising, thank you for doing it in the packaged unittest > style! You'll probably want to set r?catlee and maybe someone else when you > need a full review. Ok, I'll ping Catlee and Ted for the full review. Thanks for the help, Ben.

setuptools does not have an uninstall directive, but pip does. I think the objection to installing globally wouldn't be satisfied by uninstalling. If i understand the thinking here, each "build" (whether an actual full build or just a test run against a recent build the way Talos runs) is in a self contained directory that gets fully removed after it's done running. test/build runs therefor don't touch the rest of the box and it's safer to say that the state of the box won't change or get corrupted over time. I think the best thing to do here is install virutalenv globally and create a new virtualenv for each test run and install mozmill in that. Then it's just a matter of hosting or checking in source tarballs for mozrunner/jsbridge/mozmill that get installed in the virtualenv before the tests are run. Then when the test run is over the virtual env will get completely removed along with the build directory. For what it's worth, most of the people that I know who are running Python modules in continuous integration use virutalenv as a great way to keep the boxes clean and to keep scripts that might run in parallel on the same machine from interfering with each other.

This bug is not related to a special release of Mozmill. So removing our tracking whiteboard entry.

We should put this bug into the correct product/component. Can someone please do it? Further check bug 433852 which added easy_install to the build system for Windows.

(In reply to comment #46) > My thought here is that people will want to run mozmill tests using a similar > setup to the way that they run mochitest/xpcshell tests. Currently that isn't > possible. If you want to run mozmill tests today you must: > 1. checkout the tests from hg.mozilla.org/qa/mozmill-testss > 2. checkout mozmill source (or download from amo if running in browser) > 3. run tests In bug 500142 I am doing something similar to what you are thinking - As Thunderbird hasn't got any UI testing (apart from complicated to write/edit bloat tests), we're planning on using mozmill. Indeed we've got mozmill tests in the comm-central repository already. Most of that work is specific to TB at the moment (for us it is a priority), although some of that could be made more generic reasonably easily. The main difference is that we've got our mozmill tests in the our repository under our control. I didn't expect the QA repository to be hooked up to the build system (comment 42). My approach should end up with several things: - Developers able to run tests via "make mozmill" in ojbdir. - Developers able to run individual tests (TBD as to how yet). - Mozmill tests packaged and uploaded as part of "make package-tests" - Packaged tests able to be run by "runtest.py --list=foo.list --binary=/location/of/file" > If I edit the makefiles to package up the tests into the "standard" > objdir/_tests directory then I can at least help solve one of those steps for > people interested in running mozmill tests. I could even supply a simple > runmozmilltests.py that would run them for people in the same way that they now > run everything else. That's my thought. For this patch, we just need to put > the tests someplace when we pull them down from the qa hg repo, so putting them > in _tests seems like the right approach. Later on, I can finish out the rest > of the automation work. For the qa repository tests, can you not package them automatically via some other process on a nightly bases and then have their own run script? Then developers or anyone could download them and run them in the same manner.

Seems like work is going on on bug 516984.