Mozmill 1.3 was released a while ago, and last I heard we didn't support it. This bug will do the work necessary to upgrade our scripts to it.

A lot of stuff changed. This is not remotely backwards compatible. Should we try and conditionalize / use a different script based on the installed mozmill version? The bad news is that the newer version of mozmill is not really helping my focus problems out.

I've fixed the slight bitrot.

We need to get the maxversion in the mozmill extension's chrome.manifest bumped up.

I've submitted a pull request on github.

In an ideal world, all of our existing tests would work nicely either mozmill 1.2 and mozmill 1.3, and we could allow both trunk and 1.9.2 to support both. Sid, have you tried (or can you try) a full test-run with mozmill 1.3 and see if that's anywhere close to possible?

This is needed for a full |make mozmill| run.

(In reply to comment #5) > In an ideal world, all of our existing tests would work nicely either mozmill > 1.2 and mozmill 1.3, and we could allow both trunk and 1.9.2 to support both. > Sid, have you tried (or can you try) a full test-run with mozmill 1.3 and see > if that's anywhere close to possible? A full test run with Mozmill 1.3 + the above patch looks like it works fine on Windows. There don't seem to be any API changes that affect tests themselves. This is borne out by ctalbert's post in the newsgroup -- I don't think we use any of the APIs mentioned there. I wouldn't be in favour of supporting both Mozmill 1.2 and Mozmill 1.3, because: - Mozmill 1.3 seems more reliable. Mozmill 1.2 used to crash quite often at shutdown -- I haven't seen it with 1.3 yet. - I haven't confirmed it yet, but it seemed like an issue where anything printed to stdout got eaten up on Linux (thus making dump() from chrome code useless) looks fixed. - As asuth mentioned, the surrounding python harness code has changed a lot, and trying to support both Mozmill 1.2 and Mozmill 1.3 will require maintaining two copies of runtest.py, and testing on both each time a change is made to the harness. I think it's a good idea to keep our "test testing matrix" simple, even if our testing matrix isn't. :)

Agreed that it seems hard to justify the extra complexity that supporting both would imply. Supposing we do switch to entirely to 1.3, what does that look like? Steps that I can imagine include: * announce our plan * get this patch reviewed * simultaneously: ** update the wiki page ** update all the test tinderboxen ** land this patch Is there anything I've left out? Presumably this is less of a hassle to do before comm-1.9.2 is cut, and but not a particularly big hassle to do afterwards either. Is that actually true? Adding gozer since he'll be interested in the tinderbox bit, I suspect.

Comment on attachment 422483 [details] [diff] [review] fix runtestlist.py too r=Standard8 but I'll arrange check in time with gozer & others.

Notes: I tested the patch with: easy_install http://pypi.python.org/packages/source/m/mozrunner/mozrunner-2.4.tar.gz#md5=73f19c06bb1986e8038fb8088e6babe9 easy_install http://pypi.python.org/packages/source/j/jsbridge/jsbridge-2.3.4.tar.gz#md5=f880ab9330caf00a92146188372f3687 easy_install http://pypi.python.org/packages/source/m/mozmill/mozmill-1.4.tar.gz#d649c0634a04a9433c4792ca0a503068 I have set up instructions for deploying this on our builders at: https://wiki.mozilla.org/Thunderbird/Infrastructure/Builder_Installations#MozMill (they can also be used for dev installs). Still todo: - Post to mdat & co about upgrade; I'll do this once I've spoken to gozer about a deployment time. - Deploy (i.e. land, update buildbots) & update devmo page.

Comment on attachment 422483 [details] [diff] [review] fix runtestlist.py too Due to needing to upgrade all the builders at the same type, we're going to be landing this patch in sync with the trunk patch. a=Standard8 as its effectively NPOTB anyway.

Checked in: http://hg.mozilla.org/comm-central/rev/2ce6e173b23c http://hg.mozilla.org/releases/comm-1.9.1/rev/53f64fe1c09d

Verifying MozMill test harness fix for 3.0.2 on the basis that MozMill tests are still running fine and no-one has complained.