We should start automating litmus tests by using Mozmill.

For backup. https://litmus.mozilla.org/show_test.cgi?id=5576 is the litmus test.

From a not-yet-written Litmus test.

Notes to self: - Mozmill tests relying on modal windows (which TB has a lot of) are heavily dependent on bug 479311. - Another test at https://bugzilla.mozilla.org/show_bug.cgi?id=471510#c5

Adding bug 458352 to the deps as that includes potential scripts and discussions we've had on what to do about actually scripting getting tests running automatically.

In the case of 5576, has working support of clicking "Manage Message Filters" in Account Central. Notes: a) Templatized DOM tree support not yet implemented in Mozmill - Bug 477079 b) Modal dialogs are supported in Mozmill, but I couldn't yet make it work, so code is commented out. c) A bunch of steps are still coded to be manually clicked, some email account names and emails are assumed to be of a particular name till at least bug 458352 is fixed.

Now left blocking significantly on bug 477079 for clicking on the left folder selection tree.

I'd like to use the following to store the test cases into the tree : /mail/test/mozmill/ litmus-litmustestid/test.js and I was thinking also to store some profiles at the same location so they could be shared for our tests. Mark comments ?

As a developer, a litmus test id has even less meaning to me than a bugzilla bug number. I would suggest that we try and name things to provide as much possible info about what the test does in the file name and directory name. For example, we have a number of xpcshell tests that are based on bug numbers in mailnews/base/test/unit. I have to look inside them or visit the bug to figure out what they do. I think bug numbers and litmus test numbers are potentially useful information, but if we are including them, I propose we append them onto a more useful description of what they do, and categorize them based on the directory they go in. For example "account-dialog/deletion-settings-litmus12345.js" is useful to me. If you need each litmus test to have its own directory, there is nothing in the mozmill idiom that prevents us from having a more nested directory structure. (So "litmus/account-dialog/deletion-settings-litmus12345.js"). If the mozmill test is permanently replacing a litmus test so that we never need to manually run it again, I'm not sure we even need to put the litmus test number in the name. In that case I think it would be sufficient to just put a comment in the test, like "// litmus test #12345". Otherwise organizing tests along a strict litmus basis might create a test burden that isn't really required.

Er, in the own directory example, I glitched, I meant to say: litmus/account-dialog/deletion-settings-litmus12345/test.js And of course, in the first example, I meant account-dialog/test-deletion-settings-litmus12345.js

(In reply to comment #8) > For example, we have a number of xpcshell tests that are based on bug numbers > in mailnews/base/test/unit. I have to look inside them or visit the bug to > figure out what they do. > > I think bug numbers and litmus test numbers are potentially useful information, > but if we are including them, I propose we append them onto a more useful > description of what they do, and categorize them based on the directory they go > in. For example "account-dialog/deletion-settings-litmus12345.js" is useful to > me. If you need each litmus test to have its own directory, there is nothing > in the mozmill idiom that prevents us from having a more nested directory > structure. (So "litmus/account-dialog/deletion-settings-litmus12345.js"). So this needs to be taken into account and a description is way better than any number scheme we can come up with. Issues will end up probably like in litmus with non very descriptive Test Case (TC) - but better than numbers. Let's just drop them - as the only function I had in mind for them was to easily know which TC have been moved from litmus to mozmill. > If the mozmill test is permanently replacing a litmus test so that we never > need to manually run it again, I'm not sure we even need to put the litmus test > number in the name. In that case I think it would be sufficient to just put a > comment in the test, like "// litmus test #12345". Otherwise organizing tests > along a strict litmus basis might create a test burden that isn't really > required. Yeah that would do. Thanks for the input andrew.

Here's the mozmill test for steps 1 & 2 - step 3 is not yet done because I haven't yet have time to figure out the added complexity needed for that step. https://litmus.mozilla.org/show_test.cgi?id=5576 It runs perfectly in the mozmill harness.

Comment on attachment 393771 [details] [diff] [review] Litmus test 5576 that runs perfectly in the Mozmill harness - steps 1 & 2 I was mis-using the review flag to keep this on my radar to fix up since my understanding was gary was not available to perform further iterations on the test. I'm r-'ing the patch because the dependency on short sleeps will break on the tinderboxen. Also, I'd personally prefer that we avoid the giant automated XPath blob. We have sufficiently stable element id's that they should be directly usable and it's more clear why a test might have failed.

This bug goes back to the pool - haven't been thinking about this and don't think I'll have time to work on this.

(In reply to comment #13) > This bug goes back to the pool - haven't been thinking about this and don't > think I'll have time to work on this. Also seems like a perfect student-project, btw.

Simon might be taking this on as a student project - assigning to him.

Litmus will be replaced by MozTrap ( https://moztrap.mozilla.org ), and it will be decommissioned soon. See bug 802674.

isn't the premise of automating our manual tests still valid? And preferable to moving all manual tests to moztrap?

(In reply to Wayne Mery (:wsmwk) from comment #17) > isn't the premise of automating our manual tests still valid? > And preferable to moving all manual tests to moztrap? Yes, perhaps the premise would be, but probably in another bug since there's a lot of litmus stuff here.

filed bug 802990