Closed
Bug 815416
Opened 12 years ago
Closed 12 years ago
Excludetest list of all layout test failures in layout for b2g.json
Categories
(Testing :: Mochitest, defect)
Tracking
(firefox19 fixed, firefox20 fixed, b2g18 fixed)
RESOLVED
FIXED
mozilla20
People
(Reporter: martijn.martijn, Assigned: martijn.martijn)
References
Details
Attachments
(3 files)
(deleted),
text/plain
|
Details | |
(deleted),
text/plain
|
Details | |
(deleted),
patch
|
Details | Diff | Splinter Review |
I've attached a b2g.json file that made a succesful mochitest run inside the layout directory on the b2g emulator.
There might still be intermittent failures lurking around, since I only reran the whole mochtitest run in the layout directory, once.
Assignee | ||
Updated•12 years ago
|
Blocks: b2g-mochitests
Assignee | ||
Comment 1•12 years ago
|
||
This is an include list of all layout tests that are passing on the b2g emulator.
337 files and 32702 mochitests running in total.
Comment 2•12 years ago
|
||
Thanks Martijn. I've pushed this to try to see how it fares:
https://tbpl.mozilla.org/?tree=Try&rev=d8efad2677c3
Before we can land this, we'll need to increase the number of mochitest chunks used by B2G (I'll file a bug for this). Additionally, we should modify this to do the inverse of what it's doing, I think...that is, exclude all the tests that fail, rather than include the tests that pass. The reason for this is so that so new tests added automatically get run for B2G and don't require changes to b2g.json.
Comment 3•12 years ago
|
||
The try results are in:
chunk 1: 28 fails
chunk 2: a timeout then a crash!
chunk 3: some test timeouts followed by a buildbot timeout
Martijn, if you can add the 28 fails from chunk 1 to this list, we can at least get that running in TBPL. It may take some more runs to excise all the timeout-related failures.
Comment 4•12 years ago
|
||
The last try run tried to run *all* the tests, due to a syntax error in b2g.json. I've fixed that and pushed again: https://tbpl.mozilla.org/?tree=Try&rev=e5d6f1a973cc
Assignee | ||
Comment 5•12 years ago
|
||
(In reply to Jonathan Griffin (:jgriffin) from comment #4)
> The last try run tried to run *all* the tests, due to a syntax error in
> b2g.json. I've fixed that and pushed again:
> https://tbpl.mozilla.org/?tree=Try&rev=e5d6f1a973cc
This mochitest run looks green to me.
I see that you used an include list, instead of an exclude list. I thought you preferred the exclude list?
Comment 6•12 years ago
|
||
Awesome! Yes, I just used your patch; it would take some additional work to make an exclude list based on this include list.
Assignee | ||
Comment 7•12 years ago
|
||
Well, the first thing I posted here in this bug is an exclude list.
It should basically already be the opposite of this include list.
We should favor "include" lists for now, IMO. Reason is that I don't have buy-in that new tests added need to be B2G compliant (or added to an exclude). So we can't just blanket-include a directory or any given checkin might fail it.
As soon as I hear from overholt that they're willing to make tests B2G-compliant for r+, we can (and should) flip that so tests are opt-out instead of opt-in.
Comment 9•12 years ago
|
||
I'd still be in favor of using exclude lists. If someone adds a test that isn't B2G-compatible, it will fail a try/inbound run, and they'll either be forced to fix it or add it as an exclude.
I think that's better than having people write tests and assume it will be run on B2G, when in fact that wouldn't actually be the case.
Comment 10•12 years ago
|
||
To be more clear: most developers do not know about our mobile-specific whitelists.
Comment 11•12 years ago
|
||
Pushed the first patch (the blacklist one) to try:
https://tbpl.mozilla.org/?tree=Try&rev=1bf8acc6ae99
Comment 12•12 years ago
|
||
Almost green!
Updated list and pushed to try again:
https://tbpl.mozilla.org/?tree=Try&rev=82bc20256b6e
Comment 13•12 years ago
|
||
A green run! I'm going to land this version on inbound, if no one objects by tomorrow.
Comment 14•12 years ago
|
||
For the record, this is the patch I'm landing on inbound
Comment 15•12 years ago
|
||
Whiteboard: [automation-needed-in-aurora][automation-needed-in-b2g18]
Comment 16•12 years ago
|
||
Assignee: nobody → martijn.martijn
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla20
Comment 17•12 years ago
|
||
https://hg.mozilla.org/releases/mozilla-aurora/rev/d421c0772331
https://hg.mozilla.org/releases/mozilla-b2g18/rev/b5633541a9ab
status-b2g18:
--- → fixed
status-firefox19:
--- → fixed
status-firefox20:
--- → fixed
Whiteboard: [automation-needed-in-aurora][automation-needed-in-b2g18]
You need to log in
before you can comment on or make changes to this bug.
Description
•