Closed
Bug 860748
Opened 12 years ago
Closed 12 years ago
Cleanup exclude list for b2g mochitests on the emulator
Categories
(Testing :: Mochitest, defect)
Tracking
(Not tracked)
RESOLVED
FIXED
mozilla23
People
(Reporter: martijn.martijn, Assigned: martijn.martijn)
References
Details
(Whiteboard: [fixed-in-birch])
Attachments
(1 file, 7 obsolete files)
(deleted),
patch
|
Details | Diff | Splinter Review |
I've gone through all the tests in the existing exclude list to see if they are passing in current b2g emulator, especially since bug 788866 is fixed.
And this is on top of the patch of bug 850302, btw, so it depends on that fix.
Also, "content/html/content/test/" directory is not in the exclude list anymore (this was done because of timing out of tbpl, while trying to parse the b2g.json file)
I did a local emulator run and this was the end result:
229819 INFO Passed: 344831
229820 INFO Failed: 0
229821 INFO Todo: 7182
229822 INFO SimpleTest FINISHED
I'll also do a try-server run.
b2g.json file size before is 30.2kb and after, it's 22.3kb, so this shouldn't give problems on tbpl regarding timing out because parsing of the b2g.json file takes too long (it should become faster, even).
Comment 1•12 years ago
|
||
Don't forget to remove the trailing whitespace you've introduced.
Assignee | ||
Comment 2•12 years ago
|
||
Something that should apply against current trunk.
(In reply to :Ms2ger from comment #1)
> Don't forget to remove the trailing whitespace you've introduced.
Not sure where, but hopefully it is resolved in this patch.
Attachment #736281 -
Attachment is obsolete: true
Assignee | ||
Comment 3•12 years ago
|
||
Pushed to try-server:
mwargers-10103:mozilla-central mwargers$ hg qnew cleanup_exclude
mwargers-10103:mozilla-central mwargers$ patch -bp1 < 850302.diff
patching file testing/mochitest/runtestsb2g.py
mwargers-10103:mozilla-central mwargers$ patch -bp1 < b2g.diff
patching file testing/mochitest/b2g.json
mwargers-10103:mozilla-central mwargers$ hg qrefresh -m "Bug 860748 - Cleanup exclude list for b2g mochitests on the emulator - try: -b o -p panda,otoro,unagi -u mochitests -t none"
mwargers-10103:mozilla-central mwargers$ hg qrefresh -m "Bug 860748 - Cleanup exclude list for b2g mochitests on the emulator - try: -b o -p panda,otoro,unagi -u mochitests -t none" try
mwargers-10103:mozilla-central mwargers$ hg push -f -rtip ssh://hg.mozilla.org/try
Are you sure you want to push to remote? (y/n): y
pushing to ssh://hg.mozilla.org/try
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 4 changesets with 2 changes to 2 files (+1 heads)
remote: Looks like you used try syntax, going ahead with the push.
remote: If you don't get what you expected, check http://trychooser.pub.build.mozilla.org/ for help with building your trychooser request.
remote: Thanks for helping save resources, you're the best!
remote: You can view the progress of your build at the following URL:
remote: https://tbpl.mozilla.org/?tree=Try&rev=c98da3ad5033
remote: Trying to insert into pushlog.
remote: Please do not interrupt...
remote: Inserted into the pushlog db successfully.
mwargers-10103:mozilla-central mwargers$
Assignee | ||
Comment 4•12 years ago
|
||
With that latest patch, I got this as result, btw:
230185 INFO Passed: 344963
230186 INFO Failed: 0
230187 INFO Todo: 7246
230188 INFO SimpleTest FINISHED
INFO | automation.py | Application ran for: 2:10:02.540865
Comment 5•12 years ago
|
||
Can you make another try run, this one with -p ics_armv7a_gecko? That's the only platform we're currently running mochitests against.
Assignee | ||
Comment 6•12 years ago
|
||
Ok, done:
mwargers-10103:mozilla-central mwargers$ hg qgoto excludelist
applying excludelist
patch excludelist is empty
now at: excludelist
mwargers-10103:mozilla-central mwargers$ patch -bp1 < b2g.diff
patching file testing/mochitest/b2g.json
mwargers-10103:mozilla-central mwargers$ hg qrefresh -m "Bug 860748 - Cleanup exclude list for b2g mochitests on the emulator - try: -b o -p ics_armv7a_gecko -u mochitests -t none"
mwargers-10103:mozilla-central mwargers$ hg push -f -rtip ssh://hg.mozilla.org/try
Are you sure you want to push to remote? (y/n): y
pushing to ssh://hg.mozilla.org/try
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files (+1 heads)
remote: Looks like you used try syntax, going ahead with the push.
remote: If you don't get what you expected, check http://trychooser.pub.build.mozilla.org/ for help with building your trychooser request.
remote: Thanks for helping save resources, you're the best!
remote: You can view the progress of your build at the following URL:
remote: https://tbpl.mozilla.org/?tree=Try&rev=330f1a5cb0eb
remote: Trying to insert into pushlog.
remote: Please do not interrupt...
remote: Inserted into the pushlog db successfully.
Assignee | ||
Comment 7•12 years ago
|
||
Ok, I'm seeing failures in chunk 2:
content/events/test/test_bug615597.html
content/html/content/test/forms/test_input_sanitization.html
I can re-add those tests to the exclude list for now.
In chunk 3, some other problems are seen:
13:47:52 test_addRange.html test starts, then:
13:55:47 INFO - 27495 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/imptests/editing/selecttest/test_addRange.html | Test timed out.
13:55:47 INFO - 27496 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/imptests/editing/selecttest/test_addRange.html | Test runner timed us out.
13:55:47 INFO - 27497 INFO TEST-END | /tests/dom/imptests/editing/selecttest/test_addRange.html | finished in 455694ms
13:55:47 INFO - 27498 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/imptests/editing/selecttest/test_addRange.html | Test timed out.
I guess this test just takes too long?
A similar thing seems to happen with test_Range-insertNode.html
test_Range-cloneContents.html, test_Range-deleteContents.html, test_Range-extractContents.html seem to have more regular test failures.
I'll add those test files to the exclude list and do a new tryserver test run.
Assignee | ||
Comment 8•12 years ago
|
||
Hmm, according to bug 775227, this might be some oom problem for those time outs.
Assignee | ||
Comment 9•12 years ago
|
||
I'm also re-adding these to the exclude list, since they don't show up in the tryserver log, so weren't tested (because the run didn't complete), so I'm assuming they still fail for now:
"dom/imptests/webapps/DOMCore/tests/approved/test_Range-set.html":"",
"dom/imptests/webapps/DOMCore/tests/approved/test_Range-surroundContents.html":"",
"dom/imptests/webapps/DOMCore/tests/approved/test_Range-mutations.html":"Test timed out.",
Assignee | ||
Comment 10•12 years ago
|
||
test_input_sanitization.html is also timing out related.
I added some comments for those tests that are timing out to the exclude list.
Attachment #736299 -
Attachment is obsolete: true
Assignee | ||
Comment 11•12 years ago
|
||
The previous patch (which was made in the Linux VM) didn't apply for some unknown reason on my Mac tree, where I do the pushes from. So I had to reconstruct the patch from my Mac tree (which was annoying).
So I'm pushing this patch to the try server.
Attachment #737303 -
Attachment is obsolete: true
Assignee | ||
Comment 12•12 years ago
|
||
Pushed to try: https://tbpl.mozilla.org/?tree=Try&rev=9025aba386cb
Assignee | ||
Comment 13•12 years ago
|
||
With that try run, I get in chunk 1:
TEST-UNEXPECTED-FAIL | /tests/content/base/test/test_bug585978.html | application timed out after 330 seconds with no output
Not sure why this failure is happening, there is just a sudden long delay in the test. This test wasn't disabled in current b2g.json file, either. So I guess this is a random failure and I can ignore it for now.
I'm only seeing bug 818190 related to this test.
chunk 4:
15:36:34 INFO - 10670 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/jquery/test_jQuery.html | Test timed out.
This is in the original b2g.json file, so I'll just re-add this entry for now.
chunk 8:
15:37:27 INFO - 2466 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/geolocation/test_cachedPosition.html | uncaught exception - NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIPrefBranch.setCharPref] at http://mochi.test:8888/tests/dom/tests/mochitest/geolocation/geolocation_common.js:64
15:46:07 WARNING - TEST-UNEXPECTED-FAIL | /tests/dom/workers/test/test_closeOnGC.html | application timed out after 330 seconds with no output
15:47:16 INFO - I/GeckoDump( 797): 2466 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/geolocation/test_cachedPosition.html | uncaught exception - NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIPrefBranch.setCharPref] at http://mochi.test:8888/tests/dom/tests/mochitest/geolocation/geolocation_common.js:64
15:47:17 INFO - I/GeckoDump( 797): 3304 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/workers/test/test_closeOnGC.html | Test timed out.
I accidentally removed test_cachedPosition.html and test_closeOnGC.html in the revised patch, so those have to be re-added.
(In reply to Martijn Wargers [:mw22] (QA - IRC nick: mw22) from comment #11)
[10:34] <jgriffin> mwargers: a default B2G build is made using mozilla-b2g18, which is quite different from mozilla-central
[10:35] <jgriffin> you can make a B2G build using mozilla-central if you do BRANCH=master ./config.sh emulator
Assignee | ||
Comment 14•12 years ago
|
||
Pushed this patch to try server: https://tbpl.mozilla.org/?tree=Try&rev=077b9e7f7d51
Attachment #737306 -
Attachment is obsolete: true
Assignee | ||
Comment 15•12 years ago
|
||
Ok, only in chunk 3 a failure in dom/encoding/test/test_stringencoding.html
This one timed out.
I re-added it to the exclude list again, it passed in the previous tryserver run, but I noticed it takes a very long time to run that test file, so I guess it runs the risk of timing out.
Let me know if you want to do another tryserver run on this patch (which I hope would be the last time).
Attachment #737419 -
Attachment is obsolete: true
Attachment #737465 -
Flags: review?(jgriffin)
Comment 16•12 years ago
|
||
Comment on attachment 737465 [details] [diff] [review]
Updated patch_v3
Review of attachment 737465 [details] [diff] [review]:
-----------------------------------------------------------------
Seems good; I don't think we need another try run.
::: testing/mochitest/b2g.json
@@ -3,5 @@
> "caps": "",
> "content": "",
> "docshell": "",
> "dom": "",
> - "dom/apps": "",
Is removing 'dom/apps' intentional?
Attachment #737465 -
Flags: review?(jgriffin) → review+
Assignee | ||
Comment 17•12 years ago
|
||
Yes, because there is already "dom" mentioned, so "dom/apps" is superfluous.
I'm doing a test run locally, and if that one passes, I'll add the check-in keyword to this bug.
Assignee | ||
Comment 18•12 years ago
|
||
(In reply to Martijn Wargers [:mw22] (QA - IRC nick: mw22) from comment #15)
> Ok, only in chunk 3 a failure in dom/encoding/test/test_stringencoding.html
> This one timed out.
> I re-added it to the exclude list again, it passed in the previous tryserver
> run, but I noticed it takes a very long time to run that test file, so I
> guess it runs the risk of timing out.
I filed bug 861850 for increasing the per-test timeout for b2g mochitests.
Assignee | ||
Comment 19•12 years ago
|
||
Local test run went fine:
169910 INFO Passed: 215916
169911 INFO Failed: 0
169912 INFO Todo: 2974
169913 INFO SimpleTest FINISHED
INFO | automation.py | Application ran for: 1:46:17.218677
Assignee: nobody → martijn.martijn
Status: NEW → ASSIGNED
Assignee | ||
Updated•12 years ago
|
Keywords: checkin-needed
Updated•12 years ago
|
Attachment #737465 -
Attachment is obsolete: true
Comment 20•12 years ago
|
||
Keywords: checkin-needed
Whiteboard: [fixed-in-birch]
Comment 21•12 years ago
|
||
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla23
Assignee | ||
Comment 22•12 years ago
|
||
This is the patch that I pushed to try: https://tbpl.mozilla.org/?tree=Try&rev=4d2b426d5db4
A lot of the tests that were giving me problems in order to bug 860748, I removed in the b2g.json list. So with this patch, they shouldn't time out anymore.
Assignee | ||
Comment 23•12 years ago
|
||
Sorry, comment in the wrong bug, this was meant for bug 861850
Assignee | ||
Updated•12 years ago
|
Attachment #738581 -
Attachment is obsolete: true
You need to log in
before you can comment on or make changes to this bug.
Description
•