Open
Bug 1261197
Opened 9 years ago
Updated 1 year ago
Use StructuredOutputParser for parsing xpcshell test results
Categories
(Release Engineering :: Applications: MozharnessCore, defect)
Release Engineering
Applications: MozharnessCore
Tracking
(Not tracked)
NEW
People
(Reporter: ahal, Unassigned)
References
Details
Attachments
(2 files, 1 obsolete file)
XPCShell emits structured logs, but mozharness currently doesn't consume them. It is still parsing the old formatted logs using a set of regexes. We should make sure to use the StructuredOutputParser class instead of DesktopUnittestParser:
https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/testbase.py#502
We'll need to be careful that we don't miss failures that are hidden in unstructured output. This might mean running the same set of regexes when we come across unstructured output.
Comment 1•8 years ago
|
||
Enable structured log for xpc shell test suites. result looks fine and supposedly we can try more complicated scenarios.
Comment 2•8 years ago
|
||
Reporter | ||
Updated•8 years ago
|
Assignee: nobody → pyang
Reporter | ||
Comment 3•8 years ago
|
||
I did some testing here with the same patch I used to test reftest:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5ee267bb2352f147f638d440e75f63ae62747df8&filter-searchStr=xpcshell
https://treeherder.mozilla.org/#/jobs?repo=try&revision=ae082a11a51f8b7a519a1b2cd6af9724b9bd0dd9
The red X in the second push means that the structured log output parser returned TBPL_FAILURE whereas the old parser returned TBPL_SUCCESS. However both parsers returned an "error" log level. So I don't think that means there will be a new intermittent after landing this change (though maybe that particular failure will make the job red instead of orange or vice versa).
Anyway we should still do the normal testing in addition to the OutputParser differential patch.
Comment 4•8 years ago
|
||
That will be a new failure, and a rather ugly one if it's going to look just like that, with the autoclassify panel seeing a line of output and the failure summary panel seeing nothing.
Reporter | ||
Comment 5•8 years ago
|
||
No, it won't look like that. The stdout from that log is solely from the DesktopUnittestOutputParser, so whatever the structured one is flagging isn't visible in there (to us or to treeherder). The red X basically just tells us we need to do more testing. Good to know that would be a new intermittent though.
Comment 6•8 years ago
|
||
I have same result from those intentional failures.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=797a980944ac32a6d128234c139069c7484efa08&selectedJob=26497289
2 kinds of falures/errors triggered, however assertion failures were considered as errors (red ones), and force killed process was marked as orange.
Expected result should be opposite. Since reftest can differentiate states I think we might need to look into xpcshell.
Comment 7•8 years ago
|
||
By disabling structuredlog, the result remains. Looks like some other problem rather than this patch.
Comment 8•8 years ago
|
||
following patch by enabling structured log for xpcshell
try https://treeherder.mozilla.org/#/jobs?repo=try&revision=506cbc5d7b19c26fc185144e72044bf3f22fc462
Attachment #8767869 -
Attachment is obsolete: true
Attachment #8805023 -
Flags: review?(ahalberstadt)
Reporter | ||
Comment 9•8 years ago
|
||
Comment on attachment 8805023 [details] [diff] [review]
enable-config
Review of attachment 8805023 [details] [diff] [review]:
-----------------------------------------------------------------
Thanks, looks good. I'll land this tomorrow when after I have a chance to do the usual testing.
Attachment #8805023 -
Flags: review?(ahalberstadt) → review+
Comment 10•8 years ago
|
||
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d2ad6e385240
Use StructuredOutputParser for parsing xpcshell test results, r=ahal
I had to back this out for WinXP opt/pgo failures like https://treeherder.mozilla.org/logviewer.html#?job_id=38773982&repo=mozilla-inbound
https://hg.mozilla.org/integration/mozilla-inbound/rev/f63ab56f6202
Flags: needinfo?(pyang)
Reporter | ||
Comment 12•8 years ago
|
||
Looks like this is a pre-existing problem that was being hidden:
https://dxr.mozilla.org/mozilla-central/source/testing/xpcshell/runxpcshelltests.py#1009
Kind of scary is the fact that calling self.log.error *wasn't* turning the job red before. This patches behaviour is definitely the correct one. Seeing as that failure doesn't seem to matter, maybe we can just change the log.error to a warning. Paul, would you mind attaching a new patch and testing that it fixes this on try?
Comment 13•8 years ago
|
||
ya I guess the error wasn't found before but we can lower log level for specific log for sure.
Leave ni? for reminder.
Comment hidden (mozreview-request) |
Updated•8 years ago
|
Flags: needinfo?(pyang)
Comment 15•8 years ago
|
||
Reporter | ||
Comment 16•8 years ago
|
||
(In reply to Paul Yang [:pyang] from comment #15)
> treeherder result
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=e939ddb33e37caa4865a9fbcfea9668db25d9748
Ah, that's how this slipped through.. Looks like the xpcshell tests don't run on XP on try. Sigh, another case of things running on inbound but not try. I triggered the job in question on that push, so we'll see if it still turns orange.
Also thanks for using mozreview! :)
Reporter | ||
Comment 17•8 years ago
|
||
mozreview-review |
Comment on attachment 8808464 [details]
Bug 1261197 - Use StructuredOutputParser for parsing xpcshell test results
https://reviewboard.mozilla.org/r/91294/#review91276
Thanks!
Attachment #8808464 -
Flags: review?(ahalberstadt) → review+
Comment 18•8 years ago
|
||
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/4777ed22d16d
Use StructuredOutputParser for parsing xpcshell test results r=ahal
Comment 19•8 years ago
|
||
Backed out in https://hg.mozilla.org/integration/autoland/rev/ab5b5e1a8fc0d4f21de1a9f4d24269f922e7c201 for telling us that various different WinXP debug (and much more rarely Win7/Win8/opt) tests "logged while not in progress. Logged with data: {"status": "PASS",", like
https://treeherder.mozilla.org/logviewer.html#?job_id=6342927&repo=autoland
https://treeherder.mozilla.org/logviewer.html#?job_id=6344314&repo=autoland
https://treeherder.mozilla.org/logviewer.html#?job_id=6356669&repo=autoland
https://treeherder.mozilla.org/logviewer.html#?job_id=6359262&repo=autoland
https://treeherder.mozilla.org/logviewer.html#?job_id=6351538&repo=autoland
https://treeherder.mozilla.org/logviewer.html#?job_id=6363879&repo=autoland
https://treeherder.mozilla.org/logviewer.html#?job_id=6366573&repo=autoland
https://treeherder.mozilla.org/logviewer.html#?job_id=6365670&repo=autoland (WinXP opt)
https://treeherder.mozilla.org/logviewer.html#?job_id=6344876&repo=autoland (Win7)
https://treeherder.mozilla.org/logviewer.html#?job_id=6367458&repo=autoland (Win8)
Comment 20•8 years ago
|
||
It reminds me bug 1305583
which has same symptom, missing test_end randomly.
Reporter | ||
Comment 21•8 years ago
|
||
Xpcshell runs tests in parallel, so the START/END messages don't get logged in order.. I don't know why this would be intermittent and more prevalent on debug than opt though.
(In reply to Paul Yang [:pyang] from comment #20)
> It reminds me bug 1305583
> which has same symptom, missing test_end randomly.
I think this is a separate issue, if anything it's a missing test_start.. But it's definitely something to do with the OutputParser, as that is the only thing that changed with this patch. So I doubt it's a harness issue.
Reporter | ||
Comment 22•8 years ago
|
||
Ah, so yet again this is a case of the error existing previously but not turning the job orange:
https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-win32-debug/1478640923/mozilla-central_xp_ix-debug_test-xpcshell-bm119-tests1-windows-build38.txt.gz
Before this patch, mozharness was happily turning self.log.error(...) messages into INFO ones. I'll file a new bug to tackle this latest issue, nothing needs to change with this patch.
Comment 23•8 years ago
|
||
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/3891a7e1ba76
Use StructuredOutputParser for parsing xpcshell test results r=ahal
Comment 24•8 years ago
|
||
Backed out in https://hg.mozilla.org/integration/mozilla-inbound/rev/55eb93732d99 - I don't doubt that they deserve to look like that, but 100% of the Win8 debug runs looked like https://treeherder.mozilla.org/logviewer.html#?job_id=39408261&repo=mozilla-inbound
Comment 25•8 years ago
|
||
Well, and opt too, once I found some opt runs.
Updated•1 year ago
|
Severity: normal → S3
Comment 26•1 year ago
|
||
The bug assignee is inactive on Bugzilla, so the assignee is being reset.
Assignee: zapionator → nobody
You need to log in
before you can comment on or make changes to this bug.
Description
•