treat assertions as single tracking bugs
Categories
(Testing :: General, task, P2)
Tracking
(Not tracked)
People
(Reporter: jmaher, Unassigned, NeedInfo)
References
(Blocks 1 open bug)
Details
For the last 1.5 years we have been making our intermittent failures be treated as single tracking bug
, this is intended as a test case could have multiple failure reasons, but the same person needs to investigate the test and can look at the frequency of ALL failure types while working on the bug.
A few months ago we formalized this for crashes (example) and that gives us a summary like:
Intermittent application crashed [@ std::__1::deque<nsCocoaWindow::TransitionType, std::__1::allocator<nsCocoaWindow::TransitionType> >::front] | single tracking bug
so all failures we see with the same reason
or top level signature will be tracked by this bug.
I would like to work on assertion failures next. This is the line we typically get for an assertion failure:
Intermittent Assertion failure: !mActor, at /builds/worker/checkouts/gecko/dom/fs/api/FileSystemWritableFileStream.cpp:412
(bug)
it is hard to make that a single tracking bug- but I noticed in every case I spot checked, there is another error message shortly thereafter like:
PROCESS-CRASH | MOZ_ASSERT(!mActor) [@ mozilla::dom::FileSystemWritableFileStream::LastRelease] | dom/html/crashtests/382568-1.html
this could easily be a single tracking bug with a summary:
Intermittent MOZ_ASSERT(!mActor) [@ mozilla::dom::FileSystemWritableFileStream::LastRelease] | single tracking bug
we have the assert, the specific test name or manifest is accessible, and based on the crash signature you can find where in the code this is.
Of course there could be other formats, I would like some feedback on this.
Reporter | ||
Comment 1•1 year ago
|
||
there are MOZ_DIAGNOSTIC_ASSERTS (bug) with a summary like:
Intermittent MOZ_DIAGNOSTIC_ASSERT(mFramesDropped <= 100) (Buffers must be leaking) [@ mozilla::VideoFrameConverter::ProcessVideoFrame] | single tracking bug
fwiw, I am happy to hack on the test harnesses to make the output meet something more desirable.
one concern with some of the mozcrash reason is that it can be too long for a summary line in bugzilla, we will need to figure out how to shorten this line for the summary.
another concern is that we have multiple lines parsed by treeherder and they show up as options to file a bug for- we would need to ensure that sheriffs are using the correct one.
Description
•