Open Bug 1850769 Opened 1 year ago Updated 1 year ago

treat assertions as single tracking bugs

Categories

(Testing :: General, task, P2)

Default
task

Tracking

(Not tracked)

People

(Reporter: jmaher, Unassigned, NeedInfo)

References

(Blocks 1 open bug)

Details

For the last 1.5 years we have been making our intermittent failures be treated as single tracking bug, this is intended as a test case could have multiple failure reasons, but the same person needs to investigate the test and can look at the frequency of ALL failure types while working on the bug.

A few months ago we formalized this for crashes (example) and that gives us a summary like:
Intermittent application crashed [@ std::__1::deque<nsCocoaWindow::TransitionType, std::__1::allocator<nsCocoaWindow::TransitionType> >::front] | single tracking bug

so all failures we see with the same reason or top level signature will be tracked by this bug.

I would like to work on assertion failures next. This is the line we typically get for an assertion failure:
Intermittent Assertion failure: !mActor, at /builds/worker/checkouts/gecko/dom/fs/api/FileSystemWritableFileStream.cpp:412 (bug)

it is hard to make that a single tracking bug- but I noticed in every case I spot checked, there is another error message shortly thereafter like:
PROCESS-CRASH | MOZ_ASSERT(!mActor) [@ mozilla::dom::FileSystemWritableFileStream::LastRelease] | dom/html/crashtests/382568-1.html

this could easily be a single tracking bug with a summary:
Intermittent MOZ_ASSERT(!mActor) [@ mozilla::dom::FileSystemWritableFileStream::LastRelease] | single tracking bug

we have the assert, the specific test name or manifest is accessible, and based on the crash signature you can find where in the code this is.

Of course there could be other formats, I would like some feedback on this.

Flags: needinfo?(choller)

there are MOZ_DIAGNOSTIC_ASSERTS (bug) with a summary like:
Intermittent MOZ_DIAGNOSTIC_ASSERT(mFramesDropped <= 100) (Buffers must be leaking) [@ mozilla::VideoFrameConverter::ProcessVideoFrame] | single tracking bug

fwiw, I am happy to hack on the test harnesses to make the output meet something more desirable.

one concern with some of the mozcrash reason is that it can be too long for a summary line in bugzilla, we will need to figure out how to shorten this line for the summary.

another concern is that we have multiple lines parsed by treeherder and they show up as options to file a bug for- we would need to ensure that sheriffs are using the correct one.

Blocks: 1756895
You need to log in before you can comment on or make changes to this bug.