The attached crash information was detected while running CI tests with ThreadSanitizer on try revision cabbf59e867785cb7cdd8b59df2ab57a57bb3dd9.

This is a race on setting mChildIsAlive vs. reading it in the while loop. This could cause visibility problems when the two threads run on different cores, but also once the compiler decides to inline mChildIsAlive in the reading function. I suggest making these variables atomic if they are supposed to be read and written on multiple threads.

General information about TSan reports

Why fix races?

Data races are undefined behavior and can cause crashes as well as correctness issues. Compiler optimizations can cause racy code to have unpredictable and hard-to-reproduce behavior.

Rating

If you think this race can cause crashes or correctness issues, it would be great to rate the bug appropriately as P1/P2 and/or indicating this in the bug. This makes it a lot easier for us to assess the actual impact that these reports make and if they are helpful to you.

False Positives / Benign Races

Typically, races reported by TSan are not false positives [1], but it is possible that the race is benign. Even in this case it would be nice to come up with a fix if it is easily doable and does not regress performance. Every race that we cannot fix will have to remain on the suppression list and slows down the overall TSan performance. Also note that seemingly benign races can possibly be harmful (also depending on the compiler, optimizations and the architecture) [2][3].

[1] One major exception is the involvement of uninstrumented code from third-party libraries.
[2] http://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong
[3] How to miscompile programs with "benign" data races: https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf

Suppressing unfixable races

If the bug cannot be fixed, then a runtime suppression needs to be added in mozglue/build/TsanOptions.cpp. The suppressions match on the full stack, so it should be picked such that it is unique to this particular race. The bug number of this bug should also be included so we have some documentation on why this suppression was added.

Depends on D94287

https://hg.mozilla.org/mozilla-central/rev/bc6adafcbee0

Hmm after looking at this I'm not confident that making mChildIsAlive atomic is sufficient. This type interacts with several threads in very specific ways.

bryce, could you help push this along?

:jib, is this within your purview?

This code has been reliable for years now as far as we know, but my head hurts whenever I look at it.

Making mChildIsAlive atomic seems like a good idea to me. Open to hearing why it might be insufficient.

Sorry for the super slow reply, kept wanting to investigate this closer. So my main concern is there isn't any kind of central synchronization between the various flags like mChildIsAlive and mDestroyed and mShmemPool->Cleanup(). They all just happen in sequence, without an overlying "transaction" mechanism (such as a lock) synchronizing them.

e.g. what happens if someone checks IsShuttingDown() between mShmemPool->Cleanup() and mChildIsAlive = false? Will they make assumptions that shouldn't hold? Why Isn't the lock used in Cleanup held for this entire function?

Similarly, many lines in the code will check several flags in sequence -- e.g. mChildIsAlive && mWebRTCAlive -- and making mChildIsAlive atomic doesn't transactionally synchronize those flags. I worry other threads will observe nonsensical combinations.

It's possible that this code is specifically designed to behave correctly without such transactionality, but I have no idea how to reason about that.

Apologies if this is off base, but how viable would it be to group these flags into a single atomic? In that case, (a subset of) atomic compound checks should be possible via masks, e.g. mChildIsAlive && mWebRTCAlive -> mFlags & (ChildIsAlive | WebRTCAlive) == (ChildIsAlive | WebRTCAlive), possibly with helpers to reduce the boilerplate involved.