69 crashes with Firefox 100.0. There are 50+ different install times - does it crash only once per install?

Crash report: https://crash-stats.mozilla.org/report/index/c1ca39e3-6b99-4412-a9f3-fc4080220505

Reason: EXCEPTION_STACK_OVERFLOW

Top 10 frames of crashing thread:

0 qipcap64.dll qipcap64.dll@0x000000000002e38b 
1 qipcap64.dll qipcap64.dll@0x000000000002e469 
2 qipcap64.dll qipcap64.dll@0x000000000002e469 
3 qipcap64.dll qipcap64.dll@0x000000000002e469 
4 qipcap64.dll qipcap64.dll@0x000000000002e469 
5 qipcap64.dll qipcap64.dll@0x000000000002e469 
6 qipcap64.dll qipcap64.dll@0x000000000002e469 
7 qipcap64.dll qipcap64.dll@0x000000000002e469 
8 qipcap64.dll qipcap64.dll@0x000000000002e469 
9 qipcap64.dll qipcap64.dll@0x000000000002e469 

Hello, is this crash with the Forcepoint qipcap64.dll module loaded in Firefox related to a new version of ForcePoint? We get crash reports for both Firefox 99.0.1 and 100.0, so it looks unrelated to the new Firefox release. You can see such a crash at https://crash-stats.mozilla.org/report/index/c1ca39e3-6b99-4412-a9f3-fc4080220505

More crash reports at https://crash-stats.mozilla.org/signature/?product=Firefox&signature=qipcap64.dll&date=%3E%3D2022-04-05T00%3A00%3A00.000Z&date=%3C2022-05-05T23%3A59%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1
Based on the graph shown above, this crash signature got more frequent around April 25th.

We may need a new DLL blocklist entry for this.

I've contacted Forcepoint (I didn't notice Joyce was already needinfoed), to try and get a trial/test version to see if I can repro and test any blocking and to let them know about the problem.

The stack from the third party modules ping for the load of QIPCAP64.dll is:

0 firefox.exe!mozilla::freestanding::patched_LdrLoadDll(wchar_t*, unsigned long*, _UNICODE_STRING*, void**)+0x1ec
1 kernelbase.dll!LoadLibraryExW+0x162
2 <unknown>+0xffffffff

This doesn't really give much to go on, so I'm not sure how successful blocking the DLL might be without testing.

Setting leave-open, because we're going to remove the early beta or earlier guards.

https://hg.mozilla.org/mozilla-central/rev/f557fc59b1bf
https://hg.mozilla.org/mozilla-central/rev/84bb3b358b96

Comment on attachment 9275844 [details]
Bug 1767993: Block Forcepoint qipcap*.dll v7.7.819.1 and earlier for high crash rate. r=gcp!

Beta/Release Uplift Approval Request

• User impact if declined: Large number of crashes with qipcap DLLs
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Medium
• Why is the change risky/not risky? (and alternatives if risky): The is a risk that blocking the DLL could cause more issues, but given the high number of crashes already it is probably worth taking that risk.
  I will monitor the crash pings, so we should pick up any issues the next day when they come in.
• String changes made/needed: None
• Is Android affected?: No

Comment on attachment 9275844 [details]
Bug 1767993: Block Forcepoint qipcap*.dll v7.7.819.1 and earlier for high crash rate. r=gcp!

Approved for 101.0b6. I'm going to fold these two patches into one commit for uplift, however.

Comment on attachment 9275844 [details]
Bug 1767993: Block Forcepoint qipcap*.dll v7.7.819.1 and earlier for high crash rate. r=gcp!

Beta/Release Uplift Approval Request

• User impact if declined: Large number of crashes with qipcap DLLs
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Medium
• Why is the change risky/not risky? (and alternatives if risky): The is a risk that blocking the DLL could cause more issues, but given the high number of crashes already it is probably worth taking that risk.
  I will monitor the crash pings, so we should pick up any issues the next day when they come in.
• String changes made/needed: none
• Is Android affected?: No

https://hg.mozilla.org/releases/mozilla-beta/rev/92d2947239af

Comment on attachment 9275844 [details]
Bug 1767993: Block Forcepoint qipcap*.dll v7.7.819.1 and earlier for high crash rate. r=gcp!

Approved for 100.0.1

https://hg.mozilla.org/releases/mozilla-release/rev/a266dec236e3

Redirect a needinfo that is pending on an inactive user to the triage owner.
:haik, since the bug has high priority, high severity and recent activity, could you have a look please?

For more information, please visit auto_nag documentation.

Have an existing discussion group setup with Forcepoint. I'll try to reach out there to get help on this.

(In reply to Bob Owen (:bobowen) from comment #6)

Setting leave-open, because we're going to remove the early beta or earlier guards.

Clearing leave-open now that the EARLY_BETA_OR_EARLIER has landed.