This bug was filed from the Socorro interface and is report bp-5481ac9b-cfed-491d-9a39-b11792161025. ============================================================= Seen while looking at Nightly crash stats: http://bit.ly/2evhx1X. This crash also affects Aurora. Crashes all seem to contain the dll for Websense- QIPCAP64.dll. At the moment relatively small volume crash.

This is interesting, as it is an example of a signature where we have the 'missile' icon, but the 'yellow' flag. All crashes are happening within one minute of the browser starting up, but the StartupCrash annotation is false in 45 out of 46 reports.

This is not affecting a lot of users for now, but we might want to contact WebSense to avoid this crash exploding when we release 51.

[Tracking Requested - why for this release]: Per comment 2.

Marco, could you suggest how to follow up this issue now ?

I've contacted Forcepoint (makers of WebSense) to ask them if they can help.

full crash summary - https://crash-stats.mozilla.com/signature/?product=Firefox&signature=mozilla%3A%3Anet%3A%3AnsStandardURL%3A%3AEqualsInternal#summary Here's a better stack - https://crash-stats.mozilla.com/report/index/2ea30e23-a812-4daa-aabf-788482170103 This is a startup crash, uptimes are short. Definitely low volume though, 11 crashes on beta over 7 days, and at least one of those isn't related to websense.

The stacks in the crashes with signature "qipcap.dll@0x31100" are similar (see for example https://crash-stats.mozilla.com/report/index/fbc26a51-4a30-447b-80eb-3f3ad2170104, the crash occurs after NotifyObserver), but those are higher volume (1084 over the past week on release, which, considering throttling, means ~10000).

The crashes periodically spike. Recently, "qipcap.dll@0x31100" spiked on 29 Dec and 3 Jan. "mozilla::net::nsStandardURL::EqualsInternal" spiked on the same dates and 22 Dec. https://crash-stats.mozilla.com/signature/?product=Firefox&signature=qipcap.dll%400x31100&date=%3E%3D2016-12-04T15%3A49%3A19.000Z&date=%3C2017-01-04T15%3A49%3A19.000Z#graphs https://crash-stats.mozilla.com/signature/?product=Firefox&signature=mozilla%3A%3Anet%3A%3AnsStandardURL%3A%3AEqualsInternal&date=%3E%3D2016-12-04T15%3A49%3A09.000Z&date=%3C2017-01-04T15%3A49%3A09.000Z#graphs

We now have a machine (Win 10 x64) at Softvision with the Forcepoint (Websense) Endpoint installed. I tested today but did not manage get a crash with: Fx 47.0.2 x86, Fx 50.1.0 x86 and x64, Fx 51b10 x86, and Fx 51b11 x64. The endpoint is the latest version available: 1.5.8.2.2424. We're currently trying to get our hands on some older endpoint versions so we can try to reproduce this.

A lot of users crashing with the signature "qipcap.dll@0x31100" have multiple security software installed: (100.0% in signature vs 00.17% overall) Module "QIPCAP.dll" = true (99.30% in signature vs 00.13% overall) address = 0x68031100 (100.0% in signature vs 05.64% overall) reason = EXCEPTION_ACCESS_VIOLATION_EXEC (86.68% in signature vs 01.03% overall) Module "sysfer.dll" = true (83.41% in signature vs 00.14% overall) Module "PGHook.dll" = true (85.28% in signature vs 03.01% overall) platform_version = 10.0.10586 (100.0% in signature vs 21.44% overall) startup_crash = true (80.61% in signature vs 05.00% overall) Module "IPSEng32.dll" = true QIPCAP.dll is from WebSense. sysfer.dll and IPSEng32.dll are from Symantec. PGHook.dll is from Avecto.

(In reply to Marco Castelluccio [:marco] from comment #10) > QIPCAP.dll is from WebSense. > sysfer.dll and IPSEng32.dll are from Symantec. > PGHook.dll is from Avecto. All three seem to be the same type of Endpoint Protection software. Unfortunately it seems that getting some installers for the Symantec Endpoint Protection and the Avecto Defendpoint software may be just as complicated as it was getting the Websense Endpoint, so I don't think I can make any progress on the short term here.

(In reply to Florin Mezei, QA (:FlorinMezei) from comment #11) > (In reply to Marco Castelluccio [:marco] from comment #10) > > QIPCAP.dll is from WebSense. > > sysfer.dll and IPSEng32.dll are from Symantec. > > PGHook.dll is from Avecto. > > All three seem to be the same type of Endpoint Protection software. > Unfortunately it seems that getting some installers for the Symantec > Endpoint Protection and the Avecto Defendpoint software may be just as > complicated as it was getting the Websense Endpoint, so I don't think I can > make any progress on the short term here. Florin - as an aside, we also have a newly formed relationship with Symantec. While it may take a while like with Forcepoint - we could also pursue a test license for their endpoints too. I'll take your lead and direction here...

That would probably be helpful as well, but that's someone else's call (maybe Sylvestre, Lawrence?).

Hi all, on my work PC i have this problem (Firefox keeps crashing on startup). My IT can't provide any help What info do you need to further investigate the problem?

Hello Paolo, which version of WebSense are you using? Are you also using Symantec Endpoint Protection or Avecto Defendpoint? Can you try using the release version of Firefox (50) instead of the beta version (51)?

I'm using Websense Triton 8.2.3331 and I've reinstalled Firefox from scratch using the online setup so I think it's up to date (version 50).

(In reply to paolo.re.86 from comment #16) > I'm using Websense Triton 8.2.3331 and I've reinstalled Firefox from scratch > using the online setup so I think it's up to date (version 50). Have you submitted a crash report? Can you link it here? You just need to go to the "about:crashes" page.

(In reply to Marco Castelluccio [:marco] from comment #17) > (In reply to paolo.re.86 from comment #16) > > I'm using Websense Triton 8.2.3331 and I've reinstalled Firefox from scratch > > using the online setup so I think it's up to date (version 50). > > Have you submitted a crash report? Can you link it here? > You just need to go to the "about:crashes" page. Yes I have. My crash reports are listed here https://support.mozilla.org/en-US/questions/1152205

(In reply to paolo.re.86 from comment #16) > I'm using Websense Triton 8.2.3331 and I've reinstalled Firefox from scratch > using the online setup so I think it's up to date (version 50). This seems to be a version in between 8.2.5 (8.2.2424) and 8.3.0 (8.3.2509). I don't have any installer for any version in between these two. Paolo, do you happen to have an installer for this version available somewhere? Also, I now see that all crashes occur on Windows x86. I haven't seen any crashes on my Windows 10 x64 machine. I'll try to get a 32-bit Windows version on the machine and also try there.

(In reply to Florin Mezei, QA (:FlorinMezei) from comment #19) > (In reply to paolo.re.86 from comment #16) > > I'm using Websense Triton 8.2.3331 and I've reinstalled Firefox from scratch > > using the online setup so I think it's up to date (version 50). > > This seems to be a version in between 8.2.5 (8.2.2424) and 8.3.0 (8.3.2509). > I don't have any installer for any version in between these two. Paolo, do > you happen to have an installer for this version available somewhere? > > Also, I now see that all crashes occur on Windows x86. I haven't seen any > crashes on my Windows 10 x64 machine. I'll try to get a 32-bit Windows > version on the machine and also try there. Unfortunately I don't have any setup for Triton (it has been installed via an enterprise software installer). Btw I have crashes on Windows 10 x64 even if the crash report says Build Architecture x86.

Update: I've tried installing the x64 version of Firefox (instead of x86) and now it starts without problem. I'll do some uninstall/reinstall to check if everything is right now.

(In reply to Florin Mezei, QA (:FlorinMezei) from comment #19) > (In reply to paolo.re.86 from comment #16) > > I'm using Websense Triton 8.2.3331 and I've reinstalled Firefox from scratch > > using the online setup so I think it's up to date (version 50). > > This seems to be a version in between 8.2.5 (8.2.2424) and 8.3.0 (8.3.2509). > I don't have any installer for any version in between these two. Paolo, do > you happen to have an installer for this version available somewhere? > > Also, I now see that all crashes occur on Windows x86. I haven't seen any > crashes on my Windows 10 x64 machine. I'll try to get a 32-bit Windows > version on the machine and also try there. The architecture shown on Socorro is the architecture of the build (so Firefox 32 bit). You can figure out the architecture of the OS by looking at the "Total Virtual Memory" field (if it is larger than 2 GB, it is Windows 64 bit).

(In reply to Marco Castelluccio [:marco] from comment #22) > The architecture shown on Socorro is the architecture of the build (so > Firefox 32 bit). > You can figure out the architecture of the OS by looking at the "Total > Virtual Memory" field (if it is larger than 2 GB, it is Windows 64 bit). Indeed, I was mistaking there, and Paolo is on Win 10 x64 as well. I'll try to find this Endpoint version and see if I can reproduce.

Quick update here: we spoke to people from Forcepoint yesterday, and they said version 8.2.3331 was distributed via Tech support to some people, which is why I don't have it available. They've promised to give us access to such intermediary builds so hopefully I'll be able to test with the version here.

Track 51+ as this is related to WebSense.

There are 2822 crashes in release version 51.0.1.

I have 50.1.0 and Forcepoint 8.2.5.2424 and is working fine, but when I installl 51.0.1 it crashes. Windows 7 32bit

I was running 50.X and I was upgraded to 51.0.1 and it crashes EVERY time. I've uninstalled, gone back to 49.0.2, disabled updates. I have Forcepoint 8.2 build 2422 (policy engine version 8.1.0.151), on Win10 64bit. I've tried upgrading a couple times to 51.0.1 with the same result, instant crashes.

my corp IT asked me to try with Forcepoint 8.3 build 2515 (8.3.2515) and then upgrade to FF 51.0.1, which I've done. FF starts without immediately crashing, which is an improvement.

(In reply to f1enigma from comment #28) > I have 50.1.0 and Forcepoint 8.2.5.2424 and is working fine, but when I > installl 51.0.1 it crashes. Windows 7 32bit (In reply to andyinsdca from comment #30) > my corp IT asked me to try with Forcepoint 8.3 build 2515 (8.3.2515) and > then upgrade to FF 51.0.1, which I've done. FF starts without immediately > crashing, which is an improvement. Thanks for the reports, we are working with ForcePoint to avoid these issues in the future.

(In reply to Marco Castelluccio [:marco] from comment #31) > (In reply to f1enigma from comment #28) > > I have 50.1.0 and Forcepoint 8.2.5.2424 and is working fine, but when I > > installl 51.0.1 it crashes. Windows 7 32bit > > (In reply to andyinsdca from comment #30) > > my corp IT asked me to try with Forcepoint 8.3 build 2515 (8.3.2515) and > > then upgrade to FF 51.0.1, which I've done. FF starts without immediately > > crashing, which is an improvement. > > Thanks for the reports, we are working with ForcePoint to avoid these issues > in the future. Any update on when this will be fixed? Mozilla 52 has still the issue

(In reply to f1enigma from comment #32) > (In reply to Marco Castelluccio [:marco] from comment #31) > > (In reply to f1enigma from comment #28) > > > I have 50.1.0 and Forcepoint 8.2.5.2424 and is working fine, but when I > > > installl 51.0.1 it crashes. Windows 7 32bit > > > > (In reply to andyinsdca from comment #30) > > > my corp IT asked me to try with Forcepoint 8.3 build 2515 (8.3.2515) and > > > then upgrade to FF 51.0.1, which I've done. FF starts without immediately > > > crashing, which is an improvement. > > > > Thanks for the reports, we are working with ForcePoint to avoid these issues > > in the future. > > Any update on when this will be fixed? Mozilla 52 has still the issue Have you tried updating WebSense? The latest WebSense should be unaffected.

(In reply to Marco Castelluccio [:marco] from comment #33) > (In reply to f1enigma from comment #32) > > (In reply to Marco Castelluccio [:marco] from comment #31) > > > (In reply to f1enigma from comment #28) > > > > I have 50.1.0 and Forcepoint 8.2.5.2424 and is working fine, but when I > > > > installl 51.0.1 it crashes. Windows 7 32bit > > > > > > (In reply to andyinsdca from comment #30) > > > > my corp IT asked me to try with Forcepoint 8.3 build 2515 (8.3.2515) and > > > > then upgrade to FF 51.0.1, which I've done. FF starts without immediately > > > > crashing, which is an improvement. > > > > > > Thanks for the reports, we are working with ForcePoint to avoid these issues > > > in the future. > > > > Any update on when this will be fixed? Mozilla 52 has still the issue > > Have you tried updating WebSense? The latest WebSense should be unaffected. We are behind company firewall and application versioning has policies. So its not up to me to update the version. Do we have an indication when those problems wont affect Mozilla?

(In reply to f1enigma from comment #34) > (In reply to Marco Castelluccio [:marco] from comment #33) > > (In reply to f1enigma from comment #32) > > > (In reply to Marco Castelluccio [:marco] from comment #31) > > > > (In reply to f1enigma from comment #28) > > > > > I have 50.1.0 and Forcepoint 8.2.5.2424 and is working fine, but when I > > > > > installl 51.0.1 it crashes. Windows 7 32bit > > > > > > > > (In reply to andyinsdca from comment #30) > > > > > my corp IT asked me to try with Forcepoint 8.3 build 2515 (8.3.2515) and > > > > > then upgrade to FF 51.0.1, which I've done. FF starts without immediately > > > > > crashing, which is an improvement. > > > > > > > > Thanks for the reports, we are working with ForcePoint to avoid these issues > > > > in the future. > > > > > > Any update on when this will be fixed? Mozilla 52 has still the issue > > > > Have you tried updating WebSense? The latest WebSense should be unaffected. > > We are behind company firewall and application versioning has policies. So > its not up to me to update the version. Do we have an indication when those > problems wont affect Mozilla? ForcePoint fixed this particular crash in the latest version of WebSense. If you can't update WebSense, I'd suggest you to download and use Firefox 45 ESR (select "Firefox 45" on https://www.mozilla.org/en-US/firefox/organizations/all/).

ESR 45 and Mozilla 50.1.0 are the versions compatible with ForcePoint 2424. Its very expensive to upgrade Forcepoint. Do we have any indication on if this will be fixed in future versions of Mozilla?

(In reply to f1enigma from comment #36) > ESR 45 and Mozilla 50.1.0 are the versions compatible with ForcePoint 2424. > Its very expensive to upgrade Forcepoint. Do we have any indication on if > this will be fixed in future versions of Mozilla? The thing is that the issue can't be fixed in future versions of Firefox, as the issue is not in Firefox but in Forcepoint. I suggest using Firefox 45 ESR, at least until you are able to upgrade Forcepoint.

An update on ForcePoint and the engine required for big companies is in the region of 100kEUR. So its not something that can be easily done

(In reply to f1enigma from comment #38) > An update on ForcePoint and the engine required for big companies is in the > region of 100kEUR. So its not something that can be easily done Minor updates are that expensive too? Can you try with Firefox 53 Beta?

From my understanding there is a matrix on client and serverside part of the application. So for the 8.2.5 2424 build the engine that supports it goes to version 8.1 (server side). We are at the limit of them. In order to go beyond 8.2.5 2424 you have to update the engine side which is very expensive for companies. I will try with 53 Beta on Monday morning. Thank you for your understanding

Ive tried with 53 beta 4 and it seems that the problem does no longer exist. Ive found some other issues with websence licence like not recognising gmail.com as trusted but mail.google.com works ok. Dont know if it helps

(In reply to f1enigma from comment #41) > Ive tried with 53 beta 4 and it seems that the problem does no longer exist. > Ive found some other issues with websence licence like not recognising > gmail.com as trusted but mail.google.com works ok. > > Dont know if it helps Yes, thanks, this is useful information. I suppose you can use ESR 45 for now and then update to 53 when it is released :)

So far so good, no crashes on 53.

Confirmed fixed in FF 53.0 with Triton AP-Endpoint DLP v8.2 build 2422 and Policy Engine Version 8.2.0.92 in Windows 7 Enterprise SP1 x86_64.

websense moved to webextensions.