0 shlwapi.dll StrChrIA 1 shlwapi.dll StrStrIA 2 ws2_32.dll WSARecv (winsock) http://crash-stats.mozilla.com/report/index/0db839af-8bc4-4d5f-92e3-597482091120 Processor notes: WARNING: "[u'6']" is deficient as a name and version for an addon; WARNING: "[u'2']" is deficient as a name and version for an addon; WARNING: "[u'48']" is deficient as a name and version for an addon

Probably an LSP

Adding the space in the title so breakpad picks up the bug.

Signature StrChrIA UUID e60656f9-e02d-4563-8782-7563a2091124 Time 2009-11-24 09:05:14.693524 Uptime 54527 Last Crash 134624 seconds before submission Product Firefox Version 3.5.3 Build ID 20090824101458 Branch 1.9.1 OS Windows NT OS Version 5.1.2600 Service Pack 2 CPU x86 CPU Info GenuineIntel family 6 model 14 stepping 8 Crash Reason EXCEPTION_ACCESS_VIOLATION Crash Address 0xe564000 User Comments Processor Notes Crashing Thread Frame Module Signature [Expand] Source 0 shlwapi.dll StrChrIA 1 shlwapi.dll StrStrIA 2 @0x1066cfb 3 @0x1066d7b 4 @0x1065c2e 5 ws2_32.dll WSARecv 6 wsock32.dll recv 7 nspr4.dll _PR_MD_RECV nsprpub/pr/src/md/windows/w95sock.c:327 8 nspr4.dll SocketRead nsprpub/pr/src/io/prsocket.c:657 9 xul.dll nsSocketInputStream::Read netwerk/base/src/nsSocketTransport2.cpp:353 10 xul.dll nsHttpConnection::OnWriteSegment netwerk/protocol/http/src/nsHttpConnection.cpp:632 11 xul.dll nsHttpTransaction::WritePipeSegment netwerk/protocol/http/src/nsHttpTransaction.cpp:503 12 xul.dll nsPipeOutputStream::WriteSegments xpcom/io/nsPipe3.cpp:1137 13 xul.dll nsHttpTransaction::WriteSegments netwerk/protocol/http/src/nsHttpTransaction.cpp:529 14 xul.dll nsHttpConnection::OnSocketReadable netwerk/protocol/http/src/nsHttpConnection.cpp:664 15 xul.dll nsHttpConnection::OnInputStreamReady netwerk/protocol/http/src/nsHttpConnection.cpp:762 16 xul.dll nsSocketInputStream::OnSocketReady netwerk/base/src/nsSocketTransport2.cpp:256 17 xul.dll nsSocketTransport::OnSocketReady netwerk/base/src/nsSocketTransport2.cpp:1519 18 xul.dll nsSocketTransportService::DoPollIteration netwerk/base/src/nsSocketTransportService2.cpp:674 19 xul.dll nsSocketTransportService::OnProcessNextEvent netwerk/base/src/nsSocketTransportService2.cpp:538 20 xul.dll nsThread::ProcessNextEvent xpcom/threads/nsThread.cpp:497 21 xul.dll NS_ProcessPendingEvents_P obj-firefox/xpcom/build/nsThreadUtils.cpp:180 22 xul.dll NS_ProcessNextEvent_P obj-firefox/xpcom/build/nsThreadUtils.cpp:227 23 xul.dll nsSocketTransportService::Run netwerk/base/src/nsSocketTransportService2.cpp:581 24 xul.dll nsThread::ProcessNextEvent xpcom/threads/nsThread.cpp:510 25 xul.dll nsThread::ThreadFunc xpcom/threads/nsThread.cpp:254 26 nspr4.dll _PR_NativeRunThread nsprpub/pr/src/threads/combined/pruthr.c:426 27 nspr4.dll pr_root nsprpub/pr/src/md/windows/w95thred.c:122 28 mozcrt19.dll _callthreadstartex obj-firefox/memory/jemalloc/src/threadex.c:348 29 mozcrt19.dll _threadstartex obj-firefox/memory/jemalloc/src/threadex.c:326 30 kernel32.dll BaseThreadStart Filename Version Debug Identifier Debug Filename NPSWF32.dll 10.0.32.18 C569605C15E5448BBFD9E1FE262649B61 NPSWF32.pdb mdnsNSP.dll 1.0.6.2 E82F22B11854424A8DAE403E0F7A62A91 mdnsNSP.pdb ccL40.dll 104.0.1.17 05AB7705112C4149AC4870E12D01E6F81 ccL40.pdb I don't think any of these are related but i'd love to be proven wrong. Frames 2-4 scare me. Signature StrChrIA UUID abf3bc64-c231-431a-836a-fd47b2091122 Time 2009-11-22 09:10:13.599272 Uptime 464 Last Crash 408839 seconds before submission Product Firefox Version 3.5.3 Build ID 20090824101458 Branch 1.9.1 OS Windows NT OS Version 5.1.2600 Service Pack 2 CPU x86 CPU Info GenuineIntel family 15 model 4 stepping 3 Crash Reason EXCEPTION_ACCESS_VIOLATION Crash Address 0x5c7c000 User Comments Processor Notes WARNING: No 'client_crash_date' could be determined from the Json file Crashing Thread Frame Module Signature [Expand] Source 0 shlwapi.dll StrChrIA 1 shlwapi.dll StrStrIA 2 DA475474.x86.dll DA475474.x86.dll@0x28a3 I like this one because we can see a bad guy at the bottom. Signature StrChrIA UUID ecb0fab6-2fae-40e0-8dd2-704722091122 Time 2009-11-22 23:05:57.162820 Uptime 4641 Last Crash 4644 seconds before submission Product Firefox Version 3.5.3 Build ID 20090824101458 Branch 1.9.1 OS Windows NT OS Version 5.1.2600 Service Pack 3 CPU x86 CPU Info GenuineIntel family 15 model 4 stepping 9 Crash Reason EXCEPTION_ACCESS_VIOLATION Crash Address 0x118e2000 User Comments Processor Notes WARNING: "[u'6']" is deficient as a name and version for an addon; WARNING: "[u'2']" is deficient as a name and version for an addon; WARNING: "[u'41']" is deficient as a name and version for an addon Crashing Thread Frame Module Signature [Expand] Source 0 shlwapi.dll StrChrIA 1 shlwapi.dll StrStrIA 2 7129F7E2.x86.dll 7129F7E2 again, something that looks like a bad guy. Signature StrChrIA UUID 6f42b825-ec68-486e-9b51-59a422091119 Time 2009-11-19 06:36:28.375838 Uptime 856 Last Crash 865 seconds before submission Product Firefox Version 3.5.3 Build ID 20090824101458 Branch 1.9.1 OS Windows NT OS Version 6.1.7100 CPU x86 CPU Info GenuineIntel family 6 model 14 stepping 12 Crash Reason EXCEPTION_ACCESS_VIOLATION Crash Address 0x890d000 User Comments Processor Notes Crashing Thread Frame Module Signature [Expand] Source 0 shlwapi.dll StrChrIA 1 shlwapi.dll StrStrIA 2 avsda.dll avsda.dll@0x245d avsda.dll 9.0.1.1 B9A99DF4825F41B781F82B2E9AD4BAC91 avsda.pdb this is Avira's LSP, so I'm right in at least one instance :) this is a semi random sampling, i think i skipped only one crash from my sample because it wasn't telling me anything more than jim's original

for some reason this signature has gone crazy over the last few days report for the stack signature StrChrIA 64 crashes for StrChrIA on 20091201-crashdata 65 crashes for StrChrIA on 20091202-crashdata 52 crashes for StrChrIA on 20091203-crashdata 53 crashes for StrChrIA on 20091204-crashdata 49 crashes for StrChrIA on 20091205-crashdata 45 crashes for StrChrIA on 20091206-crashdata 53 crashes for StrChrIA on 20091207-crashdata 255 crashes for StrChrIA on 20091208-crashdata 430 crashes for StrChrIA on 20091209-crashdata 573 crashes for StrChrIA on 20091210-crashdata 623 crashes for StrChrIA on 20091211-crashdata 618 crashes for StrChrIA on 20091212-crashdata 654 crashes for StrChrIA on 20091213-crashdata 628 crashes for StrChrIA on 20091214-crashdata 684 crashes for StrChrIA on 20091215-crashdata 737 crashes for StrChrIA on 20091216-crashdata currently ranked around #27 and up 140 slots in the 7 day report.

shlwapi.dll is microsoft library which contains functions for UNC and URL paths, registry entries, and color settings. 2009 12 08 is patch tuesday. wonder if there is any connection and the new update had tickled a volume increase or a new bug?

all 232213 737 0.00317381 3.0.15 32264 120 0.00371932 3.5.5 93452 311 0.00332791 3.6b4 21927 90 0.00410453 3.6b3 675 0 3.6b2 849 3 0.00353357 3.6b1 2304 1 0.000434028 all releases 3 3.0.1 1 3.0.10 1 3.0.12 1 3.0.14 120 3.0.15 42 3.0.16 1 3.0.7 2 3.0.8 1 3.1b2 4 3.5 5 3.5.2 1 3.5.3 2 3.5.4 311 3.5.5 148 3.5.6 1 3.6b1 3 3.6b2 90 3.6b4

crash urls are pretty widely and uniformly distributed over popular browsing sites domains of sites 56 \N// 55 http://apps.facebook.com 39 http://www.facebook.com 34 about:blank// 18 http://messaging.myspace.com 18 http://home.myspace.com 17 http://www.myspace.com 16 http://viewmorepics.myspace.com 15 http://www.google.com 13 http://www.tagged.com 11 http://profile.myspace.com 10 http://twitter.com < long tail snipped>

nothing jumps out in a quick scan of http://www.microsoft.com/technet/security/bulletin/ms09-dec.mspx

Signature StrChrIA UUID 1c3122b0-5d90-427f-ad29-d52fc2091217 Time 2009-12-17 22:29:50.830579 Uptime 5379 Last Crash 38531 seconds before submission Product Firefox Version 3.5.3 Build ID 20090824101458 Branch 1.9.1 OS Windows NT OS Version 5.1.2600 Service Pack 3 CPU x86 CPU Info GenuineIntel family 15 model 4 stepping 3 Crash Reason EXCEPTION_ACCESS_VIOLATION Crash Address 0xe2a3000 User Comments Processor Notes WARNING: No 'client_crash_date' could be determined from the Json file Related Bugs Crashing Thread Frame Module Signature [Expand] Source 0 shlwapi.dll StrChrIA 1 shlwapi.dll StrStrIA 2 B3180C00.x86.dll B3180C00.x86.dll@0x28a3 Thread 3 Frame Module Signature [Expand] Source 0 ntdll.dll KiFastSystemCallRet 1 ntdll.dll ZwRemoveIoCompletion 2 B3180C00.x86.dll B3180C00.x86.dll@0x4aab 3 kernel32.dll BaseThreadStart Thread 4 Frame Module Signature [Expand] Source 0 ntdll.dll KiFastSystemCallRet 1 ntdll.dll NtReplyWaitReceivePortEx 2 B3180C00.x86.dll B3180C00.x86.dll@0x1655 3 kernel32.dll BaseThreadStart Can we blocklist any file named *.x86.dll?

more discussion of blocking *x86.dll over in bug 516112

recently running at about 600-850 crashes per day. date StrChrIAcrashes 20100101-crashdata 601 StrChrIA 20100102-crashdata 686 StrChrIA 20100103-crashdata 798 StrChrIA 20100104-crashdata 842 StrChrIA 20100105-crashdata 824 StrChrIA 20100106-crashdata 854 StrChrIA 20100107-crashdata 877 StrChrIA 20100108-crashdata 721 StrChrIA 20100109-crashdata 728 StrChrIA

also more dicussion of Crashes [@ shlwapi.dll over in Bug 556178

Still pretty high volume - sitting at #45 for 8.0. Leaving the top crash keyword for now. Is there something we can block here? Doesn't seem very actionable.

This was a bunch lower in 9.0.1. Outside the top 100. I am going to remove the topcrash keyword.

It's #16 top browser crasher in 12.0, #29 in 13.0b4, and #11 in 14.0a2.

Still high - sitting at #29 on the top crash list.

It's #24 top browser crasher in 13.0.1, but up to #11 in 14.0b10 and #6 in 14.0b11. There are no obvious correlations in 13.0.1, but in 14.0 Beta it's correlated to BitDefender 2012: StrChrIA|EXCEPTION_ACCESS_VIOLATION_READ (730 crashes) 55% (400/730) vs. 1% (462/65994) BdProvider.dll 0% (1/730) vs. 0% (1/65994) 16.0.14.1139 6% (45/730) vs. 0% (57/65994) 16.15.0.1213 4% (28/730) vs. 0% (44/65994) 16.16.0.1317 45% (325/730) vs. 1% (356/65994) 16.16.0.1339 0% (1/730) vs. 0% (4/65994) 16.17.0.1350 51% (369/730) vs. 1% (533/65994) avcuf32.dll 0% (0/730) vs. 0% (4/65994) 3.8.5601.4190 0% (0/730) vs. 0% (3/65994) 3.9.6208.4190 51% (369/730) vs. 1% (526/65994) 3.9.6339.4190 Here are the first frames of the matching stack trace: Frame Module Signature Source 0 shlwapi.dll StrChrIA 1 shlwapi.dll StrStrIA 2 BdProvider.dll BdProvider.dll@0x2777 3 BdProvider.dll BdProvider.dll@0xe444 4 ws2_32.dll send 5 nspr4.dll SocketWrite nsprpub/pr/src/io/prsocket.c:701 6 xul.dll nsSocketOutputStream::Write netwerk/base/src/nsSocketTransport2.cpp:587 7 xul.dll nsHttpConnection::OnReadSegment netwerk/protocol/http/nsHttpConnection.cpp:1190 8 xul.dll nsHttpPipeline::ReadFromPipe netwerk/protocol/http/nsHttpPipeline.cpp:682 More reports at: https://crash-stats.mozilla.com/report/list?signature=StrChrIA

This bug was #16 in 12.0 and has risen a bit in FF14. I think our best bet is to do outreach here and try the URLs in QA. Including Kev here and adding qawanted. Also tentatively tracking for release, although this wouldn't block.

Total Count URL 84 http://www.facebook.com/ 57 about:blank 11 https://www.facebook.com/login.php?login_attempt=1 11 http://www.tumblr.com/dashboard 10 http://www.facebook.com/?ref=tn_tnmn 10 http://www.meetme.com/apps/home 9 about:newtab 6 http://www.yahoo.com/ 4 http://www.facebook.com/?ref=logo 4 http://www.google.com/ 4 http://www.meetme.com/ 4 https://www.facebook.com/ 4 http://www.expressz.hu/index.do?s_=1&so_ns_rid=1&x=regions%3A9&cid=2909&parentRe 3 http://messages.meetme.com/ 3 http://www.youtube.com/ 3 http://www.odnoklassniki.ru/ 3 http://www.milliyet.com.tr/ 3 http://www.crtinv.com/products/ADDictThing/default/getData.html?d=http%3A%2F%2Fw 3 https://www.google.co.uk/ 3 http://www.cucirca.com/2008/06/19/one-tree-hill-season-5-episode-6-dont-dream-it 3 http://www.freelove.hu/index.php?id=14&sub_id=13 2 http://www.ilsole24ore.com/?refresh_ce 2 http://www.lequipe.fr/ 2 http://www.theblaze.com/ 2 http://www.tumblr.com/inbox 2 http://apps.facebook.com/warcommander/?fb_source=bookmark_apps&ref=bookmarks&cou 2 http://www.youtube.com/inbox?feature=mhee&folder=messages 2 http://facebook.com/ 2 http://www.transfermarkt.de/de/1-fsv-mainz-05/startseite/verein_39.html 2 http://www.worldstarhiphop.com/videos/ 2 http://yahoo.com/ 2 http://www.lrytas.lt/ 2 http://www.tuenti.com/#m=Home&func=index 2 http://cgi5.ebay.de/ws/eBayISAPI.dll 2 http://apps.facebook.com/avengersalliance/?fb_source=bookmark_apps&ref=bookmarks 2 http://apps.facebook.com/pioneertrail/?fb_source=bookmark_apps&ref=bookmarks&cou 2 http://apps.facebook.com/playslingo/?fb_source=bookmark_apps&ref=bookmarks&count 2 http://www.kansascity.com/sports/ 2 http://www.deviantart.com/ 2 http://www.gameduell.fr/gd/singleplayer03GameStart.do 2 https://ssl.meetme.com/login 2 http://www.profblog.org/ 2 http://www.speedtest.net/ 2 http://xfinity.comcast.net/ 2 http://globoesporte.globo.com/

(In reply to Scoobidiver from comment #17) > There are no obvious correlations in 13.0.1, but in 14.0 Beta it's > correlated to BitDefender 2012 This keeps to be the case, and I think that's something we should investigate, and possibly also to outreach to them.

I've been trying to reproduce this problem on Fx14.0b12 using the URLs listed here, doing different kinds of interactions, while having installed the BitDefender antivirus software as well as their couple of add-ons listed in AMO. No luck so far.

https://crash-stats.mozilla.com/report/list?signature=StrCmpNIA is another signature which I see in early 14.0.1 data that seems to be correlated to Bit Defender as well.

Addon correlations from Firefox 14.0.1: StrChrIA|EXCEPTION_ACCESS_VIOLATION_READ (1047 crashes) 25% (263/1047) vs. 10% (6643/66763) {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} (Adblock Plus, https://addons.mozilla.org/addon/1865) 0% (0/1047) vs. 0% (14/66763) 1.3.10 0% (0/1047) vs. 0% (2/66763) 1.3.3 0% (0/1047) vs. 0% (2/66763) 1.3.5 0% (0/1047) vs. 0% (1/66763) 2.0.1 0% (0/1047) vs. 0% (2/66763) 2.0.2 0% (4/1047) vs. 0% (131/66763) 2.0.3 0% (0/1047) vs. 0% (14/66763) 2.1 25% (259/1047) vs. 10% (6474/66763) 2.1.1 0% (0/1047) vs. 0% (1/66763) 2.1.2a.3522 0% (0/1047) vs. 0% (1/66763) 2.1.2a.3523 0% (0/1047) vs. 0% (1/66763) 2.1.2b.3528

I was able to repro this bug once on my machine. At the time of the crash a dialog came up asking me to install a root certificate. I will try to get a better set of steps or see if I can crash again. https://crash-stats.mozilla.com/report/index/bp-34503c32-9845-4f2e-aa0c-d2ff12120720

(In reply to Marcia Knous [:marcia] from comment #24) > I was able to repro this bug once on my machine. At the time of the crash a > dialog came up asking me to install a root certificate. I will try to get a > better set of steps or see if I can crash again. > > https://crash-stats.mozilla.com/report/index/bp-34503c32-9845-4f2e-aa0c- > d2ff12120720 Any updates here Marcia?

No luck reproducing on the same machine. It seems that the dialog that comes up asking me to save the root certificate doesn't work that well because I keep getting the prompt, but no crash. (In reply to Alex Keybl [:akeybl] from comment #25) > (In reply to Marcia Knous [:marcia] from comment #24) > > I was able to repro this bug once on my machine. At the time of the crash a > > dialog came up asking me to install a root certificate. I will try to get a > > better set of steps or see if I can crash again. > > > > https://crash-stats.mozilla.com/report/index/bp-34503c32-9845-4f2e-aa0c- > > d2ff12120720 > > Any updates here Marcia?

http://forum.bitdefender.com/index.php?showtopic=35932 seems to have some information, reading through it now.

I have posted in that forum asking for more information from the users that are hitting the crash.

That thread indicates it's about BD 2013 and is fixed in 15.0 Beta. It still happens in 15.0 Beta but less often according to crash stats as it's #28 top browser crasher in 13.0.1, #10 in 14.0.1 and #20 in 15.0b1 and correlations per module: * 13.0.1: no correlation to BdProvider.dll * 14.0.1: 63% (1343/2137) vs. 1% (1494/151974) BdProvider.dll 0% (1/2137) vs. 0% (1/151974) 1.0.0.177 0% (1/2137) vs. 0% (1/151974) 16.0.14.1139 1% (30/2137) vs. 0% (37/151974) 16.15.0.1213 2% (33/2137) vs. 0% (39/151974) 16.16.0.1317 24% (523/2137) vs. 0% (576/151974) 16.16.0.1339 35% (755/2137) vs. 1% (840/151974) 16.17.0.1350 * 15.0: 38% (73/192) vs. 0% (103/48561) BdProvider.dll 1% (1/192) vs. 0% (1/48561) 16.0.14.1139 0% (0/192) vs. 0% (1/48561) 16.15.0.1213 2% (4/192) vs. 0% (5/48561) 16.16.0.1317 7% (13/192) vs. 0% (22/48561) 16.16.0.1339 29% (55/192) vs. 0% (74/48561) 16.17.0.1350 It's likely related to other HTTP networking crashes in nsHttpConnection::OnReadSegment that also spiked in 14.0: bug 763261, bug 763385, bug 763386, bug 763392, bug 763393, bug 763395, and bug 763398.

https://crash-stats.mozilla.com/report/index/bp-c61a45ae-da1b-4d4c-bdcb-34e082120725 - I crashed on another machine in the lab. One thing I consistently note is that I keep getting a Bit Defender prompt to install a certificate. I click OK to close Firefox and install it, but I keep consistently getting the prompt thereafter.

The Win 7 lab machine seems to crash much more often than the other machine I tested on. This machine has crashed at least 6-7 times in the last 15 minutes. Usually when I crash I do see the same dialog referenced in Comment 30. On this Win 7 Machine I am running Bitdefender Internet Security 2013/Firefox 14.0.1. I noticed when I was on the liveleak.com site and started opening a bunch of different videos, I started crashing more often.

I got the attached after a recent crash. You can see the dialog I was referring to in previous comments. I can get this crash to reproduce fairly reliably by just clicking on links from liveleak.com.

(In reply to Marcia Knous [:marcia] from comment #32) > I can get this crash to reproduce fairly reliably by just clicking on links > from liveleak.com. People in the thread say it crashes less often with 15.0 Beta. Can you give it a try?

I took a look at the win 7 machine in the lab running firefox in a debugger. Everything looks fine on our end of things, I suspect this is caused by a bug in bitdefender.

(In reply to Nick Hurley [:hurley] from comment #34) > I suspect this is caused by a bug in bitdefender. One user says it doesn't happen in Fx 12.0 (see http://forum.bitdefender.com/index.php?s=&showtopic=35932&view=findpost&p=150695), other users say it's with Fx 15.0 Beta (see comment 29). So it's in Firefox 14.0.1.

(Bitdefender dev here) According to QA we fixed a bug very similar to mozilla bug 530074, we are putting it on update next Monday. I honestly appreciate everyone's help in identifying this.

harjoc - I appreciate the communication here.

If this still happens in connection with Bitdefender, please check that you have the latest relevant BdProvider.dll -- 16.18.0.1406 -- which was pushed this week. The security suite should update it automatically. I still see reports on crash-stats with 16.17.0.1350 which is old (and other unrelated LSP offenders) fwiw.

harjoc - can you give us any idea of the kind of behavior that tickled the BD bug? Reports seem to indicate different firefox revs had different levels of crashes, so I'm curious from a defensive pov.

I am experiencing constant crashing while using latest Bit Defender Total Security 2013 - 16.18.0.1406 and FireFox 14.01 running on windows 7. From the looks of it so far the crash seems to happen when Antiphishing is enabled. What it does is scan every link on the page and puts a green check mark next to it. The crash seems to happen most often at ajax heavy sites.

Just got another crash while doing a Quick Virus Scan as well.

Now I crashed while not even focused on the browser. It seems when I narrow it down to something, something else causes it to crash :(

E, can you provide your crash ID from about:crashes?

McManus, I asked for more info on how to reproduce, hopefully E gives us a crash ID in the mean time.

Here is a whole list of crashes within the last 5 days or so. bp-f78d66a7-bbcd-4827-9ce2-3e5152120803 bp-a7d162c5-2c55-4dc6-a7ce-0afe42120803 bp-d7d71dc9-1bf6-4ec4-84dc-aaf8c2120803 bp-25af5750-c67e-46bf-92ae-015802120803 bp-29523f92-40b1-4500-94f6-dacd22120803 bp-dc59dc26-3212-4f2b-b3f1-d30c32120803 bp-a6a9432b-e99d-411d-b9d3-2c6f72120802 bp-9577b0a1-a6e7-4e31-a5df-b37272120802 bp-3bac04a5-6a49-4408-a104-a47cc2120802 bp-7f68d372-a2de-4c9c-9ed8-113832120802 bp-ad1c3a3d-76fa-4aaa-954e-3bbc92120801 bp-bc97179f-ad3d-425f-89f2-aacae2120801 bp-a17a2bf4-34ca-480f-9b9e-882f12120731

A SUMO User who seems to be experiencing this issue. https://support.mozilla.org/en-US/questions/933874 QA, do we need to do 1:1 with them? I can step in and help out.

E, thank you for the crash ids. I looked at them all and you apparently have BdProvider.dll 16.17.0.1350 (not the latest version). I just took the liberty earlier to scrape all the crashes for "StrChrIA" for Firefox-14.0.1 from yesterday (2198 crashes). Apparently the bug reproduced on Vista and Win7, so I'm pulling the raw data for NT6.0 and NT 6.1, and looking at all the BdProvider.dll occurences (700 raw reports so far): 7 BdProvider.dll|16.15.0.1213 9 BdProvider.dll|16.16.0.1317 112 BdProvider.dll|16.16.0.1339 293 BdProvider.dll|16.17.0.1350 421 total So, 0 hits for 16.18.0.1406 (on Vista and Win7). There were 236 crashes on Firefox 15.0b2 yesterday which I guess are also mostly with BdProvider 16.17. As most users update their security product, the Bitdefender-related crash count will probably go back towards zero.

How do we update it? The reason I ask is when I go to about page on my security it says: 16.18.0.1406 but if I go to details on the bdprovider.dll, it is as you said its BdProvider 16.17

Ok nvm, all it took was a restart of the computer. Weird that Bit Defender does not prompt to restart after upgrade. Anyways I am turning on the phishing filter and will report how things go.

My Win 7 lab machine is still getting crashes, and the BitDefender about: page shows I am running 16.18.0.1406. Here is my latest crash report: https://crash-stats.mozilla.com/report/index/bp-7891fa14-b3fe-4bc8-aa30-fcd4b2120806 I did restart the computer after the update was applied. The dll version from the module section of the crash shows 16.16.0.1339 and so does the dll when I search for it on my computer. So somehow even after a restart my machine did not get updated. If I check for updates manually there are no updates found.

(In reply to Marcia Knous [:marcia] from comment #50) > My Win 7 lab machine is still getting crashes, and the BitDefender about: > page shows I am running 16.18.0.1406. This dll version is not widespread according to crash stats in 14.0.1: 1.0.0.177 0.65% 16.15.0.1213 1,20% 16.16.0.1317 1.42% 16.16.0.1339 31.70% 16.17.0.1350 64.16% 16.18.0.1406 0.87% So there's something wrong with the BD update process.

marcia, update failures happen because - the file could not be overwritten (there's an .upd file next to the .dll), or - the update download failed In both cases there should be a related message in the Events page. If not, the only suggestion so far I got from QA was to restart (I know...). I'll ask for more info if things still don't work.

Scoobidiver, it's not clear if 16.18.* is not widespread in crashes because people haven't updated to it, or because it fixes the problem. From the 1392 BdProvider-related reports from last Friday I saw 0 reports with 16.18.*. I've been trying to get some numbers from the update team to see if people are having problems updating. I'll share them once I have them.

It's #2 top browser crasher in 16.0 and is slightly correlated to Babylon, Apple's Bonjour and Avira Antivirus: StrChrIA|EXCEPTION_ACCESS_VIOLATION_READ (394 crashes) 27% (106/394) vs. 6% (3754/63152) browsemngr.dll 34% (135/394) vs. 14% (9152/63152) mdnsNSP.dll 17% (68/394) vs. 2% (1459/63152) avsda.dll StrCmpNIA|EXCEPTION_ACCESS_VIOLATION_READ (46 crashes) 22% (10/46) vs. 2% (1459/63152) avsda.dll 24% (11/46) vs. 6% (3754/63152) browsemngr.dll 30% (14/46) vs. 14% (9152/63152) mdnsNSP.dll

Can we block all version of the offending dll prior to 16.18.*? Or are LSPs immune from that?

(In reply to Jim Mathies [:jimm] from comment #55) > Can we block all version of the offending dll prior to 16.18.*? It's no longer correlated to bdprovider.dll (the BD update process is probably done for every users) so blocking those versions won't help to reduce the crash volume.

(In reply to Jim Mathies [:jimm] from comment #55) > Can we block all version of the offending dll prior to 16.18.*? Or are LSPs > immune from that? I don't know of any way we'd have to block LSPs (and we don't even do any summary/correlation reports on them, so we don't have good data if a well-identifiable LSP is behind this). Also, those DLL correlations in comment #54 are weak enough that they're not pointing to anything specific, they just say that people running into this are more likely to have those things installed - but no smoking gun here so far.

Leaving this nominated - the crash volume is too low to know for sure whether this will be a top issue.

Without hangs, it's #2 top crasher in 16.0 and #4 in 15.0.1 so it doesn't seem related to a regression in 16.0 although it spiked recently. Today's correlations are similar to those of comment 54: *15.0.1: StrChrIA|EXCEPTION_ACCESS_VIOLATION_READ (2061 crashes) 14% (294/2061) vs. 2% (3792/172261) avsda.dll (Avira AV) 16% (331/2061) vs. 4% (7355/172261) browsemngr.dll (Babylon) 12% (246/2061) vs. 6% (9621/172261) mgAdaptersProxy.dll (Adapter Proxy) 6% (127/2061) vs. 1% (1351/172261) browsemngr-15.0.dll (Babylon for Fx 15) StrCmpNIA|EXCEPTION_ACCESS_VIOLATION_READ (169 crashes) 18% (30/169) vs. 2% (3792/172261) avsda.dll (Avira AV) 28% (47/169) vs. 18% (30249/172261) mdnsNSP.dll (Bonjour) 13% (22/169) vs. 4% (7355/172261) browsemngr.dll (Babylon) *16.0: StrChrIA|EXCEPTION_ACCESS_VIOLATION_READ (423 crashes) 32% (135/423) vs. 7% (4599/65384) browsemngr.dll (Babylon) 17% (74/423) vs. 2% (1530/65384) avsda.dll (Avira AV) 23% (98/423) vs. 14% (9376/65384) mdnsNSP.dll (Bonjour) 13% (57/423) vs. 6% (4109/65384) mgAdaptersProxy.dll (Adapter Proxy) 7% (28/423) vs. 1% (825/65384) browsemngr-16.0.dll (Babylon for Fx 16) StrCmpNIA|EXCEPTION_ACCESS_VIOLATION_READ (66 crashes) 27% (18/66) vs. 7% (4599/65384) browsemngr.dll (Babylon) 14% (9/66) vs. 6% (4109/65384) mgAdaptersProxy.dll (Adapter Proxy) 21% (14/66) vs. 14% (9376/65384) mdnsNSP.dll (Bonjour)

No volume on 17 to justify tracking (yet), please re-nominate if this shows up in any significant number on 17 Beta.

i was also running into it with a test vm - https://crash-stats.mozilla.com/report/index/bp-2d73ad55-9b52-4faa-8ac9-1be3b2121017 so let me know if you need any info

(In reply to Carsten Book [:Tomcat] from comment #61) > i was also running into it with a test vm - > https://crash-stats.mozilla.com/report/index/bp-2d73ad55-9b52-4faa-8ac9- > 1be3b2121017 so let me know if you need any info This one might be caused by Babylon.

It's #2 top browser crasher in 19.0.2, #3 in 20.0b4 and 21.0a2, and #6 in 22.0a1. That higher volume is likely caused by FireJump (see http://firejump.net/) where it's correlated at 22%: StrChrIA|EXCEPTION_ACCESS_VIOLATION_READ (3995 crashes) 22% (862/3995) vs. 1% (1095/139851) firejump@firejump.net (1.0.2.5) StrCmpNIA|EXCEPTION_ACCESS_VIOLATION_READ (406 crashes) 21% (86/406) vs. 1% (1095/139851) firejump@firejump.net (1.0.2.5) StrStrIA|EXCEPTION_ACCESS_VIOLATION_READ (18 crashes) 33% (6/18) vs. 1% (1095/139851) firejump@firejump.net (1.0.2.5)

Is there an example on crash-stats where FireJump is installed?

(In reply to Jorge Villalobos [:jorgev] from comment #64) > Is there an example on crash-stats where FireJump is installed? Here are some: bp-949b0228-d9d1-498c-a4e3-b28a82130317, bp-c78ae112-c887-4033-95b5-ada572130317, bp-86073712-6b6f-4a10-ba5b-6ec562130317, bp-92c1d103-a1b8-4765-a162-dc5e02130317, bp-95b758a0-06d8-46c8-8ba1-0a3732130317, bp-49ffc639-4ec9-4967-9e5a-a26912130317, bp-fbc40951-2f8c-4f60-97de-8dfd22130317

Looking at the comments for this crash, it looks like most of them are in German. Is it possible that this is correlated to the German version of Firefox or some German software other than FireJump?

(In reply to Jorge Villalobos [:jorgev] from comment #66) > Is it possible that this is correlated to the German version of Firefox or some German > software other than FireJump? It might be indeed a geographical correlation. It's also correlated to Avira Antivirus and BrowserProtect (known as adware) with similar proportions: 22% (1073/4944) vs. 3% (4865/170379) avsda.dll 5% (249/4944) vs. 1% (1469/170379) 12.3.2.15 16% (794/4944) vs. 2% (3117/170379) 13.4.2.360 0% (1/4944) vs. 0% (8/170379) 8.0.0.5 0% (1/4944) vs. 0% (10/170379) 9.0.9.0 21% (1019/4944) vs. 7% (11388/170379) BrowserProtect.dll 0% (12/4944) vs. 0% (506/170379) 2.5.1005.80 0% (5/4944) vs. 0% (219/170379) 2.5.986.67 0% (4/4944) vs. 0% (522/170379) 2.6.1040.25 0% (11/4944) vs. 0% (474/170379) 2.6.1070.41 19% (922/4944) vs. 5% (8980/170379) 2.6.1095.52 1% (58/4944) vs. 0% (579/170379) 2.6.1125.80 0% (7/4944) vs. 0% (107/170379) 2.6.1184.107 Note that we don't have correlations per LSP.

We continue to see a high inflow of people on #firefox.de on moznet that are affected by this. There is a bug about blocklisting the extension, BUT the problem seems to be caused by: C:\Windows\System32\dhRichClient3.dll C:\Windows\System32\sqlite36_engine.dll which are tied additionally into windows through hundreds of registry entries. So blocklisting would likely NOT help.

[@ StrChrIA] seems to be spiking a bit recently on Release: https://crash-analysis.mozilla.com/rkaiser/2013-11-12/2013-11-12.firefox.25.explosiveness.html Currently at #9 in 7-day and 3-day Release top-crash reports.

StrChrIA is currently the #11 crash on v26 release, and it also has some lesser-hitting friends, StrCmpNIA and StrStrIA. StrStrIA calls both of the other two functions. In Windows 8.1, the calls got inlined, so we only see StrStrIA on the stack. In older Windows we see one of the other callees, with StrStrIA below it. v26 has user-agent locale logging, which confirms the previous observations that almost all affected users are running in German. All the crashes are running off the end of a memory page, for example trying to do a 2-byte read at address 12345FFF, where 12345FFF is valid memory but the next byte 12346000 is decommitted. Unwinding stacks by hand, I see frames like this: StrStrIA XXXX1A50 XXXX91C1 _PR_MD_RECV ...where those XXXX are two frames of dynamically-allocated code (allocation flags PAGE_EXECUTE_READWRITE, current flags PAGE_EXECUTE_READ, size 0xA000) always with the same two offsets. The return address at _PR_MD_RECV indicates that it was trying to call the |recv| API. Here's the difficulty: Other than the German language, I don't see any major correlations among recent crash reports. Yes, some users have avsda, but not enough to explain all the crashes. I don't see any suspicious extensions, modules, or LSPs across sufficiently many reports. These dynamic code pages must be coming from some external software that the minidumps don't show.

Combining the three known signatures, this is the #1 crash for users running with locale=de, with 9.5% of their crashes. https://crash-stats.mozilla.com/search/?useragent_locale=de&_facets=signature

dmajor, if we had a user who was experiencing this could we figure out the cause by logging stacks for the VirtualAlloc of the executable code pages? If we have people coming to #firefox.de with this, there are probably some technical users among them.

Recent report of this crash in the forums: https://support.mozilla.org/en-US/questions/982811

(In reply to Benjamin Smedberg [:bsmedberg] from comment #72) > dmajor, if we had a user who was experiencing this could we figure out the > cause by logging stacks for the VirtualAlloc of the executable code pages? Yes, that could likely point to the source of the allocation. I've put instructions here: https://wiki.mozilla.org/Tracing_VirtualAlloc_With_Xperf

Henrik, any chance you can investigate this?

Sorry, but I really don't have the time for anything outside of automation at the moment. There should also be other people who have a better knowledge of debugging on Windows.

Updating the rank info since it's been quite some time. @StrChrIA is still around in significant volume: * Firefox 29: #36 @ 0.33% (+0.05%) w/37 crashes * Firefox 28: #23 @ 0.56% (-0.06%) w/59 crashes * Firefox 27: #34 @ 0.29% (+0.06%) w/667 crashes * Firefox 26: #11 @ 1.08% (+0.07%) w/6632 crashes @StrCmpNIA appears to have disappeared. @StrStrIA appears to have all but disappeared with 11 crashes on Firefox 28 and none on any other version.

We contacted several affected users and the commonality between them is a program called Gutscheinfilter or G-Filter, from gutscheinfilter.de. The G-Filter installer writes two exe files in System32 and registers them as Windows services. The first is usually called gfiltersvc.exe though we have also seen gfiltersvc0.exe. The second file has a random name, chosen by taking the name of a nearby dll in System32 and mutating one character. The random file can be identified by its timestamp which is very close to that of gfiltersvc.exe. Also, the two services can be identified by their blank description in the Windows services manager. The crashes happened while the software was using a network API hook to overwrite some security-related HTTP headers. Given that behavior and the shifting filenames, this software does not seem trustworthy. Some versions of G-Filter offer an uninstaller. On others, stopping the two services and deleting their executables appears to be sufficient to remove the software. A few weeks ago this was easily the top crash in German-speaking locales, but it has now completely fallen off the charts. Perhaps antiviruses have started blocking this software, or perhaps there is a new version that doesn't crash.

This reminds me, when we find a piece of software that is obviously malware, we have in the past submitted it to anti-virus companies so they add it to their detection. In such cases, please get a hold of the offending DLL/EXE and contact me personally.

> A few weeks ago this was easily the top crash in German-speaking locales, but it has now completely > fallen off the charts. Perhaps antiviruses have started blocking this software, or perhaps there is a > new version that doesn't crash. I noted >1000 Win7 >1000 Fx35.0.1 crashes and #59 top crash I believe. Trying to assist user in Sumo thread https://support.mozilla.org/en-US/questions/1046938 Would be grateful for any assistance either in the thread or comments here. bp-84f387ee-c576-449a-b00c-d0ccc2150215 Crashing Thread 0 shlwapi.dll StrChrIA 1 shlwapi.dll StrStrIA 2 ws2_32.dll WSARecv 3 wsock32.dll recv 4 nss3.dll _PR_MD_RECV nsprpub/pr/src/md/windows/w95sock.c 5 nss3.dll SocketRead nsprpub/pr/src/io/prsocket.c 6 xul.dll nsSocketInputStream::Read(char*, unsigned int, unsigned int*) netwerk/base/src/nsSocketTransport2.cpp 7 xul.dll mozilla::net::nsHttpConnection::OnWriteSegment(char*, unsigned int, unsigned int*) netwerk/protocol/http/nsHttpConnection.cpp 8 xul.dll mozilla::net::nsHttpTransaction::WritePipeSegment(nsIOutputStream*, void*, char*, unsigned int, unsigned int, unsigned int*) netwerk/protocol/http/nsHttpTransaction.cpp 9 xul.dll nsPipeOutputStream::WriteSegments(tag_nsresult (*)(nsIOutputStream*, void*, char*, unsigned int, unsigned int, unsigned int*), void*, unsigned int, unsigned int*) xpcom/io/nsPipe3.cpp 10 xul.dll mozilla::net::nsHttpTransaction::WriteSegments(mozilla::net::nsAHttpSegmentWriter*, unsigned int, unsigned int*) netwerk/protocol/http/nsHttpTransaction.cpp 11 xul.dll mozilla::net::nsHttpConnection::OnSocketReadable() netwerk/protocol/http/nsHttpConnection.cpp 12 xul.dll mozilla::net::nsHttpConnection::OnInputStreamReady(nsIAsyncInputStream*) netwerk/protocol/http/nsHttpConnection.cpp 13 xul.dll nsSocketInputStream::OnSocketReady(tag_nsresult) netwerk/base/src/nsSocketTransport2.cpp 14 xul.dll nsSocketTransportService::DoPollIteration(bool) netwerk/base/src/nsSocketTransportService2.cpp 15 xul.dll nsSocketTransportService::Run() netwerk/base/src/nsSocketTransportService2.cpp 16 xul.dll nsThread::ProcessNextEvent(bool, bool*) xpcom/threads/nsThread.cpp 17 xul.dll nsComponentManagerImpl::CreateInstance(nsID const&, nsISupports*, nsID const&, void**) xpcom/components/nsComponentManager.cpp 18 xul.dll MessageLoop::DoWork() ipc/chromium/src/base/message_loop.cc 19 xul.dll MessageLoop::RunHandler() ipc/chromium/src/base/message_loop.cc 20 xul.dll nsThread::ThreadFunc(void*) xpcom/threads/nsThread.cpp 21 msvcr100.dll msvcr100.dll@0x8b581 22 ntdll.dll LdrpGetShimEngineInterface 23 ntdll.dll _RtlUserThreadStart

(In reply to John Hesling [:John99] from comment #80) > I noted >1000 Win7 >1000 Fx35.0.1 crashes and #59 top crash I believe. > > Trying to assist user in Sumo thread > https://support.mozilla.org/en-US/questions/1046938 Would be grateful for > any assistance either in the thread or comments here. I don't think anyone has made any progress in resolving this issue, since this bug report hasn't really been updated in almost a year. If the information in comment 78 does not help then I'm not sure what else to suggest.

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258