This bug was filed from the Socorro interface and is report bp-5c60f961-b644-4ad2-a32c-9810d2150331. ============================================================= this seems to be a new startup crash and a regression in firefox 37 - numerous comments mention that it happened immediately after updating. as per feedback of this sumo user the crash happens when you attempt to run firefox in safe mode as well: https://support.mozilla.org/en-US/questions/1054989

[Tracking Requested - why for this release]: I'm nominating this just so it shows up on the radar of release management, in case this ends up being a larger issue.

a helpful user has narrowed it down to this regression range: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=8f6965ca&tochange=a4b4cd74

Lawrence, we may want to throttle while we're figuring this one out?

Bas, this is "we failed to initialize D3D11 WARP device" where we call MOZ_CRASH.

What's the "rank" of this crash? How common is it?

sorry about clearing the need info flags before (this may have been a bug in the BMO experimental modular interface). re-adding...

This is almost certainly caused by the stardock malware intercepting Windows DLLs and messing with the calls. I wonder if this is fixed by blacklisting the wblind and wbload DLLs. But for now we should take Jeff's patch.

(In reply to Milan Sreckovic [:milan] from comment #7) > What's the "rank" of this crash? How common is it? Winthin the firs 24-48 hours, we do not have much in terms of ranks or anything, but early 37.0 release data suggests this is the #3 crash right now, with #1 being another gfx startup issue in bug 1137716, which actually was supposed to be fixed in 37 but still appears at the top with 12% of all crashes.

See https://crash-stats.mozilla.com/search/?product=Firefox&version=37.0&release_channel=release&process_type=browser&process_type=content&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature for a current search on 37.0 release crashes.

I installed Stardock window blinds on my machine and 37 causes the whole machine to hang and require a reboot. It would be interesting to see what happens for other people when window blinds is installed.

Comment on attachment 8586405 [details] [diff] [review] An emergency solution to avoid crashing Approval Request Comment [Feature/regressing bug #]: WARP (37) [User impact if declined]: Startup crashes [Risks and why]: We should not MOZ_CRASH on startup, cause that's bad.

I tried a build that should've included this fix on the machine that can reproduce it and it didn't seem to help. I wasn't able to get crash symbols so I'm not sure where the crash moved to.

The "infected" machine runs 36 fine?

(In reply to Milan Sreckovic [:milan] from comment #16) > The "infected" machine runs 36 fine? I believe so yes. My desktop (the machine that hangs) certainly ran 36 fine, I'm not as sure about that laptop that reproduces the issue.

What about nightly? I installed this stuff on my desktop, nightly seems to start fine.

I take that last comment back; managed to get it to crash.

Possible workaround for the people that are stuck with this - perhaps post in sumo - if they do have windowblinds, and they switch back to Windows Aero theme, the crash should go away. Well, at least it does for me. Fails with others, but not with Aero.

Comment on attachment 8586405 [details] [diff] [review] An emergency solution to avoid crashing Review of attachment 8586405 [details] [diff] [review]: ----------------------------------------------------------------- With this change, I don't crash on startup. ::: gfx/thebes/gfxWindowsPlatform.cpp @@ +1972,5 @@ > > if (FAILED(hr)) { > // This should always succeed... in theory. > gfxCriticalError() << "Failed to initialize WARP D3D11 device!" << hr; > + } else { I would just replace MOZ_CRASH with return instead (in the original code.) I'm not sure it's safe to continue.

(In reply to Milan Sreckovic [:milan] from comment #21) > > With this change, I don't crash on startup. I'm not sure I like where I end up, there are some artifacts (close, minimize, maximize buttons don't show up), but it doesn't crash. I don't know how much of the problem is due to windowblinds changing the "substyle".

Since the regression is about a Windows specific bug, why the automatic update for Firefox/Thunderbird is also disabled for Linux & Mac?

(In reply to Jeff Muizelaar [:jrmuizel] from comment #12) > I installed Stardock window blinds on my machine and 37 causes the whole > machine to hang and require a reboot. It would be interesting to see what > happens for other people when window blinds is installed. I was able to easily reproduce the startup crash with Firefox 37.0 and Firefox 40 Nightly, on Windows 7 x64, Windows 8 x64 and Windows 8.1 x64, using the following scenario: 1. Download and install Windowblinds (http://www.stardock.com/products/windowblinds/). 2. Start Window Blinds and apply a custom style (e.g. Lantana). 3. Attempt to start Firefox. Results: Firefox crashes before even showing the Profile Manager window (if set to show it). I tried this on two separate machines, and never did Windows freeze... I just got the crash. (In reply to Milan Sreckovic [:milan] from comment #20) > Possible workaround for the people that are stuck with this - perhaps post > in sumo - if they do have windowblinds, and they switch back to Windows Aero > theme, the crash should go away. Well, at least it does for me. Fails with > others, but not with Aero. Switching to Aero seems to indeed work around this issue on Windows 7, but doesn't seem to work on Windows 8 or 8.1 (I had to uninstall WindowBlinds here to get Firefox to start again).

Here's what's going wrong in the Window Blinds case. Window Blinds hooks D3D11CreateDevice, the hook (in wbload.dll) then calls GetModuleHandle("FIREFOX.EXE") and if this returns a valid handle, Window Blinds immediately returns failure from D3D11CreateDevice and we proceed to MOZ_CRASH.

Florin, if you rename firefox.exe to something else, does the crash go away?

(In reply to Milan Sreckovic [:milan] from comment #26) > Florin, if you rename firefox.exe to something else, does the crash go away? Yes, renaming to something else like test.exe will start Firefox without crashing.

(In reply to Florin Mezei, QA (:FlorinMezei) from comment #27) > (In reply to Milan Sreckovic [:milan] from comment #26) > > Florin, if you rename firefox.exe to something else, does the crash go away? > > Yes, renaming to something else like test.exe will start Firefox without > crashing. Thanks. Not a good workaround in general, there is other code (not written by us, things like driver workarounds and such) that depend on this name, but good to know. :lmandel is reaching out to Stardock.

Note that while Firefox starts without crashing, it does have some visual issues with different theme applied (missing titlebar) - see https://bugzilla.mozilla.org/attachment.cgi?id=8587404. Google Chrome has the same visual issues if that matters in any way.

:kairo, can you check if we have a 100% correlation between user having WindowBlinds (wbload.dll exists?) and crashes? Or is there a better way to test if WindowBlinds is installed?

Bas will land the modified patch (comment 21), with implicit r+ from Jeff (over the phone) as inbound opens.

Approval Request Comment [Feature/regressing bug #]: WARP (36) [User impact if declined]: Many crashes [Describe test coverage new/current, TreeHerder]: None yet [Risks and why]: should just fall back to the path we use on Vista/XP

Well, people at Mozilla is very busy fixing this problem, so for anyone finding an answer here, https://support.mozilla.org/en-US/questions/1055150 says OK to perform a manual update under Mac. I see machines using Ubuntu updating Firefox without problems to the latest version, so I will assume that the same is true for other Linux variants using the tarball. Many thanks to all the Mozillians for their great work.

If the patch that's asking for the approval doesn't apply because of the previous one, here's a rollup.

Comment on attachment 8587558 [details] [diff] [review] Don't MOZ_CRASH if WARP fails I reviewed this fix with Milan and Jeff. This should solve all known cases of the crash even though not all cases are definitely known. There is some risk that the fix is not complete but without any additional information our best course of action is pushing an update with this fix. We'll take this in 37.0.1 Release+ Beta+ Aurora+

(In reply to Milan Sreckovic [:milan] from comment #30) > :kairo, can you check if we have a 100% correlation between user having > WindowBlinds (wbload.dll exists?) and crashes? Or is there a better way to > test if WindowBlinds is installed? https://crash-stats.mozilla.com/report/list?product=Firefox&range_value=7&range_unit=days&date=2015-04-02&signature=gfxWindowsPlatform%3A%3AInitD3D11Devices%28%29&version=Firefox%3A37.0#tab-correlations says there there is not a 100% correlation, it's merely a 45% one - but 45% vs. 0% is still very strong.

2 questions. 1 - there i snothing in the ff37 changes about changes to hardware acceleration so what was ommited from the changes page that is causing this issue? as clearly you guys changed something sinc eff36. 2 - why havent you pulled the ff37 security page? you have disclosed security bugs which end users cannot patch now due to the update been pulled.

I tried to encompass the 3 start-up crash bugs that we fixed in 37.0.1 in a single release note: "Fixed: Start-up crash due to graphics hardware and third party software"

Verified today that Firefox 37.0.1 (BuildID=20150402191859) starts without crashing, both for a normal start and in safe mode (bug 1149864), on the same Windows 7 x64 and Windows 8 x64 machines where I reproduced the original issue with Firefox 37.0. The visual issue mentioned in comment 29 (missing titlebar) is already tracked separately in bug 1077245. The fix doesn't seem to have made the latest Nightly or Aurora yet.

I received a response from Stardock that indicated that there was an issue with WindowBlinds and the DirectX11 path in Firefox 37. They have fixed the issue in WindowBlinds 8.11 beta.

Also verified as fixed on Firefox 38 Beta 5 on Windows 7 x64 and Windows 8.1 x64.