It is a new crash signature. Crashes first appeared in 4.0b8pre/20101111 build. It is #165 top crasher in 4.0b8pre for the last 3 days. Signature nvwgf2um.dll@0xe1edf UUID 8c1ca0e0-31b8-416b-9534-4210f2101113 Time 2010-11-13 15:40:57.168406 Uptime 1359 Last Crash 112971 seconds (1.3 days) before submission Install Age 8592 seconds (2.4 hours) since version was first installed. Product Firefox Version 4.0b8pre Build ID 20101113042433 Branch 2.0 OS Windows NT OS Version 6.1.7600 CPU x86 CPU Info GenuineIntel family 6 model 30 stepping 5 Crash Reason EXCEPTION_ACCESS_VIOLATION_READ Crash Address 0x8 App Notes AdapterVendorID: 10de, AdapterDeviceID: 0a3c Frame Module Signature [Expand] Source 0 nvwgf2um.dll nvwgf2um.dll@0xe1edf 1 nvwgf2um.dll nvwgf2um.dll@0xb0f5d 2 nvwgf2um.dll nvwgf2um.dll@0x16c5dd 3 nvwgf2um.dll nvwgf2um.dll@0x3b716 4 d3d10_1core.dll d3d10_1core.dll@0x2a280 5 d3d10_1core.dll d3d10_1core.dll@0x30ca5 6 d3d10_1.dll d3d10_1.dll@0x1b927 7 d3d10_1.dll d3d10_1.dll@0x228f3 8 d3d10_1.dll d3d10_1.dll@0x1baa0 9 d3d10_1core.dll d3d10_1core.dll@0x1cd99 10 d3d10_1.dll d3d10_1.dll@0x19987 11 xul.dll mozilla::layers::ImageLayerD3D10::RenderLayer gfx/layers/d3d10/ImageLayerD3D10.cpp:209 12 xul.dll mozilla::layers::ContainerLayerD3D10::RenderLayer gfx/layers/d3d10/ContainerLayerD3D10.cpp:242 The regression range is: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=df1d1ff6b489&tochange=0f17e5f1eb01 More reports at: http://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=exact&query=&range_value=4&range_unit=weeks&hang_type=any&process_type=any&plugin_field=&plugin_query_type=&plugin_query=&do_query=1&admin=&signature=nvwgf2um.dll%400xe1edf

For signatures that start with nvwgf2um.dll, there are about 100 crashes/week on 4.0b8pre, so it is #19 top crasher. Here are some reports: http://crash-stats.mozilla.com/query/query?product=Firefox&version=Firefox%3A4.0b8pre&platform=windows&branch=2.0&range_value=1&range_unit=weeks&date=12%2F13%2F2010+07%3A53%3A19&query_search=signature&query_type=startswith&query=nvwgf2um.dll&build_id=&process_type=any&hang_type=any&do_query=1

This might be a regression from asynchronous plugin drawing, which appears in the regression window. Does anyone have a way to reproduce this?

Hrm, this looks like the technique broke somehow, I have no quick idea on what might cause this.

Here are a few of the comments in the reports: Crashes lots even though I got latest NVidia driver installed. It crashed while I was watching a WebM/HTML5 video on YouTube. I have no idea how/what happened (sg). Minefield restarted OK.

If we could get the d3d10_1core.dll symbols this would most likely be -much- easier to diagnose. Since we don't have a reproducable scenario. bjacob: Any chance you could work your aggregation magic and figure out if this is limited to some device/driver?

bas: have you tried grabbing a .dmp and seeing if the ms symbol server has d3d10_1core? it probably does...

Joe: Can you arrange that? I do not have the required permissions.

It's not limited to some particular device/driver: The top signature nvwgf2um.dll@0x131def, which is characteristic of driver 260.99, already has lots of different devices: the first 5 are: 0427 G86 [GeForce 8400M GS] 05e6 GT200b [GeForce GTX 275] 05e3 GT200b [GeForce GTX 285] 0a34 GT216 [GeForce GT 240M] 0622 G94 [GeForce 9600 GT] That's already 3 generations of NVIDIA cards. Then other signatures correspond to other driver versions: nvwgf2um.dll@0xe1edf is driver version 258.96 nvwgf2um.dll@0x9de82 is driver version 188.86, which is old...

Stack of https://crash-stats.mozilla.com/report/index/22b8e07d-c724-4a17-8573-6f1242101213 > xul.dll!nsDisplayWrapper::WrapListsInPlace(aBuilder=0x008ba5d8, aFrame=0x00000000, aLists={...}) Line 1411 C++ d3d10_1core.dll!CDevice::ID3D10Device1_SetShaderResources_<3>() d3d10_1core.dll!NMultithread::CDevice::PSSetShaderResources() d3d10_1.dll!D3D10Effects::CEffect::ApplyShaderBlock() d3d10_1.dll!D3D10Effects::CEffect::ApplyPassBlock() d3d10_1.dll!D3D10Effects::SRuntimePassBlock::Apply() xul.dll!mozilla::layers::ImageLayerD3D10::RenderLayer() Line 210 C++ xul.dll!mozilla::layers::ContainerLayerD3D10::RenderLayer() Line 265 C++ xul.dll!mozilla::layers::ContainerLayerD3D10::RenderLayer() Line 265 C++ xul.dll!mozilla::layers::LayerManagerD3D10::Render() Line 473 C++ xul.dll!mozilla::layers::LayerManagerD3D10::EndTransaction(aCallback=0x62ed6cd0, aCallbackData=0x001eca40) Line 245 C++

Just got this crash for the first time, crashed going back in a tab, was on pages I usually am on and my normal browsing behavior, the only thing different was I had a tab that was sitting on imdb.com homepage for a long time. http://crash-stats.mozilla.com/report/index/46e1aeef-04a3-49c6-9179-fa9752101217

I have this crash since yesterday's nightly. 2010-12-17 My laptop is Optimus enabled (GT420M) but I don't think that's the problem. I've seen it occur when trying to view YT vids or video streams. http://crash-stats.mozilla.com/report/index/bp-5c032834-cee7-4790-845d-2bf8f2101218

Aashish, Don't add a crash id that is specific to 64-bit build. It is not related.

It looks like the same crash signature. Why would it make a difference if it's a 64Bit or 32Bit build considering the circumstances?

> It looks like the same crash signature. Why would it make a difference if it's > a 64Bit or 32Bit build considering the circumstances? First your crash signature is not in the bug title even if it is a NVIDIA driver crash, then the stack traces are not similar (mozilla::layers::ImageLayerD3D10::RenderLayer is not inside the stack traces). You can file a new bug, but read bug 609252 comment 13 first.

Ok thanks will file a new bug.

I was able to crash several times in this stack on Windows 7 using this URL: http://www.winamp.com/?src=toolbar http://crash-stats.mozilla.com/report/index/b978ff01-75b9-41ee-9954-bb98d2101227 http://crash-stats.mozilla.com/report/index/bp-6a59ddf7-ae63-4061-b3bd-61c6b2101227

Hrm, that URL doesn't seem to cause any trouble for me (NVidia GT230M) were you doing anything specific?

My recollection is that crash happened when I loaded the site. I just crashed on my Win 7 laptop in the same stack. I had 3 tabs open and switched back to the first tab, which had NY Times loaded in it. http://crash-stats.mozilla.com/report/index/bp-9a2bcd7f-43f6-4547-bf2d-9ec372101229 is that report (In reply to comment #17) > Hrm, that URL doesn't seem to cause any trouble for me (NVidia GT230M) were you > doing anything specific?

Try installing KB2028560 update, it does fix some bugs in Direct2D and DirectWrite. From the KB article, D2d1.dll, Dwrite.dll and D3d10_1core.dll were all updated to 6.1.7600.20781. http://support.microsoft.com/kb/2028560 32bit http://download.microsoft.com/download/9/7/6/976634BE-DC4D-4B7B-9567-209811D7AF14/Windows6.1-KB2028560-v2-x86.msu 64bit http://download.microsoft.com/download/F/1/A/F1A02F0C-0F64-4E87-9BC1-EFA8CDE464FE/Windows6.1-KB2028560-v2-x64.msu

I've managed to get this crash, sadly in a release nightly without a debugger attached. I've got that update though, I'm trying to reproduce it in a debug build now, but it seems to be tricky.

Nvidia has released new GeForce 266.35 beta drivers, might want to test them and see if they fix this. Desktop drivers http://www.nvidia.com/object/win7-winvista-32bit-266.35-beta-driver.html http://www.nvidia.com/object/win7-winvista-64bit-266.35-beta-driver.html Notebook drivers http://www.nvidia.com/object/notebook-win7-winvista-266.35-beta-driver.html http://www.nvidia.com/object/notebook-win7-winvista-64bit-266.35-beta-driver.html

Here are crash reports for 4.0b8: http://crash-stats.mozilla.com/query/query?product=Firefox&version=Firefox%3A4.0b8&platform=windows&branch=2.0&range_value=1&range_unit=weeks&query_search=signature&query_type=startswith&query=nvwgf2um.dll&build_id=&process_type=any&hang_type=any&do_query=1 There are 7150 crashes/week, so it is #1 top crasher in 4.0b8 for the last week. It must be a beta9 blocker.

It is responsible of 3.6% of every 4.0b8 crashes. Assuming each crash signature is caused by a single driver version, I sorted them out by top crasher: nvwgf2um.dll@0xe1edf 8.17.12.5896 nvwgf2um.dll@0x131def 8.17.12.6099 nvwgf2um.dll@0x9de32 8.16.11.8829 nvwgf2um.dll@0x9d9b2 8.16.11.8766 nvwgf2um.dll@0xe212b 8.17.12.5896 nvwgf2um.dll@0x9de92 8.16.11.8914 nvwgf2um.dll@0x131fe9 8.17.12.6099 nvwgf2um.dll@0x9de82 8.16.11.8958 nvwgf2um.dll@0x9bec2 8.15.11.8644 nvwgf2um.dll@0x9bf22 8.15.11.8642 nvwgf2um.dll@0x9de22 8.16.11.8911 nvwgf2um.dll@0x9df92 8.16.11.8864 nvwgf2um.dll@0x96d52 8.15.11.8593 nvwgf2um.dll@0x15a9c3 8.17.12.6099 nvwgf2um.dll@0x9df02 8.16.11.8992 nvwgf2um.dll@0xc600f 8.17.11.9621 nvwgf2um.dll@0x9dd82 8.16.11.8782 nvwgf2um.dll@0x9def2 8.16.11.8988 ... It is well distributed over versions. Even recent versions (release 260) are impacted.

Bas, can you ask NVidia about this?

(In reply to comment #18) > My recollection is that crash happened when I loaded the site. > > I just crashed on my Win 7 laptop in the same stack. I had 3 tabs open and > switched back to the first tab, which had NY Times loaded in it. > > http://crash-stats.mozilla.com/report/index/bp-9a2bcd7f-43f6-4547-bf2d-9ec372101229 > is that report > > (In reply to comment #17) > > Hrm, that URL doesn't seem to cause any trouble for me (NVidia GT230M) were you > > doing anything specific? Considering you can reproduce this, is there any chance you could send me a dump with heap included for this crash?

It seems like the function we're calling in the drivers is probably PsSetShaderResources: http://msdn.microsoft.com/en-us/library/ff569210%28VS.85%29.aspx I wonder if we can call PSGetShaderResources and perhaps ID3D10ShaderResourceView::GetDesc ourselves to perhaps trigger the crash earlier.

(In reply to comment #27) > It seems like the function we're calling in the drivers is probably > PsSetShaderResources: > http://msdn.microsoft.com/en-us/library/ff569210%28VS.85%29.aspx > > I wonder if we can call PSGetShaderResources and perhaps > ID3D10ShaderResourceView::GetDesc ourselves to perhaps trigger the crash > earlier. If I could reproduce that, it's definitely the test I'd run. But we'd still need to catch that crash in a debugger to be able to find out what caused it.

This will cause us not to draw, or bind NULL as the shader resource (also drawing nothing) when texture creation fails. I don't think this is the cause since ShaderResource creation should return NULL if the texture is NULL, but some additional safety (and a debug warning) can't hurt.

I suspect this is the real problem here. When our image belongs to an old device that crashed, and we created a new one, we'll still attempt to use that old device's object with our new device, this is bad and we shouldn't do this.

Comment on attachment 502639 [details] [diff] [review] Part 1: Do not draw when texture creation fails. Have you tested this? (By simulating the failure perhaps?) Patches like this always scare me a bit because they can create more state to worry about and more paths through code. This can end up moving failures to places that are harder to diagnose. Perhaps this situation can also be logged with ReportFailure().

(In reply to comment #31) > Comment on attachment 502639 [details] [diff] [review] > Part 1: Do not draw when texture creation fails. > > Have you tested this? (By simulating the failure perhaps?) > > Patches like this always scare me a bit because they can create more state to > worry about and more paths through code. This can end up moving failures to > places that are harder to diagnose. Perhaps this situation can also be logged > with ReportFailure(). Yes, I've tested this. It makes us draw something bogus in some cases in the plugin area but it doesn't crash. I could also use ReportFailure for this.

Comment on attachment 502639 [details] [diff] [review] Part 1: Do not draw when texture creation fails. Looks fine. Perhaps add the ReportFailure at the call to CreateTexture.

Comment on attachment 502640 [details] [diff] [review] Part 2: Don't draw when ShaderResource belongs to the wrong device. This looks fine. Is there anything we could have done/can still do to prevent this mistake?

This adds a similar check for yuvImage.

http://hg.mozilla.org/mozilla-central/rev/5a1e9d51d095 http://hg.mozilla.org/mozilla-central/rev/04ae4e309307 http://hg.mozilla.org/mozilla-central/rev/c8e72631d44b Leaving open until crash-stats confirm tomorrow if this has been fixed.

Comment on attachment 502973 [details] [diff] [review] >+void >+LayerManagerD3D10::ReportFailure(const nsACString &aMsg, HRESULT aCode) >+{ >+ // We could choose to abort here when hr == E_OUTOFMEMORY. >+ nsCString msg; >+ msg.Append(aMsg); >+ msg.AppendLiteral(" Error code: "); >+ msg.AppendInt(PRUint32(aCode)); >+ NS_WARNING(msg.BeginReading()); >+} Should the whole implementation here be ifdef DEBUG? Comment on attachment 502975 [details] [diff] [review] >+ if (yuvImage->mDevice != device()) { >+ // These shader resources were created for an old device! Can't draw >+ // that here. >+ return; >+ } >+ Tabs!

Microsoft has released KB2454826 update, it replaces the older KB2028560 update and installs newer Direct2D and DirectWrite dlls which fixes more bugs. http://support.microsoft.com/kb/2454826 32bit http://download.microsoft.com/download/5/E/4/5E404378-9A5D-41AB-AFBA-1AC04F3B2A13/Windows6.1-KB2454826-x86.msu 64bit http://download.microsoft.com/download/D/B/D/DBD62263-2627-49CB-B675-AA1601EBE0BD/Windows6.1-KB2454826-x64.msu

(In reply to comment #39) > Comment on attachment 502973 [details] [diff] [review] > >+void > >+LayerManagerD3D10::ReportFailure(const nsACString &aMsg, HRESULT aCode) > >+{ > >+ // We could choose to abort here when hr == E_OUTOFMEMORY. > >+ nsCString msg; > >+ msg.Append(aMsg); > >+ msg.AppendLiteral(" Error code: "); > >+ msg.AppendInt(PRUint32(aCode)); > >+ NS_WARNING(msg.BeginReading()); > >+} > > Should the whole implementation here be ifdef DEBUG? I don't think so, longer run the intention is to connect this to some kind of runtime recording/logging system. In any case the optimizer should figure this out. > > Comment on attachment 502975 [details] [diff] [review] > >+ if (yuvImage->mDevice != device()) { > >+ // These shader resources were created for an old device! Can't draw > >+ // that here. > >+ return; > >+ } > >+ > > Tabs! Argh, that's what I get for working in cairo code. will fix.

After fixing the tabs and examining crash data, I'm going to optimistically mark this fixed!

A little oops here introduced excessive NS_WARNINGs for no reason. We should fix that.

Pushed followup for spurious warnings http://hg.mozilla.org/mozilla-central/rev/fa1a4b6abff0