This bug was filed from the Socorro interface and is report bp-77264948-764b-44dd-9363-573bf2150325 . - Ran into this on Windows 8.1 32-bit, using Firefox 37 (build ID: 20150324194038). - I don't have proper STR, but it happened during WebGl games testing (e.g. http://oos.moxiecode.com/js_webgl/monster_truck/ , http://racer.nomo.hu/ ) - Tried several other times to reproduce it, without any luck. - The crash rates increased in Firefox 37 Beta 6 and doubled in Firefox 37 Beta 7 and are affected only 32bit versions of Windows

[Tracking Requested - why for this release]: A possible WebGL regression. The severity of this depends on widespread the crash is. It might be hidden among multiple signature because it shows up as 0 kernelbase.dll RaiseException 1 msvcr120.dll _CxxThrowException f:\dd\vctools\crt\crtw32\eh\throw.cpp:152 2 msvcr120.dll msvcr120.dll@0x5da69 3 libglesv2.dll std::allocator<char>::allocate(unsigned int) c:/tools/vs2013/vc/include/xmemory0:578 4 libglesv2.dll std::vector<unsigned char, std::allocator<unsigned char> >::_Reserve(unsigned int) c:/tools/vs2013/vc/include/vector:1617 5 libglesv2.dll rx::d3d11::GenerateInitialTextureData(int, unsigned int, unsigned int, unsigned int, unsigned int, std::vector<D3D11_SUBRESOURCE_DATA, std::allocator<D3D11_SUBRESOURCE_DATA> >*, std::vector<std::vector<unsigned char, std::allocator<unsigned char> >, std::allocator<std::vector<unsigned char, std::allocator<unsigned char> > > >*) We really want to be looking to see how much this is happening from GenerateInitialTextureData.

Tracking to follow up on severity of the crash.

The stack from the crash report in comment 0 looks like this in a debugger: 0099ba20 6c379339 KERNELBASE!RaiseException+0x62 0099ba60 6c3bda6a msvcr120!_CxxThrowException+0x5b 0099ba80 6bd44388 msvcr120!operator new+0x50 0099ba88 6bdc3fb0 libGLESv2!std::allocator<char>::allocate(unsigned int _Count = 0x369f10)+0x16 [c:\tools\vs2013\vc\include\xmemory0 @ 578] 0099baa0 6bdc229d libGLESv2!std::vector<unsigned char,std::allocator<unsigned char> >::_Reserve(unsigned int _Count = 0x369f10)+0x53 [c:\tools\vs2013\vc\include\vector @ 1617] 0099bae0 6bdb2e86 libGLESv2!rx::d3d11::GenerateInitialTextureData(int internalFormat = 0n3, unsigned int width = 0, unsigned int height = 0x3b2, unsigned int depth = 1, unsigned int mipLevels = 1, class std::vector<D3D11_SUBRESOURCE_DATA,std::allocator<D3D11_SUBRESOURCE_DATA> > * outSubresourceData = 0x779547d0, class std::vector<std::vector<unsigned char,std::allocator<unsigned char> >,std::allocator<std::vector<unsigned char,std::allocator<unsigned char> > > > * outData = 0x0099bb2c)+0x153 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\d3d11\renderer11_utils.cpp @ 1032] 0099bb64 6bdb25db libGLESv2!rx::Image11::createStagingTexture(void)+0x1ee [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\d3d11\image11.cpp @ 548] 0099bba4 6bdb257e libGLESv2!rx::Image11::copyToStorageImpl(class rx::TextureStorage11 * storage11 = 0x3ce97670, struct gl::ImageIndex * index = 0x0099bc3c, struct gl::Box * region = 0x0099bc24)+0x56 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\d3d11\image11.cpp @ 143] 0099bbbc 6bda8062 libGLESv2!rx::Image11::copyToStorage2DArray(class rx::TextureStorage * storage = 0x3ce97670, struct gl::ImageIndex * index = 0x0099bc3c, struct gl::Box * region = 0x0099bc24)+0x14 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\d3d11\image11.cpp @ 107] 0099bc04 6bda7f27 libGLESv2!rx::TextureD3D_2D::commitRegion(struct gl::ImageIndex * index = 0x0099bc3c, struct gl::Box * region = 0x0099bc24)+0x49 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\textured3d.cpp @ 907] 0099bc6c 6bda7e9a libGLESv2!rx::TextureD3D_2D::updateStorageLevel(int level = 0n0)+0x7d [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\textured3d.cpp @ 865] 0099bc88 6bda7daa libGLESv2!rx::TextureD3D_2D::updateStorage(void)+0x3e [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\textured3d.cpp @ 847] 0099bc98 6bda72c3 libGLESv2!rx::TextureD3D_2D::initializeStorage(bool renderTarget = true)+0x46 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\textured3d.cpp @ 810] 0099bcd0 6bda7c74 libGLESv2!rx::TextureD3D::ensureRenderTarget(void)+0x1d [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\textured3d.cpp @ 385] 0099bcd8 6bd80811 libGLESv2!rx::TextureD3D_2D::getRenderTargetSerial(struct gl::ImageIndex * index = 0x77884684)+0x8 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\textured3d.cpp @ 723] 0099bce8 6bdb73d4 libGLESv2!rx::GetAttachmentSerial(class gl::FramebufferAttachment * attachment = 0x00000003)+0x26 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\framebuffer.cpp @ 55] 0099bd74 6bd7ddb2 libGLESv2!rx::Renderer11::applyRenderTarget(class gl::Framebuffer * framebuffer = 0x3ce7dab8)+0xb6 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\renderer\d3d\d3d11\renderer11.cpp @ 850] 0099bdac 6bd7e8ef libGLESv2!gl::Context::applyRenderTarget(unsigned int drawMode = 4, bool ignoreViewport = true)+0x2c [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\context.cpp @ 1327] 0099be5c 6bd9227a libGLESv2!gl::Context::clear(unsigned int mask = 0x4500)+0x60 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\context.cpp @ 1622] 0099beb0 538a94ef libGLESv2!glClear(unsigned int mask = 0x4500)+0x53 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\gfx\angle\src\libglesv2\libglesv2.cpp @ 652] 0099bf64 538c9439 xul!mozilla::WebGLContext::ForceClearFramebufferWithDefaultValues(unsigned int mask = 0x4500, bool * colorAttachmentsMask = 0x0099bf98)+0x1db [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\dom\canvas\webglcontext.cpp @ 1367] 0099bfa8 538a8be2 xul!mozilla::WebGLFramebuffer::CheckAndInitializeAttachments(void)+0x156 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\dom\canvas\webglframebuffer.cpp @ 891] 0099bfe0 538a8806 xul!mozilla::WebGLContext::DrawElements_check(int count = 0n6, unsigned int type = 0x1403, int64 byteOffset = 0n0, int primcount = 0n1, char * info = 0x54565990 "drawElements", unsigned int * out_upperBound = 0x0099c010)+0x2da [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\dom\canvas\webglcontextdraw.cpp @ 284] 0099c014 53856713 xul!mozilla::WebGLContext::DrawElements(unsigned int mode = 4, int count = 0n6, unsigned int type = 0x1403, int64 byteOffset = 0n0)+0x4d [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\dom\canvas\webglcontextdraw.cpp @ 317] 0099c048 52c29698 xul!mozilla::dom::WebGLRenderingContextBinding::drawElements(struct JSContext * cx = 0x4bded6d0, class JS::Handle<JSObject *> obj = class JS::Handle<JSObject *>, class mozilla::WebGLContext * self = 0x28485400, class JSJitMethodCallArgs * args = 0x00001403)+0xa8 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\obj-firefox\dom\bindings\webglrenderingcontextbinding.cpp @ 10207] 0099c08c 28485400 xul!mozilla::dom::GenericBindingMethod(struct JSContext * cx = 0x4bded6d0, unsigned int argc = 4, class JS::Value * vp = 0x0099c0c0)+0xc8 [c:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\dom\bindings\bindingutils.cpp @ 2485] It's an OOM, but it's coming by way of a thrown C++ exception. I guess since this is in ANGLE we don't get our standard infallible malloc behavior, and it's clearly built with C++ exceptions.

Dan, can you also take a look?

Assigning to Dan. Feel free to reassign if you are not going to work on this.

This is out-of-memory, right? Maybe out-of-address space on 32-bit windows. Oh, I see that Ted wrote the same at the bottom of his long call stack comment. Do we want to catch the exception from ANGLE, or do we want to disable exceptions in ANGLE.

If the allocation size is content-controllable then we ought to be able to handle OOM here. I don't know what ANGLE does in these situations, is it intended to be built with exceptions enabled and catch allocation failure? Does it have any existing policy for OOM handling?

d3d11::GenerateInitialTextureData seems to be called with width 0, height 946. Its caller, Image11::createStagingTexture, asserts that width and height are positive. That at least seems wrong, but I don't know that it's the cause.

On the other hand, this zero gets taken out (and set to 1) before we crash, so it probably isn't the cause.

Dan, any progress on this issue? We just have a few beta for 38 (btw, it is an ESR release). Thanks

I've just got free of other tasks to start looking at this. It's top of my priority list.

Digging into this, I think we are still being bitten by Bug 1134565, or something similar to it. ANGLE is throwing an exception on OOM. The ANGLE coding standard[1], in section "Other C++ Features", says: > {DONT} use C++ exceptions, they are disabled in the builds and not caught. So, here, ANGLE is throwing an exception, which it should not, and it's bubbling through our code and results in a crash. [1] https://code.google.com/p/angleproject/wiki/CodingStandard

I've traced through the code in the MSVC STL implementation and the assertion in the ANGLE coding standard about exceptions and the reality don't match. From my understanding there is no way to disable the exception thrown from std::vector when memory is exhausted because std::vector uses std:allocator, and allocate is spec'd to throw std::bad_alloc. (http://en.cppreference.com/w/cpp/memory/allocator/allocate) Milan, where should we go from here?

I need input from a build peer. Should we been using the Mozilla STL wrapper, or the system one? ANGLE's moz.build files seem keen to disable the STL wrapper.

I've modified the moz.build files for ANGLE and now allocation calls into moz_xmalloc. Simulating OOM condition in moz_xmalloc results in mozalloc_abort and program crash. (Still)

(In reply to Dan Glastonbury :djg :kamidphish from comment #15) > I've modified the moz.build files for ANGLE modified how? (In reply to Dan Glastonbury :djg :kamidphish from comment #14) > I need input from a build peer. Should we been using the Mozilla STL > wrapper, or the system one? ANGLE's moz.build files seem keen to disable the > STL wrapper. After bug 1134565, it shouldn't be. If it is, then either your tree is too old or something regressed it.

(In reply to Mike Hommey [:glandium] from comment #16) > (In reply to Dan Glastonbury :djg :kamidphish from comment #15) > > I've modified the moz.build files for ANGLE > > modified how? Removed DISABLE_STL_WRAPPING = True from the moz.build files in src/libGLESv2 and src/libEGL. > After bug 1134565, it shouldn't be. If it is, then either your tree is too > old or something regressed it. That bug moved DISABLE_STL_WRAPPINGS into the libGLESv2 & libEGL. This is what I've removed locally.

(In reply to Dan Glastonbury :djg :kamidphish from comment #17) > That bug moved DISABLE_STL_WRAPPINGS into the libGLESv2 & libEGL. This is > what I've removed locally. While it might look this way if you look at the generate_mozbuild.py change, that's actually not the case if you look at the resulting moz.build change that landed in m-c. That is, DISABLE_STL_WRAPPING was already there and was left there. It was only removed for angle itself (what ends up in libxul).

This doesn't fix the crash for user when OOM, but uses Moz infallible allocator, so is reporting the OOM via that mechanism instead of by uncaught exception.

try - https://treeherder.mozilla.org/#/jobs?repo=try&revision=f6469d93f7e0

This is too late for 38 but we could take a patch for 38 esr if it is safe.

I don't see any crashes with this signature in the last two weeks for 39. Dan, do you think that 39 is still affected?

I'd expect nobody to be using 39 at this point, since it's not aurora anymore, and there's not been a beta yet.

Liz, I think that 39 will still be affected but I defer to :glandium for the reason.

Can you request uplift? It looks like this should be uplifted to aurora and probably beta.

Comment on attachment 8599072 [details] [diff] [review] Enable STL wrapping for libGLESv2 and libEGL. Approval Request Comment [Feature/regressing bug #]: 1134565 [User impact if declined]: Very little. This change doesn't fix the OOM crash, but changes the reporting path. Instead of throwing a std::bad_alloc exception on OOM, mozalloc infallible allocator is used. [Describe test coverage new/current, TreeHerder]: Changes ANGLE allocator which is tested by all Windows runs of mochitest-gl. [Risks and why]: Low. This patch has been landed on nightly for 4 weeks with out causing issues. [String/UUID change made/needed]:

Comment on attachment 8599072 [details] [diff] [review] Enable STL wrapping for libGLESv2 and libEGL. Looks like a relatively safe change and early in the cycle. Taking it.