Android 2.3 reftests are finally running to completion on Ash and are very close to being stable enough to run on trunk trees...except for intermittent crashes in R5, usually @ jemalloc_crash, usually in a content/canvas/test/reftest/webgl-* test. I suspect this is a dup of bug 941788, which I reported when working on the Android x86 tests...but let's look at this with fresh eyes in the Android 2.3 context. https://tbpl.mozilla.org/php/getParsedLog.php?id=38208332&tree=Ash&full=1#error0 15:57:45 INFO - RESTORE PREFERENCE pref(webgl.force-enabled,false) 15:57:45 INFO - SET PREFERENCE pref(webgl.force-enabled,true) 15:57:45 INFO - REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/content/canvas/test/reftest/wrapper.html?green.png | 335 / 1051 (31%) 15:57:45 INFO - INFO | automation.py | Application ran for: 0:20:19.672006 15:57:45 INFO - INFO | zombiecheck | Reading PID log: /tmp/tmpsZtqLzpidlog 15:57:45 INFO - /data/anr/traces.txt not found 15:57:45 INFO - mozcrash INFO | Downloading symbols from: https://ftp-ssl.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/ash-android/1398109039/fennec-31.0a1.en-US.android-arm.crashreporter-symbols.zip 15:57:45 WARNING - PROCESS-CRASH | http://10.0.2.2:8854/tests/content/canvas/test/reftest/webgl-hanging-fb-test.html?aa&________ | application crashed [@ jemalloc_crash] 15:57:45 INFO - Crash dump filename: /tmp/tmpunIeul/105b0281-b84c-fc55-6792a7d5-0d5d5133.dmp 15:57:45 INFO - Operating system: Android 15:57:45 INFO - 0.0.0 Linux 2.6.29-ge3d684d #1 Mon Dec 16 22:26:51 UTC 2013 armv7l generic/sdk/generic:2.3.7/GINGERBREAD/eng.ubuntu.20140123.014351:eng/test-keys 15:57:45 INFO - CPU: arm 15:57:45 INFO - 0 CPUs 15:57:45 INFO - 15:57:45 INFO - Crash reason: SIGSEGV 15:57:45 INFO - Crash address: 0x0 15:57:45 INFO - 15:57:45 INFO - Thread 11 (crashed) 15:57:45 INFO - 0 libmozglue.so!jemalloc_crash [jemalloc.c:f85f07b1421b : 1574 + 0x4] 15:57:45 INFO - r4 = 0x46cfd040 r5 = 0x46cfd044 r6 = 0x57c81000 r7 = 0x57c82940 15:57:45 INFO - r8 = 0x57c00628 r9 = 0x00000143 r10 = 0x57c00000 fp = 0x00000000 15:57:45 INFO - sp = 0x47aa0770 lr = 0x80319ffb pc = 0x80318c52 15:57:45 INFO - Found by: given as instruction pointer in context 15:57:45 INFO - 1 libmozglue.so!arena_dalloc [jemalloc.c:f85f07b1421b : 4578 + 0x3] 15:57:45 INFO - r4 = 0x46cfd040 r5 = 0x46cfd044 r6 = 0x57c81000 r7 = 0x57c82940 15:57:45 INFO - r8 = 0x57c00628 r9 = 0x00000143 r10 = 0x57c00000 fp = 0x00000000 15:57:45 INFO - sp = 0x47aa0778 pc = 0x80319ffb 15:57:45 INFO - Found by: call frame info 15:57:45 INFO - 2 libxul.so!mozilla::dom::indexedDB::Key::~Key() [nsTSubstring.h:f85f07b1421b : 86 + 0x3] 15:57:45 INFO - r4 = 0x57ef53ec r5 = 0x0000051c r6 = 0x00000004 r7 = 0x00000148 15:57:45 INFO - r8 = 0x57ef53d0 r9 = 0x00000143 r10 = 0x518cdf7c fp = 0x00000000 15:57:45 INFO - sp = 0x47aa0798 pc = 0x4d629995 15:57:45 INFO - Found by: call frame info 15:57:45 INFO - 3 libxul.so!nsStandardURL::~nsStandardURL() [nsStandardURL.cpp:f85f07b1421b : 292 + 0xf] 15:57:45 INFO - r4 = 0x57ef53d0 r5 = 0x0000051c r6 = 0x00000004 r7 = 0x00000148 15:57:45 INFO - r8 = 0x57ef53d0 r9 = 0x00000143 r10 = 0x518cdf7c fp = 0x00000000 15:57:45 INFO - sp = 0x47aa07a0 pc = 0x4d695c57 15:57:45 INFO - Found by: call frame info 15:57:45 INFO - 4 libxul.so!nsStandardURL::~nsStandardURL() [nsStandardURL.cpp:f85f07b1421b : 292 + 0x3] 15:57:45 INFO - r4 = 0x57ef53d0 r5 = 0x0000051c r6 = 0x00000004 r7 = 0x00000148 15:57:45 INFO - r8 = 0x57ef53d0 r9 = 0x00000143 r10 = 0x518cdf7c fp = 0x00000000 15:57:45 INFO - sp = 0x47aa07a8 pc = 0x4d695c69 15:57:45 INFO - Found by: call frame info 15:57:45 INFO - 5 libxul.so!nsStandardURL::Release() [nsStandardURL.cpp:f85f07b1421b : 921 + 0x5] 15:57:45 INFO - r4 = 0x00000000 r5 = 0x0000051c r6 = 0x00000004 r7 = 0x00000148 15:57:45 INFO - r8 = 0x57ef53d0 r9 = 0x00000143 r10 = 0x518cdf7c fp = 0x00000000 15:57:45 INFO - sp = 0x47aa07b0 pc = 0x4d694e81 15:57:45 INFO - Found by: call frame info 15:57:45 INFO - 6 libxul.so!ReleaseSliceNow(unsigned int, void*) [CycleCollectedJSRuntime.cpp:f85f07b1421b : 1008 + 0x9] 15:57:45 INFO - r4 = 0x518cdf6c r5 = 0x0000051c r6 = 0x00000004 r7 = 0x00000148 15:57:45 INFO - r8 = 0x57ef53d0 r9 = 0x00000143 r10 = 0x518cdf7c fp = 0x00000000 15:57:45 INFO - sp = 0x47aa07b8 pc = 0x4d641239 15:57:45 INFO - Found by: call frame info 15:57:45 INFO - 7 libxul.so!mozilla::IncrementalFinalizeRunnable::ReleaseNow(bool) [CycleCollectedJSRuntime.cpp:f85f07b1421b : 1070 + 0xb] 15:57:45 INFO - r4 = 0x518cdf60 r5 = 0x00000000 r6 = 0x518cdf60 r7 = 0x00000001 15:57:45 INFO - r8 = 0x00989680 r9 = 0x00000000 r10 = 0x518cdf7c fp = 0x00000000 15:57:45 INFO - sp = 0x47aa07e0 pc = 0x4d640d75 15:57:45 INFO - Found by: call frame info 15:57:45 INFO - 8 libxul.so!mozilla::IncrementalFinalizeRunnable::Run() [CycleCollectedJSRuntime.cpp:f85f07b1421b : 1103 + 0x5] 15:57:45 INFO - r4 = 0x518cdf60 r5 = 0x00000000 r6 = 0x47aa083c r7 = 0x47aa086f 15:57:45 INFO - r8 = 0x46e02270 r9 = 0x00000001 r10 = 0x00000000 fp = 0x00000000 15:57:45 INFO - sp = 0x47aa0820 pc = 0x4d640db5 15:57:45 INFO - Found by: call frame info 15:57:45 INFO - 9 libxul.so!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp:f85f07b1421b : 699 + 0x5] 15:57:45 INFO - r4 = 0x46e02240 r5 = 0x00000000 r6 = 0x47aa083c r7 = 0x47aa086f 15:57:45 INFO - r8 = 0x46e02270 r9 = 0x00000001 r10 = 0x00000000 fp = 0x00000000 15:57:45 INFO - sp = 0x47aa0828 pc = 0x4d65cf5b 15:57:45 INFO - Found by: call frame info 15:57:45 INFO - 10 libxul.so!NS_ProcessNextEvent(nsIThread*, bool) [nsThreadUtils.cpp:f85f07b1421b : 263 + 0xb] 15:57:45 INFO - r4 = 0x00000000 r5 = 0x46e3e0c0 r6 = 0x4ece2eec r7 = 0x00000001 15:57:45 INFO - r8 = 0x46e01fb0 r9 = 0x47aa08f8 r10 = 0x00000000 fp = 0x00000000 15:57:45 INFO - sp = 0x47aa0868 pc = 0x4d62ffa3 15:57:45 INFO - Found by: call frame info 15:57:45 INFO - 11 libxul.so!mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) [MessagePump.cpp:f85f07b1421b : 95 + 0x7] 15:57:45 INFO - r4 = 0x46e01fa0 r5 = 0x46e3e0c0 r6 = 0x4ece2eec r7 = 0x00000001 15:57:45 INFO - r8 = 0x46e01fb0 r9 = 0x47aa08f8 r10 = 0x00000000 fp = 0x00000000 15:57:45 INFO - sp = 0x47aa0878 pc = 0x4d7812b5 15:57:45 INFO - Found by: call frame info 15:57:45 INFO - 12 libxul.so!MessageLoop::RunInternal() [message_loop.cc:f85f07b1421b : 226 + 0x5] 15:57:45 INFO - r4 = 0x46e3e0c0 r5 = 0x46e23860 r6 = 0x46e02240 r7 = 0x47aa0ac5 15:57:45 INFO - r8 = 0x47aa0908 r9 = 0x47aa08f8 r10 = 0x00000000 fp = 0x00000000 15:57:45 INFO - sp = 0x47aa08a0 pc = 0x4d776cbb 15:57:45 INFO - Found by: call frame info 15:57:45 INFO - 13 libxul.so!MessageLoop::Run() [message_loop.cc:f85f07b1421b : 219 + 0x5] 15:57:45 INFO - r4 = 0x46e3e0c0 r5 = 0x46e23860 r6 = 0x46e02240 r7 = 0x47aa0ac5 15:57:45 INFO - r8 = 0x47aa0908 r9 = 0x47aa08f8 r10 = 0x00000000 fp = 0x00000000 15:57:45 INFO - sp = 0x47aa08a8 pc = 0x4d776d6d 15:57:45 INFO - Found by: call frame info 15:57:45 INFO - 14 libxul.so!nsBaseAppShell::Run() [nsBaseAppShell.cpp:f85f07b1421b : 164 + 0x7] 15:57:45 INFO - r4 = 0x00000000 r5 = 0x46e23860 r6 = 0x46e02240 r7 = 0x47aa0ac5 15:57:45 INFO - r8 = 0x47aa0908 r9 = 0x47aa08f8 r10 = 0x00000000 fp = 0x00000000 15:57:45 INFO - sp = 0x47aa08c0 pc = 0x4dba3aaf 15:57:45 INFO - Found by: call frame info 15:57:45 INFO - 15 libxul.so!nsAppStartup::Run() [nsAppStartup.cpp:f85f07b1421b : 278 + 0x5] 15:57:45 INFO - r4 = 0x51961280 r5 = 0x47aa09d4 r6 = 0x4d64b2ad r7 = 0x47aa0ac5 15:57:45 INFO - r8 = 0x47aa0908 r9 = 0x47aa08f8 r10 = 0x00000000 fp = 0x00000000 15:57:45 INFO - sp = 0x47aa08d0 pc = 0x4e1526d7 15:57:45 INFO - Found by: call frame info 15:57:45 INFO - 16 libxul.so!XREMain::XRE_mainRun() [nsAppRunner.cpp:f85f07b1421b : 4019 + 0x5] 15:57:45 INFO - r4 = 0x47aa0910 r5 = 0x47aa09d4 r6 = 0x4d64b2ad r7 = 0x47aa0ac5 15:57:45 INFO - r8 = 0x47aa0908 r9 = 0x47aa08f8 r10 = 0x00000000 fp = 0x00000000 15:57:45 INFO - sp = 0x47aa08d8 pc = 0x4e1354f1 15:57:45 INFO - Found by: call frame info

More examples: https://tbpl.mozilla.org/php/getParsedLog.php?id=38208205&tree=Ash&full=1#error0 15:54:54 WARNING - PROCESS-CRASH | http://10.0.2.2:8854/tests/content/canvas/test/reftest/webgl-color-test.html?__&alpha&_____&_______&preserve&stencil | application crashed [@ jemalloc_crash] 15:54:54 INFO - Crash dump filename: /tmp/tmpj87pXv/3030cace-2f15-25ea-43163a46-38203150.dmp 15:54:54 INFO - Operating system: Android 15:54:54 INFO - 0.0.0 Linux 2.6.29-ge3d684d #1 Mon Dec 16 22:26:51 UTC 2013 armv7l generic/sdk/generic:2.3.7/GINGERBREAD/eng.ubuntu.20140123.014351:eng/test-keys 15:54:54 INFO - CPU: arm 15:54:54 INFO - 0 CPUs 15:54:54 INFO - 15:54:54 INFO - Crash reason: SIGSEGV 15:54:54 INFO - Crash address: 0x0 15:54:54 INFO - 15:54:54 INFO - Thread 11 (crashed) 15:54:54 INFO - 0 libmozglue.so!jemalloc_crash [jemalloc.c:f85f07b1421b : 1574 + 0x4] 15:54:54 INFO - r4 = 0x46cfd040 r5 = 0x46cfd044 r6 = 0x590c2000 r7 = 0x590c4300 15:54:54 INFO - r8 = 0x59000940 r9 = 0x4d62effb r10 = 0x59000000 fp = 0x00000000 15:54:54 INFO - sp = 0x47aa0700 lr = 0x80319ffb pc = 0x80318c52 15:54:54 INFO - Found by: given as instruction pointer in context 15:54:54 INFO - 1 libmozglue.so!arena_dalloc [jemalloc.c:f85f07b1421b : 4578 + 0x3] 15:54:54 INFO - r4 = 0x46cfd040 r5 = 0x46cfd044 r6 = 0x590c2000 r7 = 0x590c4300 15:54:54 INFO - r8 = 0x59000940 r9 = 0x4d62effb r10 = 0x59000000 fp = 0x00000000 15:54:54 INFO - sp = 0x47aa0708 pc = 0x80319ffb 15:54:54 INFO - Found by: call frame info 15:54:54 INFO - 2 libxul.so!PL_DHashTableFinish(PLDHashTable*) [pldhash.cpp:f85f07b1421b : 329 + 0x9] 15:54:54 INFO - r4 = 0x544faa9c r5 = 0x590c4440 r6 = 0x00000014 r7 = 0x590c4440 15:54:54 INFO - r8 = 0x544fab44 r9 = 0x4d62effb r10 = 0x544fabc8 fp = 0x00000000 15:54:54 INFO - sp = 0x47aa0728 pc = 0x4d62f7fb 15:54:54 INFO - Found by: call frame info 15:54:54 INFO - 3 libxul.so!mozilla::MaskLayerImageCache::~MaskLayerImageCache() [nsTHashtable.h:f85f07b1421b : 388 + 0x3] 15:54:54 INFO - r4 = 0x544faa9c r5 = 0x544faba8 r6 = 0x544faa28 r7 = 0x4ece2eec 15:54:54 INFO - r8 = 0x544fab44 r9 = 0x4d62effb r10 = 0x544fabc8 fp = 0x00000000 15:54:54 INFO - sp = 0x47aa0740 pc = 0x4d62cabd 15:54:54 INFO - Found by: call frame info 15:54:54 INFO - 4 libxul.so!nsDocument::~nsDocument() [nsBaseHashtable.h:f85f07b1421b : 52 + 0x7] 15:54:54 INFO - r4 = 0x544fa800 r5 = 0x544faba8 r6 = 0x544faa28 r7 = 0x4ece2eec 15:54:54 INFO - r8 = 0x544fab44 r9 = 0x4d62effb r10 = 0x544fabc8 fp = 0x00000000 15:54:54 INFO - sp = 0x47aa0748 pc = 0x4dd27a91 15:54:54 INFO - Found by: call frame info 15:54:54 INFO - 5 libxul.so!nsHTMLDocument::~nsHTMLDocument() [nsHTMLDocument.cpp:f85f07b1421b : 200 + 0x45] 15:54:54 INFO - r4 = 0x544fa800 r5 = 0x00000090 r6 = 0x51e64008 r7 = 0x51e64098 15:54:54 INFO - r8 = 0x0000000c r9 = 0x00000005 r10 = 0x0000000c fp = 0x00000000 15:54:54 INFO - sp = 0x47aa0778 pc = 0x4dde3735 15:54:54 INFO - Found by: call frame info https://tbpl.mozilla.org/php/getParsedLog.php?id=38207979&tree=Ash&full=1#error0 15:50:13 WARNING - PROCESS-CRASH | http://10.0.2.2:8854/tests/content/canvas/test/reftest/webgl-clear-test.html?readback&aa&alpha&________ | application crashed [@ jemalloc_crash] 15:50:13 INFO - Crash dump filename: /tmp/tmps8VDEn/1aa6459c-2ca3-2a80-4f94f5e9-214eba19.dmp 15:50:13 INFO - Operating system: Android 15:50:13 INFO - 0.0.0 Linux 2.6.29-ge3d684d #1 Mon Dec 16 22:26:51 UTC 2013 armv7l generic/sdk/generic:2.3.7/GINGERBREAD/eng.ubuntu.20140123.014351:eng/test-keys 15:50:13 INFO - CPU: arm 15:50:13 INFO - 0 CPUs 15:50:13 INFO - 15:50:13 INFO - Crash reason: SIGSEGV 15:50:13 INFO - Crash address: 0x0 15:50:13 INFO - 15:50:13 INFO - Thread 11 (crashed) 15:50:13 INFO - 0 libmozglue.so!jemalloc_crash [jemalloc.c:f85f07b1421b : 1574 + 0x4] 15:50:13 INFO - r4 = 0x00000020 r5 = 0x46cfd1f0 r6 = 0x46cfd040 r7 = 0x0000000c 15:50:13 INFO - r8 = 0x46cfd044 r9 = 0x00000000 r10 = 0x00000030 fp = 0x46cfd160 15:50:13 INFO - sp = 0x47363c00 lr = 0x803191cb pc = 0x80318c52 15:50:13 INFO - Found by: given as instruction pointer in context 15:50:13 INFO - 1 libmozglue.so!arena_bin_malloc_easy [jemalloc.c:f85f07b1421b : 3870 + 0x7] 15:50:13 INFO - r4 = 0x00000020 r5 = 0x46cfd1f0 r6 = 0x46cfd040 r7 = 0x0000000c 15:50:13 INFO - r8 = 0x46cfd044 r9 = 0x00000000 r10 = 0x00000030 fp = 0x46cfd160 15:50:13 INFO - sp = 0x47363c08 pc = 0x803191cb 15:50:13 INFO - Found by: call frame info 15:50:13 INFO - 2 libmozglue.so!arena_malloc [jemalloc.c:f85f07b1421b : 3891 + 0x5] 15:50:13 INFO - r4 = 0x00000020 r5 = 0x46cfd1f0 r6 = 0x46cfd040 r7 = 0x0000000c 15:50:13 INFO - r8 = 0x46cfd044 r9 = 0x00000000 r10 = 0x00000030 fp = 0x46cfd160 15:50:13 INFO - sp = 0x47363c18 pc = 0x8031aae5 15:50:13 INFO - Found by: call frame info 15:50:13 INFO - 3 libmozglue.so!__wrap_realloc [jemalloc.c:f85f07b1421b : 4759 + 0xb] 15:50:13 INFO - r4 = 0x00000020 r5 = 0x59818c80 r6 = 0x59800000 r7 = 0x0000000c 15:50:13 INFO - r8 = 0x00000002 r9 = 0x00000030 r10 = 0x54f1ac90 fp = 0x00000000 15:50:13 INFO - sp = 0x47363c40 pc = 0x8031b4dd 15:50:13 INFO - Found by: call frame info 15:50:13 INFO - 4 libxul.so!_cairo_array_grow_by [cairo-array.c:f85f07b1421b : 159 + 0xb] 15:50:13 INFO - r4 = 0x00000004 r5 = 0x54f1ac90 r6 = 0x00000002 r7 = 0x0000000c 15:50:13 INFO - r8 = 0x00000002 r9 = 0x4edac51c r10 = 0x54f1ac90 fp = 0x00000000 15:50:13 INFO - sp = 0x47363c98 pc = 0x4e267e47 15:50:13 INFO - Found by: call frame info 15:50:13 INFO - 5 libxul.so!_cairo_array_allocate [cairo-array.c:f85f07b1421b : 335 + 0x3] 15:50:13 INFO - r4 = 0x54f1ac90 r5 = 0x00000001 r6 = 0x47363cc4 r7 = 0x47363cec 15:50:13 INFO - r8 = 0x00000002 r9 = 0x4edac51c r10 = 0x54f1ac90 fp = 0x00000000 15:50:13 INFO - sp = 0x47363cb0 pc = 0x4e267ee5 15:50:13 INFO - Found by: call frame info 15:50:13 INFO - 6 libxul.so!_cairo_array_append_multiple [cairo-array.c:f85f07b1421b : 301 + 0x3] 15:50:13 INFO - r4 = 0x59818c98 r5 = 0x54f1ac90 r6 = 0x00000001 r7 = 0x47363cec 15:50:13 INFO - r8 = 0x00000002 r9 = 0x4edac51c r10 = 0x54f1ac90 fp = 0x00000000 15:50:13 INFO - sp = 0x47363cc0 pc = 0x4e267f1b 15:50:13 INFO - Found by: call frame info 15:50:13 INFO - 7 libxul.so!_cairo_user_data_array_set_data [cairo-array.c:f85f07b1421b : 521 + 0x7] 15:50:13 INFO - r4 = 0x59818c98 r5 = 0x59818c8c r6 = 0x00000000 r7 = 0x00000000 15:50:13 INFO - r8 = 0x00000002 r9 = 0x4edac51c r10 = 0x54f1ac90 fp = 0x00000000 15:50:13 INFO - sp = 0x47363ce0 pc = 0x4e268093 15:50:13 INFO - Found by: call frame info 15:50:13 INFO - 8 libxul.so!mozilla::gl::ReadPixelsIntoImageSurface(mozilla::gl::GLContext*, gfxImageSurface*) [GLReadTexImageHelper.cpp:f85f07b1421b : 453 + 0x5] 15:50:13 INFO - r4 = 0x00000001 r5 = 0x57dd1220 r6 = 0x5876e040 r7 = 0x5876e040 15:50:13 INFO - r8 = 0x51d5a800 r9 = 0x00000100 r10 = 0x00000100 fp = 0x00000000 15:50:13 INFO - sp = 0x47363d18 pc = 0x4d978607 15:50:13 INFO - Found by: call frame info 15:50:13 INFO - 9 libxul.so!mozilla::gl::SharedSurface_Basic::Fence() [SharedSurfaceGL.cpp:f85f07b1421b : 335 + 0x7] 15:50:13 INFO - r4 = 0x5ac14df0 r5 = 0x57dd1220 r6 = 0x00000000 r7 = 0x51d5a800 15:50:13 INFO - r8 = 0x54aa4fb0 r9 = 0x54ab9400 r10 = 0x00000000 fp = 0x00000000 15:50:13 INFO - sp = 0x47363de0 pc = 0x4d979487 15:50:13 INFO - Found by: call frame info https://tbpl.mozilla.org/php/getParsedLog.php?id=38207896&tree=Ash&full=1#error0 15:48:20 WARNING - PROCESS-CRASH | http://10.0.2.2:8854/tests/content/canvas/test/reftest/webgl-resize-test.html | application crashed [@ jemalloc_crash] 15:48:20 INFO - Crash dump filename: /tmp/tmpanIDue/6a048a14-b347-7e36-1c744318-3f7ddbac.dmp 15:48:20 INFO - Operating system: Android 15:48:20 INFO - 0.0.0 Linux 2.6.29-ge3d684d #1 Mon Dec 16 22:26:51 UTC 2013 armv7l generic/sdk/generic:2.3.7/GINGERBREAD/eng.ubuntu.20140123.014351:eng/test-keys 15:48:20 INFO - CPU: arm 15:48:20 INFO - 0 CPUs 15:48:20 INFO - 15:48:20 INFO - Crash reason: SIGSEGV 15:48:20 INFO - Crash address: 0x0 15:48:20 INFO - 15:48:20 INFO - Thread 11 (crashed) 15:48:20 INFO - 0 libmozglue.so!jemalloc_crash [jemalloc.c:f85f07b1421b : 1574 + 0x4] 15:48:20 INFO - r4 = 0x46cfd040 r5 = 0x46cfd044 r6 = 0x5938c000 r7 = 0x5938ce20 15:48:20 INFO - r8 = 0x593006a0 r9 = 0x00000005 r10 = 0x59300000 fp = 0x00000000 15:48:20 INFO - sp = 0x4746f708 lr = 0x80319ffb pc = 0x80318c52 15:48:20 INFO - Found by: given as instruction pointer in context 15:48:20 INFO - 1 libmozglue.so!arena_dalloc [jemalloc.c:f85f07b1421b : 4578 + 0x3] 15:48:20 INFO - r4 = 0x46cfd040 r5 = 0x46cfd044 r6 = 0x5938c000 r7 = 0x5938ce20 15:48:20 INFO - r8 = 0x593006a0 r9 = 0x00000005 r10 = 0x59300000 fp = 0x00000000 15:48:20 INFO - sp = 0x4746f710 pc = 0x80319ffb 15:48:20 INFO - Found by: call frame info 15:48:20 INFO - 2 libxul.so!nsCSSStyleSheet::~nsCSSStyleSheet() [mozalloc.h:f85f07b1421b : 225 + 0x5] 15:48:20 INFO - r4 = 0x5938ce20 r5 = 0x0000015c r6 = 0x57f6c008 r7 = 0x57f6c164 15:48:20 INFO - r8 = 0x0000001d r9 = 0x00000005 r10 = 0x0000000c fp = 0x00000000 15:48:20 INFO - sp = 0x4746f730 pc = 0x4e0356ab 15:48:20 INFO - Found by: call frame info 15:48:20 INFO - 3 libxul.so!SnowWhiteKiller::~SnowWhiteKiller() [nsCycleCollector.cpp:f85f07b1421b : 2386 + 0x9] 15:48:20 INFO - r4 = 0x4746f764 r5 = 0x0000015c r6 = 0x57f6c008 r7 = 0x57f6c164 15:48:20 INFO - r8 = 0x0000001d r9 = 0x00000005 r10 = 0x0000000c fp = 0x00000000 15:48:20 INFO - sp = 0x4746f738 pc = 0x4d789b71 15:48:20 INFO - Found by: call frame info

I reproduced the crash on a loaner running a Debug build with MOZ_GL_DEBUG defined.

I also tried MOZ_GL_VERBOSE, but the extra output seems to slow down the test too much -- it times out without crashing.

I can sometimes reproduce this on the emulator by running the canvas reftests (content/canvas/test/reftest). It looks like something is overrunning into jemalloc state, causing the assertion failures (and maybe a handful of other jemalloc-related crashes). Without tools like valgrind, though, it's hard to say where. I added a bunch of other assertions to see if I could narrow down where it could be happening, but no luck yet.

Running the emulator on a loaner ix, I could reproduce the crash by running R5. I could also reproduce it by running just content/canvas/test/reftest/reftest.list. I also tried various subsets of content/canvas/test/reftest/reftest.list, and was able to reproduce the crash. I also tried changing the reftest manifest to run only one test, test-webgl-clear.html, and was able to reproduce the crash. I simplified that test and was still able to reproduce the crash. In all cases, the crash rate was about the same: about 50% chance of a crash in 100 tests. Here's the simplified test: <!DOCTYPE html> <html class="reftest-wait"> <head> <meta charset="UTF-8"> <script type="text/javascript"> "use strict"; function waitForComposite(func) { var isDone = false; var doneFunc = function () { if (isDone) return; isDone = true; func(); }; window.requestAnimationFrame(doneFunc); setTimeout(doneFunc, 500); } function runTest() { var canvas = document.getElementById("canvas"); var gl = null; try { var argDict = { alpha: false, depth: false, stencil: false, antialias: false, premultipliedAlpha: false, preserveDrawingBuffer: false, }; gl = canvas.getContext("experimental-webgl", argDict); } catch(e) { dump("GYB: getContext fail!"); } if (gl) gl.finish(); else dump("GYB: no gl!!"); waitForComposite(testComplete); } function testComplete() { document.documentElement.removeAttribute("class"); } </script> </head> <body onload="runTest();"> <canvas id="canvas" width="256" height="256"></canvas> </body> </html>

*But* if I remove the default preference from reftest.list: # default-preferences pref(webgl.force-enabled,true) and instead set webgl.force-enabled = true just once at browser start - then I cannot reproduce the crash! (The default-preferences directive tells the reftest harness to set webgl.force-enabled to true before each test in that manifest, and to restore the pref at the end of each test: REFTEST TEST-START | http://10.0.2.2:8854/tests/content/canvas/test/reftest/webgl-clear-test.html?__&_____&________ RESTORE PREFERENCE pref(webgl.force-enabled,false) SET PREFERENCE pref(webgl.force-enabled,true) REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/content/canvas/test/reftest/webgl-clear-test.html?__&_____&________ | 52 / 114 (45%) RESTORE PREFERENCE pref(webgl.force-enabled,false) SET PREFERENCE pref(webgl.force-enabled,true) REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/content/canvas/test/reftest/webgl-clear-test.html?__&_____&________ | 52 / 114 (45%) REFTEST TEST-PASS | http://10.0.2.2:8854/tests/content/canvas/test/reftest/webgl-clear-test.html?__&_____&________ | image comparison (==) REFTEST INFO | Loading a blank page REFTEST TEST-END | http://10.0.2.2:8854/tests/content/canvas/test/reftest/webgl-clear-test.html?__&_____&________ ) That's not conclusive, but might suggest a cause.

Weirdness.

Recent builds on ash have not been crashing -- did someone fix this? The (potential) fix appeared in https://hg.mozilla.org/projects/ash/pushloghtml?startID=193&endID=194.

Bisecting mozilla-central builds, https://tbpl.mozilla.org/?rev=400a31da59a9 passes; https://tbpl.mozilla.org/?rev=d7c07694f339 crashes. On m-i, https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=527c58711da1 passes; fe7f450a5435 crashes.

Backing out 2c14ea7a713c (bug 1001682) and b437c670e090 (bug 1001683) restore the crash: https://tbpl.mozilla.org/?tree=Ash&rev=fb666dc79484. Thank you :jwatt!

I don't think bug 1001682 could have fixed this, actually, so I think it must have been bug 1001683.