http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1273544305.1273546243.31416.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dromaeo on 2010/05/10 19:18:25 talos-r3-leopard-011: Started Mon, 10 May 2010 19:22:43 Running test dromaeo_css: Started Mon, 10 May 2010 19:22:43 Screen width/height:1280/1024 colorDepth:24 Browser inner width/height: 1024/643 Browser outer width/height: 1024/768 NOISE: Cycle 1: loaded http://localhost/page_load_test/dromaeo/cssquery-dojo.html (next: http://localhost/page_load_test/dromaeo/cssquery-ext.html) NOISE: NOISE: __FAILbrowser non-zero return code (256)__FAIL NOISE: Cycle 1: loaded http://localhost/page_load_test/dromaeo/cssquery-dojo.html (next: http://localhost/page_load_test/dromaeo/cssquery-ext.html) NOISE: NOISE: __FAILbrowser non-zero return code (256)__FAIL Crash reason: EXC_BAD_ACCESS / KERN_PROTECTION_FAILURE Crash address: 0x14 Thread 0 (crashed) 0 XUL!nsContentList::ContentAppended(nsIDocument*, nsIContent*, nsIContent*, int) [nsINode.h : 1096 + 0x0] eip = 0x024a1710 esp = 0xbfffcb10 ebp = 0xbfffcb38 ebx = 0x024f0ef1 esi = 0x13010370 edi = 0x0ed94df0 eax = 0x00000000 ecx = 0x00000003 edx = 0x00000000 efl = 0x00210206 Found by: given as instruction pointer in context 1 XUL!nsNodeUtils::ContentAppended(nsIContent*, nsIContent*, int) [nsNodeUtils.cpp:7adb6aca38be : 139 + 0x20] eip = 0x024fd6f9 esp = 0xbfffcb40 ebp = 0xbfffcb88 Found by: previous frame's frame pointer 2 XUL!nsINode::doInsertChildAt(nsIContent*, unsigned int, int, nsAttrAndChildArray&) [nsGenericElement.cpp:7adb6aca38be : 3622 + 0x12] eip = 0x024f1143 esp = 0xbfffcb90 ebp = 0xbfffcc58 Found by: previous frame's frame pointer 3 XUL!nsGenericElement::InsertChildAt(nsIContent*, unsigned int, int) [nsGenericElement.cpp:7adb6aca38be : 3552 + 0x23] eip = 0x024f14fd esp = 0xbfffcc60 ebp = 0xbfffcc88 Found by: previous frame's frame pointer 4 XUL!nsINode::ReplaceOrInsertBefore(int, nsINode*, nsINode*) [nsGenericElement.cpp:7adb6aca38be : 4308 + 0x20] eip = 0x024f2d1f esp = 0xbfffcc90 ebp = 0xbfffcd28 Found by: previous frame's frame pointer 5 XUL!nsIDOMNode_AppendChild [nsINode.h : 1205 + 0x24] eip = 0x029f51b0 esp = 0xbfffcd30 ebp = 0xbfffce48 Found by: previous frame's frame pointer 6 libmozjs.dylib!js_Interpret [jsops.cpp:7adb6aca38be : 2199 + 0x1c] eip = 0x00268cb4 esp = 0xbfffce50 ebp = 0xbfffd138 Found by: previous frame's frame pointer

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1273552121.1273553975.27206.gz Rev3 WINNT 5.1 mozilla-central talos dromaeo on 2010/05/10 21:28:41

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1273554933.1273556414.3341.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dromaeo on 2010/05/10 22:15:33

I'm also suspicious of the sudden surge of (useless, symbol-free) crashes in test_jQuery.html in Linux 64 Mo2 starting with this same landing: http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1273545917.1273546235.31395.gz http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1273550281.1273550889.15844.gz http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1273558888.1273559341.13877.gz

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1273569835.1273572387.31953.gz Rev3 WINNT 5.1 mozilla-central talos dromaeo on 2010/05/11 02:23:55

So this could be a regression from bug 564435 or bug 564574. Perhaps I should have pushed them separately....

So the places where this happens so far seem to be: * Right after NOISE: Cycle 1: loaded http://localhost/page_load_test/dromaeo/cssquery-dojo.html (next: http://localhost/page_load_test/dromaeo/cssquery-ext.html) * Right after NOISE: Cycle 1: loaded http://localhost/page_load_test/dromaeo/cssquery-ext.html (next: http://localhost/page_load_test/dromaeo/cssquery-jquery.html) * Right after start of the dromaeo test * Right after NOISE: Cycle 1: loaded http://localhost/page_load_test/dromaeo/jslib-event-prototype.html (next: http://localhost/page_load_test/dromaeo/jslib-modify-jquery.html) In all cases, the crash seems to be due to reading memory at 0x14, which would be the mNextSibling of a null nsINode. I have been attempting to reproduce locally, with no luck so far, by running the various tests involved in a browser individually. I can't seem to figure out a sane way to run local talos without installing a web server and all that jazz, so unfortunately I'm not testing the transitions the way the tinderbox does.... jst, don't we have a replay machine set up that I could try to use to catch this?

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1273617424.1273620171.20307.gz Rev3 WINNT 6.1 mozilla-central talos dromaeo on 2010/05/11 15:37:04

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1273611852.1273618296.11300.gz Rev3 WINNT 6.1 mozilla-central talos dromaeo on 2010/05/11 14:04:12

I managed to reproduce this locally by running with gczeal=1. The underlying issue is a regression from bug 564432, but bug 564435 surfaced it. The patch in bug 565125 fixes the problem for me.

Should be fixed by checkin for bug 565125.