Retriggered: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=windows%2C7%2Cpgo%2Creftests%2Cwith%2Ce10s%2Ctest-windows7-32-pgo%2Fopt-jsreftest-e10s-1%2Cr-e10s%28j1%29&tochange=6cec6714210566cd5a156c40b1759a3c46d9d69f&fromchange=683836b692db76292cc3e4494256578afe69f24e&group_state=expanded&selectedJob=228930184

Aryx, can you please take a look? I could not add this job for those 3 pushes.

Backfilling is working for newer pushes: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=windows%2C7%2Cpgo%2Creftests%2Cwith%2Ce10s%2Ctest-windows7-32-pgo%2Fopt-jsreftest-e10s-1%2Cr-e10s%28j1%29&group_state=expanded&fromchange=5f7a214a91d721e349adae7417a4a60421431fce&selectedJob=229699285

Might be broken for the push you attempted due to the failures which that push backed out.

(In reply to Sebastian Hengst [PTO+professional development; back to normal activity 2019-02-25][:aryx] (needinfo on intermittent or backout) from comment #10)

Backfilling is working for newer pushes: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=windows%2C7%2Cpgo%2Creftests%2Cwith%2Ce10s%2Ctest-windows7-32-pgo%2Fopt-jsreftest-e10s-1%2Cr-e10s%28j1%29&group_state=expanded&fromchange=5f7a214a91d721e349adae7417a4a60421431fce&selectedJob=229699285

Might be broken for the push you attempted due to the failures which that push backed out.

Ok, so in this case how can I check where it started from?

Might not be possible and it hasn't run for months due to https://searchfox.org/mozilla-central/rev/dc0adc07db3df9431a0876156f50c65d580010cb/taskcluster/ci/test/compiled.yml#62

Oh :(

There are 61 failures in the last 7 days, all on windows7-32 pgo. Failure rate here seems to be decreasing.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=230892132&repo=mozilla-inbound&lineNumber=44013

23:40:25 INFO - REFTEST TEST-START | file:///Z:/task_1551309113/build/tests/jsreftest/tests/jsreftest.html?test=test262/language/expressions/object/setter-prop-desc.js
23:40:25 INFO - REFTEST TEST-LOAD | file:///Z:/task_1551309113/build/tests/jsreftest/tests/jsreftest.html?test=test262/language/expressions/object/setter-prop-desc.js | 11200 / 21238 (52%)
23:40:25 INFO - REFTEST TEST-PASS | file:///Z:/task_1551309113/build/tests/jsreftest/tests/jsreftest.html?test=test262/language/expressions/object/setter-prop-desc.js | item 1
23:40:25 INFO - REFTEST TEST-END | file:///Z:/task_1551309113/build/tests/jsreftest/tests/jsreftest.html?test=test262/language/expressions/object/setter-prop-desc.js
23:40:25 INFO - Assertion failure: [unhandlable oom] Failed not allocate new chunk during GC, at z:/build/build/src/js/src/vm/JSContext.cpp:1521
23:40:26 INFO - ###!!! [Parent][MessageChannel] Error: (msgtype=0x1E009A,name=PBrowser::Msg_UpdateNativeWindowHandle) Channel error: cannot send/recv
23:40:26 INFO - ###!!! [Parent][MessageChannel] Error: (msgtype=0x1E0089,name=PBrowser::Msg_Destroy) Channel error: cannot send/recv
23:40:26 INFO - A content process crashed and MOZ_CRASHREPORTER_SHUTDOWN is set, shutting down
23:40:26 INFO - ###!!! [Parent][MessageChannel] Error: (msgtype=0x34003E,name=PContent::Msg_NotifyProcessPriorityChanged) Channel error: cannot send/recv
23:40:26 INFO - [Parent 5808, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
23:40:26 INFO - [Child 1288, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
23:40:26 INFO - [Child 1288, Chrome_ChildThread] WARNING: pipe error:JavaScript error: resource://reftest/reftest.jsm, line 1558: NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIPropertyBag2.getPropertyAsAString]
23:40:26 INFO - !!! error running onStopped callback: TypeError: callback is not a function
23:40:26 INFO - [Parent 5808, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
23:40:26 INFO - [CJavaScript error: resource://reftest/reftest.jsm, line 1558: NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIPropertyBag2.getPropertyAsAString]
23:40:26 INFO - 1551310826776 Marionette TRACE Received observer notification xpcom-will-shutdown
23:40:26 INFO - 1551310826776 Marionette INFO Stopped listening on port 2828
23:40:26 INFO - 1551310826776 Marionette DEBUG Remote service is inactive
23:40:26 INFO - REFTEST INFO | Downloading symbols from: https://queue.taskcluster.net/v1/task/S3JQCBH1TLeCyiXNSe8ocA/artifacts/public/build/target.crashreporter-symbols.zip
23:40:30 INFO - REFTEST INFO | Copy/paste: Z:\task_1551309113\build\win32-minidump_stackwalk.exe c:\users\task_1551309113\appdata\local\temp\tmpwl199m.mozrunner\minidumps\590a972b-be79-43a7-b8f0-1ddac0d631cc.dmp c:\users\task_1551309113\appdata\local\temp\tmpw4pbx7
23:40:46 INFO - REFTEST INFO | Saved minidump as Z:\task_1551309113\build\blobber_upload_dir\590a972b-be79-43a7-b8f0-1ddac0d631cc.dmp
23:40:46 INFO - REFTEST INFO | Saved app info as Z:\task_1551309113\build\blobber_upload_dir\590a972b-be79-43a7-b8f0-1ddac0d631cc.extra
23:40:46 INFO - REFTEST PROCESS-CRASH | file:///Z:/task_1551309113/build/tests/jsreftest/tests/jsreftest.html?test=test262/language/expressions/object/setter-prop-desc.js (finished) | application crashed [@ js::AutoEnterOOMUnsafeRegion::crash(char const *)]
23:40:46 INFO - Crash dump filename: c:\users\task_1551309113\appdata\local\temp\tmpwl199m.mozrunner\minidumps\590a972b-be79-43a7-b8f0-1ddac0d631cc.dmp
23:40:46 INFO - Operating system: Windows NT
23:40:46 INFO - 6.1.7601 Service Pack 1
23:40:46 INFO - CPU: x86
23:40:46 INFO - GenuineIntel family 6 model 63 stepping 2
23:40:46 INFO - 8 CPUs
23:40:46 INFO -
23:40:46 INFO - GPU: UNKNOWN
23:40:46 INFO -
23:40:46 INFO - Crash reason: EXCEPTION_BREAKPOINT
23:40:46 INFO - Crash address: 0x57975991
23:40:46 INFO - Assertion: Unknown assertion type 0x00000000
23:40:46 INFO - Process uptime: 425 seconds
23:40:46 INFO -
23:40:46 INFO - Thread 0 (crashed)
23:40:46 INFO - 0 xul.dll!js::AutoEnterOOMUnsafeRegion::crash(char const *) [JSContext.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 1522 + 0x0]
23:40:46 INFO - eip = 0x57975991 esp = 0x002ee264 ebp = 0x002ee66c ebx = 0x6a7f0260
23:40:46 INFO - esi = 0x002ee264 edi = 0x5b97c7a6 eax = 0x745caa3c ecx = 0x000005f2
23:40:46 INFO - edx = 0x6acde340 efl = 0x00000216
23:40:46 INFO - Found by: given as instruction pointer in context
23:40:46 INFO - 1 xul.dll!js::AutoEnterOOMUnsafeRegion::crash(unsigned int,char const *) [JSContext.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 1535 + 0x6]
23:40:46 INFO - eip = 0x57986cc0 esp = 0x002ee674 ebp = 0x002ee67c esi = 0x5b97c7a6
23:40:46 INFO - edi = 0x00000000
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 2 xul.dll!js::TenuringTracer::moveToTenured(JSString *) [Marking.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 3190 + 0xa9]
23:40:46 INFO - eip = 0x57b3b20e esp = 0x002ee684 ebp = 0x002ee6a8 esi = 0x0000001d
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 3 xul.dll!js::gc::StoreBuffer::MonoTypeBuffer<js::gc::StoreBuffer::CellPtrEdge>::trace(js::TenuringTracer &) [Marking.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 2749 + 0x19]
23:40:46 INFO - eip = 0x57b3d339 esp = 0x002ee6b0 ebp = 0x002ee6c4 ebx = 0x6a7f0260
23:40:46 INFO - esi = 0x4c14dff4 edi = 0x6a7f0660
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 4 xul.dll!js::Nursery::doCollection(JS::GCReason,js::gc::TenureCountCache &) [Nursery.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 937 + 0x6]
23:40:46 INFO - eip = 0x57b53ab7 esp = 0x002ee6cc ebp = 0x002ee770 ebx = 0x068080e0
23:40:46 INFO - esi = 0x06808180 edi = 0x002ee740
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 5 xul.dll!js::Nursery::collect(JS::GCReason) [Nursery.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 782 + 0x9]
23:40:46 INFO - eip = 0x57b52b76 esp = 0x002ee778 ebp = 0x002ee8a8 ebx = 0x068080e0
23:40:46 INFO - esi = 0x002ee800 edi = 0x00000009
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 6 xul.dll!js::gc::GCRuntime::minorGC(JS::GCReason,js::gcstats::PhaseKind) [GC.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 7778 + 0x8]
23:40:46 INFO - eip = 0x57b3236b esp = 0x002ee8b0 ebp = 0x002ee8d0 ebx = 0x00000000
23:40:46 INFO - esi = 0x068062c0 edi = 0x61705cb8
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 7 xul.dll!js::gc::GCRuntime::gcCycle(bool,js::SliceBudget,JS::GCReason) [GC.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 7355 + 0xd]
23:40:46 INFO - eip = 0x57b31e98 esp = 0x002ee8d8 ebp = 0x002ee960 ebx = 0x068062c0
23:40:46 INFO - esi = 0x00000001 edi = 0x068062e8
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 8 xul.dll!js::gc::GCRuntime::collect(bool,js::SliceBudget,JS::GCReason) [GC.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 7559 + 0x56]
23:40:46 INFO - eip = 0x57b3296f esp = 0x002ee968 ebp = 0x002ee9f0 ebx = 0x00000004
23:40:46 INFO - esi = 0x068062c0 edi = 0x00000001
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 9 xul.dll!class JSFatInlineString * js::gc::GCRuntime::tryNewTenuredThing<JSFatInlineString,js::CanGC>(struct JSContext *, js::gc::AllocKind, unsigned int) [Allocator.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 279 + 0x97]
23:40:46 INFO - eip = 0x57b3df34 esp = 0x002ee9f8 ebp = 0x002eea8c ebx = 0x00000000
23:40:46 INFO - esi = 0x00000000 edi = 0x0680f800
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 10 xul.dll!class JSObject * js::Allocate<JSObject,js::CanGC>(struct JSContext *, js::gc::AllocKind, unsigned int, js::gc::InitialHeap, const struct js::Class *) [Allocator.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 77 + 0x1a]
23:40:46 INFO - eip = 0x57b3f6fd esp = 0x002eea94 ebp = 0x002eeb04 ebx = 0x0680f800
23:40:46 INFO - esi = 0x068062c0 edi = 0x00000000
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 11 xul.dll!static class JSObject * NewObject(struct JSContext *, class JS::Handle<js::ObjectGroup *>, js::gc::AllocKind, js::NewObjectKind, unsigned int) [JSObject.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 799 + 0x17]
23:40:46 INFO - eip = 0x579907de esp = 0x002eeb0c ebp = 0x002eeb4c ebx = 0x5b17aa40
23:40:46 INFO - esi = 0x0680f800 edi = 0x002eeb78
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 12 xul.dll!js::NewObjectWithClassProtoCommon(JSContext *,js::Class const *,JS::Handle<JSObject *>,js::gc::AllocKind,js::NewObjectKind) [JSObject.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 934 + 0xd]
23:40:46 INFO - eip = 0x57990bf1 esp = 0x002eeb54 ebp = 0x002eeb8c ebx = 0x5b17aa40
23:40:46 INFO - esi = 0x00000000 edi = 0x0680f800
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 13 xul.dll!js::NewFunctionWithProto(JSContext ,bool ()(JSContext *,unsigned int,JS::Value *),unsigned int,JSFunction::Flags,JS::Handle<JSObject *>,JS::Handle<JSAtom *>,JS::Handle<JSObject *>,js::gc::AllocKind,js::NewObjectKind) [JSFunction.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 2111 + 0x14]
23:40:46 INFO - eip = 0x5798bb71 esp = 0x002eeb94 ebp = 0x002eebc0 ebx = 0x00000000
23:40:46 INFO - esi = 0x0680f800 edi = 0x5b49cce4
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 14 xul.dll!static bool DefineAccessorPropertyById(struct JSContext *, class JS::Handle<JSObject *>, class JS::Handle<JS::PropertyKey>, const struct JSNativeWrapper & const, const struct JSNativeWrapper & const, unsigned int) [jsapi.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 1926 + 0x1e]
23:40:46 INFO - eip = 0x57acbc2d esp = 0x002eebc8 ebp = 0x002eec30 ebx = 0x00000011
23:40:46 INFO - esi = 0x0680f800 edi = 0x5b49cce4
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 15 xul.dll!JS_DefineProperties(JSContext *,JS::Handle<JSObject *>,JSPropertySpec const *) [jsapi.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 2942 + 0x16]
23:40:46 INFO - eip = 0x57ad010f esp = 0x002eec38 ebp = 0x002eece4 ebx = 0x5b49cce4
23:40:46 INFO - esi = 0x0680f800 edi = 0x002eec64
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 16 xul.dll!bool mozilla::dom::DefinePrefable<const JSPropertySpec>(struct JSContext *, class JS::Handle<JSObject *>, const struct mozilla::dom::Prefable<const JSPropertySpec> *) [BindingUtils.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 655 + 0xb]
23:40:46 INFO - eip = 0x5999e9e2 esp = 0x002eecec ebp = 0x002eed08 ebx = 0x5b490fa0
23:40:46 INFO - esi = 0x00000000 edi = 0x0680f800
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 17 xul.dll!mozilla::dom::DefineProperties(JSContext *,JS::Handle<JSObject *>,mozilla::dom::NativePropertiesN<7> const *,mozilla::dom::NativePropertiesN<7> const *) [BindingUtils.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 914 + 0xb]
23:40:46 INFO - eip = 0x570bc0f2 esp = 0x002eed10 ebp = 0x002eed28 ebx = 0x5b49c1d8
23:40:46 INFO - esi = 0x0680f800 edi = 0x002eede4
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 18 xul.dll!mozilla::dom::Window_Binding::Wrap(JSContext *,nsGlobalWindowInner *,nsWrapperCache *,JS::RealmOptions &,JSPrincipals *,bool,JS::MutableHandle<JSObject *>) [WindowBinding.cpp: : 19366 + 0x17]
23:40:46 INFO - eip = 0x571a08fa esp = 0x002eed30 ebp = 0x002eed6c ebx = 0x002eede4
23:40:46 INFO - esi = 0x0680f800 edi = 0x0d516804
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 19 xul.dll!nsGlobalWindowOuter::SetNewDocument(mozilla::dom::Document *,nsISupports *,bool) [nsGlobalWindowOuter.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 2078 + 0x238]
23:40:46 INFO - eip = 0x5719c140 esp = 0x002eed74 ebp = 0x002eee6c ebx = 0x002eede4
23:40:46 INFO - esi = 0x2556c880 edi = 0x0680f800
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 20 xul.dll!nsresult nsDocumentViewer::InitInternal(class nsIWidget *, class nsISupports *, const struct mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> & const, bool, bool, bool) [nsDocumentViewer.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 963 + 0xd]
23:40:46 INFO - eip = 0x57198070 esp = 0x002eee74 ebp = 0x002eeee0 ebx = 0x01786170
23:40:46 INFO - esi = 0x213b1de0 edi = 0x19ed7000
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 21 xul.dll!nsresult nsDocumentViewer::Init(class nsIWidget *, const struct mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> & const) [nsDocumentViewer.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 711 + 0x10]
23:40:46 INFO - eip = 0x57197e19 esp = 0x002eeee8 ebp = 0x002eeefc ebx = 0x0876d000
23:40:46 INFO - esi = 0x0a202c00 edi = 0x213b1de0
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 22 xul.dll!nsresult nsDocShell::Embed(class nsIContentViewer *, const char *, class nsISupports *) [nsDocShell.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 6251 + 0x366]
23:40:46 INFO - eip = 0x5719756b esp = 0x002eef04 ebp = 0x002eef9c
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 23 xul.dll!nsresult nsDSURIContentListener::DoContent(const class nsTSubstring<char> & const, bool, class nsIRequest *, class nsIStreamListener * *, bool *) [nsDSURIContentListener.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 182 + 0x1b4]
23:40:46 INFO - eip = 0x572538aa esp = 0x002eefa4 ebp = 0x002ef000 ebx = 0x80004005
23:40:46 INFO - esi = 0x0a202c00 edi = 0x087f9b50
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 24 xul.dll!bool nsDocumentOpenInfo::TryContentListener(class nsIURIContentListener *, class nsIChannel *) [nsURILoader.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 750 + 0xe]
23:40:46 INFO - eip = 0x5725242a esp = 0x002ef008 ebp = 0x002ef05c ebx = 0x00000000
23:40:46 INFO - esi = 0x1f084b3c edi = 0x087f9b50
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 25 xul.dll!nsresult nsDocumentOpenInfo::OnStartRequest(class nsIRequest *, class nsISupports *) [nsURILoader.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 300 + 0x16d]
23:40:46 INFO - eip = 0x572511fd esp = 0x002ef064 ebp = 0x002ef114 ebx = 0x00000000
23:40:46 INFO - esi = 0x1f084b20 edi = 0x419254f0
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 26 xul.dll!nsBaseChannel::OnStartRequest(nsIRequest *,nsISupports *) [nsBaseChannel.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 763 + 0xc]
23:40:46 INFO - eip = 0x57218b7d esp = 0x002ef11c ebp = 0x002ef154 ebx = 0x00000000
23:40:46 INFO - esi = 0x21515080 edi = 0x419254f0
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 27 xul.dll!nsresult nsInputStreamPump::OnInputStreamReady(class nsIAsyncInputStream *) [nsInputStreamPump.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 396 + 0x7b]
23:40:46 INFO - eip = 0x5721816b esp = 0x002ef15c ebp = 0x002ef1c0 ebx = 0x21515084
23:40:46 INFO - esi = 0x215150dc edi = 0x0170c040
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 28 xul.dll!nsresult mozilla::NonBlockingAsyncInputStream::AsyncWaitRunnable::Run() [NonBlockingAsyncInputStream.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 29 + 0x4b]
23:40:46 INFO - eip = 0x57473b25 esp = 0x002ef1c8 ebp = 0x002ef1e0 ebx = 0x21515084
23:40:46 INFO - esi = 0x24939340 edi = 0x26ce4c80
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 29 xul.dll!nsresult mozilla::SchedulerGroup::Runnable::Run() [SchedulerGroup.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 292 + 0x6]
23:40:46 INFO - eip = 0x58616999 esp = 0x002ef1e8 ebp = 0x002ef1f4 ebx = 0x01709100
23:40:46 INFO - esi = 0x002ef200 edi = 0x26f2a040
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 30 xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 1162 + 0x9]
23:40:46 INFO - eip = 0x57057e07 esp = 0x002ef1fc ebp = 0x002ef6d8 esi = 0x002ef200
23:40:46 INFO - edi = 0x00000000
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 31 xul.dll!NS_ProcessNextEvent(nsIThread *,bool) [nsThreadUtils.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 474 + 0x11]
23:40:46 INFO - eip = 0x5705792b esp = 0x002ef6e0 ebp = 0x002ef6fc ebx = 0x01750101
23:40:46 INFO - esi = 0x002ef6ef edi = 0x017012c0
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 32 xul.dll!void mozilla::ipc::MessagePump::Run(class base::MessagePump::Delegate *) [MessagePump.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 88 + 0xa]
23:40:46 INFO - eip = 0x57200567 esp = 0x002ef704 ebp = 0x002ef724 ebx = 0x01750101
23:40:46 INFO - esi = 0x017012b0
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 33 xul.dll!MessageLoop::RunHandler() [message_loop.cc:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 308 + 0x8]
23:40:46 INFO - eip = 0x570302d4 esp = 0x002ef72c ebp = 0x002ef75c ebx = 0x0175b400
23:40:46 INFO - esi = 0x002ef840 edi = 0x017f50d0
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 34 xul.dll!MessageLoop::Run() [message_loop.cc:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 290 + 0x5]
23:40:46 INFO - eip = 0x570575c1 esp = 0x002ef764 ebp = 0x002ef77c ebx = 0x0175b400
23:40:46 INFO - esi = 0x01709100 edi = 0x017f50d0
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 35 xul.dll!nsBaseAppShell::Run() [nsBaseAppShell.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 137 + 0xc]
23:40:46 INFO - eip = 0x57200455 esp = 0x002ef784 ebp = 0x002ef78c
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 36 xul.dll!nsAppShell::Run() [nsAppShell.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 411 + 0x6]
23:40:46 INFO - eip = 0x571ffe00 esp = 0x002ef794 ebp = 0x002ef79c esi = 0x017f50d0
23:40:46 INFO - edi = 0x0175b400
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 37 xul.dll!XRE_RunAppShell() [nsEmbedFunctions.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 908 + 0x6]
23:40:46 INFO - eip = 0x5b0fed97 esp = 0x002ef7a4 ebp = 0x002ef7b4 esi = 0x017012b0
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 38 xul.dll!mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate *) [MessagePump.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 238 + 0x5]
23:40:46 INFO - eip = 0x5899a0b8 esp = 0x002ef7bc ebp = 0x002ef7cc esi = 0x017012b0
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 39 xul.dll!MessageLoop::RunHandler() [message_loop.cc:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 308 + 0x8]
23:40:46 INFO - eip = 0x570302d4 esp = 0x002ef7d4 ebp = 0x002ef804 ebx = 0x0175b400
23:40:46 INFO - esi = 0x002ef840 edi = 0x0175b400
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 40 xul.dll!MessageLoop::Run() [message_loop.cc:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 290 + 0x5]
23:40:46 INFO - eip = 0x570575c1 esp = 0x002ef80c ebp = 0x002ef824 ebx = 0x0175b400
23:40:46 INFO - esi = 0x80004005 edi = 0x0175b400
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 41 xul.dll!XRE_InitChildProcess(int,char * * const,XREChildData const *) [nsEmbedFunctions.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 746 + 0x5]
23:40:46 INFO - eip = 0x5b0fea9a esp = 0x002ef82c ebp = 0x002ef940
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 42 xul.dll!nsresult mozilla::BootstrapImpl::XRE_InitChildProcess(int, char * *, const struct XREChildData *) [Bootstrap.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 61 + 0xe]
23:40:46 INFO - eip = 0x5b104981 esp = 0x002ef948 ebp = 0x002ef954 ebx = 0x01703040
23:40:46 INFO - esi = 0x0000001a edi = 0x01705118
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 43 firefox.exe!static int content_process_main(class mozilla::Bootstrap *, int, char * *) [plugin-container.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 49 + 0xd]
23:40:46 INFO - eip = 0x003114a1 esp = 0x002ef95c ebp = 0x002ef980
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 44 firefox.exe!static int NS_internal_main(int, char * *, char * *) [nsBrowserApp.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 265 + 0xa]
23:40:46 INFO - eip = 0x0031140a esp = 0x002ef988 ebp = 0x002ef9f0 ebx = 0x01703120
23:40:46 INFO - esi = 0x0000001b edi = 0x01703040
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 45 firefox.exe!wmain [nsWindowsWMain.cpp:55b6a8c4e0154ac41f710bf1f3f5627c68ce8d42 : 131 + 0x12]
23:40:46 INFO - eip = 0x003110e2 esp = 0x002ef9f8 ebp = 0x002efa14 ebx = 0x01703120
23:40:46 INFO - esi = 0x0000007f edi = 0x01703040
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 46 firefox.exe!static int __scrt_common_main_seh() [exe_common.inl : 288 + 0x1c]
23:40:46 INFO - eip = 0x00349dcc esp = 0x002efa1c ebp = 0x002efa5c ebx = 0x7ffdf000
23:40:46 INFO - esi = 0x6acdf0b0 edi = 0x011b2950
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 47 kernel32.dll!BaseThreadInitThunk + 0x12
23:40:46 INFO - eip = 0x763e3c45 esp = 0x002efa64 ebp = 0x002efa68
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 48 ntdll.dll!CreatePipe + 0x231
23:40:46 INFO - eip = 0x77a637f5 esp = 0x002efa70 ebp = 0x002efaa8
23:40:46 INFO - Found by: call frame info
23:40:46 INFO - 49 ntdll.dll!CreatePipe + 0x204
23:40:46 INFO - eip = 0x77a637c8 esp = 0x002efab0 ebp = 0x002efac0
23:40:46 INFO - Found by: call frame info

Steven can you assign someone to take a look at this?

Paul, is this something you could take a look at? If not, could you help find who on our team is the right person.

This is basically the same thing as bug 1529306. The rate of this failure has gone down by quite a bit since that landed. I'm not sure if the residual failure rate needs attention or not.

(In reply to Steven DeTar [:sdetar] from comment #17)

Paul, is this something you could take a look at? If not, could you help find who on our team is the right person.

I can take a look next week.

There are 21 total failures in the last 7 days, all on windows7-32 pgo.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=231428517&repo=autoland&lineNumber=8343

09:01:05 INFO - GECKO(4300) | [ACTION] NODE_PROPERTIES_LOADED - {}
09:01:05 INFO - GECKO(4300) | [GFX1-]: Failed buffer for 0, 0, 1280, 809
09:01:05 INFO - GECKO(4300) | [ACTION] COMMAND [start] - {"type":"COMMAND","command":"resume","thread":"server1.conn60.child1/context22","seqId":"420","status":"start"}
09:01:05 INFO - GECKO(4300) | [GFX1-]: Failed buffer for 0, 0, 1280, 809
09:01:05 INFO - GECKO(4300) | [ACTION] RESUME - {"type":"RESUME","thread":"server1.conn60.child1/context22","wasStepping":false}
09:01:05 INFO - GECKO(4300) | [ACTION] EVALUATE_EXPRESSIONS - {"type":"EVALUATE_EXPRESSIONS","inputs":[],"results":[]}
09:01:05 INFO - GECKO(4300) | [GFX1-]: Failed buffer for 0, 0, 1280, 809
09:01:05 INFO - GECKO(4300) | [ACTION] COMMAND [done] - {"type":"COMMAND","command":"resume","thread":"server1.conn60.child1/context22","seqId":"420","status":"done","value":{"from":"server1.conn60.child1/context22","type":"resumed"}}
09:01:05 INFO - GECKO(4300) | [ACTION] ADD_TAB - {"type":"ADD_TAB","url":"webpack3-babel6://./type-script-cjs/input.js","isOriginal":true,"sourceId":"sourceURL-http://example.com/browser/devtools/client/debugger/new/test/mochitest/examples/sourcemapped/output/webpack3-babel6/type-script-cjs.js/originalSource-0b5c1c10e7a449eca952f1295d0e315b"}
09:01:05 INFO - GECKO(4300) | [ACTION] SET_SELECTED_LOCATION - {"type":"SET_SELECTED_LOCATION","source":{"url":"webpack3-babel6://./type-script-cjs/input.js","relativeUrl":"webpack3-babel6://./type-script-cjs/input.js","id":"sourceURL-http://example.com/browser/devtools/client/debugger/new/test/mochitest/examples/sourcemapped/output/webpack3-babel6/type-script-cjs.js/originalSource-0b5c1c10e7a449eca952f1295d0e315b","isPrettyPrinted":false,"isWasm":false,"isBlackBoxed":false,"loadedState":"unloaded","introductionUrl":null,"isExtension":false,"actors":[]},"location":{"sourceId":"sourceURL-http://example.com/browser/devtools/client/debugger/new/test/mochitest/examples/sourcemapped/output/webpack3-babel6/type-script-cjs.js/originalSource-0b5c1c10e7a449eca952f1295d0e315b"}}

09:01:05 INFO - GECKO(4300) | [ACTION] LOAD_SOURCE_TEXT [start] - {"type":"LOAD_SOURCE_TEXT","sourceId":"sourceURL-http://example.com/browser/devtools/client/debugger/new/test/mochitest/examples/sourcemapped/output/webpack3-babel6/type-script-cjs.js/originalSource-0b5c1c10e7a449eca952f1295d0e315b","seqId":"421","status":"start"}
09:01:05 INFO - GECKO(4300) | Assertion failure: [unhandlable oom] Failed not allocate new chunk during GC, at z:/build/build/src/js/src/vm/JSContext.cpp:1521
09:01:06 INFO - TEST-INFO | Main app process: exit 1
09:01:06 INFO - Buffered messages logged at 08:59:31
09:01:06 INFO - Entering test bound
09:01:06 INFO - Adding a new tab with URL: http://example.com/browser/devtools/client/debugger/new/test/mochitest/examples/doc-sourcemapped.html
09:01:06 INFO - Buffered messages logged at 08:59:32
09:01:06 INFO - Tab added and finished loading
09:01:06 INFO - Opening the toolbox
09:01:06 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xmlns." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
09:01:06 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xmlns." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
09:01:06 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xmlns." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
09:01:06 INFO - Console message: [JavaScript Warning: "Removed unsafe attribute. Element: svg. Attribute: xmlns." {file: "resource://devtools/client/shared/vendor/react-dom.js" line: 5811}]
09:01:06 INFO - Toolbox opened and focused
09:01:06 INFO - Waiting on sources: parcel://./babel-bindings-with-flow/input.
09:01:06 INFO - Waiting for state change: source parcel://./babel-bindings-with-flow/input. exists
09:01:06 INFO - Buffered messages logged at 08:59:35
09:01:06 INFO - Finished waiting for state change: source parcel://./babel-bindings-with-flow/input. exists
09:01:06 INFO - Finished waiting on sources: parcel://./babel-bindings-with-flow/input.
09:01:06 INFO - TEST-PASS | devtools/client/debugger/new/test/mochitest/browser_dbg-sourcemapped-scopes.js | Original sources exist -
09:01:06 INFO - Waiting for state change: selected source
09:01:06 INFO - Finished waiting for state change: selected source
09:01:06 INFO - Waiting for ADD_BREAKPOINT to dispatch 1 time(s)
09:01:06 INFO - ADD_BREAKPOINT dispatched 1 time(s)
09:01:06 INFO - TEST-PASS | devtools/client/debugger/new/test/mochitest/browser_dbg-sourcemapped-scopes.js | One breakpoint exists -

Paul are there updates here?

There are 36 total failures in the last 7 days mostly on windows7-32 pgo.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=232937597&repo=autoland&lineNumber=52295

22:50:55 INFO - REFTEST TEST-START | file:///Z:/task_1552169738/build/tests/jsreftest/tests/jsreftest.html?test=test262/language/statements/async-generator/dstr/dflt-ary-init-iter-close.js
22:50:55 INFO - REFTEST TEST-LOAD | file:///Z:/task_1552169738/build/tests/jsreftest/tests/jsreftest.html?test=test262/language/statements/async-generator/dstr/dflt-ary-init-iter-close.js | 13025 / 21328 (61%)
22:50:55 INFO - Assertion failure: [unhandlable oom] Failed not allocate new chunk during GC, at z:/build/build/src/js/src/vm/JSContext.cpp:1456
22:50:56 INFO - ###!!! [Parent][MessageChannel] Error: (msgtype=0x1F009A,name=PBrowser::Msg_UpdateNativeWindowHandle) Channel error: cannot send/recv
22:50:56 INFO - ###!!! [Parent][MessageChannel] Error: (msgtype=0x1F0089,name=PBrowser::Msg_Destroy) Channel error: cannot send/recv
22:50:56 INFO - A content process crashed and MOZ_CRASHREPORTER_SHUTDOWN is set, shutting down
22:50:56 INFO - ###!!! [Parent][MessageChannel] Error: (msgtype=0x35003E,name=PContent::Msg_NotifyProcessPriorityChanged) Channel error: cannot send/recv
22:50:56 INFO - [Parent 6040, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
22:50:56 INFO - [JavaScript error: resource://reftest/reftest.jsm, line 1558: NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIPropertyBag2.getPropertyAsAString]
22:50:56 INFO - !!! error running onStopped callback: TypeError: callback is not a function
22:50:56 INFO - [Parent 6040, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
22:50:56 INFO - [Child 384, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
22:50:56 INFO - [Child 384, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
22:50:56 INFO - JavaScript error: resource://reftest/reftest.jsm, line 1558: NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIPropertyBag2.getPropertyAsAString]
22:50:56 INFO - 1552171856463 Marionette TRACE Received observer notification xpcom-will-shutdown
22:50:56 INFO - 1552171856463 Marionette INFO Stopped listening on port 2828
22:50:56 INFO - 1552171856463 Marionette DEBUG Remote service is inactive
22:50:56 INFO - REFTEST INFO | Downloading symbols from: https://queue.taskcluster.net/v1/task/Hh0GLa0IRMyrXSSdwemYEw/artifacts/public/build/target.crashreporter-symbols.zip
22:51:01 INFO - REFTEST INFO | Copy/paste: Z:\task_1552169738\build\win32-minidump_stackwalk.exe c:\users\task_1552169738\appdata\local\temp\tmpb1yl6v.mozrunner\minidumps\0fe1b94b-3e01-45c4-b500-546097dc9a53.dmp c:\users\task_1552169738\appdata\local\temp\tmplplmh_
22:51:16 INFO - REFTEST INFO | Saved minidump as Z:\task_1552169738\build\blobber_upload_dir\0fe1b94b-3e01-45c4-b500-546097dc9a53.dmp
22:51:16 INFO - REFTEST INFO | Saved app info as Z:\task_1552169738\build\blobber_upload_dir\0fe1b94b-3e01-45c4-b500-546097dc9a53.extra
22:51:16 INFO - REFTEST PROCESS-CRASH | file:///Z:/task_1552169738/build/tests/jsreftest/tests/jsreftest.html?test=test262/language/statements/async-generator/dstr/dflt-ary-init-iter-close.js | application crashed [@ js::AutoEnterOOMUnsafeRegion::crash(char const *)]
22:51:16 INFO - Crash dump filename: c:\users\task_1552169738\appdata\local\temp\tmpb1yl6v.mozrunner\minidumps\0fe1b94b-3e01-45c4-b500-546097dc9a53.dmp
22:51:16 INFO - Operating system: Windows NT
22:51:16 INFO - 6.1.7601 Service Pack 1
22:51:16 INFO - CPU: x86
22:51:16 INFO - GenuineIntel family 6 model 63 stepping 2
22:51:16 INFO - 8 CPUs
22:51:16 INFO -
22:51:16 INFO - GPU: UNKNOWN
22:51:16 INFO -
22:51:16 INFO - Crash reason: EXCEPTION_BREAKPOINT
22:51:16 INFO - Crash address: 0x580c8591
22:51:16 INFO - Assertion: Unknown assertion type 0x00000000
22:51:16 INFO - Process uptime: 563 seconds
22:51:16 INFO -
22:51:16 INFO - Thread 0 (crashed)
22:51:16 INFO - 0 xul.dll!js::AutoEnterOOMUnsafeRegion::crash(char const *) [JSContext.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 1457 + 0x0]
22:51:16 INFO - eip = 0x580c8591 esp = 0x002be0a8 ebp = 0x002be4b0 ebx = 0x00000000
22:51:16 INFO - esi = 0x002be0a8 edi = 0x5bff5dba eax = 0x747aaa3c ecx = 0x000005b1
22:51:16 INFO - edx = 0x67c3e340 efl = 0x00000202
22:51:16 INFO - Found by: given as instruction pointer in context
22:51:16 INFO - 1 xul.dll!js::AutoEnterOOMUnsafeRegion::crash(unsigned int,char const *) [JSContext.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 1470 + 0x6]
22:51:16 INFO - eip = 0x580d9880 esp = 0x002be4b8 ebp = 0x002be4c0 esi = 0x5bff5dba
22:51:16 INFO - edi = 0x05e01190
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 2 xul.dll!js::TenuringTracer::moveToTenuredSlow(JSObject *) [Marking.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 3010 + 0x1a]
22:51:16 INFO - eip = 0x58293b51 esp = 0x002be4c8 ebp = 0x002be4ec esi = 0x002be624
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 3 xul.dll!class mozilla::Maybe<JS::Value> js::MapGCThingTyped<`lambda at z:/build/build/src/js/src/gc/Marking.cpp:2730:43'>(const union JS::Value & const, class js::TenuringTracer::traverse<JS::Value>::<unnamed-tag> *) [Value.h:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 1323 + 0x16]
22:51:16 INFO - eip = 0x582a6df2 esp = 0x002be4f4 ebp = 0x002be514 ebx = 0x00000000
22:51:16 INFO - esi = 0x002be548 edi = 0x002be544
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 4 xul.dll!js::TenuringTracer::traceObjectSlots(js::NativeObject *,unsigned int,unsigned int) [Marking.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 2940 + 0x34]
22:51:16 INFO - eip = 0x58294304 esp = 0x002be51c ebp = 0x002be56c ebx = 0x002be624
22:51:16 INFO - esi = 0x000000de edi = 0x6060eee8
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 5 xul.dll!js::gc::StoreBuffer::SlotsEdge::trace(js::TenuringTracer &) [Marking.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 2793 + 0xa]
22:51:16 INFO - eip = 0x58293d79 esp = 0x002be574 ebp = 0x002be5c8 ebx = 0x002be624
22:51:16 INFO - esi = 0x00000001 edi = 0x000000ed
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 6 xul.dll!js::gc::StoreBuffer::MonoTypeBuffer<js::gc::StoreBuffer::SlotsEdge>::trace(js::TenuringTracer &) [Marking.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 2749 + 0xd]
22:51:16 INFO - eip = 0x58295ead esp = 0x002be5d0 ebp = 0x002be5e4 ebx = 0x0edc84b0
22:51:16 INFO - esi = 0x0edd0e10 edi = 0x0ede8000
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 7 xul.dll!js::Nursery::doCollection(JS::GCReason,js::gc::TenureCountCache &) [Nursery.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 942 + 0x6]
22:51:16 INFO - eip = 0x582ac43a esp = 0x002be5ec ebp = 0x002be690 ebx = 0x002be660
22:51:16 INFO - esi = 0x06408198 edi = 0x064080e0
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 8 xul.dll!js::Nursery::collect(JS::GCReason) [Nursery.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 783 + 0xc]
22:51:16 INFO - eip = 0x582ab414 esp = 0x002be698 ebp = 0x002be7c8 ebx = 0x064080e0
22:51:16 INFO - esi = 0x00000009 edi = 0x06408300
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 9 xul.dll!js::gc::GCRuntime::minorGC(JS::GCReason,js::gcstats::PhaseKind) [GC.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 7787 + 0x8]
22:51:16 INFO - eip = 0x5828ac1b esp = 0x002be7d0 ebp = 0x002be7f0 ebx = 0x00000000
22:51:16 INFO - esi = 0x064062c0 edi = 0x55733bf4
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 10 xul.dll!js::gc::GCRuntime::gcCycle(bool,js::SliceBudget,JS::GCReason) [GC.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 7364 + 0xd]
22:51:16 INFO - eip = 0x5828a75e esp = 0x002be7f8 ebp = 0x002be880 ebx = 0x002be818
22:51:16 INFO - esi = 0x00000001 edi = 0x064062c0
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 11 xul.dll!js::gc::GCRuntime::collect(bool,js::SliceBudget,JS::GCReason) [GC.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 7568 + 0x56]
22:51:16 INFO - eip = 0x5828b20f esp = 0x002be888 ebp = 0x002be910 ebx = 0x00000004
22:51:16 INFO - esi = 0x064062c0 edi = 0x00000001
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 12 xul.dll!class JSFatInlineString * js::gc::GCRuntime::tryNewTenuredThing<JSFatInlineString,js::CanGC>(struct JSContext *, js::gc::AllocKind, unsigned int) [Allocator.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 278 + 0x97]
22:51:16 INFO - eip = 0x58296974 esp = 0x002be918 ebp = 0x002be9ac ebx = 0x00000000
22:51:16 INFO - esi = 0x00000000 edi = 0x0640f800
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 13 xul.dll!class JSObject * js::AllocateObject<js::CanGC>(struct JSContext *, js::gc::AllocKind, unsigned int, js::gc::InitialHeap, const struct js::Class *) [Allocator.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 75 + 0x1a]
22:51:16 INFO - eip = 0x5829810d esp = 0x002be9b4 ebp = 0x002bea1c ebx = 0x0640f800
22:51:16 INFO - esi = 0x064062c0 edi = 0x00000000
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 14 xul.dll!static class JSObject * NewObject(struct JSContext *, class JS::Handle<js::ObjectGroup *>, js::gc::AllocKind, js::NewObjectKind, unsigned int) [JSObject.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 799 + 0x18]
22:51:16 INFO - eip = 0x580e3433 esp = 0x002bea24 ebp = 0x002bea6c ebx = 0x00000000
22:51:16 INFO - esi = 0x002bea98 edi = 0x0640f800
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 15 xul.dll!js::NewObjectWithClassProtoCommon(JSContext *,js::Class const *,JS::Handle<JSObject *>,js::gc::AllocKind,js::NewObjectKind) [JSObject.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 934 + 0x10]
22:51:16 INFO - eip = 0x580e3824 esp = 0x002bea74 ebp = 0x002beaac ebx = 0x5b802f00
22:51:16 INFO - esi = 0x00000000 edi = 0x0640f800
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 16 xul.dll!js::NewFunctionWithProto(JSContext ,bool ()(JSContext *,unsigned int,JS::Value *),unsigned int,JSFunction::Flags,JS::Handle<JSObject *>,JS::Handle<JSAtom *>,JS::Handle<JSObject *>,js::gc::AllocKind,js::NewObjectKind) [JSFunction.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 2094 + 0x14]
22:51:16 INFO - eip = 0x580de761 esp = 0x002beab4 ebp = 0x002beae0 ebx = 0x00000000
22:51:16 INFO - esi = 0x0640f800 edi = 0x0640f810
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 17 xul.dll!static bool DefineAccessorPropertyById(struct JSContext *, class JS::Handle<JSObject *>, class JS::Handle<JS::PropertyKey>, const struct JSNativeWrapper & const, const struct JSNativeWrapper & const, unsigned int) [jsapi.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 1908 + 0x1e]
22:51:16 INFO - eip = 0x58223be9 esp = 0x002beae8 ebp = 0x002beb50 ebx = 0x5bb4ba28
22:51:16 INFO - esi = 0x0640f800 edi = 0x0640f810
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 18 xul.dll!JS_DefineProperties(JSContext *,JS::Handle<JSObject *>,JSPropertySpec const *) [jsapi.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 2942 + 0x16]
22:51:16 INFO - eip = 0x5822809f esp = 0x002beb58 ebp = 0x002bec04 ebx = 0x5bb4ba30
22:51:16 INFO - esi = 0x0640f800 edi = 0x002beb84
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 19 xul.dll!bool mozilla::dom::DefinePrefable<const JSPropertySpec>(struct JSContext *, class JS::Handle<JSObject *>, const struct mozilla::dom::Prefable<const JSPropertySpec> *) [BindingUtils.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 655 + 0xb]
22:51:16 INFO - eip = 0x5a16fa92 esp = 0x002bec0c ebp = 0x002bec28 ebx = 0x5bb47a94
22:51:16 INFO - esi = 0x00000000 edi = 0x0640f800
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 20 xul.dll!mozilla::dom::DefineProperties(JSContext *,JS::Handle<JSObject *>,mozilla::dom::NativePropertiesN<7> const *,mozilla::dom::NativePropertiesN<7> const *) [BindingUtils.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 914 + 0xb]
22:51:16 INFO - eip = 0x5780ef12 esp = 0x002bec30 ebp = 0x002bec48 ebx = 0x5bb4b4e0
22:51:16 INFO - esi = 0x0640f800 edi = 0x002bec94
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 21 xul.dll!mozilla::dom::CreateInterfaceObjects(JSContext *,JS::Handle<JSObject *>,JS::Handle<JSObject *>,js::Class const *,JS::Heap<JSObject *> *,char const *,JS::Handle<JSObject *>,js::Class const *,unsigned int,mozilla::dom::NamedConstructor const *,JS::Heap<JSObject *> *,mozilla::dom::NativePropertiesN<7> const *,mozilla::dom::NativePropertiesN<7> const *,char const *,bool,char const * const *,bool) [BindingUtils.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 987 + 0x53]
22:51:16 INFO - eip = 0x57808599 esp = 0x002bec50 ebp = 0x002becb4 ebx = 0x0640f800
22:51:16 INFO - esi = 0x00000000 edi = 0x00000000
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 22 xul.dll!mozilla::dom::HTMLElement_Binding::CreateInterfaceObjects(JSContext *,JS::Handle<JSObject *>,mozilla::dom::ProtoAndIfaceCache &,bool) [HTMLElementBinding.cpp: : 9266 + 0x33]
22:51:16 INFO - eip = 0x57a8ea82 esp = 0x002becbc ebp = 0x002bed24 ebx = 0x62accf80
22:51:16 INFO - esi = 0x62acc388 edi = 0x62acd078
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 23 xul.dll!mozilla::dom::HTMLScriptElement_Binding::CreateInterfaceObjects(JSContext *,JS::Handle<JSObject *>,mozilla::dom::ProtoAndIfaceCache &,bool) [HTMLScriptElementBinding.cpp: : 1338 + 0x45]
22:51:16 INFO - eip = 0x57df70c6 esp = 0x002bed2c ebp = 0x002bed60 ebx = 0x62acc000
22:51:16 INFO - esi = 0x002bed44 edi = 0x72668ab0
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 24 xul.dll!mozilla::dom::HTMLScriptElement_Binding::Wrap(JSContext *,mozilla::dom::HTMLScriptElement *,nsWrapperCache *,JS::Handle<JSObject *>,JS::MutableHandle<JSObject *>) [HTMLScriptElementBinding.cpp: : 1281 + 0x4a]
22:51:16 INFO - eip = 0x57df6bed esp = 0x002bed68 ebp = 0x002bedd0 ebx = 0x6a109040
22:51:16 INFO - esi = 0x72668ab0 edi = 0x002bed7c
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 25 xul.dll!class JSObject * mozilla::dom::HTMLScriptElement::WrapNode(struct JSContext *, class JS::Handle<JSObject *>) [HTMLScriptElement.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 34 + 0x31]
22:51:16 INFO - eip = 0x57df6867 esp = 0x002bedd8 ebp = 0x002bee04 ebx = 0x0640f801
22:51:16 INFO - esi = 0x002bedf4 edi = 0x72436c84
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 26 xul.dll!nsINode::WrapObject(JSContext *,JS::Handle<JSObject *>) [nsINode.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 2624 + 0xc]
22:51:16 INFO - eip = 0x57907447 esp = 0x002bee0c ebp = 0x002bee30 esi = 0x72436c84
22:51:16 INFO - edi = 0x0640f800
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 27 xul.dll!mozilla::dom::Element::WrapObject(JSContext *,JS::Handle<JSObject *>) [Element.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 557 + 0x9]
22:51:16 INFO - eip = 0x57986eb5 esp = 0x002bee38 ebp = 0x002bee7c ebx = 0x0640f800
22:51:16 INFO - esi = 0x5b93e4d8 edi = 0x72436c84
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 28 xul.dll!XPCConvert::NativeInterface2JSObject(JS::MutableHandle<JS::Value>,xpcObjectHelper &,nsID const *,bool,nsresult *) [XPCConvert.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 960 + 0xc]
22:51:16 INFO - eip = 0x577eef22 esp = 0x002bee84 ebp = 0x002bef00 ebx = 0x00000000
22:51:16 INFO - esi = 0x5b93e4d8 edi = 0x72436c84
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 29 xul.dll!static nsresult NativeInterface2JSObject(class JS::Handle<JSObject *>, class nsISupports *, class nsWrapperCache *, const struct nsID *, bool, class JS::MutableHandle<JS::Value>) [nsXPConnect.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 579 + 0x18]
22:51:16 INFO - eip = 0x59457028 esp = 0x002bef08 ebp = 0x002bef74 ebx = 0x00000000
22:51:16 INFO - esi = 0x00fac6d4 edi = 0x72436cc8
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 30 xul.dll!nsresult nsXPConnect::WrapNativeToJSVal(struct JSContext *, class JSObject *, class nsISupports *, class nsWrapperCache *, const struct nsID *, bool, class JS::MutableHandle<JS::Value>) [nsXPConnect.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 625 + 0x9]
22:51:16 INFO - eip = 0x5781493d esp = 0x002bef7c ebp = 0x002befa8 ebx = 0x00000001
22:51:16 INFO - esi = 0x00000000 edi = 0x002befe0
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 31 xul.dll!nsresult mozilla::dom::ScriptLoader::FillCompileOptionsForRequest(const class mozilla::dom::AutoJSAPI & const, class mozilla::dom::ScriptLoadRequest *, class JS::Handle<JSObject *>, class JS::CompileOptions *) [ScriptLoader.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 2362 + 0x38]
22:51:16 INFO - eip = 0x57df670f esp = 0x002befb0 ebp = 0x002bf010 ebx = 0x0640f800
22:51:16 INFO - esi = 0x002befe0 edi = 0x72436cc8
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 32 xul.dll!nsresult mozilla::dom::ScriptLoader::AttemptAsyncScriptCompile(class mozilla::dom::ScriptLoadRequest *, bool *) [ScriptLoader.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 2010 + 0x13]
22:51:16 INFO - eip = 0x57ea41cb esp = 0x002bf018 ebp = 0x002bf104 ebx = 0x5426f400
22:51:16 INFO - esi = 0x002bf0cc edi = 0x002bf03c
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 33 xul.dll!nsresult mozilla::dom::ScriptLoadHandler::OnStreamComplete(class nsIIncrementalStreamLoader *, class nsISupports *, nsresult, unsigned int, const unsigned char *) [ScriptLoadHandler.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 417 + 0x4e]
22:51:16 INFO - eip = 0x57ea32e5 esp = 0x002bf10c ebp = 0x002bf238 ebx = 0x62aedbf0
22:51:16 INFO - esi = 0x5426f4ec edi = 0x5426f400
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 34 xul.dll!nsresult nsIncrementalStreamLoader::OnStopRequest(class nsIRequest *, nsresult) [nsIncrementalStreamLoader.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 85 + 0xf]
22:51:16 INFO - eip = 0x5799cb40 esp = 0x002bf240 ebp = 0x002bf26c ebx = 0x00000000
22:51:16 INFO - esi = 0x72458000 edi = 0x72422ca0
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 35 xul.dll!nsBaseChannel::OnStopRequest(nsIRequest *,nsresult) [nsBaseChannel.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 780 + 0xd]
22:51:16 INFO - eip = 0x57980dff esp = 0x002bf274 ebp = 0x002bf288 ebx = 0x002bf29c
22:51:16 INFO - esi = 0x62aedbd0 edi = 0x62aedc14
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 36 xul.dll!unsigned int nsInputStreamPump::OnStateStop() [nsInputStreamPump.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 655 + 0xa]
22:51:16 INFO - eip = 0x57980aa1 esp = 0x002bf290 ebp = 0x002bf2b0 esi = 0x6edf1c80
22:51:16 INFO - edi = 0x6edf1cdc
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 37 xul.dll!nsresult nsInputStreamPump::OnInputStreamReady(class nsIAsyncInputStream *) [nsInputStreamPump.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 403 + 0x5]
22:51:16 INFO - eip = 0x579673cc esp = 0x002bf2b8 ebp = 0x002bf30c ebx = 0x6edf1c84
22:51:16 INFO - esi = 0x6edf1cdc edi = 0x72689064
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 38 xul.dll!nsresult nsInputStreamReadyEvent::Run() [nsStreamUtils.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 91 + 0x7]
22:51:16 INFO - eip = 0x5796718c esp = 0x002bf314 ebp = 0x002bf320 ebx = 0x00f0b100
22:51:16 INFO - esi = 0x72422dc0 edi = 0x72675ec0
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 39 xul.dll!nsresult mozilla::SchedulerGroup::Runnable::Run() [SchedulerGroup.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 295 + 0x6]
22:51:16 INFO - eip = 0x58ddf259 esp = 0x002bf328 ebp = 0x002bf334 esi = 0x002bf340
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 40 xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 1179 + 0x9]
22:51:16 INFO - eip = 0x577a8819 esp = 0x002bf33c ebp = 0x002bf818 esi = 0x002bf340
22:51:16 INFO - edi = 0x00000000
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 41 xul.dll!NS_ProcessNextEvent(nsIThread *,bool) [nsThreadUtils.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 482 + 0x11]
22:51:16 INFO - eip = 0x577a830b esp = 0x002bf820 ebp = 0x002bf83c ebx = 0x00f50101
22:51:16 INFO - esi = 0x002bf82f edi = 0x00f012c0
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 42 xul.dll!void mozilla::ipc::MessagePump::Run(class base::MessagePump::Delegate *) [MessagePump.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 88 + 0xa]
22:51:16 INFO - eip = 0x5794e857 esp = 0x002bf844 ebp = 0x002bf864 ebx = 0x00f50101
22:51:16 INFO - esi = 0x00f012b0
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 43 xul.dll!MessageLoop::RunHandler() [message_loop.cc:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 308 + 0x8]
22:51:16 INFO - eip = 0x57780084 esp = 0x002bf86c ebp = 0x002bf89c ebx = 0x00f5b400
22:51:16 INFO - esi = 0x002bf980 edi = 0x00ff90d0
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 44 xul.dll!MessageLoop::Run() [message_loop.cc:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 290 + 0x5]
22:51:16 INFO - eip = 0x577a7fa1 esp = 0x002bf8a4 ebp = 0x002bf8bc ebx = 0x00f5b400
22:51:16 INFO - esi = 0x00f0b100 edi = 0x00ff90d0
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 45 xul.dll!nsBaseAppShell::Run() [nsBaseAppShell.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 137 + 0xc]
22:51:16 INFO - eip = 0x5794e745 esp = 0x002bf8c4 ebp = 0x002bf8cc
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 46 xul.dll!nsAppShell::Run() [nsAppShell.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 411 + 0x6]
22:51:16 INFO - eip = 0x5794e0f0 esp = 0x002bf8d4 ebp = 0x002bf8dc esi = 0x00ff90d0
22:51:16 INFO - edi = 0x00f5b400
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 47 xul.dll!XRE_RunAppShell() [nsEmbedFunctions.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 911 + 0x6]
22:51:16 INFO - eip = 0x5b7897a7 esp = 0x002bf8e4 ebp = 0x002bf8f4 esi = 0x00f012b0
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 48 xul.dll!mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate *) [MessagePump.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 238 + 0x5]
22:51:16 INFO - eip = 0x591642b8 esp = 0x002bf8fc ebp = 0x002bf90c esi = 0x00f012b0
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 49 xul.dll!MessageLoop::RunHandler() [message_loop.cc:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 308 + 0x8]
22:51:16 INFO - eip = 0x57780084 esp = 0x002bf914 ebp = 0x002bf944 ebx = 0x00f5b400
22:51:16 INFO - esi = 0x002bf980 edi = 0x00f5b400
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 50 xul.dll!MessageLoop::Run() [message_loop.cc:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 290 + 0x5]
22:51:16 INFO - eip = 0x577a7fa1 esp = 0x002bf94c ebp = 0x002bf964 ebx = 0x00f5b400
22:51:16 INFO - esi = 0x80004005 edi = 0x00f5b400
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 51 xul.dll!XRE_InitChildProcess(int,char * * const,XREChildData const *) [nsEmbedFunctions.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 749 + 0x5]
22:51:16 INFO - eip = 0x5b7894aa esp = 0x002bf96c ebp = 0x002bfa80
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 52 xul.dll!nsresult mozilla::BootstrapImpl::XRE_InitChildProcess(int, char * *, const struct XREChildData *) [Bootstrap.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 61 + 0xe]
22:51:16 INFO - eip = 0x5b78d481 esp = 0x002bfa88 ebp = 0x002bfa94 ebx = 0x00f03040
22:51:16 INFO - esi = 0x0000001a edi = 0x00f07118
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 53 firefox.exe!static int content_process_main(class mozilla::Bootstrap *, int, char * *) [plugin-container.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 49 + 0xd]
22:51:16 INFO - eip = 0x010814a1 esp = 0x002bfa9c ebp = 0x002bfac0
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 54 firefox.exe!static int NS_internal_main(int, char * *, char * *) [nsBrowserApp.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 265 + 0xa]
22:51:16 INFO - eip = 0x0108140a esp = 0x002bfac8 ebp = 0x002bfb30 ebx = 0x00f030b0
22:51:16 INFO - esi = 0x0000001b edi = 0x00f03040
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 55 firefox.exe!wmain [nsWindowsWMain.cpp:fc3a2f173e667a9ce3ca641f70d9562bdf355bb1 : 131 + 0x12]
22:51:16 INFO - eip = 0x010810e2 esp = 0x002bfb38 ebp = 0x002bfb54 ebx = 0x00f030b0
22:51:16 INFO - esi = 0x0000007f edi = 0x00f03040
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 56 firefox.exe!static int __scrt_common_main_seh() [exe_common.inl : 288 + 0x1c]
22:51:16 INFO - eip = 0x010b9edc esp = 0x002bfb5c ebp = 0x002bfb9c ebx = 0x7ffdf000
22:51:16 INFO - esi = 0x67c3f0b0 edi = 0x00a62a60
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 57 kernel32.dll!BemFreeContract + 0x135b9
22:51:16 INFO - eip = 0x76ae3c45 esp = 0x002bfba4 ebp = 0x002bfba8
22:51:16 INFO - Found by: call frame info
22:51:16 INFO - 58 ntdll.dll!BemFreeContract + 0x23169
22:51:16 INFO - eip = 0x77c537f5 esp = 0x002bfbb0 ebp = 0x002bfbe8
22:51:16 INFO - Found by: previous frame's frame pointer
22:51:16 INFO - 59 ntdll.dll!BemFreeContract + 0x2313c
22:51:16 INFO - eip = 0x77c537c8 esp = 0x002bfbf0 ebp = 0x002bfc00
22:51:16 INFO - Found by: previous frame's frame pointer

No updates, I'll make this my task this afternoon (just finishing previous task now) and tomorrow.

FYI: I'm away this Friday, back on Monday.

I'd close this as a duplicate of Bug 1472062 (which we know is difficult to fix) but it'd be good to work-around this issue so that it's no-longer occuring in testing.

Done some retriggers:
https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=9701348e4f0c2e431e0f81417e86da16b5f1cb4a&tochange=d7e2fa30d85fe1875a8c9698a35e5a771281ae8f&fromchange=56159f3c0acadb0b59437d61402c569c76e38984&selectedJob=233648313

baku: can you please take a look at this? Might be https://bugzilla.mozilla.org/show_bug.cgi?id=1531071 the culprit?

What I'd like is for the crash reporter to include information about the JavaScript heap layout at the time of the crash.

We've got a general trend of OOM-while-evicting-nursery crashes (Bug 1472062, Bug 1531071 and this bug) and although this and Bug 1531071 can occur in the same tests, there's still a general issue. I think our options are:

• Keep applying simple work-arounds each time this happens such as Bug 1531071, (eg a call to gc() at the beginning of a
  test).
• Implement the proposed solution in Bug 1472062, which is quite a bit of engineering. (It aborts nursery eviction and
  continues only with marking-through the nursery so that the tenured heap can be collected (non-incrementally) then
  reattempts nursery eviction and has to re-sweep the tenured heap due to moving.)
• Find some "better" work-around by getting an idea about what the heap looks like when these crashes happen. I think
  this requires getting some information about the heap usage/layout and virtual memory layout at the time of the crash.

I'd like #2 or #3. #3 has a chance of being easier to implement.

I will investigate #3 a little bit, but given that we're seeing this type of OOM in different places we should consider #2 a bit more seriously (we previously put it off as hard/not worth-while).

Paul, the crash dumps do already contain the virtual memory layout. I'll reach out to you with some tools that I've written before for this problem.

Hi Ted,

Yes, I expected that, I just havn't looked at it yet. What I also want was the same kind of information but at the level of the GC. So I can see what utilization is like and whether this could be a GC scheduling problem.

Jon and I decided to look at the general problem with OOMs during nursery eviction as part of Bug GCScheduling.

However it'd still be good to work around this intermittent, but I'm not sure how to gather more information. My suspicion is that there's fragmentation or garbage to be collected in some zones that aren't being collected and we're running out of virtual address space.

The tests that fail the most often appear to be:

devtools/client/webconsole/test/mochitest/browser_webconsole_multiple_windows_and_tabs.js
devtools/client/webconsole/test/mochitest/browser_webconsole_trackingprotection_errors.js
toolkit/mozapps/extensions/test/browser/browser_gmpProvider.js might be common.
and some jsreftest

I don't know about the last two but maybe the first two open multiple windows/tabs/iframes and maybe more zones. I'd like to learn more.

NI Steven, can you help me find someone from devtools to tell me more about the first two tests.

IMO this is a GC scheduling problem where we create too many zones and fail to collect them fast enough. We have seen similar things with other test suites and they have workarounds in place:

https://searchfox.org/mozilla-central/rev/7c20ad925005fbad7b8e08813115f1ec7fa1c248/layout/tools/reftest/reftest-content.js#156
https://searchfox.org/mozilla-central/rev/7c20ad925005fbad7b8e08813115f1ec7fa1c248/testing/mochitest/tests/SimpleTest/iframe-between-tests.html#14

We crash because things have gone too far; I don't think fixing this particular signature addresses the root cause.

Thanks Jan, that's what I was thinking of.

What I'm wondering is should we work around it in the most annoying cases (since they cause intermittents that may cause developers to miss other failures/crashes) or just wait (months) to fix the root cause? So it depends on how easy the work-around is, and that I don't know yet.

Paul: (response to NI comment 31) Asking around, it was suggested to try talking to Alexandre Poirot on the Dev Tools team as a starting point to learn more about the tests you listed.

Thanks Steven,

Alexandre, do you have any insight for Comment 31? Is there some behavior, maybe with windows/tabs/iframes that these tests do and others don't?

Yes, browser_webconsole_multiple_windows_and_tabs.js and browser_webconsole_trackingprotection_errors.js are both opening browser windows.

• browser_webconsole_trackingprotection_errors.js opens a new browser window at least 5 times and it also open new tab in each of these windows.
• browser_webconsole_multiple_windows_and_tabs.js opens one additional browser window. It opens 2 tabs in the already existing one and then 2 tabs in the additional one.

We don't have so many tests involving brand new browser window, but that's certainly not the only ones doing that.

But note that these OOM crashes became more frequent when we started sharing the same compartment for all DevTools.
We did that in bug 1517210 and I worked around one by putting some forced call to Cu.forceShrinkingGC():
https://searchfox.org/mozilla-central/search?q=Cu.forceShrinkingGC()&case=false&regexp=false&path=devtools%2F
There is a followup bug to try to figure this out (bug 1528986) but TBH that's outside of my expertise.

DevTools now shares the exact same compartment for everything: its commonjs modules + its chrome documents (toolbox and all the inner panels like console and inspector).
And it is important to note that DevTools shares the same compartment for all its ressources, but it is also the exact same one than browser.xul, JS XPCOM, JSMs... All system principals are now sharing the same compartment (bug 1512029 and bug 1514210). We got rid of zillions of cross compartment wrappers between each of these JS contexts! And so we also reduced the number of privileged compartments.(In reply to Jan de Mooij [:jandem] from comment #32)

IMO this is a GC scheduling problem where we create too many zones and fail to collect them fast enough. We have seen similar things with other test suites and they have workarounds in place:

"we create too many zones", I'm surprised with this conclusion. Didn't you drastically reduced the number of zones with your work on bug 1357862?
Otherwise, we recently realized that DevTools was suffering from significant leaks, that certainly doesn't help the GC to be efficient... When you close the tools, some bits of the frontend stay allocated. I don't know if these leaks stay allocated when you close the related browser window. These leaks are related to the fronted so it is privileged code running in the parent process.

(In reply to Alexandre Poirot [:ochameau] from comment #37)

"we create too many zones", I'm surprised with this conclusion. Didn't you drastically reduced the number of zones with your work on bug 1357862?

We still have a zone per top-level content document. Test harnesses usually load a ton of content pages in a single tab and we create a zone for each of them. That's intentional (bug 1527905) but it doesn't work great for such test suites.

(In reply to Jan de Mooij [:jandem] from comment #38)

(In reply to Alexandre Poirot [:ochameau] from comment #37)

"we create too many zones", I'm surprised with this conclusion. Didn't you drastically reduced the number of zones with your work on bug 1357862?

We still have a zone per top-level content document. Test harnesses usually load a ton of content pages in a single tab and we create a zone for each of them. That's intentional (bug 1527905) but it doesn't work great for such test suites.

Ah ok, yes, that makes sense.
I don't think that the two particular console tests here are creating many content documents. But some tests being run before can probably do that. As you say, some tests opens tons of tabs.

I ran these tests and didn't see a ton of zones being created (max was 40 present at once). I did see that there were twice as many zones created in an opt build vs a debug build though, which I can't explain.

(In reply to Alexandre Poirot [:ochameau] from comment #37)

Yes, browser_webconsole_multiple_windows_and_tabs.js and browser_webconsole_trackingprotection_errors.js are both opening browser windows.

• browser_webconsole_trackingprotection_errors.js opens a new browser window at least 5 times and it also open new tab in each of these windows.
• browser_webconsole_multiple_windows_and_tabs.js opens one additional browser window. It opens 2 tabs in the already existing one and then 2 tabs in the additional one.

We don't have so many tests involving brand new browser window, but that's certainly not the only ones doing that.

But note that these OOM crashes became more frequent when we started sharing the same compartment for all DevTools.
We did that in bug 1517210 and I worked around one by putting some forced call to Cu.forceShrinkingGC():
https://searchfox.org/mozilla-central/search?q=Cu.forceShrinkingGC()&case=false&regexp=false&path=devtools%2F
There is a followup bug to try to figure this out (bug 1528986) but TBH that's outside of my expertise.

Thanks, that's the kind of work-around I figured we'd need here.

IMO this is a GC scheduling problem where we create too many zones and fail to collect them fast enough. We have seen similar things with other test suites and they have workarounds in place:

"we create too many zones", I'm surprised with this conclusion. Didn't you drastically reduced the number of zones with your work on bug 1357862?

Not me.

So I was assuming that because these tests use new windows/tabs then they'd use new zones. I think that's how that works. it's more of a hypothesis than conclusion. And I sim-remembered Bug 1528986. I thought that was about there being more compartments and/or zones and therefore more (fragmented) memory. But it doesn't make sense to me that it OOMs more while using fewer compartments.

I'm confused and want to defer to Jon who may have a broader view of what's going on here. But it looks the workaround above might be best, it'd just be nice to understand why.

Otherwise, we recently realized that DevTools was suffering from significant leaks, that certainly doesn't help the GC to be efficient... When you close the tools, some bits of the frontend stay allocated. I don't know if these leaks stay allocated when you close the related browser window. These leaks are related to the fronted so it is privileged code running in the parent process.

Could be. Can we create a tests that opens and closes things (devtools and the window using devtools) 100s of times and tests to see if they get freed / memory grows too much? Is that reasonable or would it take too long to run such a test?

I'm not sure if this'll work, I'll check the crash rates in one week.

https://hg.mozilla.org/mozilla-central/rev/25c24beb21c8

I want to leave this open until IN can check that it's resolved. I forgot the leave-open keyword.

Looks like it's still failing.

Jon,

The remaining failures here are in test262. I assume there'd be a way to add a gc() between each major test or group but that seems quite 'heavy', I don't really want to have to do that.

I'm trying to think of what may have changed to cause these problems. Since it's an OOM while trying to evict the nursery I've got Bug 1407143 - don't collect the nursery every slice. And also back in 2017 we reduced the GC slice budget from 10ms to 5ms (I think, can't find the bug) without doing anything to make collections occur more often. I think this means that it'll take twice as long for a collection to complete, there may be cases where we run out of memory before an in-progress collection finishes.

Also for deciding when a collection happens based on either malloc bytes or GC bytes, I don't think the nursery is included. Since Bug 1407143 this means we'll run a slice less often because there's going to be more objects in the nursery.

What do you think? Is it reasonable to experiment with lowering the bytes threshold (and maybe malloc bytes) to get a slice to run sooner?

(In reply to Paul Bone [:pbone] from comment #50)

The remaining failures here are in test262. I assume there'd be a way to add a gc() between each major test or group but that seems quite 'heavy', I don't really want to have to do that.

Agreed, let's try and avoid that.

I'm trying to think of what may have changed to cause these problems. Since it's an OOM while trying to evict the nursery I've got Bug 1407143 - don't collect the nursery every slice.

It's possible that it triggered this issue. Note to observers: this would just have moved the OOMs from some other place, not resulted in any more OOMs overall.

And also back in 2017 we reduced the GC slice budget from 10ms to 5ms (I think, can't find the bug) without doing anything to make collections occur more often.

No, we mostly run GC in idle time now with a budget based on the amount of idle time remaining (upto some maximum). You can see the budget used in telemetry:

https://telemetry.mozilla.org/new-pipeline/evo.html#!aggregates=Median!Mean!5th%2520percentile!25th%2520percentile!75th%2520percentile!95th%2520percentile&cumulative=0&end_date=2019-04-16&include_spill=0&keys=!__none__!__none__&max_channel_version=nightly%252F68&measure=GC_BUDGET_MS&min_channel_version=nightly%252F65&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2019-03-18&trim=1&use_submission_date=0

This does show a reduction in slice budget since FF 67. I wonder what caused that?

Also for deciding when a collection happens based on either malloc bytes or GC bytes, I don't think the nursery is included.

Correct, nursery size is not included in GC bytes and 'malloc' allocations that are allocated from the nursery are not included in malloc bytes. Malloc allocations attached to GC things in the nursery are included in malloc bytes however.

Since Bug 1407143 this means we'll run a slice less often because there's going to be more objects in the nursery.

Yes that's possible. Looking at the size of the nursery over time:

https://telemetry.mozilla.org/new-pipeline/evo.html#!aggregates=Median!Mean!5th%2520percentile!25th%2520percentile!75th%2520percentile!95th%2520percentile&cumulative=0&end_date=2019-04-16&include_spill=0&keys=!__none__!__none__&max_channel_version=nightly%252F68&measure=GC_NURSERY_BYTES&min_channel_version=nightly%252F60&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2019-03-18&trim=1&use_submission_date=0

This shows we got a big increase following 63, but it has now fallen to less than the original value. So I don't think that can be the cause if this problem is still happening.

(In reply to Jon Coppeard (:jonco) from comment #51)

https://telemetry.mozilla.org/new-pipeline/evo.html#!aggregates=Median!Mean!5th%2520percentile!25th%2520percentile!75th%2520percentile!95th%2520percentile&cumulative=0&end_date=2019-04-16&include_spill=0&keys=!__none__!__none__&max_channel_version=nightly%252F68&measure=GC_BUDGET_MS&min_channel_version=nightly%252F65&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2019-03-18&trim=1&use_submission_date=0

This does show a reduction in slice budget since FF 67. I wonder what caused that?

I don't have any guesses.

I think I landed something around that time but it wouldn't have affected the budget.

Since Bug 1407143 this means we'll run a slice less often because there's going to be more objects in the nursery.

Yes that's possible. Looking at the size of the nursery over time:

https://telemetry.mozilla.org/new-pipeline/evo.html#!aggregates=Median!Mean!5th%2520percentile!25th%2520percentile!75th%2520percentile!95th%2520percentile&cumulative=0&end_date=2019-04-16&include_spill=0&keys=!__none__!__none__&max_channel_version=nightly%252F68&measure=GC_NURSERY_BYTES&min_channel_version=nightly%252F60&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2019-03-18&trim=1&use_submission_date=0

This shows we got a big increase following 63, but it has now fallen to less than the original value. So I don't think that can be the cause if this problem is still happening.

Without actually checking the dates I'd guess this increased when I landed the code to not collect the nursery every slice and dropped when I reduced the nursery's minimum size.

Okay, in that case I agree, I don't think not counting the nursery is having a sagnificant affect. and the first telemetry link shows slice budget is mostly still over 10ms. Maybe it's different in CI where these OOMs are occuring. Trying to collect more frequently may still be worth-while, not because the payoff is big but because the change is easy/small. I can at least see what it does in try..

There have been 38 failures within the last 7 days:

• 3 failures on Windows 7 opt
• 35 failures on Windows 7 Shippable opt

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=244626072&repo=mozilla-inbound&lineNumber=42820

00:42:25 INFO - REFTEST TEST-START | file:///Z:/task_1556925316/build/tests/jsreftest/tests/jsreftest.html?test=test262/built-ins/Date/prototype/toJSON/name.js
00:42:25 INFO - REFTEST TEST-LOAD | file:///Z:/task_1556925316/build/tests/jsreftest/tests/jsreftest.html?test=test262/built-ins/Date/prototype/toJSON/name.js | 7890 / 17353 (45%)
00:42:26 INFO - Assertion failure: [unhandlable oom] Failed not allocate new chunk during GC, at z:/task_1556912274/build/src/js/src/vm/JSContext.cpp:1476
00:42:26 INFO - ###!!! [Parent][MessageChannel] Error: (msgtype=0x1E009D,name=PBrowser::Msg_UpdateNativeWindowHandle) Channel error: cannot send/recv
00:42:26 INFO - ###!!! [Parent][MessageChannel] Error: (msgtype=0x1E008C,name=PBrowser::Msg_Destroy) Channel error: cannot send/recv
00:42:26 INFO - A content process crashed and MOZ_CRASHREPORTER_SHUTDOWN is set, shutting down
00:42:26 INFO - ###!!! [Parent][MessageChannel] Error: (msgtype=0x350040,name=PContent::Msg_NotifyProcessPriorityChanged) Channel error: cannot send/recv
00:42:26 INFO - [Parent 3748, Gecko_IOThread] WARNING: pipe error: 109: file z:/task_1556912274/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
00:42:26 INFO - [Child 5548, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1556912274/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
00:42:26 INFO - [Child 5548, Chrome_ChilJavaScript error: resource://reftest/reftest.jsm, line 1558: NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIPropertyBag2.getPropertyAsAString]
00:42:26 INFO - !!! error running onStopped callback: TypeError: callback is not a function
00:42:26 INFO - [Parent 3748, Gecko_IOThread] WARNING: pipe error: 109: file z:/task_1556912274/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
00:42:26 INFO - JavaScript error: resource://reftest/reftest.jsm, line 1558: NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIPropertyBag2.getPropertyAsAString]
00:42:26 INFO - [Child 3224, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1556912274/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
00:42:26 INFO - [Child 3224, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1556912274/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
00:42:26 INFO - JavaScript error: resource://reftest/reftest.jsm, line 1558: NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIPropertyBag2.getPropertyAsAString]

Paul, could you please take a look at what going on here? There are 110 failures in the last 30 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-07-21&endday=2019-08-20&tree=trunk&bug=1518138 all of them happen on windows7-32/shippable opt builds and on jsreftest-e10s test-suite
Recent failure log: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=ac0c1beee54f49efc49237a994acc7c63bf6fb7f&selectedJob=262341034

From the GC perspective the solutions to this all require a lot of engineering. We don't have a solution that's worth-while. Sorry.

Jon, what do you think, is this WONTFIX or just leave it at P5?

(In reply to pbone@mozilla.com from comment #76)
It looks like this is happening quite frequently on infra so we need to do something about it. This is OOM and there's not much we can do for that specifically. But I have a suspicion that what's causing this is that the reftest framework is leaking memory.

Given that it's mostly 32bit windows it's likely to be an address space thing. But I agree with you regarding something about the infrastructure / reftest framework.

This bug failed 31 times in the last 7 days. Occurs on windows7-32-shippable and windows7-32 on opt build types.

Recent log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=264359079&repo=autoland&lineNumber=45251

pbone: Any news about this bug?

(In reply to Dorel Luca [:dluca] from comment #80)
Hi, I'm looking into this now but I haven't been able to figure out a proper solution so far.

I think the problem is that lots of DOM windows / GC zones are being kept alive longer than expected and are living until the full GC timer fires, which is once a minute.

We currently have a hack in place to run more GC/CC during these tests (windowUtils.runNextCollectorTimer). Changing this to do a full GC/CC between every test reduces the number of live windows, but is a bit of a drastic solution.

This doesn't fix the underlying problem but makes RunNextCollectorTimer more aggressive in what it does. Testing shows this substantially reduces the max GC heap size while running these tests as well as vsize and RSS. Hopefully this will fix the problem with the tests failing while we work out a good solution for the underlying issue.

https://hg.mozilla.org/mozilla-central/rev/1bf0c1550fc1

There have been no new failures since this landed so I'm closing this bug. I filed bug 1579796 for the continuing investigation.

Looks like these failures are affecting Beta as well (but nothing on ESR68 so far at least). Can you please nominate this for Beta approval when you get a chance? Thanks!

Comment on attachment 9089867 [details]
Bug 1518138 - Make RunNextCollectorTimer trigger full GCs to address out of memory errors running JS reftests r?smaug

Beta/Release Uplift Approval Request

• User impact if declined: No user impact, but will reduce test failures in automation.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This only affects the reftest tests so there's no risk to the product.
• String changes made/needed:

Comment on attachment 9089867 [details]
Bug 1518138 - Make RunNextCollectorTimer trigger full GCs to address out of memory errors running JS reftests r?smaug

Test fix, helps us be able to rely better on automation results. OK for uplift for beta 6

https://hg.mozilla.org/releases/mozilla-beta/rev/df3bfb6c871b