https://treeherder.mozilla.org/logviewer.html#?job_id=16667007&repo=mozilla-inbound 05:20:57 WARNING - TEST-UNEXPECTED-FAIL | leakcheck | tab process: 7560 bytes leaked (AsyncLatencyLogger, AudioContext, AudioDestinationNode, AudioNode, AudioNodeEngine, ...)

This looks like another "MSG doesn't shut down good" bug.

jib: can you look at this? perhaps the second-level of AsyncShutdown use would help here. For an example, see https://treeherder.mozilla.org/logviewer.html#?job_id=20787243&repo=mozilla-inbound (mochitest-e10s-4) Paul, note it seems to be running webaudio tests.

Isn't this bug 1215265?

Looks like bug 1215265 would fix it: without https://treeherder.mozilla.org/#/jobs?repo=try&revision=8c5bfcd10286&selectedJob=16233665 with https://treeherder.mozilla.org/#/jobs?repo=try&revision=d661765d925f&selectedJob=16233660 I did a few more tries. Lets hope it still holds tomorrow.

No, it's a separate (smaller) leak: https://treeherder.mozilla.org/logviewer.html#?job_id=16252621&repo=try |<----------------Class--------------->|<-----Bytes------>|<----Objects---->| | | Per-Inst Leaked| Total Rem| 0 |TOTAL | 16 1836| 5635226 53| 14 |AsyncLatencyLogger | 44 44| 1 1| 22 |AudioContext | 200 200| 423 1| 23 |AudioDestinationNode | 144 144| 381 1| 25 |AudioNode | 88 88| 9119 1| 26 |AudioNodeEngine | 12 12| 8881 1| 28 |AudioNodeStream | 284 284| 8881 1| 83 |CondVar | 24 48| 1376 2| 99 |DOMEventTargetHelper | 40 80| 10123 2| 178 |GraphDriver | 52 52| 484 1| 248 |MediaSegment | 16 16| 6196 1| 251 |MediaStream | 144 288| 17882 2| 254 |MediaStreamGraph | 16 16| 229 1| 255 |MediaStreamGraphImpl | 328 328| 229 1| 273 |Mutex | 20 60| 5292 3| 402 |StreamBuffer | 28 28| 8941 1| 432 |Track | 20 20| 378 1| 883 |nsTArray_base | 4 128| 2470275 32|

TEST-UNEXPECTED-FAIL | leakcheck | tab process: 1836 bytes leaked (AsyncLatencyLogger, AudioContext, AudioDestinationNode, AudioNode, AudioNodeEngine, ...)

This is currently the #3 leak on OrangeFactor, and only occurs on e10s. Leak stack: 15:18:25 INFO - allocation stack: 15:18:50 INFO - #00: AsyncLatencyLogger::AddRef() [dom/media/Latency.cpp:134] 15:18:50 INFO - #01: nsCOMPtr<nsISupports>::nsCOMPtr(nsISupports*) [xpcom/glue/nsCOMPtr.h:801] 15:18:50 INFO - #02: nsObserverList::AddObserver(nsIObserver*, bool) [xpcom/glue/nsTArray.h:1590] 15:18:50 INFO - #03: nsObserverService::AddObserver(nsIObserver*, char const*, bool) [xpcom/ds/nsObserverService.cpp:252] 15:18:50 INFO - #04: nsContentUtils::RegisterShutdownObserver(nsIObserver*) [dom/base/nsContentUtils.cpp:3953] 15:18:50 INFO - #05: AsyncLatencyLogger::InitializeStatics() [xpcom/base/StaticPtr.h:128] 15:18:50 INFO - #06: nsLayoutStatics::Initialize() [layout/build/nsLayoutStatics.cpp:260] 15:18:50 INFO - #07: Initialize() [layout/build/nsLayoutModule.cpp:442] 15:18:50 INFO - #08: nsComponentManagerImpl::KnownModule::Load() [xpcom/components/nsComponentManager.cpp:899] 15:18:50 INFO - #09: nsFactoryEntry::GetFactory() [xpcom/components/nsComponentManager.cpp:1934] 15:18:50 INFO - #10: nsComponentManagerImpl::CreateInstanceByContractID(char const*, nsISupports*, nsID const&, void**) [xpcom/components/nsComponentManager.cpp:1232] 15:18:50 INFO - #11: nsComponentManagerImpl::GetServiceByContractID(char const*, nsID const&, void**) [xpcom/components/nsComponentManager.cpp:1592] 15:18:50 INFO - #12: nsGetServiceByContractID::operator()(nsID const&, void**) const [xpcom/glue/nsComponentManagerUtils.cpp:281] 15:18:50 INFO - #13: nsCOMPtr_base::assign_from_gs_contractid(nsGetServiceByContractID, nsID const&) [xpcom/glue/nsCOMPtr.cpp:103] 15:18:50 INFO - #14: NS_InitXPCOM2 [xpcom/build/XPCOMInit.cpp:726] 15:18:50 INFO - #15: XRE_InitEmbedding2 [toolkit/xre/nsEmbedFunctions.cpp:173] 15:18:50 INFO - #16: mozilla::ipc::ScopedXREEmbed::Start() [ipc/glue/ScopedXREEmbed.cpp:105] 15:18:50 INFO - #17: mozilla::dom::ContentProcess::Init() [dom/ipc/ContentProcess.cpp:82] 15:18:50 INFO - #18: XRE_InitChildProcess [toolkit/xre/nsEmbedFunctions.cpp:603] 15:18:50 INFO - #19: content_process_main(int, char**) [ipc/contentproc/plugin-container.cpp:240] 15:18:50 INFO - #20: libc.so.6 + 0x2176d 15:18:50 INFO - #21: _start 15:18:50 INFO - 418 @0x7f9ac77f1eb0 (1 references; 0 from COMPtrs) 15:18:50 INFO - allocation stack: 15:18:50 INFO - #00: mozilla::dom::AudioContext::AddRef() [dom/media/webaudio/AudioContext.cpp:76] 15:18:50 INFO - #01: mozilla::dom::AudioNode::AudioNode(mozilla::dom::AudioContext*, unsigned int, mozilla::dom::ChannelCountMode, mozilla::dom::ChannelInterpretation) [xpcom/glue/nsTArray.h:834] 15:18:50 INFO - #02: mozilla::dom::AudioDestinationNode::AudioDestinationNode(mozilla::dom::AudioContext*, bool, mozilla::dom::AudioChannel, unsigned int, unsigned int, float) [dom/media/webaudio/AudioDestinationNode.cpp:342] 15:18:50 INFO - #03: mozilla::dom::AudioContext::AudioContext(nsPIDOMWindowInner*, bool, mozilla::dom::AudioChannel, unsigned int, unsigned int, float) [mfbt/RefPtr.h:52] 15:18:50 INFO - #04: mozilla::dom::AudioContext::Constructor(mozilla::dom::GlobalObject const&, mozilla::dom::AudioChannel, mozilla::ErrorResult&) [dom/media/webaudio/AudioContext.cpp:180] 15:18:50 INFO - #05: mozilla::dom::AudioContext::Constructor(mozilla::dom::GlobalObject const&, mozilla::ErrorResult&) [dom/media/webaudio/AudioContext.cpp:167] 15:18:50 INFO - #06: mozilla::dom::AudioContextBinding::_constructor [mfbt/AlreadyAddRefed.h:138] 15:18:50 INFO - #07: js::CallJSNative(JSContext*, bool (*)(JSContext*, unsigned int, JS::Value*), JS::CallArgs const&) [js/src/jscntxtinlines.h:236] 15:18:50 INFO - #08: js::CallJSNativeConstructor(JSContext*, bool (*)(JSContext*, unsigned int, JS::Value*), JS::CallArgs const&) [js/src/jscntxtinlines.h:268] 15:18:50 INFO - #09: InternalConstruct [js/src/vm/Interpreter.cpp:568] 15:18:50 INFO - #10: Interpret [js/src/vm/Interpreter.cpp:2794] 15:18:50 INFO - #11: js::RunScript(JSContext*, js::RunState&) [js/src/vm/Interpreter.cpp:428] 15:18:50 INFO - #12: js::Invoke(JSContext*, JS::CallArgs const&, js::MaybeConstruct) [js/src/vm/Interpreter.cpp:496] 15:18:50 INFO - #13: js::fun_apply(JSContext*, unsigned int, JS::Value*) [js/src/jsfun.cpp:1277] 15:18:50 INFO - #14: js::CallJSNative(JSContext*, bool (*)(JSContext*, unsigned int, JS::Value*), JS::CallArgs const&) [js/src/jscntxtinlines.h:236] 15:18:50 INFO - #15: js::Invoke(JSContext*, JS::CallArgs const&, js::MaybeConstruct) [js/src/vm/Interpreter.cpp:478] 15:18:50 INFO - #16: Interpret [js/src/vm/Interpreter.cpp:2802] 15:18:50 INFO - #17: js::RunScript(JSContext*, js::RunState&) [js/src/vm/Interpreter.cpp:428] 15:18:50 INFO - #18: js::Invoke(JSContext*, JS::CallArgs const&, js::MaybeConstruct) [js/src/vm/Interpreter.cpp:496] 15:18:50 INFO - #19: js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value const*, JS::MutableHandle<JS::Value>) [js/src/vm/Interpreter.cpp:530] 15:18:50 INFO - #20: JS::Call(JSContext*, JS::Handle<JS::Value>, JS::Handle<JS::Value>, JS::HandleValueArray const&, JS::MutableHandle<JS::Value>) [js/src/jsapi.cpp:2892] 15:18:50 INFO - #21: mozilla::dom::EventHandlerNonNull::Call(JSContext*, JS::Handle<JS::Value>, mozilla::dom::Event&, JS::MutableHandle<JS::Value>, mozilla::ErrorResult&) [obj-firefox/dom/bindings/EventHandlerBinding.cpp:259] 15:18:50 INFO - #22: mozilla::JSEventHandler::HandleEvent(nsIDOMEvent*) [obj-firefox/dist/include/mozilla/dom/EventHandlerBinding.h:351] 15:18:50 INFO - #23: mozilla::EventListenerManager::HandleEventSubType(mozilla::EventListenerManager::Listener*, nsIDOMEvent*, mozilla::dom::EventTarget*) [dom/events/EventListenerManager.cpp:1099] 15:18:50 INFO - #24: mozilla::EventListenerManager::HandleEventInternal(nsPresContext*, mozilla::WidgetEvent*, nsIDOMEvent**, mozilla::dom::EventTarget*, nsEventStatus*) [dom/events/EventListenerManager.cpp:1268] 15:18:50 INFO - #25: mozilla::EventTargetChainItem::HandleEvent(mozilla::EventChainPostVisitor&, mozilla::ELMCreationDetector&) [dom/events/EventListenerManager.h:350] 15:18:50 INFO - #26: mozilla::EventTargetChainItem::HandleEventTargetChain(nsTArray<mozilla::EventTargetChainItem>&, mozilla::EventChainPostVisitor&, mozilla::EventDispatchingCallback*, mozilla::ELMCreationDetector&) [dom/events/EventDispatcher.cpp:315] 15:18:50 INFO - #27: mozilla::EventDispatcher::Dispatch(nsISupports*, nsPresContext*, mozilla::WidgetEvent*, nsIDOMEvent*, nsEventStatus*, mozilla::EventDispatchingCallback*, nsTArray<mozilla::dom::EventTarget*>*) [dom/events/EventDispatcher.cpp:658] 15:18:50 INFO - #28: nsDocumentViewer::LoadComplete(nsresult) [layout/base/nsDocumentViewer.cpp:998] 15:18:50 INFO - #29: nsDocShell::EndPageLoad(nsIWebProgress*, nsIChannel*, nsresult) [docshell/base/nsDocShell.cpp:7514] 15:18:50 INFO - #30: nsDocShell::OnStateChange(nsIWebProgress*, nsIRequest*, unsigned int, nsresult) [docshell/base/nsDocShell.cpp:7320] 15:18:50 INFO - #31: nsDocLoader::DoFireOnStateChange(nsIWebProgress*, nsIRequest*, int&, nsresult) [uriloader/base/nsDocLoader.cpp:1254] 15:18:50 INFO - #32: nsDocLoader::doStopDocumentLoad(nsIRequest*, nsresult) [uriloader/base/nsDocLoader.cpp:837] 15:18:50 INFO - #33: nsDocLoader::DocLoaderIsEmpty(bool) [uriloader/base/nsDocLoader.cpp:730] 15:18:50 INFO - #34: nsDocLoader::OnStopRequest(nsIRequest*, nsISupports*, nsresult) [uriloader/base/nsDocLoader.cpp:612] 15:18:50 INFO - #35: nsLoadGroup::RemoveRequest(nsIRequest*, nsISupports*, nsresult) [netwerk/base/nsLoadGroup.cpp:636] 15:18:50 INFO - #36: nsDocument::DoUnblockOnload() [dom/base/nsDocument.cpp:9122] 15:18:50 INFO - #37: nsDocument::UnblockOnload(bool) [dom/base/nsDocument.cpp:9049] 15:18:50 INFO - #38: nsDocument::DispatchContentLoadedEvents() [dom/base/nsDocument.cpp:5173] 15:18:50 INFO - #39: nsRunnableMethodImpl<void (nsDocument::*)(), true>::Run() [xpcom/glue/nsThreadUtils.h:873] 15:18:50 INFO - #40: nsThread::ProcessNextEvent(bool, bool*) [xpcom/threads/nsThread.cpp:1018] 15:18:50 INFO - #41: NS_ProcessNextEvent(nsIThread*, bool) [xpcom/glue/nsThreadUtils.cpp:297] 15:18:50 INFO - #42: mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) [ipc/glue/MessagePump.cpp:96] 15:18:50 INFO - #43: MessageLoop::RunInternal() [ipc/chromium/src/base/message_loop.cc:235] 15:18:50 INFO - #44: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:520] 15:18:50 INFO - #45: nsBaseAppShell::Run() [widget/nsBaseAppShell.cpp:158] 15:18:50 INFO - #46: XRE_RunAppShell [toolkit/xre/nsEmbedFunctions.cpp:789] 15:18:50 INFO - #47: mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) [ipc/glue/MessagePump.cpp:259] 15:18:50 INFO - #48: MessageLoop::RunInternal() [ipc/chromium/src/base/message_loop.cc:235] 15:18:50 INFO - #49: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:520] 15:18:50 INFO - #50: XRE_InitChildProcess [toolkit/xre/nsEmbedFunctions.cpp:629] 15:18:50 INFO - #51: content_process_main(int, char**) [ipc/contentproc/plugin-container.cpp:240] 15:18:50 INFO - #52: libc.so.6 + 0x2176d 15:18:50 INFO - #53: _start

FYI: padenot is going to look at this and bug 1221910 next week. We're trying to get the last bits of full duplex landed this week so we can test it more broadly and then pref it on.

This kills the entire test suite under debug builds since it happens on any various tests. Releng wants to disable the suite under e10s. Maire - need sign off on this. Can we get this fixed before the next merge?

(In reply to Ryan VanderMeulen [:RyanVM] from comment #24) > This is currently the #3 leak on OrangeFactor, and only occurs on e10s. > > Leak stack: > 15:18:25 INFO - allocation stack: > 15:18:50 INFO - #00: AsyncLatencyLogger::AddRef() > [dom/media/Latency.cpp:134] Kind of curious what the full set of AddRef/Release calls for this thing look like...though it looks like MSG doesn't use the logger (mLatencyLog) at all, so we could at least eliminate that bit of leakage.

The logger is not directly involved here; that's a side-effect. The main issue is that the MSG (with WebAudio nodes) hasn't shut down.

Please do not disable the web audio e10s tests. From orangefactor, this appears to have gotten much worse around Feb 25th. We are looking into what landed around that time to trigger the spike. We are looking at both "quick" fixes and root cause fixes. We've been actively working this on other bugs (Bug 1255737, Bug 1256510). Sorry I didn't reflect that activity in this bug. As Randell says, we believe the root cause problem is MSG failing to shutdown. We attempted to fix the problems with the MSG shutdown code in Bug 1239873, but that didn't nail all the problems. We just fixed one shutdown failure (Bug 1256510) and we're continuing to hunt. I expect to land these fixes and solve this pain before the next merge (April 18th). Updating the owner to Randell since he's taking the lead on this.

Looking good so far since bug 1267600 landed.

No OrangeFactor hits for this since bug 1267600 landed. Calling this fixed!