Ben Kelly [:bkelly, not reviewing] Reporter
	Description • 8 years ago

+++ This bug was initially created as a clone of Bug #1238707 +++

After fixing bug 1238707 another leak was revealed:

https://treeherder.mozilla.org/logviewer.html#?job_id=18936889&repo=try

I ran another push with leak stack walking enabled, and the leaked CacheEntry has the following stack:
19:00:51     INFO -  Serial Numbers of Leaked Objects:
19:00:51     INFO -  123 @1D2FB9D0 (1 references)
19:00:51     INFO -  allocation stack:
19:01:06     INFO -  #00: mozilla::net::CacheEntry::CacheEntry(nsACString_internal const &,nsIURI *,nsACString_internal const &,bool,bool,bool) [netwerk/cache2/CacheEntry.cpp:225]
19:01:06     INFO -  #01: mozilla::net::CacheStorageService::AddStorageEntry(nsACString_internal const &,nsIURI *,nsACString_internal const &,bool,bool,bool,bool,mozilla::net::CacheEntryHandle * *) [netwerk/cache2/CacheStorageService.cpp:1493]
19:01:06     INFO -  #02: mozilla::net::CacheStorageService::AddStorageEntry(mozilla::net::CacheStorage const *,nsIURI *,nsACString_internal const &,bool,mozilla::net::CacheEntryHandle * *) [netwerk/cache2/CacheStorageService.cpp:1411]
19:01:06     INFO -  #03: mozilla::net::CacheStorage::OpenTruncate(nsIURI *,nsACString_internal const &,nsICacheEntry * *) [netwerk/cache2/CacheStorage.cpp:131]
19:01:06     INFO -  #04: mozilla::net::nsHttpChannel::OpenCacheEntry(bool) [netwerk/protocol/http/nsHttpChannel.cpp:3110]
19:01:06     INFO -  #05: mozilla::net::nsHttpChannel::Connect() [netwerk/protocol/http/nsHttpChannel.cpp:380]
19:01:06     INFO -  #06: mozilla::net::nsHttpChannel::ContinueBeginConnectWithResult() [netwerk/protocol/http/nsHttpChannel.cpp:5560]
19:01:06     INFO -  #07: mozilla::net::nsHttpChannel::BeginConnect() [netwerk/protocol/http/nsHttpChannel.cpp:5463]
19:01:06     INFO -  #08: mozilla::net::nsHttpChannel::OnProxyAvailable(nsICancelable *,nsIChannel *,nsIProxyInfo *,nsresult) [netwerk/protocol/http/nsHttpChannel.cpp:5630]
19:01:06     INFO -  #09: nsAsyncResolveRequest::DoCallback() [netwerk/base/nsProtocolProxyService.cpp:253]
19:01:06     INFO -  #10: nsAsyncResolveRequest::OnQueryComplete(nsresult,nsCString const &,nsCString const &) [netwerk/base/nsProtocolProxyService.cpp:214]
19:01:06     INFO -  #11: ExecuteCallback::Run() [netwerk/base/nsPACMan.cpp:87]
19:01:06     INFO -  #12: nsThread::ProcessNextEvent(bool,bool *) [xpcom/threads/nsThread.cpp:994]
19:01:06     INFO -  #13: NS_ProcessNextEvent(nsIThread *,bool) [xpcom/glue/nsThreadUtils.cpp:297]
19:01:06     INFO -  #14: mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate *) [ipc/glue/MessagePump.cpp:98]
19:01:06     INFO -  #15: MessageLoop::RunInternal() [ipc/chromium/src/base/message_loop.cc:236]
19:01:06     INFO -  #16: MessageLoop::RunHandler() [ipc/chromium/src/base/message_loop.cc:230]
19:01:06     INFO -  #17: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:204]
19:01:06     INFO -  #18: nsBaseAppShell::Run() [widget/nsBaseAppShell.cpp:158]
19:01:06     INFO -  #19: nsAppShell::Run() [widget/windows/nsAppShell.cpp:264]
19:01:06     INFO -  #20: nsAppStartup::Run() [toolkit/components/startup/nsAppStartup.cpp:282]
19:01:06     INFO -  #21: XREMain::XRE_mainRun() [toolkit/xre/nsAppRunner.cpp:4342]
19:01:06     INFO -  #22: XREMain::XRE_main(int,char * * const,nsXREAppData const *) [toolkit/xre/nsAppRunner.cpp:4439]
19:01:06     INFO -  #23: XRE_main [toolkit/xre/nsAppRunner.cpp:4545]
19:01:06     INFO -  #24: do_main [browser/app/nsBrowserApp.cpp:220]
19:01:06     INFO -  #25: NS_internal_main(int,char * *,char * *) [browser/app/nsBrowserApp.cpp:360]
19:01:06     INFO -  #26: wmain [toolkit/xre/nsWindowsWMain.cpp:138]
19:01:06     INFO -  #27: __scrt_common_main_seh [f:/dd/vctools/crt/vcstartup/src/startup/exe_common.inl:264]
19:01:06     INFO -  #28: kernel32.dll + 0x53c45
19:01:06     INFO -  #29: ntdll.dll + 0x637f5
19:01:06     INFO -  #30: ntdll.dll + 0x637c8

19:00:51     INFO -  Leaked URLs:
19:00:51     INFO -    http://mochi.test:8888/tests/dom/workers/test/serviceworkers/fetch/synthesized-redirect-twice-real-file.txt
19:00:51     INFO -    http://mochi.test:8888/tests/dom/workers/test/serviceworkers/fetch/index.html
19:00:51     INFO -    http://mochi.test:8888/tests/dom/workers/test/serviceworkers/fetch/fetch_worker_script.js
19:00:51     INFO -    http://mochi.test:8888/tests/dom/workers/test/serviceworkers/fetch/fetch_worker_script.js
19:00:51     INFO -    http://mochi.test:8888/tests/dom/workers/test/serviceworkers/synthesized-redirect-real-file.txt
19:00:51     INFO -    http://mochi.test:8888/tests/dom/workers/test/serviceworkers/fetch/synthesized-redirect-twice-real-file.txt
19:00:51     INFO -    http://mochi.test:8888/tests/dom/workers/test/serviceworkers/fetch/fetch_worker_script.js
19:00:51     INFO -    http://mochi.test:8888/tests/dom/workers/test/serviceworkers/fetch/synthesized-redirect-twice-real-file.txt

	Ben Kelly [:bkelly, not reviewing] Reporter
	Comment 1 • 8 years ago

I managed to reproduce this on linux in an rr chaos session.  I guess rr isn't terrible.

	Ben Kelly [:bkelly, not reviewing] Reporter
	Comment 2 • 8 years ago

I'm having difficulty getting rr to replay on my machine, but at least it reproduces reliably under chaos mode.  Should make it easier to track down even without replay.

	Ben Kelly [:bkelly, not reviewing] Reporter
	Comment 3 • 8 years ago

Finally have a config which lets me replay a capture of this.  Will debug tomorrow.

	Ben Kelly [:bkelly, not reviewing] Reporter
	Comment 4 • 8 years ago

It seems the leaked URLs correspond to channels going through this could with shouldIntercept=true:

  https://dxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/HttpChannelParent.cpp?from=HttpChannelParent.cpp#516

So e10s parent-side redirection with interception.

	Ryan VanderMeulen [:RyanVM]
	Comment 5 • 8 years ago

Try confirms that we can run leak-free on Windows with just test_fetch_event.html skipped, so I'll be pushing a patch shortly that does just that. Other than that, test_notificationclick_focus.html is permafail on Android debug due to bug 1259338 but we're otherwise skip-free now!

	Ben Kelly [:bkelly, not reviewing] Reporter
	Comment 6 • 8 years ago

FWIW, I've seen chaos mode leak URLs from other tests in the dir that perform redirects.

	Pulsebot
	Comment 7 • 8 years ago

https://hg.mozilla.org/integration/mozilla-inbound/rev/1129184ec2dd

	Carsten Book [:Tomcat]
	Comment 8 • 8 years ago
bugherder

https://hg.mozilla.org/mozilla-central/rev/1129184ec2dd

	Ryan VanderMeulen [:RyanVM]
	Comment 9 • 8 years ago

https://hg.mozilla.org/releases/mozilla-aurora/rev/57c47196d35c

	Ben Kelly [:bkelly, not reviewing] Reporter
	Comment 12 • 8 years ago

I'm not currently working on this.  I think we should wait for Josh's network interception changes.

	Chris Peterson [:cpeterson]
	Comment 20 • 8 years ago

Intermittent e10s test leak

	Joel Maher ( :jmaher ) (UTC -8)
	Comment 53 • 7 years ago

On Feb 3rd this started up on win8-debug in the dom/workers/test/serviceworkers/ directory.

I did a series of retriggers here:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=win%20x64%20debug%20mochitest-e10s-4&tochange=eefc0b77e8cd3634483af8f2ab1c5c8338d85866&fromchange=0a1c94d2176286fe2316988f7641165c3090da8b&selectedJob=74150072

ni myself to follow up here on the retriggers.

It would be nice to bisect this down, that will take some time as there are many tests in this manifest:
https://dxr.mozilla.org/mozilla-central/source/dom/workers/test/serviceworkers/mochitest.ini

	Joel Maher ( :jmaher ) (UTC -8)
	Comment 54 • 7 years ago

:bkelly- can you help figure out which test is leaking or follow along and help come to a resolution of this bug?

	Ben Kelly [:bkelly, not reviewing] Reporter
	Comment 55 • 7 years ago

Andrew, can you find someone to investigate this leak?

	Andrew Overholt [:overholt]
	Comment 56 • 7 years ago

Catalin, please take a look.

	Joel Maher ( :jmaher ) (UTC -8)
	Comment 57 • 7 years ago

oh retriggering helped pinpoint the exact failure (bug 1336043):
https://hg.mozilla.org/integration/mozilla-inbound/rev/eefc0b77e8cd3634483af8f2ab1c5c8338d85866

asking :baku for info here as he authored bug 1336043.

	Cătălin Badea (:catalinb)
	Comment 58 • 7 years ago

Running test_fetch_event.html --repeat=10 triggers the leak consistently on my machine.

	Cătălin Badea (:catalinb)
	Comment 61 • 7 years ago

This might be the same leak as bug 1326136.

	Joel Maher ( :jmaher ) (UTC -8)
	Comment 62 • 7 years ago

now that bug 1326136 is fixed, I still see errors daily related to this test, it has dropped in frequency about a week ago though, not sure why.

	Joel Maher ( :jmaher ) (UTC -8)
	Comment 66 • 7 years ago

there is a leak here:
11:02:30     INFO - GECKO(1812) | nsStringStats
11:02:30     INFO - GECKO(1812) |  => mAllocCount:         276144
11:02:30     INFO - GECKO(1812) |  => mReallocCount:        57096
11:02:30     INFO - GECKO(1812) |  => mFreeCount:          276144
11:02:30     INFO - GECKO(1812) |  => mShareCount:         495693
11:02:30     INFO - GECKO(1812) |  => mAdoptCount:          28101
11:02:30     INFO - GECKO(1812) |  => mAdoptFreeCount:      28101
11:02:30     INFO - GECKO(1812) |  => Process ID: 1813, Thread ID: 140735240848128
11:02:30     INFO - GECKO(1812) | [Parent 1812] WARNING: NS_ENSURE_TRUE(mDB) failed: file /home/worker/workspace/build/src/netwerk/cache/nsDiskCacheDeviceSQL.cpp, line 1426
11:02:30     INFO - GECKO(1812) | [Parent 1812] WARNING: nsAppShell::Exit() called redundantly: file /home/worker/workspace/build/src/widget/cocoa/nsAppShell.mm, line 684
11:02:31     INFO - GECKO(1812) | [Parent 1812] WARNING: NS_ENSURE_TRUE(maybeContext) failed: file /home/worker/workspace/build/src/xpcom/threads/nsThread.cpp, line 1009
11:02:31     INFO - GECKO(1812) | [Parent 1812] WARNING: 'NS_FAILED(RemovePermissionChangeObserver())', file /home/worker/workspace/build/src/dom/notification/Notification.cpp, line 669
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: NS_ENSURE_TRUE(aObserver) failed: file /home/worker/workspace/build/src/modules/libpref/nsPrefBranch.cpp, line 715
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80070057: file /home/worker/workspace/build/src/modules/libpref/Preferences.cpp, line 1709
11:02:33     INFO - GECKO(1812) | --DOMWINDOW == 9 (0x11e192800) [pid = 1812] [serial = 4] [outer = 0x0] [url = about:blank]
11:02:33     INFO - GECKO(1812) | --DOMWINDOW == 8 (0x125c26800) [pid = 1812] [serial = 1] [outer = 0x0] [url = chrome://browser/content/hiddenWindow.xul]
11:02:33     INFO - GECKO(1812) | --DOMWINDOW == 7 (0x12a43b800) [pid = 1812] [serial = 5] [outer = 0x0] [url = about:blank]
11:02:33     INFO - GECKO(1812) | --DOMWINDOW == 6 (0x11b1e8000) [pid = 1812] [serial = 9] [outer = 0x0] [url = about:blank]
11:02:33     INFO - GECKO(1812) | --DOMWINDOW == 5 (0x12a43d800) [pid = 1812] [serial = 6] [outer = 0x0] [url = about:blank]
11:02:33     INFO - GECKO(1812) | --DOMWINDOW == 4 (0x12c6dc800) [pid = 1812] [serial = 10] [outer = 0x0] [url = about:blank]
11:02:33     INFO - GECKO(1812) | --DOMWINDOW == 3 (0x12857bc00) [pid = 1812] [serial = 11] [outer = 0x0] [url = about:newtab]
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | --DOMWINDOW == 2 (0x129f9ac00) [pid = 1812] [serial = 13] [outer = 0x0] [url = about:newtab]
11:02:33     INFO - GECKO(1812) | --DOMWINDOW == 1 (0x125c27800) [pid = 1812] [serial = 2] [outer = 0x0] [url = about:blank]
11:02:33     INFO - GECKO(1812) | --DOMWINDOW == 0 (0x11e191800) [pid = 1812] [serial = 3] [outer = 0x0] [url = chrome://browser/content/browser.xul]
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!compMgr', file /home/worker/workspace/build/src/xpcom/components/nsComponentManagerUtils.cpp, line 63
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] WARNING: '!mMainThread', file /home/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 314
11:02:33     INFO - GECKO(1812) | [Parent 1812] ###!!! ASSERTION: 2 dynamic atom(s) with non-zero refcount: http://mochi.test:8888,https://example.com: 'nonZeroRefcountAtomsCount == 0', file /home/worker/workspace/build/src/xpcom/ds/nsAtomTable.cpp, line 416
11:02:33     INFO - GECKO(1812) | #01: mozilla::VolatileBuffer::NonHeapSizeOfExcludingThis() const[/builds/slave/test/build/application/NightlyDebug.app/Contents/MacOS/XUL +0xaf800]
11:02:33     INFO - GECKO(1812) | #02: nsXPTCStubBase::Stub249()[/builds/slave/test/build/application/NightlyDebug.app/Contents/MacOS/XUL +0x14d60a]
11:02:33     INFO - GECKO(1812) | #03: workerlz4_maxCompressedSize[/builds/slave/test/build/application/NightlyDebug.app/Contents/MacOS/XUL +0x405ff17]
11:02:33     INFO - GECKO(1812) | #04: workerlz4_maxCompressedSize[/builds/slave/test/build/application/NightlyDebug.app/Contents/MacOS/XUL +0x40691fc]
11:02:33     INFO - GECKO(1812) | #05: workerlz4_maxCompressedSize[/builds/slave/test/build/application/NightlyDebug.app/Contents/MacOS/XUL +0x4069693]
11:02:33     INFO - GECKO(1812) | #06: start[/builds/slave/test/build/application/NightlyDebug.app/Contents/MacOS/firefox +0xeea]
11:02:33     INFO - GECKO(1812) | Leaked URLs:
11:02:33     INFO - GECKO(1812) |   https://example.com/tests/dom/workers/test/serviceworkers/fetch/origin/https/index-https.sjs
11:02:33     INFO - GECKO(1812) |   https://example.com/tests/dom/workers/test/serviceworkers/fetch/origin/https/index-https.sjs
11:02:33     INFO - GECKO(1812) |   http://mochi.test:8888/tests/dom/workers/test/serviceworkers/test_https_origin_after_redirect.html
11:02:33     INFO - GECKO(1812) |   http://mochi.test:8888/tests/dom/workers/test/serviceworkers/test_https_origin_after_redirect.html
11:02:33     INFO - GECKO(1812) |   http://mochi.test:8888/tests/dom/workers/test/serviceworkers/test_https_origin_after_redirect.html
11:02:33     INFO - GECKO(1812) |   https://example.org/tests/dom/workers/test/serviceworkers/fetch/origin/https/realindex.html
11:02:33     INFO - GECKO(1812) |   https://example.com/tests/dom/workers/test/serviceworkers/fetch/origin/https/index-https.sjs
11:02:33     INFO - GECKO(1812) |   http://mochi.test:8888/tests/dom/workers/test/serviceworkers/test_https_origin_after_redirect.html
11:02:33     INFO - GECKO(1812) | nsStringStats
11:02:33     INFO - GECKO(1812) |  => mAllocCount:         612616
11:02:33     INFO - GECKO(1812) |  => mReallocCount:        82475
11:02:33     INFO - GECKO(1812) |  => mFreeCount:          612404  --  LEAKED 212 !!!
11:02:33     INFO - GECKO(1812) |  => mShareCount:         501836
11:02:33     INFO - GECKO(1812) |  => mAdoptCount:          24288
11:02:33     INFO - GECKO(1812) |  => mAdoptFreeCount:      24286  --  LEAKED 2 !!!
11:02:33     INFO - GECKO(1812) |  => Process ID: 1812, Thread ID: 140735240848128
11:02:33     INFO - TEST-INFO | Main app process: exit 0
11:02:33     INFO - runtests.py | Application ran for: 0:01:56.865963
11:02:33     INFO - zombiecheck | Reading PID log: /var/folders/cf/5zhxcwh11mq5cjjb5nhm70lm00000w/T/tmp3X109Npidlog
11:02:33     INFO - ==> process 1812 launched child process 1813
11:02:33     INFO - zombiecheck | Checking for orphan process with PID: 1813
11:02:33     INFO - Stopping web server
11:02:33     INFO - Stopping web socket server
11:02:33     INFO - Stopping ssltunnel
11:02:33     INFO - TEST-INFO | leakcheck | default process: leak threshold set at 0 bytes
11:02:33     INFO - TEST-INFO | leakcheck | plugin process: leak threshold set at 0 bytes
11:02:33     INFO - TEST-INFO | leakcheck | tab process: leak threshold set at 10000 bytes
11:02:33     INFO - TEST-INFO | leakcheck | geckomediaplugin process: leak threshold set at 20000 bytes
11:02:33     INFO - TEST-INFO | leakcheck | gpu process: leak threshold set at 0 bytes
11:02:33     INFO - 
11:02:33     INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, default process 1812
11:02:33     INFO - 
11:02:33     INFO -      |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
11:02:33     INFO -      |                                      | Per-Inst   Leaked|   Total      Rem|
11:02:33     INFO -    0 |TOTAL                                 |       24    24114|17142113      375|
11:02:33     INFO -   65 |CacheEntry                            |      256      256|     529        1|
11:02:33     INFO -   68 |CacheEntryHandle                      |       32       32|    7738        1|
11:02:33     INFO -   69 |CacheFile                             |      344      344|     529        1|
11:02:33     INFO -   77 |CacheFileMetadata                     |      200      200|     529        1|
11:02:33     INFO -   78 |CacheFileOutputStream                 |      104      104|     473        1|
11:02:33     INFO -   87 |CacheStorageService                   |      304      304|       1        1|
11:02:33     INFO -   93 |CancelableRunnable                    |       48       48|   48436        1|
11:02:33     INFO -   98 |ChannelEventQueue                     |       72       72|     977        1|
11:02:33     INFO -  132 |CondVar                               |       40      400|    1132       10|
11:02:33     INFO -  149 |ConsoleReportCollector                |       64      128|    1264        2|
11:02:33     INFO -  152 |ContentParent                         |     2200     2200|       1        1|
11:02:33     INFO -  189 |DataStorage                           |      336     1008|       3        3|
11:02:33     INFO -  318 |HttpBaseChannel                       |     1328     1328|    1042        1|
11:02:33     INFO -  319 |HttpChannelParent                     |      248      248|     977        1|
11:02:33     INFO -  320 |HttpChannelParentListener             |       88       88|     942        1|
11:02:33     INFO -  339 |IdlePeriod                            |       24       72|     240        3|
11:02:33     INFO -  362 |InterceptedChannelBase                |       80       80|      17        1|
11:02:33     INFO -  379 |LoadContext                           |      104      208|    1002        2|
11:02:33     INFO -  381 |LoadInfo                              |      192      192|    4133        1|
11:02:33     INFO -  418 |Mutex                                 |       32      768|   14454       24|
11:02:33     INFO -  461 |PBrowserParent                        |      424      424|      20        1|
11:02:33     INFO -  470 |PContentParent                        |     1832     1832|       1        1|
11:02:33     INFO -  475 |PHttpChannelParent                    |       40       40|     977        1|
11:02:33     INFO -  518 |PollableEvent                         |       24       24|       1        1|
11:02:33     INFO -  563 |ReentrantMonitor                      |       40       80|    5984        2|
11:02:33     INFO -  564 |RefCountedMonitor                     |       80       80|     154        1|
11:02:33     INFO -  570 |RequestContextService                 |       88       88|       1        1|
11:02:33     INFO -  580 |Runnable                              |       40       80|  142649        2|
11:02:33     INFO -  606 |Service                               |      144      144|       1        1|
11:02:33     INFO -  623 |SharedMemory                          |       32       32|      61        1|
11:02:33     INFO -  655 |StringAdopt                           |        1        2|   24288        2|
11:02:33     INFO -  666 |TabParent                             |     1248     1248|      20        1|
11:02:33     INFO -  684 |ThirdPartyUtil                        |       32       32|       1        1|
11:02:33     INFO -  843 |ipc::MessageChannel                   |      328      328|     176        1|
11:02:33     INFO -  888 |nsAuthURLParser                       |       24       24|       2        1|
11:02:33     INFO -  944 |nsCategoryObserver                    |       96       96|       8        1|
11:02:33     INFO -  976 |nsCookiePermission                    |       64       64|       1        1|
11:02:33     INFO -  977 |nsCookieService                       |      144      144|       1        1|
11:02:33     INFO -  983 |nsDNSService                          |      160      160|       1        1|
11:02:33     INFO - 1039 |nsEffectiveTLDService                 |       40       40|       1        1|
11:02:33     INFO - 1105 |nsHttpAuthCache::OriginClearObserver  |       32       64|       2        2|
11:02:33     INFO - 1106 |nsHttpChannel                         |     1872     1872|    1042        1|
11:02:33     INFO - 1107 |nsHttpChannelAuthProvider             |      248      248|     995        1|
11:02:33     INFO - 1109 |nsHttpConnectionInfo                  |      184      184|    1004        1|
11:02:33     INFO - 1110 |nsHttpConnectionMgr                   |      296      296|       1        1|
11:02:33     INFO - 1112 |nsHttpHandler                         |      696      696|       1        1|
11:02:33     INFO - 1113 |nsHttpRequestHead                     |      136      136|    1091        1|
11:02:33     INFO - 1116 |nsIDNService                          |       96       96|       1        1|
11:02:33     INFO - 1119 |nsIOService                           |      272      272|       1        1|
11:02:33     INFO - 1147 |nsJSPrincipals                        |       24       72|    9760        3|
11:02:33     INFO - 1158 |nsLocalFile                           |      192      576|   17563        3|
11:02:33     INFO - 1167 |nsMainThreadPtrHolder<T>              |       24       96|    5329        4|
11:02:33     INFO - 1198 |nsNetworkLinkService                  |       88       88|       1        1|
11:02:33     INFO - 1203 |nsNodeWeakReference                   |       32       32|     739        1|
11:02:33     INFO - 1225 |nsPermissionManager                   |      160      160|       1        1|
11:02:33     INFO - 1232 |nsPrefBranch                          |      128      128|      57        1|
11:02:33     INFO - 1241 |nsProxyInfo                           |      104      104|     996        1|
11:02:33     INFO - 1295 |nsSiteSecurityService                 |       80       80|       1        1|
11:02:33     INFO - 1299 |nsSocketTransportService              |      256      256|       1        1|
11:02:33     INFO - 1302 |nsStandardURL                         |      264     2112|   59471        8|
11:02:33     INFO - 1308 |nsStreamConverterService              |       72       72|       1        1|
11:02:33     INFO - 1313 |nsStringBuffer                        |        8     1696|  695090      212|
11:02:33     INFO - 1359 |nsTArray_base                         |        8      264| 3750599       33|
11:02:33     INFO - 1368 |nsThread                              |      352     1056|     239        3|
11:02:33     INFO - 1387 |nsUUIDGenerator                       |       56       56|       1        1|
11:02:33     INFO - 1406 |nsWeakReference                       |       40      360|     883        9|
11:02:33     INFO - 
11:02:33     INFO - nsTraceRefcnt::DumpStatistics: 1478 entries
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 CacheEntry
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 CacheEntryHandle
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 CacheFile
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 CacheFileMetadata
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 CacheFileOutputStream
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 CacheStorageService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 CancelableRunnable
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 ChannelEventQueue
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 10 CondVar
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 2 ConsoleReportCollector
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 ContentParent
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 3 DataStorage
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 HttpBaseChannel
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 HttpChannelParent
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 HttpChannelParentListener
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 3 IdlePeriod
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 InterceptedChannelBase
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 2 LoadContext
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 LoadInfo
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 24 Mutex
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 PBrowserParent
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 PContentParent
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 PHttpChannelParent
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 PollableEvent
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 2 ReentrantMonitor
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 RefCountedMonitor
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 RequestContextService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 2 Runnable
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 Service
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 SharedMemory
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 2 StringAdopt
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 TabParent
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 ThirdPartyUtil
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 ipc::MessageChannel
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsAuthURLParser
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsCategoryObserver
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsCookiePermission
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsCookieService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsDNSService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsEffectiveTLDService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 2 nsHttpAuthCache::OriginClearObserver
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsHttpChannel
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsHttpChannelAuthProvider
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsHttpConnectionInfo
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsHttpConnectionMgr
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsHttpHandler
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsHttpRequestHead
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsIDNService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsIOService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 3 nsJSPrincipals
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 3 nsLocalFile
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 4 nsMainThreadPtrHolder<T>
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsNetworkLinkService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsNodeWeakReference
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsPermissionManager
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsPrefBranch
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsProxyInfo
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsSiteSecurityService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsSocketTransportService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 8 nsStandardURL
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsStreamConverterService
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 212 nsStringBuffer
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 33 nsTArray_base
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 3 nsThread
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 1 nsUUIDGenerator
11:02:33     INFO - TEST-INFO | leakcheck | default process: leaked 9 nsWeakReference
11:02:33    ERROR - TEST-UNEXPECTED-FAIL | leakcheck | default process: 24114 bytes leaked (CacheEntry, CacheEntryHandle, CacheFile, CacheFileMetadata, CacheFileOutputStream, ...)
11:02:33     INFO - 
11:02:33     INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, tab process 1813
11:02:33     INFO - 
11:02:33     INFO -      |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
11:02:33     INFO -      |                                      | Per-Inst   Leaked|   Total      Rem|
11:02:33     INFO -    0 |TOTAL                                 |       29        0| 6061782        0|
11:02:33     INFO - 

from this log file:
https://archive.mozilla.org/pub/firefox/tinderbox-builds/autoland-macosx64-debug/autoland_yosemite_r7-debug_test-mochitest-e10s-4-bm134-tests1-macosx-build510.txt.gz

I see an assertion, maybe that is the root cause as we might not clean up properly after hitting that?


:baku, you are ni? here as the patch author of who originally caused this to spike up a month ago.  Can you look at this bug in the next week?

	Joel Maher ( :jmaher ) (UTC -8)
	Comment 68 • 7 years ago

:overholt, can you help us find someone to look into this assuming :baku is not available.  This is a test we will probably narrow down and disable next week, it is one of the tests in:
dom/workers/test/serviceworkers (which is 109 tests, I assume it is one or two tests causing the leak)

	Joel Maher ( :jmaher ) (UTC -8)
	Comment 69 • 7 years ago

:Mavericks, can you bisect this down to help determine the one or two tests which are causing the leak?

	Andrew McCreight [:mccr8]
	Comment 70 • 7 years ago

Catalin is still assigned to this, and looked at it a month ago, so he's probably the one you should needinfo about this.

	Andrew Overholt [:overholt]
	Comment 71 • 7 years ago

(I asked Catalin on IRC to take another look)

	Cătălin Badea (:catalinb)
	Comment 72 • 7 years ago

dom/workers/test/serviceworkers/test_fetch_event.html is one of the tests where the leak happens. I'm working on finding the issue.

	Joel Maher ( :jmaher ) (UTC -8)
	Comment 73 • 7 years ago

in some try pushes, I determined that dom/workers/test/serviceworkers/test_fetch_event.html is the only leaking test in the directory, that narrows this down- :catalinb, let me know if you need help narrowing it down more or hacking on try pushes, etc.

	Joel Maher ( :jmaher ) (UTC -8)
	Comment 78 • 7 years ago

:catalinb, a few weeks ago you were looking into why dom/workers/test/serviceworkers/test_fetch_event.html is causing a leak- have you more data?  This is still leaking quite frequently.  Please let me know what if any help you need!

	Cătălin Badea (:catalinb)
	Comment 80 • 7 years ago

(In reply to Joel Maher ( :jmaher) from comment #78)
> :catalinb, a few weeks ago you were looking into why
> dom/workers/test/serviceworkers/test_fetch_event.html is causing a leak-
> have you more data?  This is still leaking quite frequently.  Please let me
> know what if any help you need!

Sorry for the late reply.

I spent a couple of days couple investigating this but couldn't find anything more conclusive than: there's a HtppChannelParent that's left hanging which causes everything else to leak. I could spend more time on this, but at this point I think it's better to wait for bug 1231222 to land, which will significantly change the fetch interception code.

	Joel Maher ( :jmaher ) (UTC -8)
	Comment 91 • 7 years ago

this has been pretty high on the failure list for the last 4 weeks, the last week really high.

:overholt, I see you as the triage owner for this component, can you help find someone from the service workers team to look into this leak issue?

	Andrew Overholt [:overholt]
	Comment 92 • 7 years ago

We're still waiting for bug 1231222 to land. Given previous investigations haven't gone anywhere, can we disable this until bug 1231222 lands?

	Joel Maher ( :jmaher ) (UTC -8)
	Comment 94 • 7 years ago

yeah, we can disable- I haven't seen much traction on bug 1231222, so lets figure out what to disable and move on.

I am trying to determine which test to disable:
https://treeherder.mozilla.org/#/jobs?repo=try&author=jmaher@mozilla.com&filter-searchStr=Windows%208%20x64%20debug%20Mochitest%20e10s%20Mochitest%20e10s%20M-e10s(4)&fromchange=2447e233a28af394e2d716f83f114834b7b6c416&tochange=e2a02d90c98786f02b98a4fb161740e5919bc9dc

hopefully a bit more work we can narrow this down and resolve this in the short term.

	Pulsebot
	Comment 97 • 7 years ago

Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/9b8d77e85563
disable test_cross_origin_url_after_redirect.html as it leaks often. r=gbrown

	Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)
	Comment 99 • 7 years ago
bugherder

https://hg.mozilla.org/mozilla-central/rev/9b8d77e85563

	Ryan VanderMeulen [:RyanVM]
	Comment 100 • 7 years ago

https://hg.mozilla.org/releases/mozilla-beta/rev/61a49526b7b9

	Hsin-Yi Tsai (she/her) [:hsinyi]
	Comment 109 • 7 years ago

Hey Tom, this is something needs investigation. Would you have bandwidth to take a look?

	Tom Tung [:tt, :ttung]
	Comment 110 • 7 years ago

Based on the previous investigation in comment 80, it's better to wait for bug 1231222 to land. I'd like to work on telemetry first.

	Joel Maher ( :jmaher ) (UTC -8)
	Comment 112 • 7 years ago

:tt, I don't see any action in bug 1231222 for the last 4 months, this is a frequent intermittent- can you help move bug 1231222 along to get it done faster?

	Tom Tung [:tt, :ttung]
	Comment 113 • 7 years ago

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #112)
> :tt, I don't see any action in bug 1231222 for the last 4 months, this is a
> frequent intermittent- can you help move bug 1231222 along to get it done
> faster?

I'm afraid that I cannot help too much for bug 1231222 since it's not easy to split the work inside. Moreover, I believe :asuth is still working on it.

	Joel Maher ( :jmaher ) (UTC -8)
	Comment 114 • 7 years ago

:tt, can you recommend someone else to work on this if you don't have time?  bug 1231222 doesn't seem to be in active development

	Andrew Sutherland [:asuth] (he/him)
	Comment 115 • 7 years ago

Sorry for not responding more promptly on that bug; we're aggressively trying to resolve all of these things, but they're a big mess, :bkelly came up with a revised plan based around bug 1391693 that simplifies the intercept situation that we're going to try and aggressively pursue.  Re: the parts I've been working on, I've been putting our 55 release regression fires (I miss Aurorora!) but should be back to this area of stuff by Monday.

	Pulsebot
	Comment 147 • 2 years ago

Pushed by jstutte@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e8ced2597f55
Enable test_cross_origin_url_after_redirect.html. r=dom-worker-reviewers,smaug