#[markdown(off)]
Filed by: vporof [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=228354795&repo=try

https://queue.taskcluster.net/v1/task/XqEpeO6UQbyy3nkMswMcxw/runs/0/artifacts/public/logs/live_backing.log

Reopening because it's not fixed:
https://treeherder.mozilla.org/logviewer.html#?job_id=245711891&repo=mozilla-central

Please don't close this as dupe of a dupe of a devtools only bug. This also happens for other test jobs, and as such wouldn't be proposed on treeherder anymore.

New test failures seem to reference a different line now:

https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1456013&startday=2019-05-27&endday=2019-06-02&tree=all

01:17:07 INFO - Assertion failure: m_refCount == 1, at z:/build/build/src/js/src/jit/ExecutableAllocator.cpp:50

https://searchfox.org/mozilla-central/rev/f8b11433159cbc9cc80500b3e579d767473fa539/js/src/jit/ExecutableAllocator.cpp#50

There are 24 total failures in the last 7 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-06-13&endday=2019-06-20&tree=trunk&bug=1527846

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=252254407&repo=mozilla-central&lineNumber=48092

There are 31 failures in the last 7 days on windows10-64-ccov, linux64-qr and linux64, all debug: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-06-29&endday=2019-07-06&tree=trunk&bug=1527846

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=254967592&repo=mozilla-central&lineNumber=49454

Whimboo any updates on this?

There are 52 total failures in the last 7 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-07-08&endday=2019-07-15&tree=trunk&bug=1527846

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=256459539&repo=autoland&lineNumber=27244

(In reply to Alexandru Michis [:malexandru] from comment #20)

There are 31 failures in the last 7 days on windows10-64-ccov, linux64-qr and linux64, all debug: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-06-29&endday=2019-07-06&tree=trunk&bug=1527846

Whimboo any updates on this?

Not sure why I'm getting asked here. I don't work on any of this code.

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] (partly away July 22th - July 26th) from comment #30)

(In reply to Alexandru Michis [:malexandru] from comment #20)

There are 31 failures in the last 7 days on windows10-64-ccov, linux64-qr and linux64, all debug: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-06-29&endday=2019-07-06&tree=trunk&bug=1527846

Whimboo any updates on this?

Not sure why I'm getting asked here. I don't work on any of this code.

You've had activity on the bug so we assumed you're working on it.

Steven, as you are the triage owner, can you please take a look at this?

Joel any suggestions here?

the pattern I see here is that this assertion shows up while this test is running: /html/cross-origin-opener/new_window_same_origin.tentative.html.

The test case is here:
https://searchfox.org/mozilla-central/source/testing/web-platform/tests/html/cross-origin-opener/new_window_same_origin.tentative.html

and it depends on a library here:
https://searchfox.org/mozilla-central/source/testing/web-platform/tests/html/cross-origin-opener/common.sub.js#20

I have also noticed that linux already accepts this as an intermittent timeout (with the recent changes for supporting multiple status)

As this test case is part of core::dom&html, lets see if :hsinyi (the component owner for the test) can shed some light.

See also bug 1486101 which failed around the same lines of code. Maybe Andrew can help here again?

I filed bug 1569655 for making this JIT leak not fatally assert. Looking at the test suites involved, there may be 2 or 3 separate leaks getting bucketed together. Probably we're just getting more and more leaks into this bucket until it passes some threshold of concern. I'll see if I can reproduce any of the leaks locally.

Per comment 36, I'll make this about the DOM leak. If I can fix that, we can file a new bug if there's anything else.

There have been 132 occurrences in the last 7 days, most on macosx1014-64, some on linux64-qr, windows10-64-ccov build type debug.

Recent failure: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=260039780&repo=autoland&lineNumber=25043

[task 2019-08-06T02:33:58.124Z] 02:33:58 INFO - GECKO(1703) | [ACTION] NODE_PROPERTIES_LOADED - {}
[task 2019-08-06T02:33:58.146Z] 02:33:58 INFO - GECKO(1703) | --DOMWINDOW == 4 (0x1114933e0) [pid = 1707] [serial = 321] [outer = 0x0] [url = about:blank]
[task 2019-08-06T02:33:58.208Z] 02:33:58 INFO - GECKO(1703) | [ACTION] REMOVE_BREAKPOINT [start] - {}
[task 2019-08-06T02:33:58.261Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live Cell 0xdb510208040 of kind FUNCTION at shutdown
[task 2019-08-06T02:33:58.263Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live Cell 0xdb510208080 of kind FUNCTION at shutdown
[task 2019-08-06T02:33:58.264Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live Cell 0xdb5102080c0 of kind FUNCTION at shutdown
[task 2019-08-06T02:33:58.264Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live Cell 0xdb510208100 of kind FUNCTION at shutdown
[task 2019-08-06T02:33:58.264Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live Cell 0xdb510208140 of kind FUNCTION at shutdown
[task 2019-08-06T02:33:58.264Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found 1230 live Cells at shutdown
[task 2019-08-06T02:33:58.264Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live Cell 0xdb510209060 of kind FUNCTION_EXTENDED at shutdown
[task 2019-08-06T02:33:58.264Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live Cell 0xdb5102090b0 of kind FUNCTION_EXTENDED at shutdown
[task 2019-08-06T02:33:58.264Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live Cell 0xdb510209100 of kind FUNCTION_EXTENDED at shutdown
[task 2019-08-06T02:33:58.264Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live Cell 0xdb510209150 of kind FUNCTION_EXTENDED at shutdown
[task 2019-08-06T02:33:58.264Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live Cell 0xdb5102091a0 of kind FUNCTION_EXTENDED at shutdown
[task 2019-08-06T02:33:58.264Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found 177 live Cells at shutdown
...
[task 2019-08-06T02:33:58.374Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found 1 live Cells at shutdown
[task 2019-08-06T02:33:58.374Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live Cell 0x6cb74c31040 of kind FAT_INLINE_STRING at shutdown
[task 2019-08-06T02:33:58.375Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live Cell 0x6cb74c31060 of kind FAT_INLINE_STRING at shutdown
[task 2019-08-06T02:33:58.375Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live Cell 0x6cb74c31080 of kind FAT_INLINE_STRING at shutdown
[task 2019-08-06T02:33:58.375Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live Cell 0x6cb74c310a0 of kind FAT_INLINE_STRING at shutdown
[task 2019-08-06T02:33:58.376Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live Cell 0x6cb74c310c0 of kind FAT_INLINE_STRING at shutdown
[task 2019-08-06T02:33:58.377Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found 177 live Cells at shutdown
[task 2019-08-06T02:33:58.377Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live Cell 0x6cb74c35028 of kind STRING at shutdown
[task 2019-08-06T02:33:58.377Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live Cell 0x6cb74c35040 of kind STRING at shutdown
[task 2019-08-06T02:33:58.378Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live Cell 0x6cb74c35058 of kind STRING at shutdown
[task 2019-08-06T02:33:58.378Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live Cell 0x6cb74c35070 of kind STRING at shutdown
[task 2019-08-06T02:33:58.378Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live Cell 0x6cb74c350b8 of kind STRING at shutdown
[task 2019-08-06T02:33:58.378Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found 87 live Cells at shutdown
[task 2019-08-06T02:33:58.378Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live RuntimeScriptData 0x124497730 with ref count 2 at shutdown
[task 2019-08-06T02:33:58.378Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live RuntimeScriptData 0x14b67b1e0 with ref count 2 at shutdown
[task 2019-08-06T02:33:58.378Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live RuntimeScriptData 0x14cd14fc0 with ref count 2 at shutdown
[task 2019-08-06T02:33:58.378Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live RuntimeScriptData 0x124497340 with ref count 2 at shutdown
[task 2019-08-06T02:33:58.379Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found live RuntimeScriptData 0x14b48c7a0 with ref count 2 at shutdown
[task 2019-08-06T02:33:58.379Z] 02:33:58 INFO - GECKO(1703) | ERROR: GC found 234 live RuntimeScriptData at shutdown
[task 2019-08-06T02:33:58.379Z] 02:33:58 INFO - GECKO(1703) | Assertion failure: m_refCount == 1, at /builds/worker/workspace/build/src/js/src/jit/ExecutableAllocator.cpp:50
[task 2019-08-06T02:33:58.379Z] 02:33:58 INFO - GECKO(1703) | [ACTION] IN_SCOPE_LINES - {}
[task 2019-08-06T02:33:58.407Z] 02:33:58 INFO - GECKO(1703) | [ACTION] MAP_SCOPES [done] - {}
[task 2019-08-06T02:33:58.434Z] 02:33:58 INFO - GECKO(1703) | [ACTION] EVALUATE_EXPRESSIONS - {"type":"EVALUATE_EXPRESSIONS","cx":{"navigateCounter":0,"thread":"server0.conn113.child1/workerTarget24/thread1","isPaused":true,"pauseCounter":1},"inputs":[],"results":[]}
[task 2019-08-06T02:33:58.439Z] 02:33:58 INFO - GECKO(1703) | [ACTION] SET_VIEWPORT - {"type":"SET_VIEWPORT","viewport":{"start":{"line":-17,"column":0},"end":{"line":34,"column":195}}}
[task 2019-08-06T02:33:58.616Z] 02:33:58 INFO - GECKO(1703) | [Parent 1703, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/workspace/build/src/ipc/glue/ProtocolUtils.cpp, line 575
[task 2019-08-06T02:33:58.616Z] 02:33:58 INFO - GECKO(1703) | ++DOCSHELL 0x100d7e800 == 9 [pid = 1703] [id = {8c244925-bf5b-a849-af39-e8dd6f52556e}]
[task 2019-08-06T02:33:58.617Z] 02:33:58 INFO - GECKO(1703) | ++DOMWINDOW == 25 (0x12e6f76a0) [pid = 1703] [serial = 638] [outer = 0x0]
[task 2019-08-06T02:33:58.617Z] 02:33:58 INFO - GECKO(1703) | ++DOMWINDOW == 26 (0x11013d400) [pid = 1703] [serial = 639] [outer = 0x12e6f76a0]
[task 2019-08-06T02:33:58.761Z] 02:33:58 INFO - GECKO(1703) | console.log: (new Error("Connection closed, pending request to server0.conn113.child1/thread19, type removeBreakpoint failed\n\nRequest stack:\nrequest@resource://devtools/shared/protocol/Front.js:166:14\ngenerateRequestMethods/</frontProto[name]@resource://devtools/shared/protocol/Front/FrontClassWithSpec.js:49:19\nremoveBreakpoint/<@resource://devtools/client/debugger/src/client/firefox/commands.js:193:41\nforEachThread/promises<@resource://devtools/client/debugger/src/client/firefox/commands.js:98:16\nforEachThread@resource://devtools/client/debugger/src/client/firefox/commands.js:95:63\nremoveBreakpoint@resource://devtools/client/debugger/src/client/firefox/commands.js:193:10\nclientRemoveBreakpoint@resource://devtools/client/debugger/src/actions/breakpoints/modify.js:54:17\nremoveBreakpoint/<@resource://devtools/client/debugger/src/actions/breakpoints/modify.js:152:69\nthunk/</</<@resource://devtools/client/debugger/src/actions/utils/middleware/thunk.js:21:45\nbindActionCreator/<@resource://devtools/client/shared/vendor/redux.js:644:12\nremoveBreakpoint@chrome://mochitests/content/browser/devtools/client/debugger/test/mochitest/helpers.js:988:22\n@chrome://mochitests/content/browser/devtools/client/debugger/test/mochitest/browser_dbg-worker-scopes.js:47:9\nAsyncTester_execTest/<@chrome://mochikit/content/browser-test.js:1346:34\nTester_execTest@chrome://mochikit/content/browser-test.js:1381:11\nnextTest/<@chrome://mochikit/content/browser-test.js:1209:14\nSimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:803:67\n", "resource://devtools/shared/protocol/Front.js", 72))
[task 2019-08-06T02:33:58.790Z] 02:33:58 INFO - GECKO(1703) | console.log: (new Error("Connection closed, pending request to server0.conn113.child1/workerTarget21/thread1, type removeBreakpoint failed\n\nRequest stack:\nrequest@resource://devtools/shared/protocol/Front.js:166:14\ngenerateRequestMethods/</frontProto[name]@resource://devtools/shared/protocol/Front/FrontClassWithSpec.js:49:19\nremoveBreakpoint/<@resource://devtools/client/debugger/src/client/firefox/commands.js:193:41\nforEachThread/promises<@resource://devtools/client/debugger/src/client/firefox/commands.js:98:16\nforEachThread@resource://devtools/client/debugger/src/client/firefox/commands.js:95:63\nremoveBreakpoint@resource://devtools/client/debugger/src/client/firefox/commands.js:193:10\nclientRemoveBreakpoint@resource://devtools/client/debugger/src/actions/breakpoints/modify.js:54:17\nremoveBreakpoint/<@resource://devtools/client/debugger/src/actions/breakpoints/modify.js:152:69\nthunk/</</<@resource://devtools/client/debugger/src/actions/utils/middleware/thunk.js:21:45\nbindActionCreator/<@resource://devtools/client/shared/vendor/redux.js:644:12\nremoveBreakpoint@chrome://mochitests/content/browser/devtools/client/debugger/test/mochitest/helpers.js:988:22\n@chrome://mochitests/content/browser/devtools/client/debugger/test/mochitest/browser_dbg-worker-scopes.js:47:9\nAsyncTester_execTest/<@chrome://mochikit/content/browser-test.js:1346:34\nTester_execTest@chrome://mochikit/content/browser-test.js:1381:11\nnextTest/<@chrome://mochikit/content/browser-test.js:1209:14\nSimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:803:67\n", "resource://devtools/shared/protocol/Front.js", 72))
[task 2019-08-06T02:33:58.791Z] 02:33:58 INFO - GECKO(1703) | console.log: (new Error("Connection closed, pending request to server0.conn113.child1/workerTarget24/thread1, type removeBreakpoint failed\n\nRequest stack:\nrequest@resource://devtools/shared/protocol/Front.js:166:14\ngenerateRequestMethods/</frontProto[name]@resource://devtools/shared/protocol/Front/FrontClassWithSpec.js:49:19\nremoveBreakpoint/<@resource://devtools/client/debugger/src/client/firefox/commands.js:193:41\nforEachThread/promises<@resource://devtools/client/debugger/src/client/firefox/commands.js:98:16\nforEachThread@resource://devtools/client/debugger/src/client/firefox/commands.js:95:63\nremoveBreakpoint@resource://devtools/client/debugger/src/client/firefox/commands.js:193:10\nclientRemoveBreakpoint@resource://devtools/client/debugger/src/actions/breakpoints/modify.js:54:17\nremoveBreakpoint/<@resource://devtools/client/debugger/src/actions/breakpoints/modify.js:152:69\nthunk/</</<@resource://devtools/client/debugger/src/actions/utils/middleware/thunk.js:21:45\nbindActionCreator/<@resource://devtools/client/shared/vendor/redux.js:644:12\nremoveBreakpoint@chrome://mochitests/content/browser/devtools/client/debugger/test/mochitest/helpers.js:988:22\n@chrome://mochitests/content/browser/devtools/client/debugger/test/mochitest/browser_dbg-worker-scopes.js:47:9\nAsync*Tester_execTest/<@chrome://mochikit/content/browser-test.js:1346:34\nTester_execTest@chrome://mochikit/content/browser-test.js:1381:11\nnextTest/<@chrome://mochikit/content/browser-test.js:1209:14\nSimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:803:67\n", "resource://devtools/shared/protocol/Front.js", 72))
[task 2019-08-06T02:33:59.071Z] 02:33:59 INFO - GECKO(1703) | --DOMWINDOW == 4 (0x108896800) [pid = 1706] [serial = 333] [outer = 0x0] [url = about:blank]
[task 2019-08-06T02:33:59.124Z] 02:33:59 INFO - GECKO(1703) | --DOMWINDOW == 3 (0x10b340200) [pid = 1706] [serial = 329] [outer = 0x0] [url = http://example.com/browser/devtools/client/debugger/test/mochitest/examples/doc-windowless-workers.html]
[task 2019-08-06T02:34:00.134Z] 02:34:00 INFO - GECKO(1703) | --DOMWINDOW == 25 (0x11013f400) [pid = 1703] [serial = 629] [outer = 0x0] [url = about:blank]

If this is getting too annoying, somebody could disable /html/cross-origin-opener/new_window_same_origin.tentative.html on OSX. That accounts for the bulk of the failures. I have looked at this a little, but I haven't managed to reproduce yet.

Hi Andrew
Latest occurence: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=260234767&resultStatus=testfailed%2Cbusted%2Cexception&revision=0178d5f513ca5e0f012c2d6a6a7107a6cae5de58 at html/cross-origin-opener/new_window_same_site_unsafe_allow_outgoing.tentative.html

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=260234767&repo=autoland&lineNumber=28640

https://hg.mozilla.org/mozilla-central/rev/0178d5f513ca

Maybe the failure is happening during shutdown, and the harness is just attributing it to the last test? The tests opens and closes lots of tabs, so I was hoping it was something the test was doing, but I guess not. I wonder if there are leak sanitizer failures for this directory.

This bug has failed 38 times in the last 7 days. Occurs on macosx1014-64, android-hw-p2-8-0-android-aarch64, linux64-qr, windows7-32 and on windows10-64-ccov on debug build types.

Recent log:
https://treeherder.mozilla.org/logviewer.html#?job_id=261312131&repo=mozilla-central

(In reply to Dorel Luca [:dluca] from comment #62)

This bug has failed 38 times in the last 7 days. Occurs on macosx1014-64, android-hw-p2-8-0-android-aarch64, linux64-qr, windows7-32 and on windows10-64-ccov on debug build types.

And if you check the graph at the top, you can see that the failures stopped on 2019-08-13. This is due to the fix of bug 1407593 which removed the assertion.

Andrew, I think the last action to do here is to re-enable the skipped test?

I expect that this failure turned into a leakcheck failure. I don't think it is worth reenabling the test until that gets figured out.