Filed by: archaeopteryx [at] coole-files.de https://treeherder.mozilla.org/logviewer.html#?job_id=172554862&repo=try https://queue.taskcluster.net/v1/task/d4ZQ30W5TNqhopKFWw0j2w/runs/0/artifacts/public/logs/live_backing.log Seen for a central-as-beta simulation. Reproduces frequently. This also failed when it landed earlier: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=f5d571a4cc16a447b38f29f499416293e8044342&selectedJob=172256043 18:45:03 INFO - TEST-INFO | Main app process: exit 0 18:45:03 ERROR - 926 ERROR TEST-UNEXPECTED-FAIL | browser/components/extensions/test/browser/test-oop-extensions/browser_ext_incognito_popup.js | leaked 1 window(s) until shutdown [url = chrome://browser/content/browser.xul] 18:45:03 ERROR - 927 ERROR TEST-UNEXPECTED-FAIL | browser/components/extensions/test/browser/test-oop-extensions/browser_ext_incognito_popup.js | leaked 1 window(s) until shutdown [url = about:blank]

This started to increase from the 10th of July, and the first failure was a bc3: https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=pending&filter-resultStatus=running&filter-resultStatus=success&filter-searchStr=windows%207%20debug%20mochitests%20with%20e10s%20test-windows7-32%2Fdebug-mochitest-browser-chrome-e10s%20m-e10s&tochange=22caf5b03354b71c909e67ec6bc71133494d3137&fromchange=a37eef374d201737232c041304435c5b96e0a046&selectedJob=187367915 Then this changed chunks bc2 with a higher frequency: https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=pending&filter-resultStatus=running&filter-resultStatus=success&filter-searchStr=windows%207%20debug%20mochitests%20with%20e10s%20test-windows7-32%2Fdebug-mochitest-browser-chrome-e10s%20m-e10s&tochange=146466f1d08ab98a22313df531b81d571553e0fb&fromchange=9cbe048f7395f806579c6e878a4166672ecd1639&selectedJob=187841850 jmaher: Can you please have a look at this? Thank you!

I see this coming from an inbound merge: https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=12f49235dfd8d8b7330bf6e722ffdc122752ec0d Ideally you should be able to retrigger on inbound in that range and find the root cause.

I did some more retriggers on the suggested range but couldn't find the root cause: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=success&filter-resultStatus=pending&filter-resultStatus=running&filter-searchStr=Windows%207%20debug%20Mochitests%20with%20e10s%20test-windows7-32%2Fdebug-mochitest-browser-chrome-e10s-2%20M-e10s(bc2)&tochange=3edc9c3ae818490ed36b8bfc8ffdfc9e222b41db&fromchange=fdb4ed369d2e00935c8bc3bcefd547f4cda44525&selectedJob=188292447

retriggering in the mozilla-inbound range was on bc2, but the failure is actually on bc6, so i did some retriggers: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=success&filter-resultStatus=pending&filter-resultStatus=running&filter-searchStr=windows%207%20debug%20mochitests%20with%20e10s%20test-windows7-32%2Fdebug-mochitest-browser-chrome-e10s-6%20m-e10s&tochange=3edc9c3ae818490ed36b8bfc8ffdfc9e222b41db&fromchange=fdb4ed369d2e00935c8bc3bcefd547f4cda44525&selectedJob=187438118 :ebalazs_, can you take a look at that above link and finish backfill/retriggers when more data comes in? If we don't find it there, then I suspect this isn't actionable.

The result from the bc6 retriggers: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=success&filter-resultStatus=pending&filter-resultStatus=running&filter-searchStr=windows%207%20debug%20mochitests%20with%20e10s%20test-windows7-32%2Fdebug-mochitest-browser-chrome-e10s-6%20m-e10s&tochange=3edc9c3ae818490ed36b8bfc8ffdfc9e222b41db&fromchange=e89192032fe21bedf44aa4447f126abf6eba61ae There are two pushes with ~ 50% failure rate: 01380eb6a6d0 and 1f836d71ddc5. So probably the culprit its one of them.

oh nice, Bug 1443590 looks to be the culprit, :dmajor, could you look at this?

Changing compilers probably altered the timing characteristics of the code in a way that exposed an existing latent issue. In terms of actually root-causing it, I'm afraid I don't have the subject-matter experience to investigate this at any reasonable speed. A DOM or Memshrink expert is likely to have better results.

jmaher: Do we have to ni someone else here? (Comment 19) Thank you.

:ddurst, I see you are the triage owner for the component this test is associated with. In comment 18, we tracked this down to a compiler change- with the current rate of failures this test will be disabled soon, is it possible you could look into this test and see if there is something we can do to make it more reliable?

Comment on attachment 8993559 [details] Bug 1452514: Correctly handle a browser window closing before its popups do. https://reviewboard.mozilla.org/r/258258/#review265404

https://hg.mozilla.org/integration/mozilla-inbound/rev/f09c12fc7f40945e4751818ef011fe800097e6e7 Bug 1452514: Correctly handle a browser window closing before its popups do. r=mixedpuppy

We are trying to build a tool to automatically classify intermittent failures, which would provide a starting point for fixing the bug, reducing the manual work for the developers. We are collecting some feedback on the results, to see if they’re good enough and where we need to improve. For this bug, the tool says that the intermittent failure is most likely a: Test Order Dependency: This includes test failures in which one or multiple test(s) have not cleaned up after itself or require a certain state to be created/set before starting to run. Also, this includes test failures in which a test is started while another one is still running, and both are accessing/requiring the same state/resource. Please check whether there are clean-up or start-up dependencies between the test cases. Once you’re done investigating and/or fixing the bug, could you tell me: - Did the tool correctly recognize the type of intermittent failure? - Did the information from the tool help your analysis, the bug fixing process, or anything in the process? (please also let us know how the tool was useful and/or what would improve the tool's usefulness for you)

New occurrences: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&group_state=expanded&selectedJob=187837217&author=cpeterson%40mozilla.com&searchStr=windows%2C7%2Cdebug%2Cmochitests%2Cwith%2Ce10s%2Ctest-windows7-32%2Fdebug-mochitest-browser-chrome-e10s-2%2Cm-e10s%28bc2%29

I'm confused. If this was resolved, then this should be fixed and a new bug opened, no?

That's the standard flow yes, however I don't remember what the circumstances were in that case.