Based on https://treeherder.mozilla.org/#/jobs?repo=holly&revision=0ee9557bcd4b it looks like we can turn on Linux debug e10s reftests. The only thing making them orange now is a small child process leak that we already whitelist in other test suites. We should just do that here and enable them.

Where is the code for reftests? They may not be set up yet to have leak thresholds configured per process type.

There's something here: http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/runreftest.py#782

Ah, I even wrote that line! I guess it is set up, somebody will just have to add a "tab" case.

Retitling this since I now want to do b-c and layout reftests in one bug. The patches in bug 1150151 make these tests green, but I'd like to get review started on the automation patch.

Can you please review and test this Armen? I think it should be pretty similar to other e10s automation changes. Here's a try push with green reftests (ignore the bc tests here): https://treeherder.mozilla.org/#/jobs?repo=try&revision=49e74e9ac0fd Here's a try push with reasonably green bc tests: https://treeherder.mozilla.org/#/jobs?repo=try&revision=58bf11b5cec8 I think the failures here are pre-existing. I'll do a push with lots more retriggers once this is closer to landing.

By running this [1], I can determine that these builders are added [2]. Before we land this, we have to make sure that your changes reach mozilla-central and after that a day or two to make sure your changes make it into the trees mentioned in [2]. You can use trigger.py: http://mozilla-ci-tools.readthedocs.org/en/latest/scripts.html#trigger-py to trigger many jobs on a try push. You can use the flag --times to say how many times. Let me know if you have issues with it. [1] https://wiki.mozilla.org/ReleaseEngineering:TestingTechniques#List_builders_added.2Fremoved [2] Builders added: + Ubuntu VM 12.04 ash debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 ash debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 ash debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 b2g-inbound debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 b2g-inbound debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 b2g-inbound debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 cedar debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 cedar debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 cedar debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 cypress debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 cypress debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 cypress debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 elm debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 elm debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 elm debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 fig debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 fig debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 fig debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 fx-team debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 fx-team debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 fx-team debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 gum debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 gum debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 gum debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 jamun debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 jamun debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 jamun debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 maple debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 maple debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 maple debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 mozilla-central debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 mozilla-central debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 mozilla-central debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 mozilla-inbound debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 mozilla-inbound debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 mozilla-inbound debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 oak debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 oak debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 oak debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 try debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 try debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 try debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 ash debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 ash debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 ash debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 b2g-inbound debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 b2g-inbound debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 b2g-inbound debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 cedar debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 cedar debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 cedar debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 cypress debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 cypress debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 cypress debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 elm debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 elm debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 elm debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 fig debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 fig debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 fig debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 fx-team debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 fx-team debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 fx-team debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 gum debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 gum debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 gum debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 jamun debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 jamun debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 jamun debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 maple debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 maple debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 maple debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 mozilla-inbound debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 mozilla-inbound debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 mozilla-inbound debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 oak debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 oak debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 oak debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 try debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 try debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 try debug test mochitest-e10s-browser-chrome-3

Also make sure that you review this and your suite meet the expectations: https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy

Thanks Armen. I don't see any extra lines for reftests in the list of builders added. Is that expected?

It is a naming collision. We should say "reftest-e10s" instead of "reftest". Builders added: + Ubuntu VM 12.04 ash debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 ash debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 ash debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 ash debug test reftest-e10s-1 + Ubuntu VM 12.04 ash debug test reftest-e10s-2 + Ubuntu VM 12.04 b2g-inbound debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 b2g-inbound debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 b2g-inbound debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 b2g-inbound debug test reftest-e10s-1 + Ubuntu VM 12.04 b2g-inbound debug test reftest-e10s-2 + Ubuntu VM 12.04 cedar debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 cedar debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 cedar debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 cedar debug test reftest-e10s-1 + Ubuntu VM 12.04 cedar debug test reftest-e10s-2 + Ubuntu VM 12.04 cypress debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 cypress debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 cypress debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 cypress debug test reftest-e10s-1 + Ubuntu VM 12.04 cypress debug test reftest-e10s-2 + Ubuntu VM 12.04 elm debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 elm debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 elm debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 elm debug test reftest-e10s-1 + Ubuntu VM 12.04 elm debug test reftest-e10s-2 + Ubuntu VM 12.04 fig debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 fig debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 fig debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 fig debug test reftest-e10s-1 + Ubuntu VM 12.04 fig debug test reftest-e10s-2 + Ubuntu VM 12.04 fx-team debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 fx-team debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 fx-team debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 fx-team debug test reftest-e10s-1 + Ubuntu VM 12.04 fx-team debug test reftest-e10s-2 + Ubuntu VM 12.04 gum debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 gum debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 gum debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 gum debug test reftest-e10s-1 + Ubuntu VM 12.04 gum debug test reftest-e10s-2 + Ubuntu VM 12.04 jamun debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 jamun debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 jamun debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 jamun debug test reftest-e10s-1 + Ubuntu VM 12.04 jamun debug test reftest-e10s-2 + Ubuntu VM 12.04 maple debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 maple debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 maple debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 maple debug test reftest-e10s-1 + Ubuntu VM 12.04 maple debug test reftest-e10s-2 + Ubuntu VM 12.04 mozilla-central debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 mozilla-central debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 mozilla-central debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 mozilla-central debug test reftest-e10s-1 + Ubuntu VM 12.04 mozilla-central debug test reftest-e10s-2 + Ubuntu VM 12.04 mozilla-inbound debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 mozilla-inbound debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 mozilla-inbound debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 mozilla-inbound debug test reftest-e10s-1 + Ubuntu VM 12.04 mozilla-inbound debug test reftest-e10s-2 + Ubuntu VM 12.04 oak debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 oak debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 oak debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 oak debug test reftest-e10s-1 + Ubuntu VM 12.04 oak debug test reftest-e10s-2 + Ubuntu VM 12.04 try debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 try debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 try debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 try debug test reftest-e10s-1 + Ubuntu VM 12.04 try debug test reftest-e10s-2 + Ubuntu VM 12.04 x64 ash debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 ash debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 ash debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 ash debug test reftest-e10s-1 + Ubuntu VM 12.04 x64 ash debug test reftest-e10s-2 + Ubuntu VM 12.04 x64 b2g-inbound debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 b2g-inbound debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 b2g-inbound debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 b2g-inbound debug test reftest-e10s-1 + Ubuntu VM 12.04 x64 b2g-inbound debug test reftest-e10s-2 + Ubuntu VM 12.04 x64 cedar debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 cedar debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 cedar debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 cedar debug test reftest-e10s-1 + Ubuntu VM 12.04 x64 cedar debug test reftest-e10s-2 + Ubuntu VM 12.04 x64 cypress debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 cypress debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 cypress debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 cypress debug test reftest-e10s-1 + Ubuntu VM 12.04 x64 cypress debug test reftest-e10s-2 + Ubuntu VM 12.04 x64 elm debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 elm debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 elm debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 elm debug test reftest-e10s-1 + Ubuntu VM 12.04 x64 elm debug test reftest-e10s-2 + Ubuntu VM 12.04 x64 fig debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 fig debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 fig debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 fig debug test reftest-e10s-1 + Ubuntu VM 12.04 x64 fig debug test reftest-e10s-2 + Ubuntu VM 12.04 x64 fx-team debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 fx-team debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 fx-team debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 fx-team debug test reftest-e10s-1 + Ubuntu VM 12.04 x64 fx-team debug test reftest-e10s-2 + Ubuntu VM 12.04 x64 gum debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 gum debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 gum debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 gum debug test reftest-e10s-1 + Ubuntu VM 12.04 x64 gum debug test reftest-e10s-2 + Ubuntu VM 12.04 x64 jamun debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 jamun debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 jamun debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 jamun debug test reftest-e10s-1 + Ubuntu VM 12.04 x64 jamun debug test reftest-e10s-2 + Ubuntu VM 12.04 x64 maple debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 maple debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 maple debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 maple debug test reftest-e10s-1 + Ubuntu VM 12.04 x64 maple debug test reftest-e10s-2 + Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 mozilla-central debug test reftest-e10s-1 + Ubuntu VM 12.04 x64 mozilla-central debug test reftest-e10s-2 + Ubuntu VM 12.04 x64 mozilla-inbound debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 mozilla-inbound debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 mozilla-inbound debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 mozilla-inbound debug test reftest-e10s-1 + Ubuntu VM 12.04 x64 mozilla-inbound debug test reftest-e10s-2 + Ubuntu VM 12.04 x64 oak debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 oak debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 oak debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 oak debug test reftest-e10s-1 + Ubuntu VM 12.04 x64 oak debug test reftest-e10s-2 + Ubuntu VM 12.04 x64 try debug test mochitest-e10s-browser-chrome-1 + Ubuntu VM 12.04 x64 try debug test mochitest-e10s-browser-chrome-2 + Ubuntu VM 12.04 x64 try debug test mochitest-e10s-browser-chrome-3 + Ubuntu VM 12.04 x64 try debug test reftest-e10s-1 + Ubuntu VM 12.04 x64 try debug test reftest-e10s-2

Doh. Thanks.

https://hg.mozilla.org/build/buildbot-configs/rev/702ae244be8d Try run: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b630a13cf3bd

In production: https://hg.mozilla.org/build/buildbot-configs/rev/702ae244be8d

in bug 1155423, we have a r-e10s-1 intermittent which started on April 16th (shortly after this was deployed in production. I spent some time this weekend retriggering jobs and realized that the frequency of this vs when it should have found a root cause was out of alignment- I went back a full day on inbound and it is still showing up: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=Ubuntu%20VM%2012.04%20x64%20mozilla-inbound%20debug%20test%20reftest-e10s-1&fromchange=a365d588c8df&tochange=c75413a564b2 Some questions: * did we run r-e10s-1 on linux64 prior to this patch? Or was it just r-e10s ? * how much testing was done on this specific patch? (when I do buildbot changes, I verify buildernames, not necessarily tests) * any general thoughts on the intermittent we are seeing?

> I spent some time this weekend retriggering jobs and realized that the frequency of this vs > when it should have found a root cause was out of alignment Could you clarify what you mean by this? Before this bug we did not run any DEBUG reftests. We did run r-e10s in linux and linux64 opt. We didn't run r-e10s-1. At some point I retriggered reftests a bunch of times, but probably not enough for this to show up. From your treeherder link, it seems to happen less than 5% of the time (I counted 25 failures in about 640 retriggers, although that's a guess since I don't know how to get treeherder to show that data). I moved the bug on the intermittent to graphics since it seems to be a bug in our vsync code. Perhaps they'll figure something out.

yeah, I had some bad english there. We got an original report of linux debug r-e10s-1 failing, after retriggers, it reproduced easily, so I went back in time. The further back in time we go, we start to get to cases where we have ran the test more times than what it would take to reproduce, therefore increasing the likelihood that the failure is not related to the code landing nearby the original reporting. I suspect this is just an issue with debug r-e10s tests since when the buildbot changes in this bug landed we ran those tests for the first time. While this is a low failure rate, it is one of the more frequent intermittent failures filed in the last month. Thanks for the info billm and trying to get this to the right people to look at it.

Can this bug be closed, Bill? It looks like we are running these e10s tests now.