Blake Kaplan (:mrbkap) (inactive) Reporter
	Description • 9 years ago

As we're getting closer to having e10s ride the trains, having good automated test coverage is going to get more and more important. We currently only run mochitests with e10s on Linux and I'd like to extend that to OSX and Windows.

I understand that there's a cost in machine time to this, so we might want to figure out a throttling strategy to not cause too much drag on treeherder. What needs to be done to make this happen?

I have a try run at https://treeherder.mozilla.org/#/jobs?repo=try&revision=1cad3dcad6ff

	Ryan VanderMeulen [:RyanVM] Assignee
	Comment 1 • 9 years ago

The main issue I see here is capacity. OSX 10.10 is our smallest slave pool by a large margin (the next-smallest pool - OSX 10.6 - has ~50% more capacity). RelOps has new OSX hardware in the budget for this year, but it's difficult to say how long it will take for it to be ordered, delivered, installed, validated for production duty, etc.

Joel Maher and Kim Moir have done a lot of work on SETA (Search for Extraneous Test Automation) to reduce the amount of test load by leveraging historical data to force-coalesce certain test suites. However, we need said data before it can be used to reduce load :). Already we see OSX 10.10 test backlog on a daily basis under typical load conditions. And we've already had to resort to ghastly hacks like not running 10.10 tests by default on Try to even have things be as good as they currently are.

Windows isn't in much greater shape, but isn't quite as dire as 10.10. We have a small ray of hope of increasing the Windows test slave capacity in the relative near future by pushing Windows builds into the cloud and repurposing those machines as test slaves instead. However, I don't know what the current status of that work is.

I'm sorry to be a stick in the mud about this. I can fully appreciate why you want to see this done, but I'm afraid that adding another 10 jobs per platform (assuming opt+debug) per push is going to cause tremendous pain from a test backlog standpoint.

I've added a few RelEng/RelOps people to the CC list in case they want to chime in as well.

	Amy Rich [:arr] [:arich]
	Comment 2 • 9 years ago

FYI the order for the new OS X machines has been placed, but it will likely be at least a month before we see any capacity increase there, and that will be gradual as we perform rolling replacements of the existing legacy hardware.

The windows build machines can not be re-purposed as test machines (they run on very different, very old hardware that can't even support our needs for tests), so there won't be any increase in capacity coming from that quarter.

	Joel Maher ( :jmaher ) (UTC -8)
	Comment 3 • 9 years ago

We can crank up SETA again now that we have more tools in place.  How many new OSX machines did we order?  We keep adding jobs, I think we should be targeting 50% capacity growth.  With that said, do we have a pending order to increase our windows capacity?

	Amy Rich [:arr] [:arich]
	Comment 4 • 9 years ago

Based on growth numbers from the past 4 years, we've ordered 200 minis (roughly double what we have now for 10.10), but that capacity has to last us 3 years. We're also shifting the try load from 10.6 to 10.10, so there will be a pretty immediate increase in load due to that.

There are no plans or budget to increase windows test hardware capacity this year.

	Joel Maher ( :jmaher ) (UTC -8)
	Comment 5 • 9 years ago

thanks for the info :arr.  Sounds like we need to look more into selective test running per commit with things like SETA or other tools if we want to turn this stuff on.

	Blake Kaplan (:mrbkap) (inactive) Reporter
	Comment 6 • 9 years ago

I figured that this wouldn't be as easy as disabling a couple of tests and flipping a switch :-)

That being said, this is definitely something that needs to happen, preferably before e10s hits the beta channel. There's no way that it makes sense to ship e10s without full automated test coverage on all of the platforms we ship on.

Out of curiosity, what do the numbers look like for Windows, which is probably the most important platform to run tests on?

	Amy Rich [:arr] [:arich]
	Comment 7 • 9 years ago

:ryanvm, do you have the numbers that :mdbkap asked for in comment 6, or do we need to ask someone else for those?

	Ryan VanderMeulen [:RyanVM] Assignee
	Comment 8 • 9 years ago

All I know is what slave health tells me - 172 WinXP, 171 Win7, 180 Win8.

	Ryan VanderMeulen [:RyanVM] Assignee
	Comment 9 • 9 years ago

Note that those are the absolute numbers, not accounting for any disabled/loaner/decommed/whatever slaves not taking jobs.

	Amy Rich [:arr] [:arich]
	Comment 10 • 9 years ago

Oh, sorry, yes, I could have provided those, but I presumed that :mrbkap was asking about utilization and available capacity to run jobs.

	Ryan VanderMeulen [:RyanVM] Assignee
	Comment 11 • 9 years ago

Besides "awful" I don't have a great answer. We're constantly backlogged on Windows as it is. Even with inbound closed for a good portion of the day today, XP currently has 386 pending, 7 has 659, and 8 has 534.

We're going to need some seriously aggressive SETA to withstand turning on 10 more jobs per platform.

	Chris AtLee [:catlee]
	Comment 12 • 9 years ago

I think our only real option is to enable the tests and suffer worse wait times / coalescing. 

The real question is how to minimize the impact.

Can we start by running the e10s tests infrequently? Or conversely, run the non-e10s tests infrequently?

	Ryan VanderMeulen [:RyanVM] Assignee
	Comment 13 • 9 years ago

We have lots of new OSX hardware, where do we stand on standing up more e10s test coverage now?

	Jonathan Griffin (:jgriffin)
	Comment 14 • 9 years ago

(In reply to Ryan VanderMeulen [:RyanVM] from comment #13)
> We have lots of new OSX hardware, where do we stand on standing up more e10s
> test coverage now?

We haven't started; I've been prioritizing Windows but can start looking at things on OSX.

	Jonathan Griffin (:jgriffin)
	Comment 15 • 8 years ago

Here's what a current try run looks like with e10s enabled: https://treeherder.mozilla.org/#/jobs?repo=try&revision=544469683953&selectedJob=17293727

It's not too bad; we can get to green by selectively disabling pretty easily, I hope.

	Pulsebot
	Comment 16 • 8 years ago

https://hg.mozilla.org/integration/mozilla-inbound/rev/d04d0ed7d5a4
https://hg.mozilla.org/integration/mozilla-inbound/rev/98b74336b85f

	Carsten Book [:Tomcat]
	Comment 17 • 8 years ago
bugherder

https://hg.mozilla.org/mozilla-central/rev/d04d0ed7d5a4
https://hg.mozilla.org/mozilla-central/rev/98b74336b85f

	Ryan VanderMeulen [:RyanVM] Assignee
	Comment 18 • 8 years ago

Quick status update:

I've landed test disabling patches for a lot of the deps from this bug and things are starting to shape up to a point where we could conceivably have them running soon in some capacity in production. Mochitest-JP is broken across the board (and my understanding is that's the case in general for them on e10s). Mochitest-push is failing 20-30% of the time on opt/debug, hitting two different known intermittents. Mochitest-other is broken, but that's the case for e10s in general.

OPT: Mochitests are mostly green at this point, except for a never-ending parade of devtools intermittent whack-a-mole. Some websocket tests seem to be pretty orange-prone (see deps). Crashtests and jsreftests are green and stable. Reftests are permafail on 10.6 due to APZ issues related to scrollbar drawing. If push came to shove, we could probably just annotate those away, but kats is actively investigating them. Reftests are better on 10.10, but still fail about 20% of the time.

DEBUG: Mochitest-plain is mostly green, save for a smattering of gfx leaks across different directories and a semi-frequent service worker leak-the-world that affects Windows too. Crashtests and mochitest-gl are currently perma-assert and/or crashing. I could annotate around the crashtest failures if need-be, but it looks like the gfx team is already working on a fix anyway. Reftests are in the same boat as their opt counterparts. Devtools have been a never-ending game of docShell leaks, and disabling one test just makes another one pop up instead.

	Ryan VanderMeulen [:RyanVM] Assignee
	Comment 19 • 8 years ago

The new OSX 10.10 test machines are online now. I think it's time to get OSX debug e10s tests enabled across the board instead of only on Ash. There's on devtools-specific permafail that will need addressing (bug 1268319), but I'll happily skip that directory on e10s if that's all that's blocking at this point.

	Ryan VanderMeulen [:RyanVM] Assignee
	Comment 20 • 8 years ago

We're already running all the opt suites we care about in production, so we only need to turn on the remaining debug tests. I'm targeting 48+ since that's the version we're targeting for wider release.

Builders added:
+ Rev7 MacOSX Yosemite 10.10.5 fx-team debug test crashtest-e10s
+ Rev7 MacOSX Yosemite 10.10.5 fx-team debug test jsreftest-e10s
+ Rev7 MacOSX Yosemite 10.10.5 fx-team debug test mochitest-e10s-1
+ Rev7 MacOSX Yosemite 10.10.5 fx-team debug test mochitest-e10s-2
+ Rev7 MacOSX Yosemite 10.10.5 fx-team debug test mochitest-e10s-3
+ Rev7 MacOSX Yosemite 10.10.5 fx-team debug test mochitest-e10s-4
+ Rev7 MacOSX Yosemite 10.10.5 fx-team debug test mochitest-e10s-5
+ Rev7 MacOSX Yosemite 10.10.5 fx-team debug test mochitest-e10s-devtools-chrome-1
+ Rev7 MacOSX Yosemite 10.10.5 fx-team debug test mochitest-e10s-devtools-chrome-2
+ Rev7 MacOSX Yosemite 10.10.5 fx-team debug test mochitest-e10s-devtools-chrome-3
+ Rev7 MacOSX Yosemite 10.10.5 fx-team debug test mochitest-e10s-devtools-chrome-4
+ Rev7 MacOSX Yosemite 10.10.5 fx-team debug test mochitest-e10s-devtools-chrome-5
+ Rev7 MacOSX Yosemite 10.10.5 fx-team debug test mochitest-e10s-devtools-chrome-6
+ Rev7 MacOSX Yosemite 10.10.5 fx-team debug test mochitest-e10s-devtools-chrome-7
+ Rev7 MacOSX Yosemite 10.10.5 fx-team debug test mochitest-e10s-devtools-chrome-8
+ Rev7 MacOSX Yosemite 10.10.5 fx-team debug test mochitest-gl-e10s
+ Rev7 MacOSX Yosemite 10.10.5 fx-team debug test mochitest-media-e10s
+ Rev7 MacOSX Yosemite 10.10.5 fx-team debug test reftest-e10s
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test crashtest-e10s
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test jsreftest-e10s
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-1
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-2
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-3
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-4
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-5
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-devtools-chrome-1
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-devtools-chrome-2
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-devtools-chrome-3
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-devtools-chrome-4
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-devtools-chrome-5
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-devtools-chrome-6
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-devtools-chrome-7
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-devtools-chrome-8
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-gl-e10s
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-media-e10s
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test reftest-e10s
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test crashtest-e10s
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test jsreftest-e10s
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test mochitest-e10s-1
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test mochitest-e10s-2
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test mochitest-e10s-3
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test mochitest-e10s-4
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test mochitest-e10s-5
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test mochitest-e10s-devtools-chrome-1
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test mochitest-e10s-devtools-chrome-2
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test mochitest-e10s-devtools-chrome-3
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test mochitest-e10s-devtools-chrome-4
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test mochitest-e10s-devtools-chrome-5
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test mochitest-e10s-devtools-chrome-6
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test mochitest-e10s-devtools-chrome-7
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test mochitest-e10s-devtools-chrome-8
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test mochitest-gl-e10s
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test mochitest-media-e10s
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-central debug test reftest-e10s
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound debug test crashtest-e10s
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound debug test jsreftest-e10s
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound debug test mochitest-e10s-1
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound debug test mochitest-e10s-2
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound debug test mochitest-e10s-3
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound debug test mochitest-e10s-4
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound debug test mochitest-e10s-5
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound debug test mochitest-e10s-devtools-chrome-1
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound debug test mochitest-e10s-devtools-chrome-2
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound debug test mochitest-e10s-devtools-chrome-3
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound debug test mochitest-e10s-devtools-chrome-4
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound debug test mochitest-e10s-devtools-chrome-5
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound debug test mochitest-e10s-devtools-chrome-6
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound debug test mochitest-e10s-devtools-chrome-7
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound debug test mochitest-e10s-devtools-chrome-8
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound debug test mochitest-gl-e10s
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound debug test mochitest-media-e10s
+ Rev7 MacOSX Yosemite 10.10.5 mozilla-inbound debug test reftest-e10s
+ Rev7 MacOSX Yosemite 10.10.5 try debug test crashtest-e10s
+ Rev7 MacOSX Yosemite 10.10.5 try debug test jsreftest-e10s
+ Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-1
+ Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-2
+ Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-3
+ Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-4
+ Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-5
+ Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-devtools-chrome-1
+ Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-devtools-chrome-2
+ Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-devtools-chrome-3
+ Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-devtools-chrome-4
+ Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-devtools-chrome-5
+ Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-devtools-chrome-6
+ Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-devtools-chrome-7
+ Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-e10s-devtools-chrome-8
+ Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-gl-e10s
+ Rev7 MacOSX Yosemite 10.10.5 try debug test mochitest-media-e10s
+ Rev7 MacOSX Yosemite 10.10.5 try debug test reftest-e10s

	Ryan VanderMeulen [:RyanVM] Assignee
	Comment 21 • 8 years ago

Comment on attachment 8752290 [details] [diff] [review]
enable remaining OSX debug e10s suites on Gecko 48+

https://hg.mozilla.org/build/buildbot-configs/rev/343ac36a1864

	Release Engineering SlaveAPI Service
	Comment 22 • 8 years ago

In production: https://hg.mozilla.org/build/buildbot-configs/rev/343ac36a1864

	Ryan VanderMeulen [:RyanVM] Assignee
	Comment 23 • 8 years ago

\m/