In our recent meeting [1] about e10s testing, we concluded that: 1. Before e10s is enabled by default, we will run Talos tests on PGO, non-PGO, Linux e10s, and Windows e10s+PGO. To reduce test load, jmaher suggests we only test e10s on Win7, Win8, 32-bit Linux, and 64-bit Linux. 2. After e10s is enabled by default in Nightly, will we test e10s and non-e10s on every push? 3. After e10s is riding the trains (Aurora), we will test e10s on every push but non-e10s on the reduced testing frequency (above). [1] https://etherpad.mozilla.org/uM2IdRZFJf

Ok, working on enabling e10s talos suites for PGO builds only to start with.

disable e10s talos suites on holly; enable e10s suites on m-c for pgo builds.

Comment on attachment 8526084 [details] [diff] [review] pgo_talos-buildbot-configs.diff Review of attachment 8526084 [details] [diff] [review]: ----------------------------------------------------------------- ::: mozilla-tests/config.py @@ +1738,5 @@ > BRANCHES['mozilla-central']['build_branch'] = "1.9.2" > BRANCHES['mozilla-central']['pgo_strategy'] = 'periodic' > +BRANCHES['mozilla-central']['xperf-e10s_tests'] = (1, False, TALOS_TP_NEW_OPTS, WIN7_ONLY) > +BRANCHES['mozilla-central']['tp5o-e10s_tests'] = (1, False, TALOS_TP_NEW_OPTS, NO_WINXP) > +BRANCHES['mozilla-central']['g1-e10s_tests'] = (1, False, TALOS_TP_NEW_OPTS, NO_WINXP) I suspect this NO_WINXP could be adjusted in the future to match non e10s (ALL_TALOS_PLATFORMS)- there could be a reason for not doing this on xp which I am not aware of right now. @@ +1750,5 @@ > + 'other-e10s_nol64', > + 'other-e10s_l64', > + 'svgr-e10s', > + 'dromaeojs-e10s', > + 'chromez-e10s'] will this remove the non e10s stuff from pgo? In many ways that wouldn't be a big deal;

Comment on attachment 8526084 [details] [diff] [review] pgo_talos-buildbot-configs.diff Review of attachment 8526084 [details] [diff] [review]: ----------------------------------------------------------------- ::: mozilla-tests/config.py @@ +1750,5 @@ > + 'other-e10s_nol64', > + 'other-e10s_l64', > + 'svgr-e10s', > + 'dromaeojs-e10s', > + 'chromez-e10s'] No, but maybe that means we should change the name of this configuration key. What this means is that these suites will be run only for PGO builds, not for regular opt builds. The list of builders added/removed by these changes: --- old.txt 2014-11-20 11:10:31.000000000 -0500 +++ new.txt 2014-11-20 11:26:13.000000000 -0500 +mozilla-central Windows 8 64-bit mozilla-central pgo talos other-e10s_nol64 ScriptFactory +mozilla-central Windows 8 64-bit mozilla-central pgo talos svgr-e10s ScriptFactory +mozilla-central Windows 8 64-bit mozilla-central pgo talos dromaeojs-e10s ScriptFactory +mozilla-central Windows 8 64-bit mozilla-central pgo talos chromez-e10s ScriptFactory +mozilla-central Windows 8 64-bit mozilla-central pgo talos tp5o-e10s ScriptFactory +mozilla-central Windows 8 64-bit mozilla-central pgo talos g1-e10s ScriptFactory +mozilla-central Windows 7 32-bit mozilla-central pgo talos other-e10s_nol64 ScriptFactory +mozilla-central Windows 7 32-bit mozilla-central pgo talos xperf-e10s ScriptFactory +mozilla-central Windows 7 32-bit mozilla-central pgo talos svgr-e10s ScriptFactory +mozilla-central Windows 7 32-bit mozilla-central pgo talos dromaeojs-e10s ScriptFactory +mozilla-central Windows 7 32-bit mozilla-central pgo talos chromez-e10s ScriptFactory +mozilla-central Windows 7 32-bit mozilla-central pgo talos tp5o-e10s ScriptFactory +mozilla-central Windows 7 32-bit mozilla-central pgo talos g1-e10s ScriptFactory +mozilla-central Windows XP 32-bit mozilla-central pgo talos other-e10s_nol64 ScriptFactory +mozilla-central Ubuntu HW 12.04 x64 mozilla-central pgo talos other-e10s_l64 ScriptFactory +mozilla-central Ubuntu HW 12.04 x64 mozilla-central pgo talos svgr-e10s ScriptFactory +mozilla-central Ubuntu HW 12.04 x64 mozilla-central pgo talos dromaeojs-e10s ScriptFactory +mozilla-central Ubuntu HW 12.04 x64 mozilla-central pgo talos chromez-e10s ScriptFactory +mozilla-central Ubuntu HW 12.04 x64 mozilla-central pgo talos tp5o-e10s ScriptFactory +mozilla-central Ubuntu HW 12.04 x64 mozilla-central pgo talos g1-e10s ScriptFactory +mozilla-central Ubuntu HW 12.04 mozilla-central pgo talos other-e10s_nol64 ScriptFactory +mozilla-central Ubuntu HW 12.04 mozilla-central pgo talos svgr-e10s ScriptFactory +mozilla-central Ubuntu HW 12.04 mozilla-central pgo talos dromaeojs-e10s ScriptFactory +mozilla-central Ubuntu HW 12.04 mozilla-central pgo talos chromez-e10s ScriptFactory +mozilla-central Ubuntu HW 12.04 mozilla-central pgo talos tp5o-e10s ScriptFactory +mozilla-central Ubuntu HW 12.04 mozilla-central pgo talos g1-e10s ScriptFactory -holly Windows 8 64-bit holly talos other-e10s_nol64 ScriptFactory -holly Windows 8 64-bit holly talos svgr-e10s ScriptFactory -holly Windows 8 64-bit holly talos dromaeojs-e10s ScriptFactory -holly Windows 8 64-bit holly talos chromez-e10s ScriptFactory -holly Windows 8 64-bit holly talos tp5o-e10s ScriptFactory -holly Windows 8 64-bit holly talos g1-e10s ScriptFactory -holly Rev5 MacOSX Mountain Lion 10.8 holly talos other-e10s_nol64 ScriptFactory -holly Rev5 MacOSX Mountain Lion 10.8 holly talos svgr-e10s ScriptFactory -holly Rev5 MacOSX Mountain Lion 10.8 holly talos dromaeojs-e10s ScriptFactory -holly Rev5 MacOSX Mountain Lion 10.8 holly talos chromez-e10s ScriptFactory -holly Rev5 MacOSX Mountain Lion 10.8 holly talos tp5o-e10s ScriptFactory -holly Rev5 MacOSX Mountain Lion 10.8 holly talos g1-e10s ScriptFactory -holly Rev4 MacOSX Snow Leopard 10.6 holly talos other-e10s_nol64 ScriptFactory -holly Rev4 MacOSX Snow Leopard 10.6 holly talos svgr-e10s ScriptFactory -holly Rev4 MacOSX Snow Leopard 10.6 holly talos dromaeojs-e10s ScriptFactory -holly Rev4 MacOSX Snow Leopard 10.6 holly talos chromez-e10s ScriptFactory -holly Rev4 MacOSX Snow Leopard 10.6 holly talos tp5o-e10s ScriptFactory -holly Rev4 MacOSX Snow Leopard 10.6 holly talos g1-e10s ScriptFactory -holly Windows 7 32-bit holly talos other-e10s_nol64 ScriptFactory -holly Windows 7 32-bit holly talos xperf-e10s ScriptFactory -holly Windows 7 32-bit holly talos svgr-e10s ScriptFactory -holly Windows 7 32-bit holly talos dromaeojs-e10s ScriptFactory -holly Windows 7 32-bit holly talos chromez-e10s ScriptFactory -holly Windows 7 32-bit holly talos tp5o-e10s ScriptFactory -holly Windows 7 32-bit holly talos g1-e10s ScriptFactory -holly Windows XP 32-bit holly talos other-e10s_nol64 ScriptFactory -holly Ubuntu HW 12.04 x64 holly talos other-e10s_l64 ScriptFactory -holly Ubuntu HW 12.04 x64 holly talos svgr-e10s ScriptFactory -holly Ubuntu HW 12.04 x64 holly talos dromaeojs-e10s ScriptFactory -holly Ubuntu HW 12.04 x64 holly talos chromez-e10s ScriptFactory -holly Ubuntu HW 12.04 x64 holly talos tp5o-e10s ScriptFactory -holly Ubuntu HW 12.04 x64 holly talos g1-e10s ScriptFactory -holly Ubuntu HW 12.04 holly talos other-e10s_nol64 ScriptFactory -holly Ubuntu HW 12.04 holly talos svgr-e10s ScriptFactory -holly Ubuntu HW 12.04 holly talos dromaeojs-e10s ScriptFactory -holly Ubuntu HW 12.04 holly talos chromez-e10s ScriptFactory -holly Ubuntu HW 12.04 holly talos tp5o-e10s ScriptFactory -holly Ubuntu HW 12.04 holly talos g1-e10s ScriptFactory

(In reply to Chris Peterson (:cpeterson) from comment #0) > 2. After e10s is enabled by default in Nightly, will we test e10s and > non-e10s on every push? I can't see any reason to continue testing non-e10s after we enable it by default, unless we think there's a significant chance that we'll turn it off by default again.

(In reply to Ben Hearsum [:bhearsum] from comment #6) > (In reply to Chris Peterson (:cpeterson) from comment #0) > > 2. After e10s is enabled by default in Nightly, will we test e10s and > > non-e10s on every push? > > I can't see any reason to continue testing non-e10s after we enable it by > default, unless we think there's a significant chance that we'll turn it off > by default again. e10s is currently enabled by default on Nightly, but it will not ride to Aurora. We need continue testing non-e10s on mozilla-central to avoid surprise regressions when mozilla-central is merged to Aurora where e10s is no longer enabled by default.

Comment on attachment 8526084 [details] [diff] [review] pgo_talos-buildbot-configs.diff Review of attachment 8526084 [details] [diff] [review]: ----------------------------------------------------------------- catlee, thanks for writing this patch and answering my questions. I look forward to this going live and us getting closer to e10s for everything. Regarding e10s vs non e10s testing, users can disable this feature, I can see us testing non e10s on nightlies or in some limited fashion for a few release cycles after we have already released e10s on the release channel.

In production: https://hg.mozilla.org/build/buildbot-configs/rev/e3fc54de6e7e

In production: https://hg.mozilla.org/build/buildbotcustom/rev/60e31b9c1457

to wrap up this bug: * turn off the winxp stuff - we are perma fail there * determine if it is simple to get osx running (we don't do pgo for osx, so it didn't get scheduled) some actions or reasons related to the above mentioned two items would be nice to see so we can close this bug out

(In reply to Joel Maher (:jmaher) from comment #11) > to wrap up this bug: > * turn off the winxp stuff - we are perma fail there > * determine if it is simple to get osx running (we don't do pgo for osx, so > it didn't get scheduled) catlee: are these tasks on your plate? Do you need a developer from the e10s team to investigate the WinXP perma fail?

alright, winxp tests will be ready to go here soon- we run all suites on xp except for dromaeo and g1. I am not sure how to get osx tests up and running on this schedule.

ok, winxp fix is on m-c, likewise all the other tests should be converted (session restore is still broken though).

what are the next steps here? Are we ready to run this more frequently? I don't recall any regressions specific to e10s talos, so maybe there isn't a need to go forward. My impression is that we need to get osx running e10s, once that is in place- we can have a better idea of what is working and what differences we have on our existing tests between e10s/non-e10s.

Is this still blocked by getting OSX running e10s?

we have osx running e10s tests. We still need to get e10s scheduled on inbound/fx-team, even if we don't run it there. Then at least we have the ability to backfill.

ok, we are finding more and more e10s only regressions, this needs to get resolved one way or another to avoid wasting days of time pushing to try and waiting. this adds all (but xperf) talos e10s jobs to inbound[+pgo] and fx-team[+pgo]. I know this isn't ideal for buildername limits and available machines, but without anybody adding a patch to solve this you get what you get! ideally we could force coalescing with SETA or some other means, and then backfill as needed which would be 100% acceptable.

Thanks catlee. let me get this in and then we can determine ways to force coalescing if needed.

In production: https://hg.mozilla.org/build/buildbot-configs/rev/ffd6aedf3243

as these are permafail for months, lets disable them, ideally I will enable all of this on try server so folks can debug there. For now lets free up some hours of osx 10.10 machine time. here is the builder differences: Builders removed - Rev5 MacOSX Yosemite 10.10 fx-team talos g1-osx-e10s - Rev5 MacOSX Yosemite 10.10 mozilla-central talos g1-osx-e10s - Rev5 MacOSX Yosemite 10.10 mozilla-inbound talos g1-osx-e10s

Comment on attachment 8644536 [details] [diff] [review] disable osx g1 tests for talos e10s (1.0) Review of attachment 8644536 [details] [diff] [review]: ----------------------------------------------------------------- not a huge fan of commenting out in this file, since this file is already pretty crufty, but assuming its temporary and you're tracking it I'm ok with it

In production: https://hg.mozilla.org/build/buildbot-configs/rev/574f0cc2646e