modifications to these tests have resulted in us creating useful tests, these have been running side by side for a while, time to do the replacement!

this will need to ride the trains.

Comment on attachment 779790 [details] [diff] [review] switch tsvg/tscroll with 'x' versions for trunk (1.0) Review of attachment 779790 [details] [diff] [review]: ----------------------------------------------------------------- LGTM.

landed, will wait for the reconfig tomorrow: https://hg.mozilla.org/build/buildbot-configs/rev/5c56fd7fe000 https://hg.mozilla.org/build/buildbot-configs/rev/970e83957e9f

STOP THE PRESS! While I think this replacement is good, I also think we need a wider consensus before deprecating talos tests. I'll post to dev.platform and inform of this coming change, and let's act according to the feedback there. Meanwhile we can unhide the x-tests on tbpl and make them fully active - if they're not already (e.g. email-report regressions to devs, visible by default on tbpl etc).

backed out due to bug 888899 being backed out and we need this fixed on osx first: https://hg.mozilla.org/build/buildbot-configs/rev/e916c85b249e https://hg.mozilla.org/build/buildbot-configs/rev/7fae5a3a155f

Following some reservations about comment 5 from Callek on IRC: - The main reason for the delay is that "ASAP iterations mode" (which is the main difference the X tests bring) is not fully stable yet (bug 888899 to make it work on OSX, bug 880036 for temporarily hanged Firefox when it can't keep up, bug 884955 to overcome this hang). - As for my relation to this bug and my request to back it out: I'm the one who noticed that the old tests are not optimal due to their iterations mode, added the X tests (modified versions of the old tests), and suggested to joel on IRC to deprecate the old tests. Unfortunately, it got file and landed before I noticed that it was filed. So I requested to back it out as soon as I could, before this reaches production. - I still think that exposing this to a wider audience is in place before deprecating talos tests which were our benchmark for a long while, as sub-optimal as they are.

Shouldn't this be needed? +++ b/mozilla/project_branches.py @@ -267,16 +267,19 @@ PROJECT_BRANCHES = { }, 'cedar': { 'mozharness_tag': 'default', 'mozharness_talos': True, 'lock_platforms': True, 'enable_talos': True, 'talos_suites': { 'xperf': 1, + 'otherx': 1, + 'svgx': 1, + 'rafx': 1, }, On another note, are we making the replacing *only* on mozilla-central? Are these new suites only supposed to run on m-c?

this new suites should run on all trunk based branches. we will roll out to aurora on the next merge and so forth.

Comment on attachment 779790 [details] [diff] [review] switch tsvg/tscroll with 'x' versions for trunk (1.0) (In reply to Joel Maher (:jmaher) from comment #9) > this new suites should run on all trunk based branches. we will roll out to > aurora on the next merge and so forth. If so, the currently r+'ed patch would not do that. Changing to r-. I've also noticed that we're not disabling them but changing the coalescing preference: +BRANCHES['mozilla-central']['svgr_tests'] = (1, False, {}, ALL_TALOS_PLATFORMS) +BRANCHES['mozilla-central']['other_tests'] = (1, False, {}, ALL_TALOS_PLATFORMS) This does *not* disable the tests but sets coalescing to False for these jobs. http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/config_common.py#46 Sorry. My review was wrong.

In production

(In reply to Rail Aliiev [:rail] from comment #11) > In production But not really since it was backed out (/me goes to make maintenance page clear)

Before we remove the old tests, we should run the old and new ones in parallel, to be able to assess the effectiveness of the new ones. While I can see the new tests on desktop platforms (tscrollx, tsvgx at T(T) when ?showall=1), On mobile platforms I can see the new tsvgx_nochrome (also T(T)) but not a variant of the new scroll test, and also not the old tscroll. Can we turn on both the old and the new tscroll tests on mobile before we deprecate the old ones?

tscroll is not supported on mobile. So we have a few weeks of side by side data as it stands.

Bug 888899 part 2 - which is in inbound but haven't landed yet - changes the prefs which are used to set ASAP mode (which tsvgx and tscrollx use). If the prefs which are used for tsvgx and tscrollx don't match, then they will give different results. This is expected, and is the main reason for the delay in deprecating the old tests.

landed on talos: http://hg.mozilla.org/build/talos/rev/4063ef2a221e This needs a talos deployment before it is resolved.

Update from an email thread with clint, taras, joel and myself about a week ago: we decided to land the new tests fully production on all trees, and keep the old tests for 3 weeks on m-c only and hidden. Right now it's still the other way around (old tests fully production, new tests only on m-c and hidden).

Updated the Talos wiki sections of tsvg/x, tscroll/x and tsvg_opacity (tsvg_opacity included reference to ASAP even though it wasn't touched and doesn't use ASAP mode). Would appreciate some review/English-editing. https://wiki.mozilla.org/index.php?title=Buildbot%2FTalos%2FTests&diff=702982&oldid=699505

I have done a few minor edits as well as adding a section for tart: https://wiki.mozilla.org/Buildbot/Talos/Tests

Looks good. Thanks!