We currently have a limited set of talos tests that can run with the new web render software fallback backend. When using mach try chooser --full and selecting 'shippable' builds, these tests show up as (test)-swr variants. Currently we have the following tests available on try:

talos-other-swr
talos-g4-swr
talos-g1-swr
talos-bcv-swr
talos-svgr-swr

We would like to add swr runs for the following tests:
talos-sessionrestore-manywindows
talos-perf-reftest
talos-g5
talos-g3
talos-tp5o
talos-damp
talos-chromez
talos-tabswitch
talos-webgl
talos-xperf

In testing on Try, it looks like the set of available -swr tests only show up for Linux64. I'd like to expand that list to Windows (10/7), and MacOS.

It also appears that most of the result tracking (perfherder etc) aren't seeing the -swr configuration as a separate line, and results are being lost/merged.

Andrew, can you please help us get this fixed?

Hey Mihai, Joe Walker suggested I ping you on this. Hopefully you can help get this prioritized?

Apologies for the delay, I'll take a look this afternoon. Was leaving the needinfo as a reminder, but should have left a comment.

(In reply to Jim Mathies [:jimm] from comment #1)

In testing on Try, it looks like the set of available -swr tests only show up for Linux64. I'd like to expand that list to Windows (10/7), and MacOS.

To clarify, is it necessary to run on both Windows 7 and 10? We're trying to be a little less gratuitous about where things run these days.

(In reply to Andrew Halberstadt [:ahal] from comment #5)

(In reply to Jim Mathies [:jimm] from comment #1)

In testing on Try, it looks like the set of available -swr tests only show up for Linux64. I'd like to expand that list to Windows (10/7), and MacOS.

To clarify, is it necessary to run on both Windows 7 and 10? We're trying to be a little less gratuitous about where things run these days.

We'd like to be able to run talos for both on Try. It doesn't have to be easily accessible (hidden behind mach's --full list maybe?) We can also consider removing the win7 option down the road once we've shipped.

Sounds good, it'll definitely be available on try using --full at least. But I guess perf tasks are only run every 20th push on autoland anyway, so on secound thought it's not a big deal.

Btw, looks like there are a couple of perma-fails on MacOS:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c1fa80a94a3b3066fd2e7852e1f56a1c64c96920

I'll adjust my patch to exclude those specific tasks so we can go ahead and land the others. The perftest team would be the ones to ask about fixing the issue if you need a hand with them.

The T-swr(o) failure there is bug 1666055, which I'll try to land today!

The webgl one looks like a new issue, I can debug soon.

This will add the following tasks to mozilla-central:

test-linux64-qr/opt-talos-chrome-swr-e10s
test-linux64-qr/opt-talos-damp-swr-e10s
test-linux64-qr/opt-talos-g3-swr-e10s
test-linux64-qr/opt-talos-g5-swr-e10s
test-linux64-qr/opt-talos-perf-reftest-swr-e10s
test-linux64-qr/opt-talos-sessionrestore-many-windows-swr-e10s
test-linux64-qr/opt-talos-tabswitch-swr-e10s
test-linux64-qr/opt-talos-tp5o-swr-e10s
test-linux64-qr/opt-talos-webgl-swr-e10s
test-linux64-shippable-qr/opt-talos-chrome-swr-e10s
test-linux64-shippable-qr/opt-talos-damp-swr-e10s
test-linux64-shippable-qr/opt-talos-g3-swr-e10s
test-linux64-shippable-qr/opt-talos-g5-swr-e10s
test-linux64-shippable-qr/opt-talos-perf-reftest-swr-e10s
test-linux64-shippable-qr/opt-talos-sessionrestore-many-windows-swr-e10s
test-linux64-shippable-qr/opt-talos-tabswitch-swr-e10s
test-linux64-shippable-qr/opt-talos-tp5o-swr-e10s
test-linux64-shippable-qr/opt-talos-webgl-swr-e10s
test-macosx1014-64-shippable-qr/opt-talos-bcv-swr-e10s
test-macosx1014-64-shippable-qr/opt-talos-chrome-swr-e10s
test-macosx1014-64-shippable-qr/opt-talos-damp-swr-e10s
test-macosx1014-64-shippable-qr/opt-talos-g1-swr-e10s
test-macosx1014-64-shippable-qr/opt-talos-g4-swr-e10s
test-macosx1014-64-shippable-qr/opt-talos-g5-swr-e10s
test-macosx1014-64-shippable-qr/opt-talos-perf-reftest-swr-e10s
test-macosx1014-64-shippable-qr/opt-talos-sessionrestore-many-windows-swr-e10s
test-macosx1014-64-shippable-qr/opt-talos-svgr-swr-e10s
test-macosx1014-64-shippable-qr/opt-talos-tp5o-swr-e10s
test-windows10-64-qr/opt-talos-bcv-swr-e10s
test-windows10-64-qr/opt-talos-chrome-swr-e10s
test-windows10-64-qr/opt-talos-damp-swr-e10s
test-windows10-64-qr/opt-talos-g1-swr-e10s
test-windows10-64-qr/opt-talos-g4-swr-e10s
test-windows10-64-qr/opt-talos-g5-swr-e10s
test-windows10-64-qr/opt-talos-other-swr-e10s
test-windows10-64-qr/opt-talos-perf-reftest-swr-e10s
test-windows10-64-qr/opt-talos-sessionrestore-many-windows-swr-e10s
test-windows10-64-qr/opt-talos-svgr-swr-e10s
test-windows10-64-qr/opt-talos-tabswitch-swr-e10s
test-windows10-64-qr/opt-talos-tp5o-swr-e10s
test-windows10-64-qr/opt-talos-webgl-swr-e10s
test-windows10-64-qr/opt-talos-xperf-swr-e10s
test-windows10-64-shippable-qr/opt-talos-bcv-swr-e10s
test-windows10-64-shippable-qr/opt-talos-chrome-swr-e10s
test-windows10-64-shippable-qr/opt-talos-damp-swr-e10s
test-windows10-64-shippable-qr/opt-talos-g1-swr-e10s
test-windows10-64-shippable-qr/opt-talos-g4-swr-e10s
test-windows10-64-shippable-qr/opt-talos-g5-swr-e10s
test-windows10-64-shippable-qr/opt-talos-other-swr-e10s
test-windows10-64-shippable-qr/opt-talos-perf-reftest-swr-e10s
test-windows10-64-shippable-qr/opt-talos-sessionrestore-many-windows-swr-e10s
test-windows10-64-shippable-qr/opt-talos-svgr-swr-e10s
test-windows10-64-shippable-qr/opt-talos-tabswitch-swr-e10s
test-windows10-64-shippable-qr/opt-talos-tp5o-swr-e10s
test-windows10-64-shippable-qr/opt-talos-webgl-swr-e10s
test-windows10-64-shippable-qr/opt-talos-xperf-swr-e10s

The damp task also appears to fail very frequently on Linux:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7fabcead08662aa98e536786648fbf638900f04c

I disabled damp on Linux and webgl on MacOS in my patch. If you end up fixing those issues, they can be re-enabled by adding the platform back to the variants key of the appropriate task stanza in taskcluster/ci/test/talos.yml.

The damp crash looks like it might be bug 1671055, which just landed.

webgl should be fixed by bug 1671061, which is up for review now.

I rebased on top of latest central + the patches from bug 1671061 but both damp and webgl still appear to be failing:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f8b5384ac0e1990925cff26cc0845e45519caa4d

So I'll land the current incarnation of the patch. Whoever ends up fixing them can re-enable them alongside the fix like this:
https://hg.mozilla.org/try/rev/4b0f82be26fd3a59c0e20416a96e0fca029d92e8

https://hg.mozilla.org/mozilla-central/rev/729d218d2958

(In reply to Andrew Halberstadt [:ahal] from comment #13)

I rebased on top of latest central + the patches from bug 1671061 but both damp and webgl still appear to be failing:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f8b5384ac0e1990925cff26cc0845e45519caa4d

So I'll land the current incarnation of the patch. Whoever ends up fixing them can re-enable them alongside the fix like this:
https://hg.mozilla.org/try/rev/4b0f82be26fd3a59c0e20416a96e0fca029d92e8

Thanks for working on this!

It looks like your try push got artifacts builds, so I don't think the c++ code changes you included actually got compiled into the binaries being tested.

Here's a try push from latest central, with the variants enabled: https://treeherder.mozilla.org/#/jobs?repo=try&revision=3c911029b7d0a59391227b14eea978aa6e497f36

Ah, oops! I see you made another change on top of mine in there, feel free to throw it on phab and I can review if you like.

https://hg.mozilla.org/mozilla-central/rev/2481a9cb04ac