We want to enable all mochitests for stylo sooner than later, so that we can catch improvement and regression on a larger range. At the same time, we would need to retire stylo-failures.md and use fails-if instead for style system tests as well. stylo-failures.md was used because many tests in layout/style/test could actually contain lots of different failures, so a coarse-grained "fails-if" isn't good enough to catch improvement and regression. But now, majority of failures in stylo-failures.md have been fixed.

I'm not going to enable all... at least browser-chrome and a11y still seem to fail a lot. We can try them later.

Comment on attachment 8894444 [details] Bug 1383992 part 1 - Disable failing mochitests. https://reviewboard.mozilla.org/r/165628/#review170698

Comment on attachment 8894445 [details] Bug 1383992 part 2 - Enable some more mochitests for stylo. https://reviewboard.mozilla.org/r/165630/#review170746 ::: taskcluster/ci/test/test-sets.yml:95 (Diff revision 1) > + - mochitest-browser-screenshots > + - mochitest-chrome > + - mochitest-clipboard > + - mochitest-gpu > + - mochitest-media > + - mochitest-webgl my concern here is we will be increasing the load on osx- this is a platform that is heavily restrained on resources- do you have estimates on how much runtime this will increase? ::: taskcluster/ci/test/tests.yml:677 (Diff revision 1) > extra-options: > - --mochitest-suite=chrome > tier: > by-test-platform: > windows7-32.*: 2 > + linux64-stylo/debug: 3 will opt be run on as tier1 on all trees? ::: taskcluster/ci/test/tests.yml:690 (Diff revision 1) > virtualization: hardware > instance-size: xlarge > run-on-projects: > by-test-platform: > windows.*: ['try'] > + linux64-stylo/debug: ['try'] will opt be run on all trees? ::: taskcluster/ci/test/tests.yml:728 (Diff revision 1) > extra-options: > - --mochitest-suite=plain-clipboard,chrome-clipboard,browser-chrome-clipboard > tier: > by-test-platform: > windows7-32.*: 2 > + linux64-stylo/debug: 3 this is tier-1 for linux64-stylo/opt?

Comment on attachment 8894445 [details] Bug 1383992 part 2 - Enable some more mochitests for stylo. https://reviewboard.mozilla.org/r/165630/#review170746 > my concern here is we will be increasing the load on osx- this is a platform that is heavily restrained on resources- do you have estimates on how much runtime this will increase? Let's drop the whole stylo test set from osx then? > will opt be run on as tier1 on all trees? Yes, because we don't have problem on running opt build with these tests. They currently fail on debug build because of bug 1384701. See bug 1387993. If you are concerned about that we are having different test coverage on debug vs. opt, I guess we can universally disable these tests?

Comment on attachment 8894445 [details] Bug 1383992 part 2 - Enable some more mochitests for stylo. https://reviewboard.mozilla.org/r/165630/#review170746 > Let's drop the whole stylo test set from osx then? So all mochitests runs for 362 mins in total for debug build, and 188 mins for opt build, from the try push. The replaced tests currently take 24 mins and 9 mins respectively (from a recent m-c commit). That means around 338 mins and 179 mins increment. Probably doesn't sound fine?

Discussed with bholley, if we are concerned about the load on mac platform, probably we can only enable them on m-c for mac? That means we would regress some of our test coverage on mac (since we are retiring mochitest-style and mochitest-chrome-style which currently run on autoland), but that's probably not a big problem. jmaher, would you be happy with that? Tomcat, would sheriff be fine with that?

We went through a bit of trouble to get the current set of Stylo tests running on macOS, so it would be a bit sad to disable them... Maybe we can pre-seed SETA for macOS, like we are doing for Windows (bug 1386806)?

KWierso answered sheriff's point. It seems as far as a task is tier-2, it is fine to only run on m-c and try.

The updated patch moves all the new mochitests into tier-2 and make them only run on m-c and try for macosx64-stylo platform.

yes, m-c+try is good for osx; other platforms that we have tests running in AWS have better capacity, but talos and macosx* are hardware limited.

Comment on attachment 8894445 [details] Bug 1383992 part 2 - Enable some more mochitests for stylo. https://reviewboard.mozilla.org/r/165630/#review171194 looks much better

Comment on attachment 8894445 [details] Bug 1383992 part 2 - Enable some more mochitests for stylo. https://reviewboard.mozilla.org/r/165630/#review171492 ::: testing/mozharness/configs/unittests/linux_unittest.py (Diff revision 2) > "mochitest-devtools-chrome-coverage": ["--flavor=browser", "--subsuite=devtools", "--chunk-by-runtime", "--timeout=1200"], > "jetpack-package": ["--flavor=jetpack-package"], > "jetpack-package-clipboard": ["--flavor=jetpack-package", "--subsuite=clipboard"], > "jetpack-addon": ["--flavor=jetpack-addon"], > "a11y": ["--flavor=a11y"], > - "plain-style": ["--failure-pattern-file=stylo-failures.md", "layout/style/test", "dom/smil/test", "dom/animation/test"], Please also remove the same entries from `mac_unittest.py` and `win_taskcluster_unittest.py`.

We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again. hg error in cmd: hg rebase -s c12dd8a82fb9 -d c95150c0fe9d: rebasing 412513:c12dd8a82fb9 "Bug 1383992 part 1 - Disable failing mochitests. r=heycam" merging dom/base/test/mochitest.ini rebasing 412514:1a79deb8189a "Bug 1383992 part 2 - Enable some more mochitests for stylo. r=jmaher" (tip) merging taskcluster/ci/test/test-sets.yml merging taskcluster/ci/test/tests.yml merging testing/mozharness/configs/unittests/linux_unittest.py warning: conflicts while merging taskcluster/ci/test/tests.yml! (edit, then use 'hg resolve --mark') unresolved conflicts (see hg resolve, then hg rebase --continue)

Pushed by xquan@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/76dafa6645cb part 1 - Disable failing mochitests. r=heycam https://hg.mozilla.org/integration/autoland/rev/baf282c4df0c part 2 - Enable some more mochitests for stylo. r=jmaher

Pushed by xquan@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/d0e338649b01 followup - Fix tests.yml.

Pushed by xquan@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/b886db98d2bc followup - Update mochitest expectation.

https://hg.mozilla.org/mozilla-central/rev/76dafa6645cb https://hg.mozilla.org/mozilla-central/rev/baf282c4df0c https://hg.mozilla.org/mozilla-central/rev/d0e338649b01 https://hg.mozilla.org/mozilla-central/rev/b886db98d2bc