Bug 1344132 enable us to run stylo-to-gecko test using reftest.list. But it is not enabled on the try server. So we need to mark stylo-specific fails inside reftest.list, remove all the reftest-stylo.list and change the try configuration to run reftest.list instead of reftest-stylo.list.

The reftest-stylo.list thing is creating too much confusion, so I guess it's time to do this. These patches depends on Bug 1344132. I'll land the two together when they both received r+. There are some rules I use to prepare the patch: * Things that clearly fail on stylo-vs-gecko test are marked as "fails-if(stylo)" * Stylo-specific crashes or intermittents are marked as "skip-if(stylo)", we'll try them periodically to see if they are fixed or not. * Things that are marked as "fails" or "fails-if(foo)" but is passing in stylo-vs-gecko test will be changed to "fails-if(!stylo)" and "fails-if(foo&&!stylo)" * Some tests (e.g. reftest-sanity) uses the same test file against different ref files. We only test the first one and mark the rest as "skip-if(stylo)" * We put the "skip-if(stylo)" and "fails-if(stylo)" at the end of the list, so it will override other conditions * Tests in the form "foo.html#id" will break the test because it only scrolls but not reload. This is tricker to fix so I'll create a follow-up bug for that, there are only ~6 of them in the whole reftest, so we won't lose a lot of test coverage

Comment on attachment 8859004 [details] Bug 1351548 - Add stylo-vs-gecko expectations to reftest.lists Hi Bobby, I usually ask heycam for the reftest stuff, but he is on PTO so would you mind review it for me?

Comment on attachment 8859004 [details] Bug 1351548 - Add stylo-vs-gecko expectations to reftest.lists https://reviewboard.mozilla.org/r/131024/#review133998 ::: layout/reftests/font-face/reftest.list:72 (Diff revision 1) > HTTP(..) == unicoderange-4.html unicoderange-4-ref.html > > # Dynamic changes > # we need to skip these because of the bug that's causing order-2.html to fail > -HTTP(..) == enable-sheet-1.html enable-sheet-1-ref.html > -skip HTTP(..) == enable-sheet-2.html multiple-in-family-1-ref.html > +skip-if(stylo) HTTP(..) == enable-sheet-1.html enable-sheet-1-ref.html > +skip skip-if(stylo) HTTP(..) == enable-sheet-2.html multiple-in-family-1-ref.html This is kind of redundant I suppose, but we can fix it up later.

Comment on attachment 8859005 [details] Bug 1351548 - Switch from reftest-stylo.list to reftest.list on https://reviewboard.mozilla.org/r/131026/#review134004

Comment on attachment 8859006 [details] Bug 1351548 - Remove reftest-stylo.lists https://reviewboard.mozilla.org/r/131028/#review134006

Nice! I didn't review every line of the diffs but this seems to be doing the right thing, and CI will give us the canonical results anyway. Let's get this landed - it's one step closer to victory!

Pushed by slyu@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/1f9d0f8e65b9 Add stylo-vs-gecko expectations to reftest.lists. r=bholley https://hg.mozilla.org/integration/mozilla-inbound/rev/4d8dfee7ebf3 Switch from reftest-stylo.list to reftest.list on linux64-stylo. r=bholley https://hg.mozilla.org/integration/mozilla-inbound/rev/aebb0ceeb20a Remove reftest-stylo.lists. r=bholley

Backed out bug 1344132 and bug 1351548 for failing reftest reftest-opaque-layer-fail.html and many reftest failures for Stylo builds: Bug 1344132 https://hg.mozilla.org/integration/mozilla-inbound/rev/1d3a3ae57b06ae9ef71cf670727ee099aa1e248c Bug 1351548 https://hg.mozilla.org/integration/mozilla-inbound/rev/6232431e65ef49c045f822c4c1c1335f9ff6d2a9 https://hg.mozilla.org/integration/mozilla-inbound/rev/f75640e4a0cc3deb4cc7d7b1d1aa544916ad5c60 https://hg.mozilla.org/integration/mozilla-inbound/rev/e4b8f7c680ad1d88d2eeefb2ad63c656b435fc8e https://hg.mozilla.org/integration/mozilla-inbound/rev/1f9d0f8e65b91a5fee031950fb3940be44381194#l142.144 has > fails fails-if(stylo) As far as I know, the "fails-if" will overwrite the "fails". For stylo, there are more assertions than expected. Please also check the other reftest failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=aebb0ceeb20ae6031f456f5893db0ba4ec96d9e9&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable

Pushed by slyu@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/46fc8d459483 Add stylo-vs-gecko expectations to reftest.lists. r=bholley https://hg.mozilla.org/integration/mozilla-inbound/rev/3940a4760c84 Switch from reftest-stylo.list to reftest.list on linux64-stylo. r=bholley https://hg.mozilla.org/integration/mozilla-inbound/rev/e1ceacb7312d Remove reftest-stylo.lists. r=bholley

Pushed by slyu@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/107d824d3657 Update stylo reftest expectations. r=me

https://hg.mozilla.org/mozilla-central/rev/46fc8d459483 https://hg.mozilla.org/mozilla-central/rev/3940a4760c84 https://hg.mozilla.org/mozilla-central/rev/e1ceacb7312d https://hg.mozilla.org/mozilla-central/rev/107d824d3657

Pushed by kgupta@mozilla.com: https://hg.mozilla.org/mozilla-central/rev/1c6fa11d1fed Follow-up to fix incorrect changes in cset 46fc8d459483. r=me a=Tomcat

FYI ^ there were some errors when merging the reftest.list files. Two of the four errors were showing up as R4 failures on m-c because of TEST-UNEXPECTED-PASS. The other two were fuzzy passes so they are green regardless of whether they are marked fails-if or not (we really should fix that...). You might want to recheck that cset to make sure there weren't other errors, I only looked for webrender mismatches.

This accidentally re-enabled the test disabled in bug 1351548. Please check that no other changes to test expectations got reverted by accident.

This bug is 1351548.

Thank you. The test that got re-enabled is bug 1352942.