These are all currently hidden due to bug 1223198, which will be fixed by moving off GTK 3.4. There are some new failures due theme and font differences: https://treeherder.mozilla.org/#/jobs?repo=try&revision=c330de71e670bc92bee559dd1d6d6561dd33e8c8&exclusion_profile=false&selectedJob=28924857

Comment on attachment 8799604 [details] bug 1309086 run taskcluster desktop reftests and crashtests on Ubuntu 16.04 https://reviewboard.mozilla.org/r/84742/#review83374 this is great, thanks for pushing this forward. Any indication of crashtest or jsreftest could be moved over as well? I assume you tested on opt/debug/asan on your try pushes?

Comment on attachment 8799604 [details] bug 1309086 run taskcluster desktop reftests and crashtests on Ubuntu 16.04 https://reviewboard.mozilla.org/r/84742/#review83374 I added the same change to crashtests. It probably makes sense for jsreftest too, but I'm limiting this to testsuites about which I know something. Passing on taskcluster, but not on buildbot opt tests, which will not be unhidden because bug 1223198 will not be fixed on buildbot. https://treeherder.mozilla.org/#/jobs?repo=try&revision=61310e5e5bea92484677c0e6b44bdbed63a53ce4&exclusion_profile=false

Comment on attachment 8800168 [details] bug 1309086 reftest annotation changes for move to Ubuntu 16.04 https://reviewboard.mozilla.org/r/85160/#review85210 r=me, with one bug-number annotation added: ::: layout/reftests/css-grid/reftest.list:82 (Diff revision 1) > == grid-item-align-003.html grid-item-align-003-ref.html > == grid-item-justify-001.html grid-item-justify-001-ref.html > == grid-item-justify-002.html grid-item-justify-002-ref.html > == grid-item-stretch-001.html grid-item-stretch-001-ref.html > == grid-item-self-baseline-001.html grid-item-self-baseline-001-ref.html > -skip-if(!gtkWidget) == grid-item-content-baseline-001.html grid-item-content-baseline-001-ref.html # depends on exact Ahem baseline font metrics which seems to differ between platforms > +random == grid-item-content-baseline-001.html grid-item-content-baseline-001-ref.html # depends on exact Ahem baseline font metrics which seems to differ between platforms I filed bug 1310792 on these. Please add "bug 1310792" to the end of the message on the first line here.

> +random == grid-item-content-baseline-001.html grid-item-content-baseline-001-ref.html # depends on exact Ahem baseline font metrics which seems to differ between platforms Are there NO Linux platforms at all where these tests still pass?

Comment on attachment 8800168 [details] bug 1309086 reftest annotation changes for move to Ubuntu 16.04 I object to the layout/reftests/css-grid/reftest.list changes until comment 8 has been answered.

(same; my r+ was based on the assumption that this affected all linux tinderboxen. If there are some which are unaffected & still pass, we should absolutely craft a more specific annotation that keeps them running [and still link to bug 1310792 in the manifest, so we've got the failure-everywhere-else tracked].)

(In reply to Mats Palmgren (:mats) from comment #8) > Are there NO Linux platforms at all where these tests still pass? They still pass on Ubuntu 12.04 platforms. Some tinderboxen would still be running reftests on 12.04 with buildbot after changes here, but their results are already hidden behind "Excluded jobs". There are no linux32 reftests running on taskcluster (pending resolution of either bug 1253315 or bug 1209932), and it is easy enough to annotate these reftests as passing on linux32. Would you be happy to live with that for the sake of getting some (i.e. taskcluster) reftests unhidden on linux64, where all reftests are also currently hidden? https://archive.mozilla.org/pub/firefox/try-builds/ktomlinson@mozilla.com-e6a4ea63fc2642452e48c26cd108c67fa8597af8/try-linux/try_ubuntu32_vm_test-reftest-1-bm05-tests1-linux32-build95.txt.gz https://treeherder.mozilla.org/#/jobs?repo=try&revision=e6a4ea63fc2642452e48c26cd108c67fa8597af8&exclusion_profile=false&selectedJob=29303317 Are you also happy to run these reftests on other platforms to catch any assertion failures should they start occurring?

Comment on attachment 8800168 [details] bug 1309086 reftest annotation changes for move to Ubuntu 16.04 (In reply to Daniel Holbert [:dholbert] from comment #10) > (same; my r+ was based on the assumption that this affected all linux > tinderboxen. If there are some which are unaffected & still pass, we should > absolutely craft a more specific annotation that keeps them running There is not currently anything available in the sandbox to distinguish between Ubuntu 12.04 and 16.04 or between buildbot and taskcluster, but I assume the intent is to decommission builtbot tests where they have moved to taskcluster, so any new attribute would become obsolete. Therefore this patch implements the proposal of comment 11 to annotate differently on i686 where there are no taskcluster tests. Requesting review again because there are hidden tinderboxen unaffected by the test system changes proposed here.

(In reply to Karl Tomlinson (:karlt) from comment #11) > There are no linux32 reftests running on taskcluster [...] Boo. And it looks like the (non-taskcluster) Linux32 reftest jobs on mozilla-inbound are currently hidden (behind "excluded") for some reason. > Would you be happy to live with that for the > sake of getting some (i.e. taskcluster) reftests unhidden on linux64, where > all reftests are also currently hidden? Based on your logs, it does indeed look like that keeps this test running on (hidden) Linux32 reftest -- which is better than the current state of things, I think. > Are you also happy to run these reftests on other platforms to catch any > assertion failures should they start occurring? The switch from "skip-if" to "random-if", you mean? That seems fine, yeah.

Comment on attachment 8800168 [details] bug 1309086 reftest annotation changes for move to Ubuntu 16.04 Your Try run from comment 11 shows "unexpected-pass" for all non-taskcluster Linux x64 opt reftest runs, it looks like. So, some of the annotations still need a bit of adjustment, it seems.

css-grid/reftest.list looks better now, thanks!

(In reply to Daniel Holbert [:dholbert] from comment #15) > Your Try run from comment 11 shows "unexpected-pass" for all non-taskcluster > Linux x64 opt reftest runs, it looks like. Yes. Those are hidden and will remain hidden because of bug 1223198. The fix for that bug is to use an OS without the bug, which taskcluster enables us to do. With these tests running on taskcluster, there will be no need to keep running them on buildbot. > So, some of the annotations still need a bit of adjustment, it seems. There is not currently anything available in the sandbox to distinguish between Ubuntu 12.04 and 16.04 or between buildbot and taskcluster. When taskclaster and buildbot are running the same tests on different 64-bit OSes, the results are different. The taskcluster tests are the ones I'm aiming to unhide, and so the annotations are designed for those machines. I guess an attribute could be added, but I don't know how its value would be determined, and it seems unnecessary when the buildbot linux64 tests are obsoleted by the taskcluster tests.

Comment on attachment 8800168 [details] bug 1309086 reftest annotation changes for move to Ubuntu 16.04 OK, just wanted to make sure those oranges weren't an oversight. r=me, thanks!

Pushed by ktomlinson@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/b25ba1be8195 run taskcluster desktop reftests and crashtests on Ubuntu 16.04 r=jmaher https://hg.mozilla.org/integration/autoland/rev/4a6af5f7d9bb reftest annotation changes for move to Ubuntu 16.04 r=dholbert

(In reply to Karl Tomlinson (:karlt) from comment #17) > I guess an attribute could be added, but I don't know how its value would be > determined, and it seems unnecessary when the buildbot linux64 tests are > obsoleted by the taskcluster tests. FWIW, if we wanted to add this we could presumably just read the bits out of /etc/lsb-release: $ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=14.04 DISTRIB_CODENAME=trusty DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS" mozinfo sets an `os_version` attribute for most platforms, but it's using `platform.linux_distribution()` on Linux which seems to be returning useless results (the contents of /etc/debian_version on my Ubuntu).

https://hg.mozilla.org/mozilla-central/rev/b25ba1be8195 https://hg.mozilla.org/mozilla-central/rev/4a6af5f7d9bb

Looks like things are much happier now, other than the 1302389-scrolled-rect* tests. https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-searchStr=linux%20x64%20reftest&group_state=expanded&filter-tier=1&filter-tier=2&filter-tier=3&exclusion_profile=false

I'm trying to run 32-bit linux reftests on 16.04 and hitting many of the same problems encountered in this bug; I'll need to relax some of the annotations to skip/fail on 32-bit linux also.