https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=asan&tochange=d7844fe50165&fromchange=92a73b885c87&group_state=expanded Dunno whether we're slow about drawing form controls, and thus our slowest platform is sometimes too slow, or we now sometimes just don't draw them at all, but switching to GTK+ 3 has produced a ton of very frequent failures in ASan tests, which at a glance all seem to have form controls in common.

Most of these are very subtle differences in native rendering. Perhaps whether alpha-recovery gets used depends on clips or what the native theme draws depends on offsets somehow. Maybe Lee has some ideas. bug 1210135, bug 1222402, and css-required-radio.html in https://treeherder.mozilla.org/logviewer.html#?job_id=17028401&repo=mozilla-inbound are exceptions with larger differences in native rendering. Bug 1203867, with clipped text seems unrelated to native rendering, unless dimensions are changing. roc was seeing some odd behaviour with buttons. Don't know why these would be asan-only. I wonder whether asan explicitly initializes memory to different values.

I tried disabling various different paths of nsNativeThemeGTK::DrawWidgetBackground(). Although they didn't obviously clear up these kind of failures, they did trigger some similar failures more reliably. Disabling snapping [1][3] caused a new permanent fail in the scrollbar thumb: > 551463-1.html | image comparison (==), max difference: 15, number of differing pixels: 117 Disabling the "defined(MOZ_X11) && defined(CAIRO_HAS_XLIB_SURFACE)" fast-path [2][4][5] in DrawThemeWithCairo() caused a permanent fail in textarea background: > wordwrap-03.html | image comparison (==), max difference: 1, number of differing pixels: 44 and a new frequent failure with mostly missing tree header row background > tree-row-outline-1.xul | image comparison (==), max difference: 61, number of differing pixels: 13248 [1] https://dxr.mozilla.org/mozilla-central/rev/d3d286102ba7f8801e9dfe12d534f49554ba50c0/widget/gtk/nsNativeThemeGTK.cpp#1087 [2] https://dxr.mozilla.org/mozilla-central/rev/d3d286102ba7f8801e9dfe12d534f49554ba50c0/widget/gtk/nsNativeThemeGTK.cpp#859 [3] https://treeherder.mozilla.org/#/jobs?repo=try&revision=f2cdfafd3275 [4] https://treeherder.mozilla.org/#/jobs?repo=try&revision=095143f3b3f5 [5] https://treeherder.mozilla.org/#/jobs?repo=try&revision=a8d9c74da361

Andrew, is there something you can help here? Karl do you want/need help?

Sure, I can give it a shot if Karl doesn't mind me taking this. I looked at this issue a while back and it seemed nontrivial, but I'm sure some digging in rr will turn up something.

Thanks Andrew and Milan. Yes, any help would be much appreciated. The various code paths in DrawWidgetBackground() may lead to different tessellation algorithms, so it is reasonable to fuzz tests with small differences. Some look similar, but differences up to 16/255 sound quite large for just different tessellation algorithms. It's a bit disturbing that this only happens with ASAN builds, but fuzzing the test is still reasonable if the differences are such that they are not visible enough to be worth investigating. The failures with larger differences, such as the native widget not being draw at all, are more concerning. It may be worth logging whether the X11 error path is hit. https://dxr.mozilla.org/mozilla-central/rev/d3d286102ba7f8801e9dfe12d534f49554ba50c0/widget/gtk/nsNativeThemeGTK.cpp#1178 But requesting access to a VM and installing rr on it is probably the best strategy. (In reply to Karl Tomlinson (ni?:karlt) from comment #1) > roc was seeing some odd behaviour with buttons. This may also be worth checking out to see whether it is easy to reproduce. He was seeing buttons being mis-drawn on scrolling. He probably had FC21 at the time. data:text/html,<div%20style="height:120%"><button%20style="height:100px;width:100px">

It appears most of the issues here are caused by the unico theming engine intermittently not being used for rendering, falling back to gtk's default theme engine. In the attached screenshot of gtk3-widget-factory using the fallback theme engine on Ubuntu 12.04, note the flat checkboxes, which similar to what we see when layout/reftests/forms/input/checkbox/indeterminate-selector.html fails. Other artifacts similar to what we're seeing on the test infrastructure are the black bordered scrollbar troughs and text fields inset differently. I suspect that ASAN's memory poisoning is triggering this, particularly the malloc/free implementation's poisoning (the only effect ASAN should have on system libraries). I considered that ASAN's malloc/free implementation violated the expectations of glib's slice allocator (chromium recommends to disable it on ASAN builds), but setting G_SLICE=always-malloc didn't seem to have any impact on the intermittents (see https://treeherder.mozilla.org/#/jobs?repo=try&revision=54c1954fd6a0).

You could try running some of these tests under rr chaos mode. If they catch a failure, you'll be able to figure it out. I haven't tested ASAN under rr so there might be some issues there, but if so they should not be hard to fix.

I vote we just stop running reftests on ASan - I've lost all interest in their results, and I will star any number of failures, 2 or 10 or 100, as whatever happens to be the first failure which is already filed and thus suggested.

This is blocking enabling e10s reftests on ASAN. The failure rate is ~50% on R1 and 15-20% on R2.

One option here may be to run these tests with GTK_THEME=Adwaita in the environment. Maybe even Raleigh, as that was the default GTK theme for 3.4. I'm guessing some reftests expected results may need tweaking. If it turns out to be a bug in unico, then that may be our best option.

(In reply to Karl Tomlinson (ni?:karlt) from comment #10) > One option here may be to run these tests with GTK_THEME=Adwaita in the > environment. > Maybe even Raleigh, as that was the default GTK theme for 3.4. > I'm guessing some reftests expected results may need tweaking. > > If it turns out to be a bug in unico, then that may be our best option. GTK_THEME was introduced in GTK+ commit 1e70ad1cd126942b38e83c2d01712e8dc7ca675e, which didn't make it in until 3.11.2. I'd imagine the best way to do this would be by setting the "gtk-theme-name" setting at runtime. I don't think adding a custom settings.ini will work, as XSettings will override that. https://developer.gnome.org/gtk3/stable/GtkSettings.html#GtkSettings--gtk-theme-name Using the default theming engine sounds good to me though- I think it should resolve this issue. I'll see if I can put together a patch.

Forcing the fallback GTK3 theme (Raleigh) appears to clean up the cases of test failures caused by intermittent changes in the theming engine used, but new intermittents (such as layout/reftests/text-overflow/single-value.html) appear to be introduced by the change- mostly small compositing issues with scrollbar elements, from what I can tell. These may be fuzzed reasonably, unlike the theming engine variances we currently obtain which may remarkably vary. https://treeherder.mozilla.org/#/jobs?repo=try&revision=08c0609e9810 (the reftests consistently failing across R1 and R2 may be viably fuzzed, IMO- it would be interesting to see the cause now that we can deterministically reproduce it) I'm not sure if it's worth it to switch to a much less commonly used fallback theming engine in this case just to trade one type of intermittent for another (albeit a much more manageable kind).

In addition, the Raleigh theme appears to not draw resizers, so we would have to disable tests for those were we to switch.

Thanks for the investigation. I've hidden ASan reftest on every tree other than esr38/esr45. Not only are their results useless, but they are starting to devalue the entire suite - I find myself ignoring a reftest failure because I expect this bustage, only to realize it was on Linux64 debug rather than ASan.

I wonder how hard it would be to just disable these image comparison failures on ASan builds. I'd think that we really only care about leaks and things like use-after-frees on Asan builds, not whether an image is off by a pixel or two.

That's a really interesting idea (though IIRC, we have other tests outside of reftests that are also affected by this root problem - see |skip-if = asan| hits in dxr). What are your thoughts about that, David?

Are the ASAN tests run on the same OS version / configuration / etc. as other tests?

Yes, they run on the same machine pool as our other Linux64 tests.

(In reply to Ryan VanderMeulen [:RyanVM] from comment #16) > That's a really interesting idea (though IIRC, we have other tests outside > of reftests that are also affected by this root problem - see |skip-if = > asan| hits in dxr). What are your thoughts about that, David? In some ways it's a reasonable idea, but I'm somewhat hesitant to add such a configuration to reftest for fear that it would be mistakenly used where not intended. It also seems like this should be something that somebody should just be able to reproduce and debug the problem and fix it.

Also, does somebody have a pointer to an example log showing one of these failures? All the logs in this bug seem to be expired.

(In reply to Ryan VanderMeulen [:RyanVM] from comment #21) > https://treeherder.mozilla.org/logviewer.html#?job_id=155009&repo=ash That one looks like it's intermittently using an entirely different GTK theme from one test to the next!

> That one looks like it's intermittently using an entirely different GTK > theme from one test to the next! That's consistent with Andrew's observation that it looks like Unico is intermittently not used. Looking at the source code, I'm not clear what if anything ensures the engine is resolved from the style before gtk_render_handle() is called. This function uses priv->theming_engine without doing anything to ensure it is initiallize appropriately. gtk_style_context_init sets up the default engine: priv->theming_engine = g_object_ref ((gpointer) gtk_theming_engine_load(/* name = */NULL)); File-static style_data_lookup() is where this can be set to a different engine. It is called from gtk_style_context_get_section(), gtk_style_context_get_property(), _gtk_style_context_peek_property(), _gtk_style_context_peek_style_property(), gtk_style_context_lookup_icon_set(), gtk_style_context_notify_state_change(), gtk_style_context_get_font(), but I'm don't see that any of these are necessarily called. If this is really an issue, it is fixed in https://git.gnome.org/browse/gtk+/commit/gtk/gtkstylecontext.c?id=6ac754cdd1b5a273b0c0930866211c9b4bc450f4 which is in https://git.gnome.org/browse/gtk+/log/gtk/gtkstylecontext.c?h=gtk-3-6&qt=grep&q=engine We can force resolution of the engine through calling a suitable function. Tried this at https://treeherder.mozilla.org/#/jobs?repo=try&revision=1926fdba0fb3&exclusion_profile=false&filter-tier=1 https://treeherder.mozilla.org/#/jobs?repo=try&revision=9e0918755c83&filter-tier=1&filter-tier=3&exclusion_profile=false but didn't reproduce without the patch either https://treeherder.mozilla.org/#/jobs?repo=try&revision=5967042fab02&exclusion_profile=false&filter-tier=1

The much more common issue producing large-magnitude differences now seems to be the ends of scrollbars. This comparison of scrollbar troughs, just above the empty resizer space in layout/reftests/w3c-css/submitted/flexbox/flexbox-overflow-horiz-001.html recorded in https://treeherder.mozilla.org/logviewer.html#?job_id=18239274&repo=try "Reference" on the right is the outlier. On vertical scrollbars the antialiasing can look very different. From a distance it gives the feel of a one pixel difference in position, but up close it is clearly not a pixel shift and looks like much less smoothing than normal in the aa. gtk3drawing uses integer coordinates and sizes, so the only way I could imagine non-integer offsets is through the cairo transformation matrix. The scrollbars do not have the dark borders that would be expected if Unico were not loaded.

> The scrollbars do not have the dark borders that would be expected if Unico > were not loaded. But this is a similar comparison with Ambience modified to use a background like the -unico-border-gradient that unico uses. data:text/html,<textarea style="overflow:scroll;height:100px;background:purple"></textarea> I suspect the background is sometimes being drawn without Unico, but Unico is used for the borders.

(In reply to Karl Tomlinson (ni?:karlt) from comment #25) > But this is a similar comparison with Ambience modified to use a background > like the -unico-border-gradient that unico uses. I mean s/background/border-color/

~/moz/rr/rr% git diff diff --git a/src/record_syscall.cc b/src/record_syscall.cc index e0eb6d5..8aae71b 100644 --- a/src/record_syscall.cc +++ b/src/record_syscall.cc @@ -1803,7 +1803,7 @@ static void prepare_exit(RecordTask* t, int exit_code) { static void prepare_mmap_register_params(RecordTask* t) { Registers r = t->regs(); - if (t->session().enable_chaos() && + if (false && t->session().enable_chaos() && !(r.arg4_signed() & (MAP_FIXED | MAP_32BIT)) && r.arg1() == 0) { // No address hint was provided. Randomize the allocation address. size_t len = r.arg2(); % cat obj-asan/dist/bin/asan-options #!/bin/sh export ASAN_OPTIONS=handle_segv=false:detect_leaks=false exec rr record --chaos /home/karl/moz/dev/obj-asan/dist/bin/firefox ${1+"$@"} % while MOZCONFIG=../mozconfig-asan DISPLAY=:1 LD_LIBRARY_PATH=/home/karl/downloads/ubuntu/precise/usr/lib/x86_64-linux-gnu:/home/karl/downloads/ubuntu/precise/lib/x86_64-linux-gnu ./mach mochitest --e10s accessible/tests/mochitest/scroll/test_zoom_text.html --keep-open=false --appname=obj-asan/dist/bin/asan-options; do done The random selection of themes is caused by the bugs fixed in https://git.gnome.org/browse/gtk+/commit/gtk/gtkbitmaskprivateimpl.h?id=77912a65e2aa6eb9cfee994fa93bbfbc6b9f8538 https://git.gnome.org/browse/gtk+/commit/gtk/gtkbitmaskprivateimpl.h?id=e8f2eeac9221d91ba32447bbcbdc5ba057647f8e which are in https://git.gnome.org/browse/gtk+/log/gtk/gtkbitmaskprivateimpl.h?h=3.6.2 The bug exists in https://git.gnome.org/browse/gtk+/log/gtk/gtkbitmaskprivateimpl.h?h=3.4.0 When allocated addresses of bit masks have bits of the low 32 bits in common, the intersection returns true. gtk_css_style_provider_lookup() uses _gtk_bitmask_intersects() as an optimization, and so returning true incorrectly causes no problems. We only see the bugs when allocated addresses have none of the low 32 bits in common. This is more likely with ASAN as allocations are larger. Unico registers 26 custom properties. If the unico engine is not loaded, with Raleigh, for example, there should be only 51 properties. Bit masks will not be allocated and so the buggy path will not be taken. I don't know why we are not seeing the resizer with Raleigh. gtk3-widget-factory version 3.4.3 draws a resizer for the window, so I assume we could get a resizer out of Raleigh. That may be easier than using g_mem_set_vtable to over-allocate to ensure that all sizeof(GtkBitmask) == 16 allocations have a single bit set in common.

To prevent Unico from loading, the theme needs to be selected before nsLookAndFeel::Init() runs because that happens early #0 nsLookAndFeel::Init (this=<optimized out>) at /mnt/ssd1/karl/moz/dev/widget/gtk/nsLookAndFeel.cpp:1098 #1 0x00007f40025caf98 in nsXPLookAndFeel::GetInstance () at /mnt/ssd1/karl/moz/dev/widget/nsXPLookAndFeel.cpp:264 #2 0x00007f40025cb5b3 in mozilla::LookAndFeel::GetInt (aID=<optimized out>, aResult=<optimized out>) at /mnt/ssd1/karl/moz/dev/widget/nsXPLookAndFeel.cpp:911 #3 0x00007f4000fdaf86 in mozilla::LookAndFeel::GetInt (aDefault=0, aID=mozilla::LookAndFeel::eIntID_UseAccessibilityTheme) at /mnt/sda11/karl/obj-opt/dist/include/mozilla/LookAndFeel.h:561 #4 nsChromeRegistryChrome::CheckForOSAccessibility (this=<optimized out>) at /mnt/ssd1/karl/moz/dev/chrome/nsChromeRegistryChrome.cpp:160 #5 0x00007f4002c7bf60 in ScopedXPCOMStartup::SetWindowCreator ( this=<optimized out>, native=<optimized out>) at /mnt/ssd1/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:1597 #6 0x00007f4002c7f9f4 in XREMain::XRE_mainRun (this=<optimized out>) at /mnt/ssd1/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:4086 #7 0x00007f4002c8063b in XREMain::XRE_main (this=<optimized out>, argc=<optimized out>, argv=<optimized out>, aAppData=<optimized out>) at /mnt/ssd1/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:4445 #8 0x00007f4002c80864 in XRE_main (argc=<optimized out>, argv=<optimized out>, aAppData=<optimized out>, aFlags=<optimized out>) at /mnt/ssd1/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:4551 #9 0x0000000000405262 in do_main (argc=argc@entry=3, argv=argv@entry=0x7ffd86312128, envp=envp@entry=0x7ffd86312148, xreDirectory=0x7f400ab586c0) at /mnt/ssd1/karl/moz/dev/browser/app/nsBrowserApp.cpp:220 #10 0x00000000004049d5 in main (argc=3, argv=0x7ffd86312128, envp=0x7ffd86312148) at /mnt/ssd1/karl/moz/dev/browser/app/nsBrowserApp.cpp:360 before #0 nsAppShell::Init (this=<optimized out>) at /mnt/ssd1/karl/moz/dev/widget/gtk/nsAppShell.cpp:204 #1 0x00007f40025f1f8f in nsAppShellInit () at /mnt/ssd1/karl/moz/dev/widget/nsAppShellSingleton.h:45 #2 0x00007f4000fc0851 in nsComponentManagerImpl::KnownModule::Load ( this=<optimized out>) at /mnt/ssd1/karl/moz/dev/xpcom/components/nsComponentManager.cpp:870 #3 0x00007f4000fc181f in nsFactoryEntry::GetFactory (this=<optimized out>) at /mnt/ssd1/karl/moz/dev/xpcom/components/nsComponentManager.cpp:1896 #4 0x00007f4000fc1b16 in nsComponentManagerImpl::CreateInstance ( this=<optimized out>, aClass=..., aDelegate=<optimized out>, aIID=..., aResult=<optimized out>) at /mnt/ssd1/karl/moz/dev/xpcom/components/nsComponentManager.cpp:1118 #5 0x00007f4000fc3024 in nsComponentManagerImpl::GetService ( this=<optimized out>, aClass=..., aIID=..., aResult=<optimized out>) at /mnt/ssd1/karl/moz/dev/xpcom/components/nsComponentManager.cpp:1363 #6 0x00007f4000fea415 in CallGetService (aCID=..., aIID=..., aResult=<optimized out>) at /mnt/ssd1/karl/moz/dev/xpcom/glue/nsComponentManagerUtils.cpp:56 #7 0x00007f4000fea451 in nsGetServiceByCIDWithError::operator() ( this=<optimized out>, aIID=..., aInstancePtr=<optimized out>) at /mnt/ssd1/karl/moz/dev/xpcom/glue/nsComponentManagerUtils.cpp:265 #8 0x00007f4000fe1630 in nsCOMPtr_base::assign_from_gs_cid_with_error ( this=<optimized out>, aGS=..., aIID=...) at /mnt/ssd1/karl/moz/dev/xpcom/glue/nsCOMPtr.cpp:92 #9 0x00007f4002c470a5 in nsCOMPtr<nsIAppShell>::operator= (aRhs=..., this=<optimized out>) at /mnt/sda11/karl/obj-opt/dist/include/nsCOMPtr.h:631 #10 nsAppStartup::Init (this=<optimized out>) at /mnt/ssd1/karl/moz/dev/toolkit/components/startup/nsAppStartup.cpp:157 #11 0x00007f4002c7551a in nsAppStartupConstructor (aOuter=<optimized out>, aIID=..., aResult=<optimized out>) at /mnt/ssd1/karl/moz/dev/toolkit/components/build/nsToolkitCompsModule.cpp:67 #12 0x00007f4000fc1f32 in nsComponentManagerImpl::CreateInstanceByContractID ( this=<optimized out>, aContractID=<optimized out>, aDelegate=<optimized out>, aIID=..., aResult=<optimized out>) at /mnt/ssd1/karl/moz/dev/xpcom/components/nsComponentManager.cpp:1203 #13 0x00007f4000fc3246 in nsComponentManagerImpl::GetServiceByContractID ( this=<optimized out>, aContractID=<optimized out>, aIID=..., aResult=<optimized out>) at /mnt/ssd1/karl/moz/dev/xpcom/components/nsComponentManager.cpp:1560 #14 0x00007f4000fea48f in CallGetService (aContractID=<optimized out>, aIID=..., aResult=<optimized out>) at /mnt/ssd1/karl/moz/dev/xpcom/glue/nsComponentManagerUtils.cpp:69 #15 0x00007f4000fea4a7 in nsGetServiceByContractID::operator() ( this=<optimized out>, aIID=..., aInstancePtr=<optimized out>) at /mnt/ssd1/karl/moz/dev/xpcom/glue/nsComponentManagerUtils.cpp:280 #16 0x00007f4000fe1693 in nsCOMPtr_base::assign_from_gs_contractid ( this=<optimized out>, aGS=..., aIID=...) at /mnt/ssd1/karl/moz/dev/xpcom/glue/nsCOMPtr.cpp:103 #17 0x00007f4002c7bf87 in nsCOMPtr<nsIWindowCreator>::nsCOMPtr (aGS=..., this=<optimized out>) at /mnt/sda11/karl/obj-opt/dist/include/nsCOMPtr.h:540 #18 ScopedXPCOMStartup::SetWindowCreator (this=<optimized out>, native=<optimized out>) at /mnt/ssd1/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:1599 #19 0x00007f4002c7f9f4 in XREMain::XRE_mainRun (this=<optimized out>) at /mnt/ssd1/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:4086 #20 0x00007f4002c8063b in XREMain::XRE_main (this=<optimized out>, argc=<optimized out>, argv=<optimized out>, aAppData=<optimized out>) at /mnt/ssd1/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:4445 #21 0x00007f4002c80864 in XRE_main (argc=<optimized out>, argv=<optimized out>, aAppData=<optimized out>, aFlags=<optimized out>) at /mnt/ssd1/karl/moz/dev/toolkit/xre/nsAppRunner.cpp:4551 #22 0x0000000000405262 in do_main (argc=argc@entry=3, argv=argv@entry=0x7ffd86312128, envp=envp@entry=0x7ffd86312148, xreDirectory=0x7f400ab586c0) at /mnt/ssd1/karl/moz/dev/browser/app/nsBrowserApp.cpp:220 #23 0x00000000004049d5 in main (argc=3, argv=0x7ffd86312128, envp=0x7ffd86312148) at /mnt/ssd1/karl/moz/dev/browser/app/nsBrowserApp.cpp:360 With that, and bug 1261277 and bug 1264170 fixed, the reftests differences all seem fuzzable, but there are some other tests failing: docshell/test/browser/browser_timelineMarkers-02.js | markers includes Paint dom/tests/mochitest/general/test_offsets.xul | scrollHeight - got 49, expected 51 toolkit/content/tests/chrome/test_scaledrag.xul | After mouseup and drag right on slider, scale has correct value - got 1, expected 2 https://treeherder.mozilla.org/#/jobs?repo=try&revision=16cc938b9356&exclusion_profile=false

Moving to ubuntu 16.04 images is another option here. This can be configured on try using the last two changesets at https://treeherder.mozilla.org/#/jobs?repo=try&revision=b6afe6747bb1

Intermittent e10s test failure

The whole testsuite has been hidden.

(In reply to Phil Ringnalda (:philor) from comment #8) > I vote we just stop running reftests on ASan - I've lost all interest in > their results, and I will star any number of failures, 2 or 10 or 100, as > whatever happens to be the first failure which is already filed and thus > suggested. This isn't going to be popular, since unlike ASan people actually expect to see results for reftests on opt and debug Linux32 and Linux64, but I'm ready to hide them too, for the exact same reason.

P4, eh? Okay, which do we want, of the two available choices: * no visible reftests run on any flavor of linux, opt/debug/asan, linux32/linux64, or * a bug1223198 condition in the sandbox, true for any flavor of linux we currently run, and a random-if(bug1223198) added to every single reftest which displays a form control or a scrollbar, so that we will have visible Linux reftests which are a lie, looking like reftests have run when in fact only a small percentage have run?

Clear enough.

All Linux reftests hidden on all trees except mozilla-esr45.

This is pretty unfortunate. Linux is usually the only platform we run tests on before landing patches, because it is faster than other platforms. Hopefully this can be resolved as soon as possible.

I ran across this bug, if we have all the reftests hidden for linux32 opt/debug linux64 opt/pgo, we are wasting a lot of effort. I have been looking to get linux32 tests running on taskcluster (building a docker image) and with the tests perma failing, I am considering ignoring these which is sad and only makes the problem worse. Can we get a decision here as to if we care about these tests and either fix them this year or turn them off on our branches and leave them available on try in case somebody has time in the future? as a note, asan reftests (on taskcluster/docker/ubuntu 16.04) seem to be pretty green.

:karlt, I see you are the module owner, can you help address my comments in comment 37 ?

FWIW, I'm turning off all buildbot-based Linux reftests (except ESR45) over in bug 1317568 :)

hmm, with that information, I will not plan to run reftests on linux32 taskcluster and let someone else figure out how to get them running. To be honest there are about 8 failures, so it isn't much, when there is interest to get them running again.

This bug should either be called fixed or wontfix depending on your perspective. Buildbot jobs are never going to be fixed because Taskcluster has since gotten them green and unhidden (bug 1311916) on Linux64, and that's obviously the only path forward we care about. Assuming we still care about Linux32 enough to support getting jobs ported over from Buildbot to TC on it, my suggestion would be to open a new bug similar to bug 1309086 for Linux32 and track getting them enabled there.

This bug still affects a few mochitests, but bug 1290183 is the solution for them also, and so, yes, there is no need to keep this bug open. Buildbot linux64 reftests are obsolete and so can be disabled. They were still useful while hidden for finding regressions holding back bug 1311916, but are know longer useful. I don't know the issues preventing making linux64 builds and reftests tier 1 (bug 1243025), or whether it matters if the tc reftests are tier 1 or 2. I don't know the long term direction with bug 1287099 versus bug 1209932. Having buildbot linux32 reftests running on m-c-merged branches may be useful for finding causes of regressions before bug 1287099 allows them to be unhidden, but I expect there are few regressions affecting only linux32 reftests. Bug 1319109 may prove me wrong.