there have been a few bugs which have minor regression on 10.6 (bug 990140, bug 990233, bug 985575) which have brought up the question of how much value do we get out of running talos tests on 10.6. Andreas brings up a valid point in https://bugzilla.mozilla.org/show_bug.cgi?id=990233#c10. Currently for OSX we run on 10.6 and 10.8 (we disabled 10.7 testing completely). I know 10.9 is in the works, but no idea on the time horizon for enabling performance and unittests. Over the last year, a lot of work has been done to make/rewrite tests which are useful (tart, cart, tsvgx, tscrollx). We know most of the talos tests provide good value (others like dromaeo have been questioned, see bug 987136). The question becomes, given a regression on 10.6- do we care about it? I suspect the answer will be yes if it is large (>20%), yes if it affects other platforms as well, and probably no if it is <5% and only affects 10.6. I would like to ensure we are running and monitoring tests which provide value and make good use of our resources.

We know of at least one thing that running talos on 10.6 is good for, giving us coverage that is useful for other platforms: 10.6 OpenGL libraries have fewer optimizations than the OpenGL libraries of more recent OSX versions. For that reason, we have seen real gfx/gl regressions where we were making unneeded OpenGL calls, caught by Talos on 10.6 slaves and not on any other platform. That includes at least bug 950113, and possibly bug 990233. We have reasons to suspect that the 10.6 slaves are not the only machines where OpenGL libraries are un-optimized in this way, and that the regressions found on 10.6 would also exist on many other devices, especially mobile devices where driver quality is typically lower and where we have relatively little test coverage.

More to the point of the question being asked: if a regression is 10.6 only, and is caused by a patch touching code around gfx/gl or gfx/layers/opengl, then yes we probably do care.

(In reply to Joel Maher (:jmaher) from comment #0) > I would like to ensure we are running and monitoring tests which provide > value and make good use of our resources. Just adding that these resources include not only IT resources but also very real developers and development time. A regression which should be handled could be followed with a backout, someone[s] to look and re-assess the code, code changes, try runs, reassessments, etc. So working on unnecessary ones does impede development, sometimes very meaningfully. Following comment 2, for now, I'd suggest that we keep the 10.6 tests running at the same trees/frequency as before, but not try to pursue small-ish regressions unless they're clearly a result of gfx/layers/gl patches. Let's relax this constraint in the future if we become aware of other code areas at which a 10.6-only regression could be indicative of a bigger issue.

For now I will continue to: * file bugs for all regressions * be explicit that they are 10.6 only when that is the case * ask if the patches in question are gfx/layers/gl to see if it is worth investigating or not. Is there anything else we should do for 10.6 only bugs when they are found?

Sounds good to me. Small 10.6-only bugs which are not clear gfx/gl/etc patches will only get filed/logged, closed as wontfix, and depend on this bug?

sold!

Note that if we had more extensive Talos coverage on mobile devices that we care about the most (typical b2g hardware) then my reason for caring about 10.6 would evaporate!

(In reply to Benoit Jacob [:bjacob] from comment #7) > Note that if we had more extensive Talos coverage on mobile devices that we > care about the most (typical b2g hardware) then my reason for caring about > 10.6 would evaporate! What would be a good OS to catch these types of regressions? (instead of 10.6) Linux? Windows? Can a thread be started to find one (few)? Anything that we could run on EC2? Out of all talos suites, which ones are useful to catch these gfx/gl regressions? I want to find a way out of 10.6 since we will not support it forever.

(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #8) > (In reply to Benoit Jacob [:bjacob] from comment #7) > > Note that if we had more extensive Talos coverage on mobile devices that we > > care about the most (typical b2g hardware) then my reason for caring about > > 10.6 would evaporate! > > What would be a good OS to catch these types of regressions? (instead of > 10.6) Linux? Windows? In an ideal world, we would have B2G running on actual B2G hardware; failing that, Android running on B2G-like hardware. (The most important aspect of hardware being the GPU, i.e. preferably an Adreno GPU). If we can't run tests on real mobile GPUs then I suppose that all other options (emulators, software renderers, desktop GPUs) each have a slight chance of catching various kinds of relevant bugs, like here OSX 10.6 luckily catches bugs of the form "accidentally making redundant OpenGL calls" as thankfully the 10.6 GL libraries are unoptimized in this respect. So I'm not saying that the current 10.6 slaves are perfect, and so, I'm not saying that we have to wait until a perfect solution (B2G on real B2G hardware) exists to phase out 10.6 slaves. > Out of all talos suites, which ones are useful to catch these gfx/gl > regressions? Out of the top of my head, at least tpaint, tscroll, tresize. Really anything that will cause a lot of visual changes in the browser window (where scrolling and resizing do count as visual change). > I want to find a way out of 10.6 since we will not support it forever. Sure, and I'm not saying that you should. I just wanted to provide one data point.

with bug 1118329 resolved, I am not sure there is much left to do here. We are running about half of the tests on osx 10.6 now, ones that have been deemed important.

it was a year ago when we had discussed that 10.6 was useful for mobile platforms due to the GPU and unoptimized openGL. Right now we have a reduced set of tests running on 10.6, graphics only ones. The landscape changes every few months, I think enough time has elapsed that we should revisit whether or not we should be running osx 10.6 talos tests. In fact we have dozens of osx 10.6 timeouts on talos in bug 934310 (600+ to date), and it would be nice to just solve the frustration for the sheriffs and reduce the unecessary testing we do. That is if it makes sense to. :bjacob, can you help us figure out if now 12 months later we have the same requirements on mobile devices to test on 10.6 as we did back then?

IMO we can remove 10.6 completely. Benoit no longer works with Mozilla for some months now, though he might still want to reply. Jeff, what are your thoughts on this? In a nutshell, do you think that leaving some graphics tests running on OS X 10.6 has some value in detecting real performance issues which are harder to detect on newer versions of OS X due to better optimizations? Do you know anyone else who might have a useful opinion on this subject?

No longer at Mozilla / not qualified to answer this question at the moment.

(In reply to Avi Halachmi (:avih) from comment #12) > IMO we can remove 10.6 completely. > > Jeff, what are your thoughts on this? In a nutshell, do you think that > leaving some graphics tests running on OS X 10.6 has some value in detecting > real performance issues which are harder to detect on newer versions of OS X > due to better optimizations? I do not think it provides commensurate value to also test webgl on 10.6 on CI. 10.6 has the same structure as 10.8 (and 10.10). > > Do you know anyone else who might have a useful opinion on this subject? :jrmuizel (already needinfo'd)

ping for update here

I think we can do away with 10.6 talos testing. Given performance regressions on 10.6, it's unlikely that we'd spend much or any effort investigating them.

this is a buildbot-config patch to remove talos from osx10.6 and the random windows 8 talos tests on mozilla-release (I think these got on there by accident when we enabled windows 8). Quite possibly we could disable talos on 10.6 easier or in a cleaner way.

Comment on attachment 8595430 [details] [diff] [review] remove osx 10.6 talos, also win8x64 on mozilla-release (1.00 Review of attachment 8595430 [details] [diff] [review]: ----------------------------------------------------------------- r+ with fixing get_talos_slave_platforms(). I think you can fix it by adding this: PLATFORMS['macosx64']['talos_slave_platforms'] = ['yosemite'] which gets processed here: http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla-tests/config_common.py#l73 ::: mozilla-tests/config.py @@ +226,5 @@ > platform_config[slave_platform]['try_slaves'] = sorted(TRY_SLAVES[slave_platform]) > else: > platform_config[slave_platform]['try_slaves'] = platform_config[slave_platform]['slaves'] > > ALL_TALOS_PLATFORMS = get_talos_slave_platforms(PLATFORMS, platforms=('linux', 'linux64', 'win32', 'macosx64', 'win64')) You can probably modify get_talos_slave_platforms to not give you snowleopard. @@ +232,5 @@ > NO_WINXP = [platform for platform in ALL_TALOS_PLATFORMS if platform != 'xp-ix'] > WIN7_ONLY = ['win7-ix'] > WIN8_ONLY = ['win8_64'] > LINUX64_ONLY = get_talos_slave_platforms(PLATFORMS, platforms=('linux64',)) > NO_LINUX64 = get_talos_slave_platforms(PLATFORMS, platforms=('linux', 'win32', 'macosx64', 'win64')) Same in here.

thanks! updated and landed: https://hg.mozilla.org/build/buildbot-configs/rev/63762f3232e5

Could you please give me another patch to address the comment below? I still see this line (which you don't need): http://hg.mozilla.org/build/buildbot-configs/rev/63762f3232e5#l1.12 and this line: http://hg.mozilla.org/build/buildbot-configs/rev/63762f3232e5#l1.23 You don't need this: http://hg.mozilla.org/build/buildbot-configs/rev/63762f3232e5#l2.13 if you add this: PLATFORMS['macosx64']['talos_slave_platforms'] = ['yosemite']

Thanks armenzg, as per IRC I have landed a change: https://hg.mozilla.org/build/buildbot-configs/rev/83a60eef2545 This looks like the right way and we have now cleaned up a lot of extra builders!

In production: https://hg.mozilla.org/build/buildbot-configs/rev/63762f3232e5

In production: https://hg.mozilla.org/build/buildbot-configs/rev/83a60eef2545