As part of bug #1549555 and bug #1518930 I was looking at various tests that were run on each platform.

Currently, reftest is run on almost every platform, but reftest-no-accel and reftest-gpu are only run on a handful of platforms combined.

Given that each chunk (after re-balancing) consumes approximately 20-30 minutes each, the amount of time that the CI system consumes in order to run these tests is not trivial.

What conditions are each of the specialized reftest variants testing? Do they continue to be relevant in 2019 and beyond, or perhaps could they be reduced in scope and/or sunsetted?

:catlee - I've seen your name attached to a bunch of reftest related bugs. Do you know the specifics of what the no-accel and gpu variants are testing, that are separate from the baseline reftest?

:jgilbert, would you know the answer to the above question? I don't know the history of no-accel and gpu reftest jobs. Maybe you would know who could help get an answer.

btw, reftest-gpu (tc config) was added by part AI of bug 1381577. reftest-no-accel config goes back to buildbot (out of tree).

(In reply to Edwin Gao (:egao) from comment #1)

:catlee - I've seen your name attached to a bunch of reftest related bugs. Do you know the specifics of what the no-accel and gpu variants are testing, that are separate from the baseline reftest?

no, I don't know, sorry.

reftest-no-accel is needed on every platform that we normally have a gpu-accelerated fast-path, but also support a non-gpu-accelerated slow-path.
reftest-gpu should just be the complement of that: Platforms where we don't require a gpu for reftest, but we need to test the gpu-accelerated path most of our users will take.
There are platforms where we always have a gpu (Android, maybe Mac?) where we probably don't have to split these out.

:jgilbert, thanks for the reply, that is a good explanation.

here is what we have running currently:
linux64: reftest, wpt-reftest, reftest-no-accel
win7: reftest, wpt-reftest, reftest-no-accel, reftest-gpu
win10: reftest, wpt-reftest
osx: reftest, wpt-reftest
android: reftest, wpt-reftest

I would think we need to duplicate the wpt-reftests as no-accel and gpu as well.

:jgilbert, can you look over the above list in this comment and let us know if there are any adjustments (add, edit, remove) needed?

That looks like what we want, thanks!
I don't know about wpt-reftest vis a vis whether we want/need no-accel/gpu there.
:jrmuizel?

ping to help finish this bug...

I think yes, the wpt-reftests should also be duplicated the same way regular reftests are. At least in an ideal world assuming no constraints on testing machinery.

(Stealing and responding to needinfo to get this bug off the triage dashboard for the reftest component)