For all I know, our current Canvas2D mochitests only test 100x50 canvases. We need to expand them to test at least two different sizes. This is generally valuable, but we also have a very specific reason to want to test two different sizes: we want to selectively turn on GPU acceleration of canvas rendering (e.g. Skia/GL) based on the canvas size, and when we do that, we will want our tests to cover both large GPU-accelerated canvases and small non-GPU-accelerated canvases. Otherwise, one of the two rendering paths would be untested. Snorp, tentatively assigning to you based on today's Skia/GL meeting.

Skia/GL is now turned on on B2G (bug 905227 landed) so the present bug is our last blocker to have Skia/GL/B2G not only landed but correctly covered by tests. The preference to use to adjust whether the canvas mochitests will use Skia/GL, is: gfx.canvas.min-size-for-skia-gl It is a 'hidden' preference, don't look for it in preference files. By default (when that preference isn't set) the min size for skia GL is 128 so mochitests don't use Skia/GL. We need to run them again with that preference set to 0.

Dear #ateam, we need to run all the mochitests in content/canvas/test/Makefile.in twice, with two different values of a preference. We can't just manually edit these tests, as there are many and the list will keep growing. What can we do?

Just to check that it will succeed, here is a try run with the non-default value of the pref that we want to test and currently don't on m-c. https://tbpl.mozilla.org/?tree=Try&rev=0086288d922c

I don't know that there's an easier way to do this than to run a separate mochitest job with a pref set via the --extra-prefs command-line arg. We'd have to verify this command-line arg works with B2G; I don't know that we've ever tried to use it before.

There are crossorigin and webgl subdirectories as well. Would you want all of those run with multiple prefs as well? what about the tests that are not prefixed with test_2d.* such as test_mozGetAsFile.html? One way to do this would be to edit all the files once and provide a framework for running them internally by changing the prefs. This could be done by reading in a common file which has the preferences defined and we could iterate over all the preference chunks, apply/test, repeat. Can we change the pref dynamically and run the test case, otherwise we would need to do something similar but with makefile hacks? The only other way I see is what jgriffin mentioned with passing a parameter into the harness to set the prefs for the entire test run. This is ugly because these tests run as a chunk of the entire set of mochitests. We would be multiplying our runs of other tests as well. For Android we split out webgl into its own suite because it was a high risk for OOM. Maybe we split out all canvas tests into their own suite to make this easier? Do we need all these prefs for every changeset? Would running these on PGO only builds be sufficient? Maybe on nightly only builds? It seems like we would be adding a lot of test time to support the entire matrix of prefs/tests.

I was going to say exactly what Joel said. The only two ways I know of doing this in our current setup are by setting up a separate test run with --set-pref (like mochitest-ipcplugins), or changing your tests to have either a sub-harness that runs the whole set of tests with both pref values, or each test runs its tests internally with both pref values.

Thanks guys. I will do it the sub-harness way (a la WebGL) then. I was sort of considering doing that, but wanted to check if there was another way that would be preferred by the specialists. Joel: To answer your questions: this is only about the content/canvas/src test without subdirectories (only affects Canvas 2D, and not very important for the crossorigin tests). This only applies to a single boolean preference, so we only want to run these tests twice. The reason why we want to is that we effectively have two completely different canvas 2d rendering paths (GL for large canvases when we can, software for small canvaeses or when we can't have GL). For that same reason, we actually need to have this checked on every changeset (even if I fully agree that it would be nice to run many things less frequently than on every changeset --- just not this particular thing).