It looks like we're going to need to split the manifest when running Gaia UI tests against emulators on TBPL due to us hitting the 2 hour build limit. Rather than split up the manifest files manually and have to maintain these separate files, it would be good to control this via a command line option. I'm thinking something like --chunk=1:4 where the tests selected to run would be split into four groups, as equal in number as possible, and the first group would be run. Subsequent (or parallel) jobs using --chunk=2:4 etc would run the remaining groups. So long as all jobs are running on the same revision then the groups can be trusted not to overlap or omit tests.

Comment on attachment 8388459 [details] [diff] [review] Add a command line option to split the tests and run a specified chunk. v1.0 Review of attachment 8388459 [details] [diff] [review]: ----------------------------------------------------------------- ::: testing/marionette/client/marionette/runner/base.py @@ +975,5 @@ > if self.marionette.check_for_crash(): > break > > def run_test_sets(self): > + run_chunk, total_chunks = [int(s) for s in self.chunk.split(':')] Can we use this_chunk instead of run_chunk to be consistent with mochitest and other harnesses?

Carried r+

Before landing this, I notice that mochitest uses --this-chunk and --total-chunks command line options. Do we want to simply use that rather than introduce a strictly formatted string? My initial thinking was that it would be better to keep the number of command line options to a minimum, but the two arguments would be less error prone and would be consistent with mochitests.

dave and I discussed and agreed it made sense to use the same argument names as mochitest, for consistency's sake.

Backed out for Gu bustage. https://hg.mozilla.org/integration/fx-team/rev/8b9f265aae5a https://tbpl.mozilla.org/php/getParsedLog.php?id=36389340&tree=Fx-Team

And other bustage: https://tbpl.mozilla.org/php/getParsedLog.php?id=36389402&tree=Fx-Team Please make sure this passes on Try before re-requesting checkin.

Carrying r+ running on try: https://tbpl.mozilla.org/?tree=Try&rev=7fb1e0c74d74

Sorry for that silly mistake Ryan, try looks good now.