Kim, we'd like to add a new "linux64-stylo-sequential" platform to run some more Stylo tests: Stylo tests can run in either parallel or sequential mode (primarily for single-core machines). In bug 1318187, we will make parallel mode the default for all Stylo tests on inbound, autoland, and central. To make sure we're still testing sequential mode, we'd like to create a new "linux64-stylo-sequential" platform that only runs on central and try. linux64-stylo-sequential would run reftests, mochitests, and crashtests for both debug and opt builds. We don't need to run sequential reftests on non-e10s. e10s debug reftests e10s debug mochitests e10s debug crashtests e10s opt reftests e10s opt mochitests e10s opt crashtests non-e10s debug mochitests non-e10s debug crashtests non-e10s opt mochitests non-e10s opt crashtests

I have a few questions for you So my understanding is that you are asking for a new standalone linux64-stylo-sequential build with the tests listed above. Is there a mozconfig for this build? I didn't see one in tree. I assume when you say try, you meant that the build can run on try but not by default? Also, I'm going to check on the state of our AWS budget these days.

(In reply to Kim Moir [:kmoir] from comment #1) > I have a few questions for you > > So my understanding is that you are asking for a new standalone > linux64-stylo-sequential build with the tests listed above. Is there a > mozconfig for this build? I didn't see one in tree. I don't think there is - it would be a normal stylo build with an environmental variable set. Nathan Froyd knows the details there. > I assume when you say > try, you meant that the build can run on try but not by default? That should be fine. > > Also, I'm going to check on the state of our AWS budget these days. Since these only run on m-c, they should be effectively budget-neutral.

(In reply to Bobby Holley (:bholley) (busy with Stylo) from comment #2) > (In reply to Kim Moir [:kmoir] from comment #1) > > I have a few questions for you > > > > So my understanding is that you are asking for a new standalone > > linux64-stylo-sequential build with the tests listed above. Is there a > > mozconfig for this build? I didn't see one in tree. > > I don't think there is - it would be a normal stylo build with an > environmental variable set. Nathan Froyd knows the details there. This platform can use the regular stylo mozconfig, and most (all?) of the config file bits should just be copy-and-pastes of the linux64-stylo config file bits. Once all the necessary config file bits are in place, we'd just need to modify the taskcluster transform that's going to land as part of bug 1318187. It looks like: @transforms.add def parallel_stylo_tests(config, tests): """Ensure that any stylo tests running with e10s enabled also test parallel traversal in the style system.""" for test in tests: if not test['test-platform'].startswith('linux64-stylo/'): yield test continue e10s = test['e10s'] # We should have already handled 'both' in an earlier transform. assert e10s != 'both' if not e10s: yield test continue test['mozharness'].setdefault('extra-options', [])\ .append('--parallel-stylo-traversal') yield test as that bug is trying to run stylo tests in parallel on e10s tests, and sequentially on non-e10s. The transform would then add the appropriate mozharness option if the test was linux64-stylo only; linux64-stylo-sequential tests would be passed through unmodified. Or we can sequence things such that the linux64-stylo-sequential configs are stood up first, and then bug 1318187 flips the linux64-stylo tests to use parallel stylo. I'm fine either way.

garndt: I talked to catlee and he suggested talking to you about this in light of the taskcluster AWS budget since this is a request to add an additional stylo build + tests on m-c and by request on try. Is there an issue with enabling these given current budget constraints?

Per discussion, we don't need a whole new platform for this: it is just a variation of how the tests are executed. Also, since we're only talking tests on Linux (for now), cost should not be an issue, as the cost of a Linux test worker is literally fractions of a penny per hour and the total cost for these extra tasks will be just a few cents per job. We'll spend more money discussing it here and implementing it than it will cost in CI in the next several months. If we're really that budget constrained, we have far bigger things to worry about.

gps does have a point that adding these suites do not add much to the cost per push. The worker types used for those suites range from $0.01/hr to $0.04/hr on average. Quick calculations show it might be < 10 cents per push unless I misplaced a zero. For things like this, there really is no concern about adding them or not. It is good to know when we do add more jobs because over time 10 cent additions do add up if you do it a 100 times.

Kim, do you need any additional info to run the tests in Stylo sequential mode using the existing linux64-stylo builds? Based on gps's comment 5, I don't think we need a new linux64-stylo-sequential mozconfig or build.

I think so. Are changes needed in treeherder so the new tests appear correctly?

(In reply to Kim Moir [:kmoir] from comment #8) > I think so. Are changes needed in treeherder so the new tests appear > correctly? Sorry, I don't know. I assume the new tests would need a new test name abbreviation on Treeherder, but I don't know if both the tests and Treeherder need to know about the test name abbreviations.

fixes linting errors in https://treeherder.mozilla.org/#/jobs?repo=try&revision=86b69ce91bca1e0a8cb5197ca84e5e0f27264ecf&selectedJob=96966850

Pushed by kmoir@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/c21bd752dc66 Run Stylo tests in sequential mode r=aki