For code coverage builds, we should ignore SCHEDULES and always run all tests.

the specific concern is that we skip jsreftests often (good for saving resources) and end up with variable code coverage when analyzing a full run. I assume we can fix this with a transform?

Are these the cron-driven coverage builds? If so, they should just include all of the relevant tests in the target task set, rather than trying to adjust the optimization settings.

these are regular test sets scheduled on mozilla-central only (i.e. every m-c build, not a cron task): https://searchfox.org/mozilla-central/source/taskcluster/ci/test/test-platforms.yml#129 https://searchfox.org/mozilla-central/source/taskcluster/ci/test/test-platforms.yml#185

Hm, so the idea would be to optimize these normally on all branches where they run, but on mozilla-central to never optimize them? Which will mean always performing the ccov build on each platform, then running the jsreftest tasks? I'm just trying to get an idea of the cost and E2E time..

(In reply to Dustin J. Mitchell [:dustin] from comment #4) > Hm, so the idea would be to optimize these normally on all branches where > they run, but on mozilla-central to never optimize them? Which will mean > always performing the ccov build on each platform, then running the > jsreftest tasks? I'm just trying to get an idea of the cost and E2E time.. I think we would need to never optimize them on mozilla-central only for the ccov builds. The other builds should be unaffected.

OK, then I think implementing that by setting optimization with a `by: branch: ..` would be OK.

We have two options: 1) define the optimization in the yaml file, disabling them for ccov builds; 2) modify the transform we have for ccov builds and set the optimizations there. The advantage of the first solution is that it is cleaner (even though it requires modifying many yaml files) and less likely to regress. The advantage of the second solution is that we don't have to remember to add the `optimization` to any new yaml file.

Comment on attachment 8952485 [details] Never optimize test suites for coverage builds as far as I can tell this looks good. My knowledge of tasks has become less useful with the schedules work, it seems to have overflowed what I can process and understand! I would have thought that just |test['optimization'] = None| would have solved this. Maybe there is transform logic that requires the removal of 'schedules-component' and 'when' clauses as well.

Comment on attachment 8952484 [details] [diff] [review] Make it possible to overwrite 'optimization' in tests transforms Review of attachment 8952484 [details] [diff] [review]: ----------------------------------------------------------------- Looks great. Sorry for the delay in review!!

Pushed by mcastelluccio@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/4cbab279837a Make it possible to overwrite 'optimization' in tests transforms. r=dustin https://hg.mozilla.org/integration/mozilla-inbound/rev/eeb50080d6de Never optimize test suites for coverage builds. r=jmaher

https://hg.mozilla.org/mozilla-central/rev/4cbab279837a https://hg.mozilla.org/mozilla-central/rev/eeb50080d6de

Needinfoing me so I remember to verify everything is working as it should.