The AWSY tests are currently running as tier 2 due to frequent network issues, which may be related to the bug 1517323, which has been fixed for Raptor. Once bug 1567137 is resolved, let's review the failure rate and see if we can promote these tests to tier 1.

See bug 1540886 for more background on the failures we were seeing.

It looks like we haven't seen any intermittents like bug 1540886 or bug 1541183 in around 4 months. I think we can promote this test to tier 1.

Pushed to try: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&searchStr=sy&revision=f62838cbd0f50a5c4244a94a4b31dfcef4facbf4

Roping in Nathan, David, and Gabriele - apologies for casting the net wide, and without much on-IRC heads-up (and thanks for your help on bug 1549272, Gabriele, David!)

The few failures [0][1] (earlier retriggers have passed, since, and there are new ones pending) seem mostly in variants of bug 1543552 and bug 1377988, and also seem limited (in my few runs) to Linux x64 CCov debug, which is also currently tier 2.

Thoughts/comments, both on general approach with awsy tp6 (before, and later, with Fission), and specific to the patch and intended coverage, here?

The need-info for you, Nathan, is to help load-balance me bugging Gabriele and David elsewhere :-)

Cheers!

[0] https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&author=sdonner%40mozilla.com&selectedJob=267490692
[1] https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&author=sdonner%40mozilla.com&selectedJob=267517907

(In reply to Stephen Donner [:stephend] from comment #5)

Roping in Nathan, David, and Gabriele - apologies for casting the net wide, and without much on-IRC heads-up (and thanks for your help on bug 1549272, Gabriele, David!)

The few failures [0][1] (earlier retriggers have passed, since, and there are new ones pending) seem mostly in variants of bug 1543552 and bug 1377988, and also seem limited (in my few runs) to Linux x64 CCov debug, which is also currently tier 2.

It's not clear to me that we should even be running talos tests against coverage builds in the first place, so I wouldn't worry too much about that...but the awsy-tp6 tests are not supposed to be running against coverage builds:

https://hg.mozilla.org/try/rev/4074a1a78d95bdc4f3f9eaca8396952b05f28c3a#l1.5

so why are they running against coverage builds? Is this a casing issue (CCov vs. ccov) in the job name?

(In reply to Nathan Froyd [:froydnj] from comment #6)

It's not clear to me that we should even be running talos tests against coverage builds in the first place, so I wouldn't worry too much about that...but the awsy-tp6 tests are not supposed to be running against coverage builds:

https://hg.mozilla.org/try/rev/4074a1a78d95bdc4f3f9eaca8396952b05f28c3a#l1.5

so why are they running against coverage builds? Is this a casing issue (CCov vs. ccov) in the job name?

From what I /think/ I've discovered so far, the job-defaults here[2] for awsy (which I think awsy-tp6 inherits, and only overrides when needed) point to awsy_script.py. That (appears to) unilaterally import and set up CodeCoverageMixin[3], but that's just a slightly-better-than-wild guess.

[2] https://searchfox.org/mozilla-central/rev/e18057a9613ffda06dfd3640209ca234ed7dc37d/taskcluster/ci/test/awsy.yml#6
[3] https://searchfox.org/mozilla-central/rev/e18057a9613ffda06dfd3640209ca234ed7dc37d/testing/mozharness/scripts/awsy_script.py#30-32

I'll land the tier-1 promotion bit, and file the ccov issue separately, and cross-reference here.

https://hg.mozilla.org/mozilla-central/rev/96844e92a0f6