STR:

1. update m-c to current tip
2. ./mach try fuzzy --full
3. filter for e.g. 'mac 'awsy

ER:
see both qr and non-qr jobs

AR:
no non-qr jobs

In Fx83 WebRender will be enabled for 90%+ of the macOS population, so for performance purposes we should mostly just care about the QR setup. In the interests of saving resources we switched mac CI to mostly running with WebRender enabled.

Set release status flags based on info from the regressing bug 1673071

(In reply to Kartikaya Gupta (email:kats@mozilla.staktrace.com) from comment #1)

In Fx83 WebRender will be enabled for 90%+ of the macOS population, so for performance purposes we should mostly just care about the QR setup. In the interests of saving resources we switched mac CI to mostly running with WebRender enabled.

It makes sense that WebRender should be our primary configuration, but should we maintain a subset of tests running with WebRender disabled? A performance regression for ~10% of our users on macOS would still affect a large number of users.

Jeff, thoughts?

If we can afford to have some of both that's great. To make a decision here we really need to know how much running both will cost. Dave can you get an estimate for that?

(In reply to Jeff Muizelaar [:jrmuizel] from comment #5)

If we can afford to have some of both that's great. To make a decision here we really need to know how much running both will cost. Dave can you get an estimate for that?

The macOS hardware is more limited, but I suspect we could at least run AWSY. We also have the ability to run some tests less frequently, which could be an option here. I'm not able to answer the question of cost, however I suspect it's more an issue of device pool capacity. Mihai, can you help here?

Agreed with Joel to 302 this to him. thank you!

we typically don't have a queue for osx, and we do have a pool that will free up in another week. In short we have capacity to schedule at least 10 jobs, and probably 20 by next week. These will run periodically on autoland and be sheriffed I assume?

(In reply to Joel Maher ( :jmaher ) (UTC -0800) from comment #8)

we typically don't have a queue for osx, and we do have a pool that will free up in another week. In short we have capacity to schedule at least 10 jobs, and probably 20 by next week. These will run periodically on autoland and be sheriffed I assume?

I would suggest AWSY + high value page load tests running on the autoland backstop.

Let me know if you want me to look into doing this, or if somebody else is going to.

Is this a wontfix?

(In reply to Ryan VanderMeulen [:RyanVM] from comment #11)

Is this a wontfix?

No, assigning to Kartikaya based on comment 10.

:sparky :bebe looks like kats won't be able to work on this, could one of you take a look?

(In reply to Dave Hunt [:davehunt] [he/him] ⌚GMT from comment #9)

I would suggest AWSY + high value page load tests running on the autoland backstop.

I wrote a patch that adds back AWSY and raptor-tp6 for non-QR macOS. But I'm not totally sure what you mean by the "autoland backstop". It looks like raptor-tp6 only runs on m-c by default, and awsy runs on autoland/m-c/m-b/m-r. I can trim that down a bit if you want.

(But also, if the patch is going to be significantly more complicated than what I wrote, it would be better if somebody else picks it up)

(In reply to Kartikaya Gupta (email:kats@mozilla.staktrace.com) from comment #15)

(In reply to Dave Hunt [:davehunt] [he/him] ⌚GMT from comment #9)

I would suggest AWSY + high value page load tests running on the autoland backstop.

I wrote a patch that adds back AWSY and raptor-tp6 for non-QR macOS. But I'm not totally sure what you mean by the "autoland backstop". It looks like raptor-tp6 only runs on m-c by default, and awsy runs on autoland/m-c/m-b/m-r. I can trim that down a bit if you want.

Thanks kats, I appreciate you staying involved! I suspect it might be necessary to get :sparky to help here as he's recently migrated macOS to browsertime. Also, it looks like we might need to do some more work to target the high value tests more easily.

I'm thinking we should implement a by-tier key and use it here in the run-on-projects settings for these kinds of cases: https://searchfox.org/mozilla-central/source/taskcluster/ci/test/browsertime-desktop.yml#98-106

Alternatively, we could create a new test group, but then we have the issue that we have two nearly duplicated sets of definitions for these tests which is something we've been trying to avoid.

Another option is to use a transform to modify the run-on-projects settings on tier-1 tests.

:davehunt, I think it might be a good idea for :kimberlythegeek to finish this off since it will get more complicated.

(In reply to Greg Mierzwinski [:sparky] from comment #18)

I'm thinking we should implement a by-tier key and use it here in the run-on-projects settings for these kinds of cases: https://searchfox.org/mozilla-central/source/taskcluster/ci/test/browsertime-desktop.yml#98-106

Alternatively, we could create a new test group, but then we have the issue that we have two nearly duplicated sets of definitions for these tests which is something we've been trying to avoid.

Another option is to use a transform to modify the run-on-projects settings on tier-1 tests.

I like the idea of a by-tier if that will work given that the tiers themselves are set by app/subtest.

:davehunt, I think it might be a good idea for :kimberlythegeek to finish this off since it will get more complicated.

:kimberlythegeek could you sync with :sparky on moving this forward?

Apologies that this slipped through before everyone went on PTO, I will touch base with him as soon as possible.

https://hg.mozilla.org/mozilla-central/rev/ba821d51cce8