Brief Description of the request (required):

We are aiming to ship WebRender in 68 for Windows 10 desktop, with AMD graphics cards. We would like to conduct an experiment, similar to what we did for 67 with. I also plan on requesting a gradual rollout, will link to that bug once it is made. Will also fill out the request in experimenter.

Business purpose for this request (required):

To verify there is no impact on performance when we ship our WebRender milestone in 68

Requested timelines for the request or how this fits into roadmaps or critical decisions (required):

Would like to start this a week after 68 releases

Links to any assets (e.g Start of a PHD, BRD; any document that helps describe the project):

Name of Data Scientist (If Applicable):

tdsmith

Please note if it is found that not enough information has been given this will delay the triage of this request.

Hi Jessie,

1. Can you link to the gradual rollout bug?
2. Can you review and revise the list of objectives you've defined in Experimenter against the AMD dashboard at https://metrics.mozilla.com/webrender/dashboard_amd.html? We will fail some of those criteria as they're currently defined, so either we should reconsider whether we're ready for a rollout or revise the go/no-go criteria. (I noticed tab switch time and crash rates -- I also just filed bug 1562354 to make sure we're including more recent beta builds in those plots, but the tab-switch regression reproduces on nightly.)

I've scheduled us for the lightning advising meeting for July 3 to give the Normandy team a heads-up about the experiment and rollout; you and telin should have just received invitations.

Thanks!

Here is the link to the rollout bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1562966

For the tab-switching metric, I think we are okay to revise our objectives and remove that. It did not result in any meaningful or actionable outcomes in previous experiments - I will revise the experimenter request.

We are looking at the crash rates right now, will confirm as soon as I can.

(In reply to Tim Smith 👨‍🔬 [:tdsmith] from comment #1)

Hi Jessie,

1. Can you link to the gradual rollout bug?
2. Can you review and revise the list of objectives you've defined in Experimenter against the AMD dashboard at https://metrics.mozilla.com/webrender/dashboard_amd.html? We will fail some of those criteria as they're currently defined, so either we should reconsider whether we're ready for a rollout or revise the go/no-go criteria. (I noticed tab switch time and crash rates -- I also just filed bug 1562354 to make sure we're including more recent beta builds in those plots, but the tab-switch regression reproduces on nightly.)

I've scheduled us for the lightning advising meeting for July 3 to give the Normandy team a heads-up about the experiment and rollout; you and telin should have just received invitations.

Thanks!

Following up on this - looking at the most recent dashboard it looks like things have improved with the crash rates.

For the power analysis, I'll assume the population target calculated for 66 still makes sense: https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/81426/command/81545

Indeed, the last few WR release experiments have been well-powered with 5% of Nvidia profiles.

We observe fewer AMD cards compared to Nvidia cards in telemetry, by a factor of 3. Looking at the last week of June: https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/140224/command/140249

Another question is what fraction of eligible chipsets are in qualified machines. Machines with batteries are not qualified for WR. Empirically, Nvidia chipsets are about twice as likely to be in qualified machines (84% in both beta and release) than AMD chipsets are (42% in beta), which adds another factor of 2.

So we need 5% * 3 * 2 = 30% of qualified AMD machines to enroll to be well-powered.

30% is a big number, but only 2.7% of active client_ids in a recent week met the Windows 10 + AMD graphics filters, and we expect only half of those to be desktops, so the final enrolled population will be 0.4% of desktop WAU.

Are there any recent updates for this bug on this experiment?

No. AFAICT the power analysis is the only data science that got done on this experiment; the only Databricks notebook that mentions this slug is irrelevant.

The WebRender team decided to abandon the rollout in favor of just delivering the feature, so this didn't have to inform any "live" decisions: https://bugzilla.mozilla.org/show_bug.cgi?id=1562966#c9

Work for the DS team is now tracked in Jira. You can search with the Data Science Jira project for the corresponding ticket.