Closed Bug 1580917 Opened 5 years ago Closed 5 years ago

[Experiment] Pref-Flip Experiment: WebRender Intel Performance 70 Fx 70.0 Release

Categories

(Shield :: Shield Study, task, P3)

Tracking

(firefox70- affected)

RESOLVED FIXED
Tracking Status
firefox70 - affected

People

(Reporter: experimenter, Assigned: jbonisteel)

References

()

Details

User Story

Experiment Type: Pref Flip Experiment

    What is the preference we will be changing

gfx.webrender.all.qualified

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment enabled 50%:

Value: true

Control branch
        
- Treatment disabled 50%:

Value: false

This is the group that we will disable WebRender for so we can compare performance against those that do have WebRender
        

    What version and channel do you intend to ship to?

5% of Release Firefox 70.0

    Are there specific criteria for participants?

Prefs: IMPORTANT: This is a pref flip experiment, please keep it separate from the Rollout (https://experimenter.services.mozilla.com/experiments/webrender-intel-rollout/).
Experiments: none

Any additional filters:
1. Windows 10
2. normandy.telemetry.main.environment.system.gfx.features.wrQualified.status == 'available'
3. Intel Graphics cards, Windows 10 no battery, small screen: (<= 1920x1200)

Sgua NOTE:  Filtering on normandy.telemetry.main.environment.system.gfx.features.wrQualified.status == 'available' and an Intel graphics card should incorporate the screen size and battery limits; those are baked into the wrQualified assessment.

In comments with Jessie we agreed to filter to single monitors.
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Oct 29, 2019 - Nov 26, 2019 (28 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

overall crash reports ≤ 5% increase in crash rate
OOM crash reports ≤ 5% increase in crash rate
shutdown crashes ≤ 5% increase in crash rate
CANVAS_WEBGL_SUCCESS ≤ 5% regression in median of fraction "True" per user
COMPOSITE_TIME Median per-user fraction of slow frames < 0.5% (absolute)
CONTENT_FRAME_TIME_VSYNC ≤ 5% regression in median of per-user fraction of slow events
CONTENT_FULL_PAINT_TIME ≤ 5% regression in fraction of slow paints (> 16 ms), ≤ 5% regression in median of per-user means
DEVICE_RESET_REASON ≤ 5% increase in reset rate
FX_PAGE_LOAD_MS_2 ≤ 5% regression in median of per-user means

Crash (/submission/reset) rates will be measured as events per 1,000 usage hours. A "slow event" means 16 ms or 200% vsync.

Do you plan on surveying users at the end of the experiment?  No.
Strategy and Insights can help create surveys if needed

    Who is the owner of the data analysis for this experiment?

sguha@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

Signed off as GREEN, for more info see https://experimenter.services.mozilla.com/experiments/webrender-intel-performance-70/#comment496.

Green

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/webrender-intel-performance-70/

WebRender Intel Performance 70

The goal of this study is to hold back a small population of eligible users from enabling WebRender, another milestone of which is riding the trains to release 70, in order to assess its effect on performance and engagement metrics. Our specific target for 70 is desktop, Windows 10 with Intel graphics cards and small screens.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

More information: https://experimenter.services.mozilla.com/experiments/webrender-intel-performance-70/

[Tracking Requested - why for this release]:

WebRender Intel Performance
Release 70

We have finished testing the WebRender Intel Performance experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

  • We haven’t found any issues during testing.

Testing Summary:

Tested Platforms:

  • Windows 10 x64

Tested Firefox versions:

  • Firefox Release 70

Regards,

Robert Martin

User Story: (updated)
Summary: [Experiment]: Pref-Flip: WebRender Intel Performance 70 → [Experiment] Pref-Flip Experiment: WebRender Intel Performance 70 Fx 70.0 Release
User Story: (updated)
User Story: (updated)
Start Date: 2019-10-29 End Date: 2019-11-26

I'm going to leave this open since it's yet to launch, but untrack it. Following along in Experimenter instead.

User Story: (updated)
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.