Closed Bug 1585039 Opened 5 years ago Closed 2 years ago

[meta][android] Add support to raptor-browsertime for measuring cpu, memory, and power

Categories

(Testing :: Raptor, enhancement, P3)

Product:
Testing
Component:
Raptor
Version:
Version 3
Type:
enhancement

Tracking

(firefox109 fixed)

Status:
RESOLVED FIXED
Milestone:
109 Branch
Tracking Flags:
Tracking Status
firefox109 --- fixed

People

(Reporter: rwood, Assigned: afinder)

References

(Blocks 2 open bugs)

Details

(Keywords: meta, Whiteboard: [fxp])

Attachments

(1 file, 2 obsolete files)

Bug 1585039 [meta][android] Add support to raptor-browsertime for measuring cpu r=#perftest
(deleted), text/x-phabricator-request
Details

Add the ability to measure cpu, memory, and power during raptor-browsertime tests.

Blocks: browsertime-ci-android
Summary: Add support to raptor-browsertime for measuring cpu, memory, and power → Add support to raptor-browsertime android for measuring cpu, memory, and power
Priority: P2 → P4
Assignee: nobody → gmierz2

Setting to P3 and unassigning myself. This is a lower priority task at the moment.

Assignee: gmierz2 → nobody
Priority: P4 → P3

Mass-removing myself from cc; search for 12b9dfe4-ece3-40dc-8d23-60e179f64ac1 or any reasonable part thereof, to mass-delete these notifications (and sorry!)

Summary: Add support to raptor-browsertime android for measuring cpu, memory, and power → [meta][android] Add support to raptor-browsertime for measuring cpu, memory, and power
Depends on: 1642486

Power testing migration has started in bug 1642486.

:davehunt, so Memory usage information is already gathered with the profiler so we will have to extract it in browsertime, but CPU is done very differently from both power and memory. I'm wondering if we should wait for CPU usage data to become available through the profiler rather than porting our code to browsertime (which it sounds like it's being worked on now-ish by gerald)?

Flags: needinfo?(dave.hunt)

I like the idea of doing this all through the profiler. Measuring CPU isn't a high priority, so perhaps we can afford to wait. I suggest opening a dependency and making sure it's blocked on the bug to bring CPU usage into the profiler.

Flags: needinfo?(dave.hunt)
Blocks: 1511347
No longer blocks: 1511347

We can now gather CPU usage information using glean in browsertime. Here's a patch that does this: https://treeherder.mozilla.org/jobs?repo=try&revision=1b5e8c125671c5c67c766045a920e5582e5b946a

Assignee: nobody → afinder
Status: NEW → ASSIGNED
Severity: normal → S3
Attachment #9295065 - Attachment is obsolete: true
Attachment #9302052 - Attachment is obsolete: true
Flags: needinfo?(afinder)
Flags: needinfo?(afinder)
Flags: needinfo?(afinder)
Flags: needinfo?(afinder)
Pushed by afinder@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/1953e8420be4 [meta][android] Add support to raptor-browsertime for measuring cpu r=perftest-reviewers,sparky,AlexandruIonescu

https://hg.mozilla.org/mozilla-central/rev/1953e8420be4

Status: ASSIGNED → RESOLVED
Closed: 2 years ago
status-firefox109: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 109 Branch
Blocks: 1804448
Regressions: 1804529

== Change summary for alert #36330 (as of Wed, 07 Dec 2022 00:29:38 GMT) ==

Regressions:

Ratio Test Platform Options Absolute values (old vs new)
31% reddit-billgates-ama.members PerceptualSpeedIndex windows10-64-shippable-qr cold fission webrender 371.85 -> 486.17
26% reddit-billgates-ama.members ContentfulSpeedIndex windows10-64-shippable-qr cold fission webrender 315.15 -> 396.75
24% reddit-billgates-ama.members FirstVisualChange windows10-64-shippable-qr cold fission webrender 133.33 -> 164.75
21% reddit-billgates-ama.members PerceptualSpeedIndex linux1804-64-shippable-qr cold fission webrender 321.06 -> 387.62
19% reddit-billgates-post-1.comments PerceptualSpeedIndex windows10-64-shippable-qr cold fission webrender 211.38 -> 251.79
17% reddit-billgates-post-2.hot LastVisualChange windows10-64-shippable-qr cold fission webrender 452.88 -> 528.88
17% facebook-nav.groups FirstVisualChange windows10-64-shippable-qr cold fission webrender 200.00 -> 233.00
16% facebook-nav.friends FirstVisualChange windows10-64-shippable-qr cold fission webrender 142.79 -> 166.04
16% facebook-nav.friends SpeedIndex windows10-64-shippable-qr cold fission webrender 142.79 -> 166.04
16% reddit-billgates-post-1.posts ContentfulSpeedIndex linux1804-64-shippable-qr cold fission webrender 203.29 -> 235.42
... ... ... ... ...
3% welcome SpeedIndex linux1804-64-shippable-qr fission warm webrender 1,108.12 -> 1,142.33
3% welcome PerceptualSpeedIndex linux1804-64-shippable-qr fission warm webrender 1,118.75 -> 1,152.33
3% ebay PerceptualSpeedIndex linux1804-64-shippable-qr fission warm webrender 923.00 -> 950.33
2% wikipedia fcp linux1804-64-shippable-qr bytecode-cached fission warm webrender 44.00 -> 45.00
2% ebay SpeedIndex linux1804-64-shippable-qr fission warm webrender 1,150.00 -> 1,175.00

Improvements:

Ratio Test Platform Options Absolute values (old vs new)
76% reddit-billgates-post-2.hot FirstVisualChange macosx1015-64-shippable-qr cold fission webrender 286.67 -> 70.00
69% reddit-billgates-post-2.hot FirstVisualChange windows10-64-shippable-qr cold fission webrender 291.60 -> 90.38
67% reddit-billgates-post-2.top FirstVisualChange linux1804-64-shippable-qr cold fission webrender 295.00 -> 98.33
66% reddit-billgates-post-2.hot ContentfulSpeedIndex macosx1015-64-shippable-qr cold fission webrender 286.67 -> 96.21
66% reddit-billgates-post-2.top SpeedIndex linux1804-64-shippable-qr cold fission webrender 296.17 -> 101.71
... ... ... ... ...
2% twitter loadtime linux1804-64-shippable-qr bytecode-cached fission warm webrender 575.04 -> 563.25

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=36330

Regressions: 1806203
Regressions: 1806422
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: