Run YouTube playback performance tests muted on Android devices
Categories
(Testing :: Raptor, task, P1)
Tracking
(firefox69 fixed)
Tracking | Status | |
---|---|---|
firefox69 | --- | fixed |
People
(Reporter: whimboo, Assigned: whimboo)
References
(Depends on 1 open bug, Blocks 1 open bug)
Details
Attachments
(2 files)
We want to run the upcoming YouTube playback performance tests (bug 1548845) also on Android devices for apps which are based on GeckoView. Sadly those tests won't work because autoplay is blocked. And as long as bug 1547717 hasn't been fixed, we need a workaround.
I would propose that for those apps we simply run the suite with audio muted, which results in no blocking of autoplay.
Note, the performance numbers might be slightly different, which I will check next on desktop at least.
Assignee | ||
Comment 1•5 years ago
|
||
As discussed last Friday in the streaming meeting using that page with muted audio shouldn't have an impact on the performance results. As such I will go ahead soon and also enable jobs on Android.
Assignee | ||
Comment 2•5 years ago
|
||
Chris, on which Android devices it would be good to get those tests running in CI? Do we have some which have HW decoding for H265 and VP9?
Assignee | ||
Comment 3•5 years ago
|
||
I assume running these tests on emulated devices doesn't make that much sense. So I wonder which hardware mobile devices we have in our CI which could be used to run the playback performance tests on. As best with hardware video decoding for the above codecs. It will also affect the question I raised for Chris.
Florin and/or Dave, could you give some feedback?
Assignee | ||
Comment 4•5 years ago
|
||
At least when checking Treeherder I can see Moto G5, and Pixel 2:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=raptor%2Candroid
Assignee | ||
Comment 5•5 years ago
|
||
As confirmed with Florin on IRC those are the two supported real devices. There are no FireTV stick 4k yet in CI.
Comment 6•5 years ago
|
||
The Pixel2 is probably a good candidate. The Moto G5 doesn't have hardware VP9, so won't do well in the tests. It would be good to have a reasonable lower spec device as well as the Pixel to run the tests over.
Assignee | ||
Comment 7•5 years ago
|
||
Perfect. So I will enable CI builds for both of them, but only on mozilla-central for now. Once we know more about the noise level we can then decide to also let them run on integration branches with alerting turned on.
Assignee | ||
Comment 8•5 years ago
|
||
Assignee | ||
Comment 9•5 years ago
|
||
Here you can find the perfherder data for the G5 and Pixel 2:
https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=try,2049061,1,10&series=try,2049350,1,10
As you can see there is quite a difference between both devices. G5 shows a lot of noise as already expected.
Assignee | ||
Comment 10•5 years ago
|
||
Adding those tests with the premise to run all the videos as muted,
at least as long as bug 1547717 hasn't been fixed.
Assignee | ||
Comment 11•5 years ago
|
||
Depends on D32040
Comment 12•5 years ago
|
||
Comment 13•5 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/c04159a7f012
https://hg.mozilla.org/mozilla-central/rev/53ead6f38bd7
Assignee | ||
Comment 14•5 years ago
|
||
Chris, do you have an idea why the tests on Moto G5 are faster than the ones on the Google Pixel 2? Given the missing hardware decoders I would expect a longer duration. Also the G5 jobs are even faster than desktop!
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=ytp%2Candroid
Comment 15•5 years ago
|
||
(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #14)
Chris, do you have an idea why the tests on Moto G5 are faster than the ones on the Google Pixel 2? Given the missing hardware decoders I would expect a longer duration. Also the G5 jobs are even faster than desktop!
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=ytp%2Candroid
I'm seeing the Pixel2 scoring about 20 minutes, and the G5 scoring about 55 minutes, so that's the opposite of what you reported here? Am I misreading the results? I assume taking more minutes to run means it's slower?
Assignee | ||
Comment 16•5 years ago
|
||
I wonder where you see the 55 minutes for G5? When I check on Treeherder all the jobs are still below of 25 minutes and as such faster than the Pixel 2 which is between 25 and 30 minutes.
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=ytp%2Candroid
Comment 17•5 years ago
|
||
(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #16)
I wonder where you see the 55 minutes for G5? When I check on Treeherder all the jobs are still below of 25 minutes and as such faster than the Pixel 2 which is between 25 and 30 minutes.
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=ytp%2Candroid
Pixel 2:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=ytp%2Candroid&selectedJob=250252517
Says: raptor-youtube-playback-geckoview-live pgo: 21.54
Moto G5:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=ytp%2Candroid&selectedJob=250252438
raptor-youtube-playback-geckoview-live pgo: 59.15
Click on the performance tab? I could be looking in the wrong spot?
Assignee | ||
Comment 18•5 years ago
|
||
Oh, now I see the confusion! My question was not about the mean value of dropped frames for that test suite, but the run time. This value can be seen at the bottom left panel.
It's basically not important but I would expect that a phone with a better hardware would also finish the job faster.
Comment 19•5 years ago
|
||
Thanks for clearing that up.
If you open a log for a MotoG5 run, and search for "u'decodedFrames': 0", you'll see a lot of hits, all on the 1440p and 2160p. You don't get hits on the Pixel2 log. So what I think's happening is that on the MotoG5 the in-built decoder is either refusing to decode, or we're expressing that we can't decode, resolutions greater than 1080p and the test is detecting that and not running these tests.
So I think the Moto G5 runs are faster because they're not running the 1440p and 2160p cases.
Assignee | ||
Comment 20•5 years ago
|
||
Interesting. Is that something which should be highlighted in the perf results? Right now this is a case where we don't actually see valid results. Imagine if we don't play any video at all, it would have a mean of 0 dropped frames, which gives totally a wrong assumption of the healthiness of the test.
At least for the health dashboard we report 100% failure rate if 0 decoded frames are detected. But not sure what to do with the plain decoded frames value.
Comment 21•5 years ago
|
||
The tests where we can't decode on that device, we'll likely never be able to decode on that device, as we're limited by the hardware. So there's not much point in tracking those tests on that device. I would suggest having some kind of "expected fail" for the > 1080p cases on the MotoG5.
Assignee | ||
Comment 22•5 years ago
|
||
(In reply to Chris Pearce [:cpearce (GMT+13)] from comment #21)
The tests where we can't decode on that device, we'll likely never be able to decode on that device, as we're limited by the hardware. So there's not much point in tracking those tests on that device. I would suggest having some kind of "expected fail" for the > 1080p cases on the MotoG5.
Thanks! Alexandru, could you or Arnold please have a look at this? A new bug blocking bug 1552738 would be good.
Comment 23•5 years ago
|
||
(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #22)
(In reply to Chris Pearce [:cpearce (GMT+13)] from comment #21)
The tests where we can't decode on that device, we'll likely never be able to decode on that device, as we're limited by the hardware. So there's not much point in tracking those tests on that device. I would suggest having some kind of "expected fail" for the > 1080p cases on the MotoG5.
Thanks! Alexandru, could you or Arnold please have a look at this? A new bug blocking bug 1552738 would be good.
I'm on it. I created the bug https://bugzilla.mozilla.org/show_bug.cgi?id=1552484
Updated•5 years ago
|
Description
•