We are seeing add-on errors when attempting to load the raptor webext with the geckoview example app. Even a slimmed down raptor webext shows the same issues.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e47b8ad301b6c311e2c7a18e59fc9b107fcf8fd4&filter-tier=1&filter-tier=2&filter-tier=3 fails using mozinstall to install geckoview_example.apk

Comment on attachment 9006595 [details] [diff] [review] motog5-raptor-test-platforms.patch Review of attachment 9006595 [details] [diff] [review]: ----------------------------------------------------------------- LGTM

Comment on attachment 9006596 [details] [diff] [review] motog5-raptor-test-sets.patch Review of attachment 9006596 [details] [diff] [review]: ----------------------------------------------------------------- LGTM

Comment on attachment 9006597 [details] [diff] [review] motog5-raptor-test-platform-assignments.patch Review of attachment 9006597 [details] [diff] [review]: ----------------------------------------------------------------- Also seeing the same error on try (mozinstall failing to install the geckoview_example app). ::: testing/mozharness/configs/raptor/android_hw_config.py @@ +1,2 @@ > +import os > +#import platform nit: remove commented out line @@ +1,4 @@ > +import os > +#import platform > + > +#VENV_PATH = '%s/build/venv' % os.getcwd() nit: remove commented out line ::: testing/mozharness/mozharness/mozilla/testing/raptor.py @@ +133,5 @@ > self.test = None > self.gecko_profile = self.config.get('gecko_profile') > self.gecko_profile_interval = self.config.get('gecko_profile_interval') > + self.test_packages_url = self.config.get('test_packages_url') > + self.test_suite = self.config.get('test_suite') Where is self.test_suite used (somewhere in the land of bitbar?)

No where that I know. I added it to the last patch when I added the test_packages_url to fix the problem there. We can remove it. These patches+reviews were mostly to get the patches to you and gbrown so you could look at them and figure out the problems while I'm unavailable.

Ok great thanks :bc!

I'm running into some trouble around argument passing. In particular, I see: https://dxr.mozilla.org/mozilla-central/rev/c2e3be6a1dd352b969a45f0b85e87674e24ad284/testing/mozharness/mozharness/mozilla/testing/raptor.py#396 def run_tests(self, args=None, **kw): but run_tests(self) is just a mozharness script "step" function, which, afaik, doesn't take parameters. Regardless of options passed to the script, I find kw = {}. On the other hand, I see a similar run_tests signature in Talos. :rwood - Any insight?

In other news, I don't see any code to install geckoview after the apk is downloaded. I'm going to re-use the apk installation code from android_hardware_unittest.py, first refactoring it so that code can be shared between the various mozharness scripts for android.

This almost worked: https://treeherder.mozilla.org/#/jobs?repo=try&revision=72fa647736c48253baa668108411c48f90219801

1. It would be nice to dump the logcat to see what the device is doing. 2. The control server appears to be working. How can we tell the mitmproxy is working? We can ask bitbar if there are any networking issues which might be a problem. 3. Perhaps this is motog5 related. I could move some p2 devices into the p2 perf device group and we could enable speedometer on them as well.

we shouldn't have mitmproxy for speedometer, but we do have 2 different ports we serve data from (control server, data server). I agree that the p2 devices would be worth trying

updated for rwood's comments.

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #16) > we shouldn't have mitmproxy for speedometer, but we do have 2 different > ports we serve data from (control server, data server). I agree that the p2 > devices would be worth trying Yeah mitmproxy won't start unless you try to run tp6.

(In reply to Geoff Brown [:gbrown] from comment #13) > In other news, I don't see any code to install geckoview after the apk is > downloaded. I'm going to re-use the apk installation code from > android_hardware_unittest.py, first refactoring it so that code can be > shared between the various mozharness scripts for android. I'm assuming that uses MozDevice ADBAndroid.install_app? (I'll check)

(In reply to Geoff Brown [:gbrown] from comment #12) > I'm running into some trouble around argument passing. In particular, I see: > > https://dxr.mozilla.org/mozilla-central/rev/ > c2e3be6a1dd352b969a45f0b85e87674e24ad284/testing/mozharness/mozharness/ > mozilla/testing/raptor.py#396 > > def run_tests(self, args=None, **kw): > > but run_tests(self) is just a mozharness script "step" function, which, > afaik, doesn't take parameters. > > Regardless of options passed to the script, I find kw = {}. > > On the other hand, I see a similar run_tests signature in Talos. > > :rwood - Any insight? I'm a bit confused, how are you invoking Raptor? You shoudn't call run_tests directly...

(In reply to Robert Wood [:rwood] from comment #19) > (In reply to Geoff Brown [:gbrown] from comment #13) > > In other news, I don't see any code to install geckoview after the apk is > > downloaded. I'm going to re-use the apk installation code from > > android_hardware_unittest.py, first refactoring it so that code can be > > shared between the various mozharness scripts for android. > > I'm assuming that uses MozDevice ADBAndroid.install_app? (I'll check) Oh I see there's a mozharness android mixin cool

Not looking for a review really. These patches are to enable the p2 testing. We can deal with reviews when we are ready to go live.

oops, mixed a couple of patches up. I'll fix up but applying them all will fix the issue.

(In reply to Robert Wood [:rwood] from comment #20) > > I'm a bit confused, how are you invoking Raptor? You shoudn't call run_tests > directly... NVM I see you got it to work with: 01:08:44 INFO - Calling ['/builds/worker/workspace/build/venv/bin/python', u'/builds/worker/workspace/build/tests/raptor/raptor/raptor.py', u'--test', 'raptor-speedometer', u'--symbolsPath', 'https://queue.taskcluster.net/v1/task/A4EzNSaGSDOpjHKUrRMfHA/artifacts/public/build/target.crashreporter-symbols.zip', u'--app', 'geckoview', u'--branchName', 'try', u'--binary', 'org.mozilla.geckoview_example', u'--log-tbpl-level=debug'] with output_timeout 3600 Is there any way that we can run this and watch the device live to see what is going on? On your run in comment 14 looks like it is all starting but then it times out. Either it is really really slow and it is timing out before the test is done, or the test isn't starting for some reason... the adb reverse is there so it should be finding the served-out speedometer source.

try run without the fixes for installing. https://treeherder.mozilla.org/#/jobs?repo=try&revision=dec04f0407e405f000f9b54a9d0c9a8d8c934d94&filter-tier=1&filter-tier=2&filter-tier=3 p2 aarch64 does not work with --artifact. do this to exclude it ./mach try fuzzy --full --query "'test-android-hw arm7 raptor-speedometer-geckoview" --artifact live log doesn't work unfortunately due to issues with taskcluster-worker. If you dump the logcat, you'll be able to see more about what the device is doing.

With logcat artifact: https://treeherder.mozilla.org/#/jobs?repo=try&revision=f50647b61baffead6f91d093b317c3148c35782a

Thanks :gbrown for adding the logcat artifact. Retriggered some. Speedometer is running on the device but is intermittently failing with: 09-06 07:37:00.601 14720 14735 E GeckoConsole: [JavaScript Error: "Error: Load timeout for modules: app/page/app,jquery 09-06 07:37:00.601 14720 14735 E GeckoConsole: http://requirejs.org/docs/errors.html#timeout" {file: "http://localhost:49235/Speedometer/resources/todomvc/dependency-examples/flight/flight/node_modules/requirejs/require.js" line: 168}] 09-06 07:37:00.601 14720 14735 E GeckoConsole: makeError@http://localhost:49235/Speedometer/resources/todomvc/dependency-examples/flight/flight/node_modules/requirejs/require.js:168:17 09-06 07:37:00.601 14720 14735 E GeckoConsole: checkLoaded@http://localhost:49235/Speedometer/resources/todomvc/dependency-examples/flight/flight/node_modules/requirejs/require.js:698:23 09-06 07:37:00.601 14720 14735 E GeckoConsole: checkLoaded/checkLoadedTimeoutId<@http://localhost:49235/Speedometer/resources/todomvc/dependency-examples/flight/flight/node_modules/requirejs/require.js:720:25 On the run above in comment 30, you'll see the green run doesn't have that error. Nice to see it does work, but for some reason there's this intermittent.

Comment on attachment 9006902 [details] [diff] [review] p2-raptor-test-platform-assignments.patch Review of attachment 9006902 [details] [diff] [review]: ----------------------------------------------------------------- Great thanks

Just adding :gbrown's latest try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=305337495d7d7a3ca65f9b04d161825eb88008ca On one of the failures I noticed the weather app was doing an update (not sure if that caused speedometer to stop/timeout). The bitbar folks have since 'disabled Time and Weather system app from all Moto G5s' so perhaps that will help.

This works for me, with :bc's patches. https://treeherder.mozilla.org/#/jobs?repo=try&revision=f6a0f768436a746ba37d6506cf12d992664b1368

I did a bunch of tests using Geoff's patches <https://treeherder.mozilla.org/#/jobs?repo=try&revision=866fd6509fbd8f19af719e597ed7106ffaed3be2&filter-tier=1&filter-tier=2&filter-tier=3> Interesting fact 1: p2 x86_64 is slower than x86. Interesting fact 2: p2s are pretty reliable with no oranges out of 10 and seem to have pretty consistent values. Interesting fact 3: motog5 failures are intermittent but too common. If we can't figure it out we could alleviate the problem by reducing the timeout and keeping them as tier 3. This seems to be the issue 09-06 20:29:35.977 5012 5027 E GeckoConsole: [JavaScript Error: "Error: Load timeout for modules: app/page/app,jquery 09-06 20:29:35.977 5012 5027 E GeckoConsole: http://requirejs.org/docs/errors.html#timeout" {file: "http://localhost:38347/Speedometer/resources/todomvc/dependency-examples/flight/flight/node_modules/requirejs/require.js" line: 168}] 09-06 20:29:35.977 5012 5027 E GeckoConsole: makeError@http://localhost:38347/Speedometer/resources/todomvc/dependency-examples/flight/flight/node_modules/requirejs/require.js:168:17 09-06 20:29:35.977 5012 5027 E GeckoConsole: checkLoaded@http://localhost:38347/Speedometer/resources/todomvc/dependency-examples/flight/flight/node_modules/requirejs/require.js:698:23 09-06 20:29:35.977 5012 5027 E GeckoConsole: checkLoaded/checkLoadedTimeoutId<@http://localhost:38347/Speedometer/resources/todomvc/dependency-examples/flight/flight/node_modules/requirejs/require.js:720:25 Is that time out configurable?

interesting observation and awesome try push :) looking at this, I first thought we were hitting http://requirejs.org, but that is just an error message- we are really localhost. Digging a bit deeper, I think we are timing out to load the jquery which is 320K in size- is it possible we are running out of memory on the g5? I looked at a logcat artifact uploaded (which is really useful btw): https://queue.taskcluster.net/v1/task/J7Hid377SKmdfTdLBUBRbQ/runs/0/artifacts/public/test_info/logcat-ZY322M554N.log There are a handful of Javascript Errors, most don't seem to be related. In fact we have a 20 minute timeout inside the browser, but our normal runtime is <10 minutes- I am not sure if extending the timeout will help or not. Another thing I noticed is the screenshot isn't getting uploaded, small details, but that might have a clue. I see a few messages in the logcat indicating that gecko is in the background, worth looking into to see if we can disable more services. We could get speedometer on P2 running 'now' and sort out the G5 when we have more time (i.e. after getting Unity-webgl and battery tests up)

I was able to reproduce the same intermittent locally on a Moto G5 that I borrowed - once via raptor and twice via running the speedometer (1.0) benchmark via wifi directly on the speedometer benchmark site in the geckoview_example app. There was lots of memory still available on the device even when the intermittent did happen. After that I wanted to get extra info (i.e. adb shell dumpsys meminfo) and I couldn't repro the failure again. Tried for a couple of hours and couldn't repro it again - it now it seems to work great hah. I think we should go ahead and land the patches and then revisit this again later and see how it runs on the G5. I am hesitant to file a geckoview bug until we can reproduce it consistently.

Ok, so are we ready to land? Do you guys want to land your patches separately but at the same time?

Let's wait until we have everyone together on irc this afternoon and we can decide/coordinate.

The patches as is need to land together as a group since gbrown's depends on the existence of testing/mozharness/configs/raptor/android_hw_config.py and we will get installation failures without gbrown's patch. Or we could refactor a couple and move the android_hw_config.py creation to gbrown's patch and land it first, followed by the others. Or we could just live with installation failures for one push if we landed gbrown's separately. I would vote for gbrown landing everything but am open to other approaches.

Comment on attachment 9006890 [details] [diff] [review] motog5-raptor-test-platform-assignments.patch gbrown already reviewed but the check in comment references both him and rwood.

Comment on attachment 9007097 [details] [diff] [review] support apk install, logcat, etc Most of the android.py code is copied from android_hardware_unittest.py, or android_emulator_unittest.py. I plan to remove duplicate code from those scripts and have all the android mozharness scripts use android.py -- one day, in another bug.

Comment on attachment 9007097 [details] [diff] [review] support apk install, logcat, etc Review of attachment 9007097 [details] [diff] [review]: ----------------------------------------------------------------- r+. Looks ok to me. Couple of style issues that aren't important enough to call out. I would really love for use to be able to use mozdevice, mozscreenshot and other stuff from mozbase in mozharness. Maybe we can think about that for the future. ::: testing/mozharness/mozharness/mozilla/testing/android.py @@ +188,5 @@ > + (self.installer_path, self.device_name), > + EXIT_STATUS_DICT[TBPL_RETRY]) > + return install_ok > + > + def screenshot(self, prefix): This only works on the emulator. We added the capability in bug 1482878 to also do hardware using dump_device_screen. Is there anyway for us to get mozscreenshot into mozharness? Maybe a followup? ::: testing/mozharness/mozharness/mozilla/testing/raptor.py @@ +487,5 @@ > env=env) > + > + if self.app == "geckoview": > + self.logcat_stop() > + self.screenshot("raptor-screenshot-end") See comment in the AndroidMixin

(In reply to Bob Clary [:bc:] from comment #43) > This only works on the emulator. We added the capability in bug 1482878 to > also do hardware using dump_device_screen. Thanks, I was wondering about that. I'll remove the screenshot calls from raptor, and we can follow up on device screenshot for raptor later. > Is there anyway for us to get mozscreenshot into mozharness? Maybe a > followup? That is possible, but awkward (standard way is to duplicate the mozbase project in mozharness). Anyway, yes, let's consider that in a followup.

Once we get these last few reviews done, I'll check one more time on try, then land all the patches...hopefully soon!

Comment on attachment 9006902 [details] [diff] [review] p2-raptor-test-platform-assignments.patch Let's put the Moto G5 as tier 3 due to the intermittent, and the Google Pixel 2 as tier 2 since it seems to be stable. Thanks!

(In reply to Robert Wood [:rwood] from comment #46) > Comment on attachment 9006902 [details] [diff] [review] > p2-raptor-test-platform-assignments.patch > > Let's put the Moto G5 as tier 3 due to the intermittent, and the Google > Pixel 2 as tier 2 since it seems to be stable. Thanks! Oh I see that the raptor.yml is also updated in the patch in comment 41 (and has the G5 tier 3, default tier 2 which is what we want).

gbrown: want to go ahead and handle the tier change that rwood requested for p2-raptor-test-platform-assignments.patch before you push it ?

(In reply to Bob Clary [:bc:] from comment #49) Yes - will do.

Comment on attachment 9007097 [details] [diff] [review] support apk install, logcat, etc Review of attachment 9007097 [details] [diff] [review]: ----------------------------------------------------------------- Awesome thanks! ::: testing/mozharness/mozharness/mozilla/testing/android.py @@ +99,5 @@ > + install_ok = False > + if int(self.sdk_level) >= 23: > + cmd = ['install', '-r', '-g', self.installer_path] > + else: > + cmd = ['install', '-r', self.installer_path] nit: maybe log a message here indicating that we are unable to set permissions on the android app at install time. Just in case down the road we have failures caused by permissions not set. @@ +139,5 @@ > + ) > + if not os.path.exists(manifest_path): > + self.fatal("Could not retrieve manifest needed to retrieve " > + "artifacts from %s" % manifest_path) > + self.tooltool_fetch(manifest_path, nit: suggest add a comment here indicating that tooltool fetch is found in the TestingMixin (since there is no import/inheritance for it).

Updated to not change tier again.

Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/90884f7b97d8 add test platform for android-hw-g5-7-0-arm7-api-16/opt - android-hw-arm7-raptor, r=gbrown,rwood https://hg.mozilla.org/integration/mozilla-inbound/rev/50259b3caa6b add test set for android-hw-arm7-raptor - raptor-speedometer-geckoview, r=gbrown,rwood. https://hg.mozilla.org/integration/mozilla-inbound/rev/d4be0ae6ae4e add raptor-speedometer-geckoview for android-hw-g5 test platforms, r=gbrown,rwood. https://hg.mozilla.org/integration/mozilla-inbound/rev/56d714aedeb0 add raptor-speedometer-geckoview for android-hw-p2 test platforms, r=gbrown,rwood. https://hg.mozilla.org/integration/mozilla-inbound/rev/638d02ffcc6f add test platform for android-hw-p2-8-0-*/opt - android-hw-*-raptor, r=gbrown,rwood https://hg.mozilla.org/integration/mozilla-inbound/rev/1d75f4dbb1e8 add test set for android-hw-aarch64-raptor - raptor-speedometer-geckoview, r=gbrown,rwood. https://hg.mozilla.org/integration/mozilla-inbound/rev/251d13633524 Add geckoview installation and android diagnostics to raptor; r=bc,rwood

https://hg.mozilla.org/mozilla-central/rev/90884f7b97d8 https://hg.mozilla.org/mozilla-central/rev/50259b3caa6b https://hg.mozilla.org/mozilla-central/rev/d4be0ae6ae4e https://hg.mozilla.org/mozilla-central/rev/56d714aedeb0 https://hg.mozilla.org/mozilla-central/rev/638d02ffcc6f https://hg.mozilla.org/mozilla-central/rev/1d75f4dbb1e8 https://hg.mozilla.org/mozilla-central/rev/251d13633524