Geoff Brown [:gbrown] Assignee
	Description • 9 years ago

We want to run most of our test suites on an emulator running Android 4.3, as developed in bug 1062365. Of course some tests fail in that environment; this bug will update manifests to mark tests as failing or skip them on Android 4.3.

	Ryan VanderMeulen [:RyanVM]
	Comment 6 • 9 years ago

"TODO: NEW BUG"

I <3 you so much.

	Dan Minor [:dminor]
	Comment 9 • 9 years ago

Comment on attachment 8579050 [details] [diff] [review]
xpcshell manifest updates for Android 4.3

Review of attachment 8579050 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm!

	Geoff Brown [:gbrown] Assignee
	Comment 10 • 9 years ago

mochitest-gl: https://hg.mozilla.org/integration/mozilla-inbound/rev/71357797bf19
xpcshell: https://hg.mozilla.org/integration/mozilla-inbound/rev/51146c3e7939

	Ryan VanderMeulen [:RyanVM]
	Comment 11 • 9 years ago

mochitest-gl patch backed out for mochitest-1 and mochitest-gl bustage.
https://hg.mozilla.org/integration/mozilla-inbound/rev/07a6b1f3448a

https://treeherder.mozilla.org/logviewer.html#?job_id=7809348&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=7808601&repo=mozilla-inbound

	Geoff Brown [:gbrown] Assignee
	Comment 12 • 9 years ago

mochitest-gl changes for 2.3 will apply to 4.3 too. Synced up the configs again:

https://hg.mozilla.org/build/mozharness/rev/d99684a19ced

	Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)
	Comment 13 • 9 years ago

https://hg.mozilla.org/mozilla-central/rev/51146c3e7939

	Geoff Brown [:gbrown] Assignee
	Comment 14 • 9 years ago

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #11)
> mochitest-gl patch backed out for mochitest-1 and mochitest-gl bustage.

There is a syntax error in that patch:

03-20 16:46:04.789 W/GeckoConsole( 2258): [JavaScript Error: "ReferenceError: invalid assignment left-hand side" {file: "http://mochi.test:8888/tests/dom/canvas/test/webgl-conformance/mochi-single.html?conformance/attribs/gl-enable-vertex-attrib.html" line: 124 column: 16 source: "        failSet['conformance/extensions/oes-texture-float.html' = true;
03-20 16:46:04.789 W/GeckoConsole( 2258): "}]

(missing trailing ']').

	James Willcox (:snorp) (jwillcox@mozilla.com) (he/him)
	Comment 16 • 9 years ago

Comment on attachment 8581714 [details] [diff] [review]
robocop.ini updates for android 4.3

Review of attachment 8581714 [details] [diff] [review]:
-----------------------------------------------------------------

Shouldn't it really be "skip-if android_emulator"?

	Geoff Brown [:gbrown] Assignee
	Comment 17 • 9 years ago

Certainly the majority of robocop annotations seem to apply to both 2.3 and 4.3. It's interesting (to me anyway!) that many of the tests fail differently on 2.3 than on 4.3 -- different failures, different bugs, different causes? Or just poor tests, susceptible to timing variations?

We do not currently have a way for the harness to detect if the tests are running on emulator. I suppose we could introduce an alias mechanism so "android_emulator" expands to "android_version == 11 || android_version == 18" but I'm not sure that would be a significant improvement.

	Geoff Brown [:gbrown] Assignee
	Comment 18 • 9 years ago

robocop - https://hg.mozilla.org/integration/mozilla-inbound/rev/cef88d7252a1

	Jordan Lund (:jlund)
	Comment 19 • 9 years ago

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/production

	Geoff Brown [:gbrown] Assignee
	Comment 21 • 9 years ago

reftests - https://hg.mozilla.org/integration/mozilla-inbound/rev/e4214ff58dab

	Ryan VanderMeulen [:RyanVM]
	Comment 22 • 9 years ago

https://hg.mozilla.org/mozilla-central/rev/cef88d7252a1
https://hg.mozilla.org/mozilla-central/rev/e4214ff58dab

	Geoff Brown [:gbrown] Assignee
	Comment 23 • 9 years ago

mochitest-gl re-land with missing ] added (see Comment 14):
 
https://treeherder.mozilla.org/#/jobs?repo=try&revision=3141adb89ad0
https://hg.mozilla.org/integration/mozilla-inbound/rev/a6619aed2c9d

	Ryan VanderMeulen [:RyanVM]
	Comment 24 • 9 years ago

https://hg.mozilla.org/mozilla-central/rev/a6619aed2c9d

	Geoff Brown [:gbrown] Assignee
	Comment 27 • 9 years ago

Promising!

https://treeherder.mozilla.org/#/jobs?repo=try&revision=92c69392f2a2

	Ryan VanderMeulen [:RyanVM]
	Comment 28 • 9 years ago

Should there be 4.3 jobs on that run?

	Geoff Brown [:gbrown] Assignee
	Comment 29 • 9 years ago

mochitest - https://hg.mozilla.org/integration/mozilla-inbound/rev/6d9dd4432988

	Geoff Brown [:gbrown] Assignee
	Comment 30 • 9 years ago

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #28)
> Should there be 4.3 jobs on that run?

Ha! There are! The "Android 2.3 API9 opt" tests in that try run are actually using the build from "Android 4.0 API11+ opt" and running with the Android 4.3 mozharness configuration, which runs Android 4.3 in the emulator. How? https://hg.mozilla.org/try/rev/92c69392f2a2 points mozharness to my user repo, which has hacks to replace the apk and tests.zip urls and modify the configuration file name.

As usual, the logs tell the story:

http://ftp.mozilla.org/pub/mozilla.org/mobile/try-builds/gbrown@mozilla.com-92c69392f2a2/try-android-api-9/try_ubuntu64_vm_mobile_test-xpcshell-1-bm121-tests1-linux64-build91.txt.gz

16:59:54     INFO - Run as scripts/scripts/android_emulator_unittest.py --cfg android/androidarm_4_3.py --test-suite xpcshell-1 --blob-upload-branch try --download-symbols ondemand
...
17:00:00     INFO -  INFO - File AVDs-armv7a-android-4.3.1_r1-build-2015-03-16.tar.gz fetched from http://tooltool.pvt.build.mozilla.org.proxxy1.srv.releng.use1.mozilla.com/build/ as /home/cltbld/.android/tmp7iIFYw
...
17:01:23     INFO - Downloading http://ftp.mozilla.org.proxxy1.srv.releng.use1.mozilla.com/pub/mozilla.org/firefox/try-builds/gbrown@mozilla.com-92c69392f2a2/try-android-api-11/fennec-39.0a1.en-US.android-arm.tests.zip to /builds/slave/test/build/fennec-39.0a1.en-US.android-arm.tests.zip
...
17:02:07     INFO - Downloading http://ftp.mozilla.org.proxxy1.srv.releng.use1.mozilla.com/pub/mozilla.org/firefox/try-builds/gbrown@mozilla.com-92c69392f2a2/try-android-api-11/fennec-39.0a1.en-US.android-arm.apk to /builds/slave/test/build/fennec-39.0a1.en-US.android-arm.apk

	Ryan VanderMeulen [:RyanVM]
	Comment 31 • 9 years ago

https://hg.mozilla.org/mozilla-central/rev/6d9dd4432988