I was unable to test the new tests enabled in bug 1352333 for Android 5.1/sdk 22/Nexus 6 until we landed the patch and we were able to run tests on try. Several tests in Cdm1, Mdm3 and Mdm4 appear to be perma failing. I have initiated a try run to determine whether we can achieve green by disabling specific tests or if we should exclude these tests totally from the Nexus 6 devices.

From the try run at https://treeherder.mozilla.org/#/jobs?repo=try&revision=88022e5d1b3a3e29ea1896b0f175b09b2c869a28&exclusion_profile=false&filter-tier=1&filter-tier=2&filter-tier=3&selectedJob=93644366 the Android 5.1/Nexus 6 failures are: bug 1329532 bug 1066090 debug |TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_BufferedSeek.html | Test timed out. opt |TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_BufferedSeek.html | Test timed out. bug 1358640 debug |TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_EndedEvent.html | Test timed out. opt |TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_EndedEvent.html | Test timed out. bug 1341519 debug |TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_FrameSelection.html | Test timed out. opt |TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_FrameSelection.html | Test timed out. bug 1342247 debug |TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_HaveMetadataUnbufferedSeek.html | Test timed out. opt |TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_HaveMetadataUnbufferedSeek.html | Test timed out. bug 1341146 debug |TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_MediaSource.html | Test timed out. opt |TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_MediaSource.html | Test timed out. bug 1225758 debug |TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_MediaSource_memory_reporting.html | Test timed out. opt |TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_MediaSource_memory_reporting.html | Test timed out. bug 1359010 debug |TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_OnEvents.html | Test timed out. opt |TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_OnEvents.html | Test timed out. bug 1293896 bug 1342683 debug |TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_SplitAppendDelay.html | Test timed out. opt |TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_SplitAppendDelay.html | Test timed out. bug 1211999 debug |TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_SplitAppend.html | Test timed out. opt |TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_SplitAppend.html | Test timed out. bug 1359012 debug |TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_TruncatedDuration.html | Test timed out. opt |TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_TruncatedDuration.html | Test timed out. bug 1358718 debug | REFTEST TEST-UNEXPECTED-FAIL | dom/media/test/crashtests/795892-1.html | load failed: timed out waiting for reftest-wait to be removed opt | REFTEST TEST-UNEXPECTED-FAIL | dom/media/test/crashtests/795892-1.html | load failed: timed out waiting for reftest-wait to be removed

This first try run is with a skip-if(Android) in the Cdm1 test. https://treeherder.mozilla.org/#/jobs?repo=try&revision=5c9c603dcbebdf278ca1dc6955e4301ffcfab0a9 gbrown pointed out how to do this for a single sdk version using skip-if(AndroidVersion=='22'). This second try run is for Cmd1 only with AndroidVersion. It is properly skipped for the Nexus 6 and run for the others. Win! \o/ https://treeherder.mozilla.org/#/jobs?repo=try&revision=a78ac19c1574feb6481587985120607df7ccb015

Comment on attachment 8860990 [details] [diff] [review] bug-1358876.patch Review of attachment 8860990 [details] [diff] [review]: ----------------------------------------------------------------- thanks for doing this!

Pushed by bclary@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/df17c800d8f2 Autophone - disable perma orange tests on Android 5.1, r=jmaher.

AndroidVersion failed on Linux. I probably should have done skip-if(Android&&AndroidVersion=='22'). Aryx is going to back me out and I'll do another try run before request review again.

Backed out for crash test bustage: https://hg.mozilla.org/integration/mozilla-inbound/rev/8d2645bc72849e499daa26972a99b4307a395d95 Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=df17c800d8f2c36d218f92e9fd2f65a9013f6c43&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=93782942&repo=mozilla-inbound >LOG ERROR | EXCEPTION: ReferenceError: AndroidVersion is not defined [

bad platforms in the last try. once more with feeling... https://treeherder.mozilla.org/#/jobs?repo=try&revision=8a63ee84b8618afc8ed40685c4b0372e7d0a5605

The linux crash tests are green. OSX are still pending, but this looks better.

Comment on attachment 8861113 [details] [diff] [review] bug-1358876-v2.patch Review of attachment 8861113 [details] [diff] [review]: ----------------------------------------------------------------- sorry I missed that the first time around!

Pushed by bclary@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/3301d5530ddd Autophone - disable perma orange tests on Android 5.1, r=jmaher.

I have more work to do to disable all of the offending Android 5.1 failures. Will have a patch ready for review by Monday.

Comment on attachment 8863783 [details] [diff] [review] bug-1358876-followup.patch Review of attachment 8863783 [details] [diff] [review]: ----------------------------------------------------------------- looks good!

Pushed by bclary@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/035af9053c26 Autophone - disable additional perma orange tests on Android 5.1, r=jmaher