https://docs.google.com/spreadsheets/d/11LE-ITakYnH9XIiKzSgFntOazXPxDrx9jtZ-tk71Kls/edit?usp=sharing setSinkId seems to not be defined, despite the wpt metadata setting media.setsinkid.enabled to true.

This seems similar to the failures I see in bug 1499866, where things that should be enabled aren't getting enabled, maybe it's something in the harness? The log output includes >> 1:04.72 INFO Setting pref media.setsinkid.enabled (true) << but maybe that isn't actually making it into the profile on Fennec?

If it matters, these are being run on builds from fairly current mozilla-central. I assume these builds get the EARLY_BETA_OR_EARLIER treatment, but I haven't looked to confirm that. Don't know if that could make a difference for some of these features...

(In reply to Wes Kocher (:KWierso) from comment #1) > This seems similar to the failures I see in bug 1499866, where things that > should be enabled aren't getting enabled, maybe it's something in the > harness? > > The log output includes >> 1:04.72 INFO Setting pref media.setsinkid.enabled > (true) << but maybe that isn't actually making it into the profile on Fennec? Can you run 1 test and then when it is finished and the emulator is left open can you have a look at the about:config for that preferece to make sure it is set. Part of me wonders if we are setting the profile on the host but not actually making sure we do it for mobile...

The preferences are set using Chrome js executed via marionette [1], so in theory it ought to work on mobile. It's possible that it doesn't of course. You could also try reading back the pref after setting it and logging the value you get. The pref setting is happening at [2], so adding something like logging.info("%s has value " % (name, self.prefs.get(name))) After that should allow us to check consistency. [1] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/executors/executormarionette.py#209 [2] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/executors/executormarionette.py#209

(In reply to David Burns :automatedtester from comment #3) > Can you run 1 test and then when it is finished and the emulator is left > open can you have a look at the about:config for that preferece to make sure > it is set. Part of me wonders if we are setting the profile on the host but > not actually making sure we do it for mobile... When I do that, after a test that does NOT explicitly set that pref, media.setsinkid.enabled is 'false', but when the test explicitly DOES set the pref, it is 'true'. So the preference IS getting set correctly. There goes that theory... Alex, is this expected to not work on Android, maybe?

The setSinkId method should be available but the enumerate devices does not work on android so it return 0 devices, so: These 2 failures are not expected. I am testing on my android device and the method is there. It must be a failure of the test harness. FAIL setSinkId on default audio output should always work - audio.setSinkId is not a function @http://web-platform.test:8000/audio-output/setSinkId.html:12:19 Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:1561:20 promise_test/tests.promise_tests</<@http://web-platform.test:8000/resources/testharness.js:591:31 promise_test/tests.promise_tests<@http://web-platform.test:8000/resources/testharness.js:590:20 FAIL setSinkId fails with NotFoundError on made up deviceid - audio.setSinkId is not a function @http://web-platform.test:8000/audio-output/setSinkId.html:14:55 Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:1561:20 promise_test/tests.promise_tests</<@http://web-platform.test:8000/resources/testharness.js:591:31 promise_test/tests.promise_tests<@http://web-platform.test:8000/resources/testharness.js:590:20 The following error is expected, so over all, we expect that the test will fail in android with that error. FAIL List device, setSinkId should on the default, the rest of the devices will get a NotAlowedError - assert_not_equals: media device list includes at least one audio output device got disallowed value 0 @http://web-platform.test:8000/audio-output/setSinkId.html:20:3 The same for the other test.

The feature landed one week ago in case you don't test the latest. Before that the _function does not exist_ error is expected. See also Bug 934425.

Okay, so running on a build from a recent (last few hours) pull from autoland, I get these errors: Unexpected Results ------------------ /audio-output/setSinkId.html FAIL List device, setSinkId should on the default, the rest of the devices will get a NotAlowedError - assert_not_equals: media device list includes at least one audio output device got disallowed value 0 @http://web-platform.test:8000/audio-output/setSinkId.html:20:3 /audio-output/setSinkId.https.html FAIL List device, setSinkId should on the default, the rest of the devices will get a NotAlowedError - assert_not_equals: media device list includes at least one audio output device got disallowed value 0 @https://web-platform.test:8443/audio-output/setSinkId.https.html:22:3

(In reply to Wes Kocher (:KWierso) from comment #9) > Created attachment 9019564 [details] > Bug 1499871 - Fix spelling of "NotAllowedError" r?achronop (This doesn't fix the failures, but the misspelling annoyed me enough to correct it.)

(In reply to Wes Kocher (:KWierso) from comment #8) > Okay, so running on a build from a recent (last few hours) pull from > autoland, I get these errors: > > Unexpected Results > ------------------ > /audio-output/setSinkId.html > FAIL List device, setSinkId should on the default, the rest of the devices > will get a NotAlowedError - assert_not_equals: media device list includes at > least one audio output device got disallowed value 0 > @http://web-platform.test:8000/audio-output/setSinkId.html:20:3 > /audio-output/setSinkId.https.html > FAIL List device, setSinkId should on the default, the rest of the devices > will get a NotAlowedError - assert_not_equals: media device list includes at > least one audio output device got disallowed value 0 > @https://web-platform.test:8443/audio-output/setSinkId.https.html:22:3 That looks good. It's the expected failure on android (for now).

Pushed by wkocher@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/121ccc3d4437 Fix spelling of "NotAllowedError" r=achronop

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/13711 for changes under testing/web-platform/tests

Upstream PR merged

Pushed by wkocher@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/ca744888101e Update test metadata for wpt audio-output on fennec r=achronop

These tests started passing again on try, and started doing the same for me locally after I force-updated the android avd. Patch incoming to update the expectation data.

At some point, these started passing again on CI. Force-updating the Android AVD makes these tests start passing locally as well. Let's just remove the android-specific failing expectations since they don't seem necessary any more.

It's Bug 1482150 that has just been landed. Those tests are not on try yet, right? I did not see a change there.

Correct, nothing runs without https://hg.mozilla.org/try/rev/29da107e44d72adf184b5cbcce9a53a157f3cc41 applied, and then only on try with you specifically asking for them to run. Sample failure: https://treeherder.mozilla.org/logviewer.html#?job_id=210768951&repo=try

Pushed by wkocher@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/91ad57be0886 Mark the wpt audio-output expectations as passing everywhere r=jgraham