SoundCloud published an HTML 5 <audio> interoperability test. We're failing 8 tests: 1. Seeking to unbuffered position with seamless playback 2. Supports MP3 format (bug 562730) 3. Consistent timeupdate interval (15ms - 250ms) 4. Property "played" (bug 462959) 5. Property "playbackRate" (bug 495040) 6. Property "defaultPlaybackRate" (bug 495040) 7. Property "currentTime" 8. Event "ratechange" (bug 495040) Of the ones we don't already have bugs for: 1. Test is broken (serves unseekable Ogg and expects to seek, also could fail intermittently depending on media download speed). 3. Expects delta time between timeupdate events to be >= 15ms and <= 250ms, in Firefox the test sees a delta time of 260-285ms. This is a real bug. 7. Test is broken (serves unseekable Ogg and expects to seek, can pass intermittently depending on media download speed).

Filed bug 700210 for #3. I've attempted to contact the owners about the broken tests.

When I say "unseekable Ogg", what I mean is that the server does not support range requests, so Firefox is only able to seek in buffered parts of the file. #1 is unreliable because it attempts to seek to audio.buffered.end(0) + 60, but depending on download speed it's possible for audio.buffered.end(0) to equal audio.duration, so the test not complete. #7 is unreliable because it can intermittently pass depending on media download speed, as a seek into a buffered area will succeed but a seek into an unbuffered area will fail due to the first issue mention in this comment.

Additionally, it's not clear to me what #1 intends to test, but seeking to an unbuffered area will not be seamless as there will be network and decoding delays involved in the seek. If it's expected that the old playback location continues to play until the seek completes, this isn't the case in Firefox nor in any implementation I'm aware of. The test doesn't (and can't) detect this behaviour, anyway. What the test appears to be checking is whether playback continues automatically after seeking into an unbuffered area--if this is the intended action the test should probably be renamed to reflect what is being tested.

Hi Matthew! I must say you caught us by surprise since we're today doing a final work marathon on this project to be released tomorrow. Anyway your input is very valuable and the Github repo (http://github.com/soundcloud/areweplayingyet) is open now. If you have any suggestions to improve a test just create a Pull Request. Going to fix some tests now! ;)

After running the full test on Firefox Beta 8 (beta channel), the count is still at 0 at the bottom of website.

Antoine as I wrote above, you shouldn't consider anything final until Wednesday. Thank you for your feedback.

The seeking issue is resolved, so #7 now passes. I think #1 should pass now, but I suspect the test is timing out before it has a chance to complete. The tests have a default timeout of 10s, which may be too low when running some of them over a slowish international link. Two new tests are failing: AAC support (we don't, no bug to reference, but the reasoning is roughly the same as for MP3), and 'stalled' (which should pass, not sure what's going on there).

Hi Matthew! Let's talk about the #1. Do you understand the use case? If implemented right, it allows fast playback of _any_ part of an audio file (without the need to buffer everything until that point) and then if no more user interaction happens it should continue playback smoothly. About AAC support... You should now more about that than me but didn't it go open source recently? The stalled event is not working on Firefox... The loadstart event is triggered, then we halt the transfer of data for 3500ms (the spec requires 3000 but we gave some space) where stalled should be triggered. Cheers, Tomás

Apple release their ALAC (Apple Lossless Audio Codec, equivalent to FLAC) under an open source license, but AFAIK, AAC is still encumbered with patents.

@Paul Do you think ALAC could be a good way of erradicating the problem of having different audio formats for different browsers?

Tomas, ALAC is unfortunately a lossless audio codec, and is intended for very high quality/high bitrate streams, so it might not be very useful for our purpose, which is streaming media.

@Paul, acknowledged. So what are your ideas on how this could be solved?

@Matthew: The first test was indeed timing out. Have look at it now! http://areweplayingyet.org/support-seeking-unbuffered-position

Tomás, the codec issue is hard. Apple and MS could support Ogg like Google, Opera, and Mozilla. In some years the patents on mp3 will expire and we can add that to Firefox. A shim that does audio decompression in js might work in the meantime. Fortunately, the <audio> element lets you specify multiple <source> elements!

Can anyone look at this test source and validate that is valid (passing or not)? Webkit people are saying that seeking should only happen after the canplay event was triggered but this makes Firefox fail.

It _made_ Firefox fail because canplay is triggered again after the seek so there would be an infinite loop. Passing now.

Requiring the canplay event before seeking corresponds to HAVE_FUTURE_DATA. It should be possible to seek as soon as you reach HAVE_METADATA (and receive the loadedmetadata event).

Matthew can you please link me to the spec part that defines that? Thanks

Here: http://www.whatwg.org/specs/web-apps/current-work/multipage/video.html#seeking Note that it's an error to seek while in readyState HAVE_NOTHING. All other states are valid, including HAVE_METADATA. Here: http://www.whatwg.org/specs/web-apps/current-work/multipage/video.html#the-ready-states Under HAVE_METADATA: "The API will no longer raise an exception when seeking." Also note in that section that canplay is fired when the readyState is HAVE_FUTURE_DATA. So this change is wrong, and should be reverted: https://github.com/soundcloud/areweplayingyet/commit/37d10a57d420807cc6838f8e94d56f5198b01798

It's probably better to have these discussions in the AreWePlayingYet issue tracker so they're more visible to other interested parties. This bug is intended to track Firefox bugs, and also track notes on the tests before the AWPY issue tracker was opened up. I'll file AWPY test bugs there from now on.

Loop test fails on Linux (with PulseAudio) due to bug 582513.

Just stumbles into AWPY today, so please bear with me. These still fail on Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:14.0) Gecko/20120314 Firefox/14.0a1 ID:20120314031139 http://areweplayingyet.org/support-seeking-unbuffered-position http://areweplayingyet.org/support-hot-swapping-src http://areweplayingyet.org/support-consistent-playback-delay http://areweplayingyet.org/property-seeking http://areweplayingyet.org/property-played http://areweplayingyet.org/property-playbackRate http://areweplayingyet.org/property-defaultPlaybackRate http://areweplayingyet.org/event-suspend http://areweplayingyet.org/event-stalled http://areweplayingyet.org/event-seeking http://areweplayingyet.org/event-ratechange http://areweplayingyet.org/event-pause http://areweplayingyet.org/event-loadedmetadata http://areweplayingyet.org/attribute-preload-metadata Changing "Platform" accordingly. Should new bugs be filled for the failing new tests?

The failing tests need to be reviewed to confirm they're valid tests, and if so, why they're failing. That can be done separately for each test if someone would like to file individual bugs that block this one.

Hey guys, We made minors update to AWPY: - Introduced check for Opus support http://areweplayingyet.org/support-format-opus :) - Split the test 'support-seeking-unbuffered-position' into 3 tests to take into account different codecs and removed the former one. (you can read more about the motivations: https://github.com/soundcloud/areweplayingyet/pull/41) * http://areweplayingyet.org/support-seeking-unbuffered-position-ogg * http://areweplayingyet.org/support-seeking-unbuffered-position-mp3 * http://areweplayingyet.org/support-seeking-unbuffered-position-aac We will also introduce an equivalent test for Opus. Or if you want to contribute you can propose a patch :)

In this day, Firefox nightly 20.0a1 (2012-11-26) have 42 test passed/47. Firefox fail for: * Seeking to unbuffered position with continuous playback after seeking (MP3) * Seeking to unbuffered position with continuous playback after seeking (AAC) * Supports MP3 format * Supports AAC format => Normal: problems with patent. It's work if you choose Ogg. * Event "stalled" I can't find a bug for it in bigzilla. Firefox 17.0 have 39/47. The difference it's for: * Property "playbackRate" * Property "defaultPlaybackRate" * Event "ratechange"

(In reply to vulcain from comment #26) > * Event "stalled" I can't find a bug for it in bigzilla. I think it's bug 702870.

Firefox 17 on android 4.1 have 38/47: * Seeking to unbuffered position with continuous playback after seeking (AAC) * Seeking to unbuffered position with continuous playback after seeking (MP3) * Seeking to unbuffered position with continuous playback after seeking (Ogg) * Support MP3 format * Consistent time between play() and actual playback (< 20ms) * Property "playbackRate" * Property "defaultPlaybackRate" * Event "stalled" * Event "ratechange" with Firefox 20.0a nightly on Android 4.1 :42/47 * Seeking to unbuffered position with continuous playback after seeking (AAC) * Seeking to unbuffered position with continuous playback after seeking (Ogg) * Support MP3 format * Consistent time between play() and actual playback (< 20ms) * Event "stalled"

For information on Linux 3.2.0-34-generic x86_64: Opera 12.11 have 39/47 (fail for seek all format, don't support Opus, mp3, aac; Event "abort"; Event "stalled", Seeking to unbuffered position with continuous playback after seeking with ogg, mp3, aac). Chromium 20.0.1132.47 have 19/47 Google Chrome 23.0.1271.91 have 43/47 and fail for Event "stalled" and Seeking to unbuffered position with continuous playback after seeking for mp3, aac, Ogg. Amazing, i don't remember that Chrome could play opus codec. On Android 4.1, Opera mobile 12.10 have 35/47 Chrome have 29/47, Dolphin Browser 9.0.3 have 29/47, Android Browser 4.1.2 have 29/47.

With Firefox 48 on Windows 10: * Event "stalled" * Event "abort" None of the previous reports mention this "abort" failure, I am not sure if it is a new test or a regression.

Firefox 48 on android 4.4 have 42/47: * Consistent time between play() and actual playback (< 20ms) * Property "buffered" * Event "stalled" * Event "canplaythrough" * Event "abort" None of the previous Android reports mention these "buffered", "canplaythrough" or "abort" failures, I am not sure if they are new tests or regressions.

[update] With Firefox 48 to 51 on Windows 10 we have 44/47: * Property "buffered" (I missed this failure on #c30 above) * Event "stalled" * Event "abort" None of the previous Android reports mention these "buffered", or "abort" failures, I am not sure if they are new tests or regressions.