Matthew Gregan [:kinetik] Reporter
	Description • 13 years ago

SoundCloud published an HTML 5 <audio> interoperability test.  We're failing 8 tests:

1. Seeking to unbuffered position with seamless playback
2. Supports MP3 format (bug 562730)
3. Consistent timeupdate interval (15ms - 250ms)
4. Property "played" (bug 462959)
5. Property "playbackRate" (bug 495040)
6. Property "defaultPlaybackRate" (bug 495040)
7. Property "currentTime"
8. Event "ratechange" (bug 495040)

Of the ones we don't already have bugs for:

1. Test is broken (serves unseekable Ogg and expects to seek, also could fail
   intermittently depending on media download speed).
3. Expects delta time between timeupdate events to be >= 15ms and <= 250ms, in Firefox
   the test sees a delta time of 260-285ms.  This is a real bug.
7. Test is broken (serves unseekable Ogg and expects to seek, can pass intermittently
   depending on media download speed).

	Matthew Gregan [:kinetik] Reporter
	Comment 1 • 13 years ago

Filed bug 700210 for #3.

I've attempted to contact the owners about the broken tests.

	Matthew Gregan [:kinetik] Reporter
	Comment 2 • 13 years ago

When I say "unseekable Ogg", what I mean is that the server does not support range requests, so Firefox is only able to seek in buffered parts of the file.

#1 is unreliable because it attempts to seek to audio.buffered.end(0) + 60, but depending on download speed it's possible for audio.buffered.end(0) to equal audio.duration, so the test not complete.

#7 is unreliable because it can intermittently pass depending on media download speed, as a seek into a buffered area will succeed but a seek into an unbuffered area will fail due to the first issue mention in this comment.

	Matthew Gregan [:kinetik] Reporter
	Comment 3 • 13 years ago

Additionally, it's not clear to me what #1 intends to test, but seeking to an unbuffered area will not be seamless as there will be network and decoding delays involved in the seek.

If it's expected that the old playback location continues to play until the seek completes, this isn't the case in Firefox nor in any implementation I'm aware of.  The test doesn't (and can't) detect this behaviour, anyway.

What the test appears to be checking is whether playback continues automatically after seeking into an unbuffered area--if this is the intended action the test should probably be renamed to reflect what is being tested.

	Tomás Seart
	Comment 4 • 13 years ago

Hi Matthew!

I must say you caught us by surprise since we're today doing a final work marathon on this project to be released tomorrow.
Anyway your input is very valuable and the Github repo (http://github.com/soundcloud/areweplayingyet) is open now. If you have any suggestions to improve a test just create a Pull Request.

Going to fix some tests now! ;)

	Antoine Turmel [:GeekShadow]
	Comment 5 • 13 years ago

After running the full test on Firefox Beta 8 (beta channel), the count is still at 0 at the bottom of website.

	Tomás Seart
	Comment 6 • 13 years ago

Antoine as I wrote above, you shouldn't consider anything final until Wednesday. Thank you for your feedback.

	Matthew Gregan [:kinetik] Reporter
	Comment 7 • 13 years ago

The seeking issue is resolved, so #7 now passes.  I think #1 should pass now, but I suspect the test is timing out before it has a chance to complete.  The tests have a default timeout of 10s, which may be too low when running some of them over a slowish international link.

Two new tests are failing: AAC support (we don't, no bug to reference, but the reasoning is roughly the same as for MP3), and 'stalled' (which should pass, not sure what's going on there).

	Tomás Seart
	Comment 8 • 13 years ago

Hi Matthew!

Let's talk about the #1. Do you understand the use case? If implemented right, it allows fast playback of _any_ part of an audio file (without the need to buffer everything until that point) and then if no more user interaction happens it should continue playback smoothly.

About AAC support... You should now more about that than me but didn't it go open source recently?

The stalled event is not working on Firefox... The loadstart event is triggered, then we halt the transfer of data for 3500ms (the spec requires 3000 but we gave some space) where stalled should be triggered.

Cheers,
Tomás

	Paul Adenot (:padenot)
	Comment 9 • 13 years ago

Apple release their ALAC (Apple Lossless Audio Codec, equivalent to FLAC) under an open source license, but AFAIK, AAC is still encumbered with patents.

	Tomás Seart
	Comment 10 • 13 years ago

@Paul Do you think ALAC could be a good way of erradicating the problem of having different audio formats for different browsers?

	Paul Adenot (:padenot)
	Comment 11 • 13 years ago

Tomas, ALAC is unfortunately a lossless audio codec, and is intended for very high quality/high bitrate streams, so it might not be very useful for our purpose, which is streaming media.

	Tomás Seart
	Comment 12 • 13 years ago

@Paul, acknowledged. So what are your ideas on how this could be solved?

	Tomás Seart
	Comment 13 • 13 years ago

@Matthew: The first test was indeed timing out. Have look at it now! http://areweplayingyet.org/support-seeking-unbuffered-position

	Ralph Giles (:rillian)
	Comment 14 • 13 years ago

Tomás, the codec issue is hard. Apple and MS could support Ogg like Google, Opera, and Mozilla. In some years the patents on mp3 will expire and we can add that to Firefox. A shim that does audio decompression in js might work in the meantime.

Fortunately, the <audio> element lets you specify multiple <source> elements!

	Tomás Seart
	Comment 15 • 13 years ago

Can anyone look at this test source and validate that is valid (passing or not)? Webkit people are saying that seeking should only happen after the canplay event was triggered but this makes Firefox fail.

	Tomás Seart
	Comment 16 • 13 years ago

http://areweplayingyet.org/support-seeking-unbuffered-position

	Tomás Seart
	Comment 17 • 13 years ago

It _made_ Firefox fail because canplay is triggered again after the seek so there would be an infinite loop. Passing now.

	Matthew Gregan [:kinetik] Reporter
	Comment 18 • 13 years ago

Requiring the canplay event before seeking corresponds to HAVE_FUTURE_DATA.  It should be possible to seek as soon as you reach HAVE_METADATA (and receive the loadedmetadata event).

	Tomás Seart
	Comment 19 • 13 years ago

Matthew can you please link me to the spec part that defines that? Thanks

	Matthew Gregan [:kinetik] Reporter
	Comment 20 • 13 years ago

Here: http://www.whatwg.org/specs/web-apps/current-work/multipage/video.html#seeking

Note that it's an error to seek while in readyState HAVE_NOTHING.  All other states are valid, including HAVE_METADATA.

Here: http://www.whatwg.org/specs/web-apps/current-work/multipage/video.html#the-ready-states

Under HAVE_METADATA: "The API will no longer raise an exception when seeking."

Also note in that section that canplay is fired when the readyState is HAVE_FUTURE_DATA.

So this change is wrong, and should be reverted: https://github.com/soundcloud/areweplayingyet/commit/37d10a57d420807cc6838f8e94d56f5198b01798

	Matthew Gregan [:kinetik] Reporter
	Comment 21 • 13 years ago

It's probably better to have these discussions in the AreWePlayingYet issue tracker so they're more visible to other interested parties.  This bug is intended to track Firefox bugs, and also track notes on the tests before the AWPY issue tracker was opened up.  I'll file AWPY test bugs there from now on.

	Matthew Gregan [:kinetik] Reporter
	Comment 22 • 13 years ago

Loop test fails on Linux (with PulseAudio) due to bug 582513.

	alex_mayorga
	Comment 23 • 12 years ago

Just stumbles into AWPY today, so please bear with me.

These still fail on Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:14.0) Gecko/20120314 Firefox/14.0a1 ID:20120314031139

http://areweplayingyet.org/support-seeking-unbuffered-position
http://areweplayingyet.org/support-hot-swapping-src
http://areweplayingyet.org/support-consistent-playback-delay
http://areweplayingyet.org/property-seeking
http://areweplayingyet.org/property-played
http://areweplayingyet.org/property-playbackRate
http://areweplayingyet.org/property-defaultPlaybackRate
http://areweplayingyet.org/event-suspend
http://areweplayingyet.org/event-stalled
http://areweplayingyet.org/event-seeking
http://areweplayingyet.org/event-ratechange
http://areweplayingyet.org/event-pause
http://areweplayingyet.org/event-loadedmetadata
http://areweplayingyet.org/attribute-preload-metadata

Changing "Platform" accordingly.

Should new bugs be filled for the failing new tests?

	Matthew Gregan [:kinetik] Reporter
	Comment 24 • 12 years ago

The failing tests need to be reviewed to confirm they're valid tests, and if so, why they're failing.  That can be done separately for each test if someone would like to file individual bugs that block this one.

	Yves Van Goethem
	Comment 25 • 12 years ago

Hey guys,
We made minors update to AWPY:
 - Introduced check for Opus support http://areweplayingyet.org/support-format-opus :)
 - Split the test 'support-seeking-unbuffered-position' into 3 tests to take into account different codecs and removed the former one. (you can read more about the motivations: https://github.com/soundcloud/areweplayingyet/pull/41)
  * http://areweplayingyet.org/support-seeking-unbuffered-position-ogg
  * http://areweplayingyet.org/support-seeking-unbuffered-position-mp3
  * http://areweplayingyet.org/support-seeking-unbuffered-position-aac

We will also introduce an equivalent test for Opus.
Or if you want to contribute you can propose a patch :)

	vulcain
	Comment 26 • 12 years ago

In this day, 
Firefox nightly 20.0a1 (2012-11-26) have 42 test passed/47. Firefox fail for:
* Seeking to unbuffered position with continuous playback after seeking (MP3)
* Seeking to unbuffered position with continuous playback after seeking (AAC)
* Supports MP3 format
* Supports AAC format
=> Normal: problems with patent. It's work if you choose Ogg.
* Event "stalled"  I can't find a bug for it in bigzilla.


Firefox 17.0 have 39/47. The difference it's for:
* Property "playbackRate"
* Property "defaultPlaybackRate"
* Event "ratechange"

	Sid
	Comment 27 • 12 years ago

(In reply to vulcain from comment #26)
> * Event "stalled"  I can't find a bug for it in bigzilla.
I think it's bug 702870.

	vulcain
	Comment 28 • 12 years ago

Firefox 17 on android 4.1 have 38/47:
* Seeking to unbuffered position with continuous playback after seeking (AAC)
* Seeking to unbuffered position with continuous playback after seeking (MP3)
* Seeking to unbuffered position with continuous playback after seeking (Ogg)
* Support MP3 format
* Consistent time between play() and actual playback (< 20ms)
* Property "playbackRate"
* Property "defaultPlaybackRate"
* Event "stalled"
* Event "ratechange"

with Firefox 20.0a  nightly on Android 4.1 :42/47
* Seeking to unbuffered position with continuous playback after seeking (AAC)
* Seeking to unbuffered position with continuous playback after seeking (Ogg)
* Support MP3 format
* Consistent time between play() and actual playback (< 20ms)
* Event "stalled"

	vulcain
	Comment 29 • 12 years ago

For information on Linux 3.2.0-34-generic x86_64:
Opera 12.11 have 39/47 (fail for seek all format, don't support Opus, mp3, aac; Event "abort"; Event "stalled", Seeking to unbuffered position with continuous playback after seeking with ogg, mp3, aac). 
Chromium 20.0.1132.47 have 19/47
Google Chrome 23.0.1271.91 have 43/47 and fail for Event "stalled" and Seeking to unbuffered position with continuous playback after seeking for mp3, aac, Ogg. Amazing, i don't remember that Chrome could play opus codec.

On Android 4.1, Opera mobile 12.10 have 35/47 Chrome have 29/47, Dolphin  Browser 9.0.3 have 29/47, Android Browser 4.1.2 have 29/47.

	Alexandre Folle de Menezes
	Comment 30 • 8 years ago

With Firefox 48 on Windows 10:
* Event "stalled"
* Event "abort"

None of the previous reports mention this "abort" failure, I am not sure if it is a new test or a regression.

	Alexandre Folle de Menezes
	Comment 31 • 8 years ago

Firefox 48 on android 4.4 have 42/47:
* Consistent time between play() and actual playback (< 20ms)
* Property "buffered"
* Event "stalled"
* Event "canplaythrough"
* Event "abort"

None of the previous Android reports mention these "buffered", "canplaythrough" or "abort" failures, I am not sure if they are new tests or regressions.

	Alexandre Folle de Menezes
	Comment 32 • 8 years ago

[update] With Firefox 48 to 51 on Windows 10 we have 44/47:
* Property "buffered" (I missed this failure on #c30 above)
* Event "stalled"
* Event "abort"

None of the previous Android reports mention these "buffered", or "abort" failures, I am not sure if they are new tests or regressions.