David Flanagan [:djf] Reporter
	Description • 10 years ago

On mobile devices (such as low-end FirefoxOS devices) there is often only a single hardware unit for decoding H264 video, with no software decoder as a fallback. This means that only a single app can be playing a video at a time. If every app and web page displays only one video at a time, and if every app and web page listens for visiblitychange events and pauses their video when hidden, then things mostly work.

But the introduction of WebRTC calls through the Hello app has broken this for FirefoxOS. If the user tries to play a video while on a video call, H264 decoding hardware will be in use and the video they want to play will not play but will not fail in any user-visible or programmer-visible way. See bug 1079519

There are two things that I'd like to address this:

1) the <video> element should have some kind of "please wait" or "video hardware in use" icon or animation that it can display in this situation to give some kind of feedback to the user that their video cannot be displayed. I think of this as something like the broken image icon displayed by the <img> element.

2) I'd like new Web API of some sort so that web pages and apps can detect this condition and notify the user in some more informative way that the video decoder is already in use. media elements already fire a confusing array of events and already have a readyState property, but I'm not sure if any of these are suitable for indicating this "hardware not currently available" condition. Most of the existing API is about waiting for video data to arrive over the network, we don't seem to have anything that we can use when playback has to wait for access to the hardware.

	David Flanagan [:djf] Reporter
	Comment 1 • 10 years ago

CC'ing Sotaro who wrote the existing code for sharing access to the video hardware in FirefoxOS, and Russ who maintains the FirefoxOS video app.

	David Flanagan [:djf] Reporter
	Comment 2 • 10 years ago

I'm nominating this bug for 2.2 because it is a followup to bug 1079519 which was a last-minute 2.1+ blocker.

	Sotaro Ikeda [:sotaro] Assignee
	Comment 3 • 10 years ago

roc, cpearce, do you have ideas about how to deliver "hw video decoder(some resources) is not available" kind of event. to gaia(JS) side?

	Sotaro Ikeda [:sotaro] Assignee
	Comment 4 • 10 years ago

jesup, when WebRTC is going to implement releasing hw video codec if an application goes to background?

	Chris Pearce [:cpearce (Not reading bugmail)]
	Comment 5 • 10 years ago

We could dispatch a "waiting" event to the media element. The W3C specs for MSE and EME say we should do this when playback using these APIs is blocked due to insufficient data or lack of a decryption key. This seems a similar situation.

In the EME case there's additionally an HTMLMediaElement.waitingFor [1] attribute added to the spec with values to indicate what the media load algorithm is blocked waiting for. We could add another value to the MediaWaitingFor enumeration "hardware-decoder", or somesuch.

[1] https://dvcs.w3.org/hg/html-media/raw-file/default/encrypted-media/encrypted-media.html#idl-def-MediaWaitingFor

	Robert O'Callahan (:roc) (email my personal email if necessary)
	Comment 6 • 10 years ago

I agree with Chris. We can't really expect third-party content to handle any kind of new event or state. So we need to do something automatic that fits into the current HTML5 media element model. Putting the media element into HAVE_CURRENT_DATA readyState and firing a waiting event is probably the right way to do that. Adding a new value to MediaWaitingFor makes sense for those few apps (e.g. the ones we write) that want to detect this specific situation. (That should be proposed to a spec body, though.)

	Randell Jesup [:jesup] (needinfo me)
	Comment 7 • 10 years ago

(In reply to Sotaro Ikeda [:sotaro] from comment #4)
> jesup, when WebRTC is going to implement releasing hw video codec if an
> application goes to background?

First, we have to make a design decision that video transmission *and* reception should end if it's backgrounded.  I could see that decision being made, but right now that's not how it works, and there may be some counter-arguments (or alternative UI's, where a video call continues as reception-only in a small overlay while you look something up). 

Second, if/when the first is decided, considerable new code will need to be written to dynamically release the resource and then re-acquire it when foregrounded, and deal with error conditions, and also force a reception error to ask the sender for an IDR.  Since this is very out of tune with the current model used by the webrtc.org layer, we'd likely have to implement it at the OMX codec layer and fake that the codec was still active in some way.  Note that the OMX codec layer is very isolated from anything that would know about foreground/background currently.

	Sotaro Ikeda [:sotaro] Assignee
	Comment 8 • 10 years ago

(In reply to Chris Pearce (:cpearce) from comment #5)
> We could dispatch a "waiting" event to the media element. The W3C specs for
> MSE and EME say we should do this when playback using these APIs is blocked
> due to insufficient data or lack of a decryption key. This seems a similar
> situation.
> 
> In the EME case there's additionally an HTMLMediaElement.waitingFor [1]
> attribute added to the spec with values to indicate what the media load
> algorithm is blocked waiting for. We could add another value to the
> MediaWaitingFor enumeration "hardware-decoder", or somesuch.
> 
> [1]
> https://dvcs.w3.org/hg/html-media/raw-file/default/encrypted-media/encrypted-
> media.html#idl-def-MediaWaitingFor

cpearce, latest document seems not have this attribute. Is there no problem to use it to fix this bug?

	Chris Pearce [:cpearce (Not reading bugmail)]
	Comment 9 • 10 years ago

I don't think we should use something that's been removed from the spec.

I filed bug 1130906 to remove our implementation of HTMLMediaElement.waitingFor. It would only likely be used by applications using EME anyway, which are likely to be uncommon, so it probably was not going to have good suppport.

We can still fire a "waiting" event, as we still roughly fit the conditions describing when it's fired:

"Playback has stopped because the next frame is not available, but the user agent expects that frame to become available in due course."
http://dev.w3.org/html5/spec-author-view/video.html#event-media-waiting

That would not let us distinguish between waiting due to hardware loss and waiting for other reasons. So we may need to either re-add a waitingFor attribute, or make the "waiting" event a non-simple event with a code field.

	Sotaro Ikeda [:sotaro] Assignee
	Comment 10 • 10 years ago

(In reply to Chris Pearce (:cpearce) from comment #9)
> 
> That would not let us distinguish between waiting due to hardware loss and
> waiting for other reasons. So we may need to either re-add a waitingFor
> attribute, or make the "waiting" event a non-simple event with a code field.

cpearce, code field seems reasonable choice, is there something we need to do before adding code field?

	Chris Pearce [:cpearce (Not reading bugmail)]
	Comment 11 • 10 years ago

You'll probably want to convince someone on the DOM team that it's a good idea; I expect you will need a DOM peer to review the patch as you'll need to change WebIDL.

	Stephany Wilkes
	Comment 12 • 10 years ago

B2G triage is blocking based on DJF's comment #2 and importance of WebRTC in recent initiatives.

	Kai-Chih Hu [:khu]
	Comment 13 • 10 years ago

It doesn't have any update for 10 days. Could you share the latest update with us? Thank you.

	Sotaro Ikeda [:sotaro] Assignee
	Comment 14 • 10 years ago

I posted a question to dev-webapi@lists.mozilla.org and waited the response. But I could not receive a valid response :-(
https://groups.google.com/forum/#!topic/mozilla.dev.webapi/za8KpzSc70g

I am going to think about how to do it.

	Sotaro Ikeda [:sotaro] Assignee
	Comment 15 • 10 years ago

Current "waiting" and "waitingforkey" events' spec are the following.

* waiting
 - Playback has stopped because the next frame is not available, 
   but the user agent expects that frame to become available in due course. 
 - readyState is equal to or less than HAVE_CURRENT_DATA, and paused is false. 
   Either seeking is true,
    or the current playback position is not contained in any of the ranges in buffered.
 - It is possible for playback to stop for other reasons without paused being false,
   but those reasons do not fire this event

 http://dev.w3.org/html5/spec-author-view/video.html#event-media-waiting


* waitingforkey
  - Playback is blocked waiting for a key.
  - The element is potentially playing and its readyState is equal to HAVE_FUTURE_DATA or greater.
  -  It should only be called when the HTMLMediaElement object is potentially playing
      and its readyState is equal to HAVE_FUTURE_DATA or greater.

  http://w3c.github.io/encrypted-media/#dom-evt-waitingforkey

	Sotaro Ikeda [:sotaro] Assignee
	Comment 16 • 10 years ago

In Media elements, only readyState and networkState exit as resource related states.

* readyState
 + Rexpresses the current state of the element with respect to rendering the current playback position

 - HAVE_NOTHING
 - HAVE_METADATA
 - HAVE_CURRENT_DATA
 - HAVE_FUTURE_DATA
 - HAVE_ENOUGH_DATA
  http://www.w3.org/html/wg/drafts/html/master/semantics.html#dom-media-readystate


* networkState
 + the current state of network activity for the element

 - NETWORK_EMPTY
 - NETWORK_IDLE
 - NETWORK_LOADING
 - NETWORK_NO_SOURCE
  http://www.w3.org/html/wg/drafts/html/master/semantics.html#dom-media-networkstate

	Sotaro Ikeda [:sotaro] Assignee
	Comment 17 • 10 years ago

During dormant state, decoder is released and majority of HTMLMediaElement APIs tell a lie about it's state. But readyState still notify the correct value.

	Sotaro Ikeda [:sotaro] Assignee
	Comment 18 • 10 years ago

(In reply to Sotaro Ikeda [:sotaro] from comment #17)
> During dormant state, decoder is released and majority of HTMLMediaElement
> APIs tell a lie about it's state. But readyState still notify the correct
> value.

It is not correct. If MediaDecoder is set to dormant during playback, HTMLMediaElement receive MediaDecoderOwner::NEXT_FRAME_AVAILABLE, then the readyState is set to HAVE_CURRENT_DATA.

	Sotaro Ikeda [:sotaro] Assignee
	Comment 19 • 10 years ago

I talked with :ehsan about this problem. The following are feedback.

- "waiting" should not be used for this.

- "waiting" expects that the video frame will be available in near future.
  But in hardware decoder case, if another app does not release hardware video decoder,
  the video frames will not be available forever.

- "waiting" uses generic Event object. If we use "waiting" for it.
  We need to create a new Event Object for it.
  It changes the prototype chain of the "waiting" event and could affect to the current Web pages.

- Therefore instead, custom event should be used for it.
  And we should limit it only to certified app.

	David Flanagan [:djf] Reporter
	Comment 20 • 10 years ago

Note that the bug that was blocked on this was resolved with a workaround instead. I still very much want this to be fixed, but maybe it does not need to block.  Even if this is resolved in gecko for 2.2, I do not expect there to be time to make corresponding gaia changes to use the new gecko feature.

	Sotaro Ikeda [:sotaro] Assignee
	Comment 21 • 10 years ago

Thanks for the comment. bajaj, can you re-think about b2g2.2+?

	bhavana bajaj [:bajaj]
	Comment 22 • 10 years ago

[Tracking Requested - why for this release]:

(In reply to Sotaro Ikeda [:sotaro] from comment #21)
> Thanks for the comment. bajaj, can you re-think about b2g2.2+?

sure, the workaround should be sufficient here especially at this time, so I'll remove the blocking flag

	Sotaro Ikeda [:sotaro] Assignee
	Comment 23 • 7 years ago

The bug was for b2g.