Chris Jones [:cjones] inactive; ni?/f?/r? if you need me Reporter
	Description • 12 years ago

See 791164.  Our current video pipeline relies on the media cache and decoder being "close to" content, which is awesome because it lets us isolate CPU decoders, and it lets us implement things like texturing videos in WebGL.

This model breaks down when we want to use hardware decoders with privileged interfaces.

The simplest thing we can do here is probably a "decoder service" bridge, a la bug 785592.  Things are a bit more difficult for video decoding because we have to deliver the raw encoded bytes to the decoder.  We can do this with shmem, I guess.

	[:fabrice] Fabrice Desré
	Comment 1 • 12 years ago

Chris, do we still need want to do that for 1.0.x/1.1 ?

	Chris Jones [:cjones] inactive; ni?/f?/r? if you need me Reporter
	Comment 2 • 12 years ago

We need this to be done ASAP but it's not trivial.

	Sotaro Ikeda [:sotaro]
	Comment 3 • 12 years ago

I have implemented it in Bug 803471. I am not sure a way used in the bug is acceptable for FirefoxOS.

	cajbir (:cajbir)
	Comment 4 • 12 years ago

I like the approach in bug 803471 and hope we can go forward with it. What are your thoughts on it Chris Jones?

	Chris Jones [:cjones] inactive; ni?/f?/r? if you need me Reporter
	Comment 5 • 12 years ago

I'm not familiar with that code.  Should we do the architecture review here there or somewhere else?

My main questions are how that approach will deal with
 - either
   * per-app-process media caches
   * overhead of copying decoder stream data from app processes to mediaserver process
 - or
   * how we can let mediaserver read the streams in-process and cache there
 - keeping zero-copy output from HW decoder
 - path that frames will take from HW decoder to b2g compositor thread
 - "readback" of video frames by content to blit to <canvas> or texture for WebGL, e.g.
 - mediating multiple media elements attempting to use a HW decoder

I would personally prefer to create something analogous to the mediaserver as a "native gecko process", to get off the android treadmill, but the current HW decoder setup equals camera as the biggest architectural mistakes we made in v1.0.0, so I'm certainly happy to take the "fixes" in multiple steps.

	Sotaro Ikeda [:sotaro]
	Comment 6 • 12 years ago

(In reply to Chris Jones [:cjones] [:warhammer] from comment #5)

This approach do not require change to gecko's media framework. All things work transparently.

> My main questions are how that approach will deal with
>  - either
>    * per-app-process media caches

per-app-process media caches works same as before. MediaExtractor(parser) is still in app process.

>    * overhead of copying decoder stream data from app processes to
> mediaserver process

Additional stream data copying do not happen when stream data is delivered between app process and media server process. Video data is delivered by using GraphicBuffer. Other stream data is delivered by using IMemory(ashmem).

>  - or
>    * how we can let mediaserver read the streams in-process and cache there

media caches are not delivered to mediaserver process. They are parsed by MediaExtractor in app process. parsed data is delivered to mediaserver.

>  - keeping zero-copy output from HW decoder

Video out frames from hw codecs are delivered to app process by using GraphicBuffer. Sw codec output frames are delivered to app process by using IMemory.

>  - path that frames will take from HW decoder to b2g compositor thread

I wrote the diagram, route is almost same as before. Changes are local to OMXCodec and camera hw.

>  - "readback" of video frames by content to blit to <canvas> or texture for
> WebGL, e.g.

It works same as before.

>  - mediating multiple media elements attempting to use a HW decoder

It is same as before. Current FirefoxOS do not have hw codec mediator. I am going to think about it in Bug 831747.

	Sotaro Ikeda [:sotaro]
	Comment 7 • 12 years ago

current FirefoxOS diagram around nsMediaOmxDecoder

	Sotaro Ikeda [:sotaro]
	Comment 8 • 12 years ago

diagram around nsMediaOmxDecoder after applying the change.
new Objects are in different color.

	Sotaro Ikeda [:sotaro]
	Comment 9 • 12 years ago

current FirefoxOS's diagram around camera

	Sotaro Ikeda [:sotaro]
	Comment 10 • 12 years ago

camera diagram after applying change.
new objects are in different color.

	Robert O'Callahan (:roc) (email my personal email if necessary)
	Comment 11 • 12 years ago

These are great diagrams!

The design looks very non-invasive to me, so I like it!

	Chris Jones [:cjones] inactive; ni?/f?/r? if you need me Reporter
	Comment 12 • 12 years ago

I agree, this looks like about the most minimal change we can make to fix the architectural mistake.

	Paul Theriault [:pauljt] (no longer reading bugmail)
	Comment 13 • 7 years ago

Closing FxOS to as part of triage for sandbox tasks.