See 791164. Our current video pipeline relies on the media cache and decoder being "close to" content, which is awesome because it lets us isolate CPU decoders, and it lets us implement things like texturing videos in WebGL. This model breaks down when we want to use hardware decoders with privileged interfaces. The simplest thing we can do here is probably a "decoder service" bridge, a la bug 785592. Things are a bit more difficult for video decoding because we have to deliver the raw encoded bytes to the decoder. We can do this with shmem, I guess.

Chris, do we still need want to do that for 1.0.x/1.1 ?

We need this to be done ASAP but it's not trivial.

I have implemented it in Bug 803471. I am not sure a way used in the bug is acceptable for FirefoxOS.

I like the approach in bug 803471 and hope we can go forward with it. What are your thoughts on it Chris Jones?

I'm not familiar with that code. Should we do the architecture review here there or somewhere else? My main questions are how that approach will deal with - either * per-app-process media caches * overhead of copying decoder stream data from app processes to mediaserver process - or * how we can let mediaserver read the streams in-process and cache there - keeping zero-copy output from HW decoder - path that frames will take from HW decoder to b2g compositor thread - "readback" of video frames by content to blit to <canvas> or texture for WebGL, e.g. - mediating multiple media elements attempting to use a HW decoder I would personally prefer to create something analogous to the mediaserver as a "native gecko process", to get off the android treadmill, but the current HW decoder setup equals camera as the biggest architectural mistakes we made in v1.0.0, so I'm certainly happy to take the "fixes" in multiple steps.

(In reply to Chris Jones [:cjones] [:warhammer] from comment #5) This approach do not require change to gecko's media framework. All things work transparently. > My main questions are how that approach will deal with > - either > * per-app-process media caches per-app-process media caches works same as before. MediaExtractor(parser) is still in app process. > * overhead of copying decoder stream data from app processes to > mediaserver process Additional stream data copying do not happen when stream data is delivered between app process and media server process. Video data is delivered by using GraphicBuffer. Other stream data is delivered by using IMemory(ashmem). > - or > * how we can let mediaserver read the streams in-process and cache there media caches are not delivered to mediaserver process. They are parsed by MediaExtractor in app process. parsed data is delivered to mediaserver. > - keeping zero-copy output from HW decoder Video out frames from hw codecs are delivered to app process by using GraphicBuffer. Sw codec output frames are delivered to app process by using IMemory. > - path that frames will take from HW decoder to b2g compositor thread I wrote the diagram, route is almost same as before. Changes are local to OMXCodec and camera hw. > - "readback" of video frames by content to blit to <canvas> or texture for > WebGL, e.g. It works same as before. > - mediating multiple media elements attempting to use a HW decoder It is same as before. Current FirefoxOS do not have hw codec mediator. I am going to think about it in Bug 831747.

current FirefoxOS diagram around nsMediaOmxDecoder

diagram around nsMediaOmxDecoder after applying the change. new Objects are in different color.

current FirefoxOS's diagram around camera

camera diagram after applying change. new objects are in different color.

These are great diagrams! The design looks very non-invasive to me, so I like it!

I agree, this looks like about the most minimal change we can make to fix the architectural mistake.

Closing FxOS to as part of triage for sandbox tasks.