On mozilla-b2g18 this video freezes after a couple of minutes, not always on the same spot: https://www.youtube.com/watch?v=V1IBH1bz5-c I attached a log. It's a long >1 hour video and when I run it there are several long pauses for buffering. Eventually I loose audio then the whole video freezes. It seems like there's some problems with buffering and the hardware decoder.

I think bug 862096 might be related.

and/or bug 846571

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #2) > and/or bug 846571 bug 846571 is using sw codec, but this bug uses hw codec. bug 846571 is not related, I think.

Sorry, comment #3 was my misunderstanding. At first it try to play video by using OMXCodec, But it fails to create hw codec. Then do fallback to WebM.

Nominate to tef. Because I confirmed the bug on unagi.

I re-confirm about comment #5 again.

comment #4 was totally my misunderstanding. It uses hw codec.

Freezing happened also Firefox OS v1.0.1 on unagi. In my case, the video freeze about 10 seconds or more, then move again. video stop happens often during playback.

I also confirmed the freeze as in comment #0 on Firefox OS v1.0.1 on unagi.

It seems that there are two type of freeze - [1] long freeze that made by nsMediaCacheStream::Read()'s delay - [2] short freeze that unrelated to nsMediaCacheStream::Read() It seems that [2] makes the video playback freeze. Need to investigate about it more.

After when the video playback started [2] is almost ignorable, though as time progress blocking time of nsMediaCacheStream::Read() becomes significant. Each call is going to take more than 200ms. Sometimes more than 1 sec.

Temporary patch for the logout.

Cnecked the decode's latency by using attachment 739575 [details] [diff] [review] in Bug 863441. Big latency at nsMediaCacheStream::Read() happens sometimes, though it is not often as in Bug 863441.

I think there are two issues here: [1] The video stutters (buffers?) every minute or so for long periods of time [2] Eventually [1] seems to put the HW decoder in a bad state I can look into why the pauses break the HW Decoder [2] @Sotaro maybe you can help me figure out why the buffering pauses are so bad?

Is it possible this is unagi-specific? QA, can you test on a buri device?

Video freeze happens from timeout at OMXCodec::waitForBufferFilled_l(). default timeout value is 3 sec. http://androidxref.com/4.0.4/xref/frameworks/base/media/libstagefright/OMXCodec.cpp#3427

Follwoing a log of the timeout. There is enough out buffer but no input buffer. From the log, slow input causes the timeout. > E OMXCodec: [OMX.qcom.video.decoder.avc] Timed out waiting for output buffers: 0/11

Bug 864181 is for extend tme timeout at OMXCodec::waitForBufferFilled_l(). The bug could mitigate the problem.

Need to figure out why data input becomes so slow.

extend a timeout at OMXCodec::waitForBufferFilled_l() from 3 sec to 20 sec. By applying the patch, whole video freezes is prevented. Though several long pauses for buffering happens often.

It seems that long pauses comes from nsMediaCache. I do not know about nsMediaCache, though it is difficult to fix for v1.0.1.

It is better to split long pauses problem to a different bug.

doublec, how do you think about comment #22 ?

Sotaro, AFAICT the OMX codec should only time out this way when there are decoded output buffers that are never returned back to the decoder. In theory even if you run out of input buffers that should mean that the hw decoder will stop outputting buffers, but still expects all pending buffers to come back soon. I think if this wasn't the case we would always hit the time out on Pause.

Hi diego, I can not understand what you want to say. Can you explain more about it? OMXCodec is android's class. I can not say what OMXCodec should not do. The timeout is used in OMXCodec not to block OMXCodec::read() long time. OMXCodec::read() is sync API, therefore there is a such timeout not to block client forever, I think. If OMXCodec::read() is called when data input is very slow, OMXCodec::read() could fail because of the timeout. It is OMXCodec's implementation. > I think if this wasn't the case we would always hit the time out on Pause. Calling OMXCodec::pause() is added by you in Bug 860760. When OMXCodec is in pause state, OmxDecoder do not call OMXCodec::read(). So there is no problem of timeout.

And when client calls OMXCodec::read(), OMXCodec's state needs to be EXECUTING or RECONFIGURING. http://androidxref.com/4.0.4/xref/frameworks/base/media/libstagefright/OMXCodec.cpp#3904

(In reply to Sotaro Ikeda [:sotaro] from comment #27) > And when client calls OMXCodec::read(), OMXCodec's state needs to be > EXECUTING or RECONFIGURING. Sorry, OMXCodec::pause() does not affect to OMXCodec's state.

(In reply to Andrew Overholt [:overholt] from comment #16) > Is it possible this is unagi-specific? > > QA, can you test on a buri device? Using Buri device, partner build: Gaia 5399410b5c217c4fc31fb49ea31dc039c31adaab BuildID 20130415035434 Version 18.0 No freezing or stuttering observed - over 4 minutes into the video it is still playing fine. I am connected to a wifi access point within the office

Right now there are a lot of difference between our build on unagi and partner build on buri. It is not only hardware but also software. Partner build uses Hardware composer to render video. But our build do not use it. Access speed to internal flash access might also different. I have no way to build buri's ROM similar to partners'. It is very inefficient and irritating me.

I got partner build and confirmed on Buri. Using Buri device, partner build: OS version 1.0.1.0-prerelease Platform Version 18.0 BuildID 20130419170628 Sometimes frame jitterd litte bit and video freezes after a couple of minutes. The way to freeze video seems different from unagi. There are no log of timeout at OMXCodec::waitForBufferFilled_l().

When I playbacked the video again, I see more long pause and timeout at timeout at OMXCodec::waitForBufferFilled_l() happens. It seems that partner build already extend the timeout from 3 sec. > E OMXCodec: [OMX.qcom.audio.decoder.aac] Timed out waiting for output buffers: 0/2

I also find another bug. Bottom position of the video is unstable. sometimes stretched a bit. It is a similar bug to Bug 850566. Bug 850566 is for GL rendering. Same video stretching problem happens in hw composer path.

It is just an assumption. If it is correct need to create a new bug. I already confirmed that video stretch happens in videos of Bug 850566.

When video freezes, I see not timeout log at OMXCodec::waitForBufferFilled_l() on buri's partner build. It is different from unagi.

I feel that jittering of video come from use of internal Flash for MediaCache.

doublec, is it possible not to use internal Flash for MediaCache?

cpearce will be able to better advise on whether internal flash can be avoided for MediaCache.

We discussed storing the media cache on the SD card if it's present back in bug 785662. I think the conclusion back then was that it's a good idea, but we need to handle losing access to the SD card (and thus MediaCache) when either the SD card is removed, or when the device is plugged into a computer.

cpearce, are there a possibility use only RAM for MediaCache? Android's stagefright uses only RAM for media playback via http.

Write to Flash/SD sometimes requires significant cpu usage. And affect to media playback's performance.

Bug 865091 blocks the STR of this bug.

I notice that I can not play youtube video by video app today.

Yes, we could use an in-memory only cache for B2G. Fennec may also benefit from this. It would not be hard to do, I think you can do it by implementing FileBlockCache to read/write to memory instead of to file. But how much memory can we spare to dedicate to storing video/audio data? 10MB? 20MB? We'd want to ensure that we limited the total memory allocated for media cache across all processes, else we'd run out of memory. Or perhaps your FileBlockCache could remote to a central media cache process?

Just stating the obvious here: P1

(In reply to Sotaro Ikeda [:sotaro] from comment #36) > I feel that jittering of video come from use of internal Flash for > MediaCache. My assumption of comment #36 was incorrect. I locally implemented RAMBlockCache, but the same problem happens.

Oops sorry, wrong bug. Updating priority

v1.0.1 devices can not use RAMBlockCache. RAM is very scarce resource. In the use cases of the youtube video playback, total MemFree is around 10M, even when running applications on the devices was minimum. And sometimes MemFree goes down to 8MB.

During playback, I see total MemFree than 1MB. I checked on buri device.

After that the video app was killed by lowmemory kiler. Then total MemFree went back around 80M(include Home, Usage, pre-allocated app process's RAM).

Intermittent freeze come from nsMediaCacheStream::Read(). Now, I think this comes from MediaCache miss. If the miss happens, it takes longer time until receiving the data. On b2g, media cache file size is 4MB max. http://mxr.mozilla.org/mozilla-b2g18/source/b2g/app/b2g.js#272 But in desktop it is 500MB max. http://mxr.mozilla.org/mozilla-b2g18/source/content/media/nsMediaCache.cpp#644

Could we also cache the block after the read cursor, to reduce the chance of a cache miss? That would only increase memory usage by 1 block (32KB I think). I'm not sure caching one extra block would be enough though.

I patched content/media/omx so the OmxDecoder doesn't always keep a reference to the most recent audio and video buffer. That improves the stability so that the video still plays with stutter, but plays further. However, the OMX decoder still eventually times out waiting for buffers. I think we're leaking media buffers somewhere.

Turns out in stagefright's video player pauses the decoder when the cache runs low. I think we'll have to do the same in gonk's omx decoder. https://www.codeaurora.org/gitweb/quic/la/?p=platform/frameworks/base.git;a=blob;f=media/libstagefright/AwesomePlayer.cpp;h=da1adc142812afd7e6387557ba21e0ca40a89090;hb=refs/heads/b2g/ics_strawberry#l775 I'll give it a shot.

gecko's media framework already have similar pause mechanism. Following functions are used for it. - nsBuiltinDecoderStateMachine::GetUndecodedData() - MediaOmxReader::GetBuffered() http://mxr.mozilla.org/mozilla-b2g18/source/content/media/nsBuiltinDecoderStateMachine.cpp http://mxr.mozilla.org/mozilla-b2g18/source/content/media/omx/nsMediaOmxReader.cpp#284

Interesting. According to this mechanism, if run low on undecoded data, playback should stop and buffering should start, which is what we want: http://mxr.mozilla.org/mozilla-b2g18/source/content/media/nsBuiltinDecoderStateMachine.cpp#2308 But I don't see this happening. Maybe it never gets to do this check since it's stuck getting data from the media cache?

Intermittent long video freeze happens because of MediaCahe miss. I confirmed about it by adding custom log. Following the MediaCache and reading status. One block size is 32Kbyte. Audio read is at block 1160 and video read is at block 873. Read position difference is 8MB. But total media cache size is 4MB. - audio read MediaStreamSource::readAt() offset 38011300 size 258 offsetBlock 1160 - video read MediaStreamSource::readAt() offset 28615228 size 1601 offsetBlock 873 <MediaCache> - startBlock 875 endBlock 933 - startBlock 1064 endBlock 1068 - startBlock 1069 endBlock 1072 - startBlock 1073 endBlock 1075 - startBlock 1076 endBlock 1080 - startBlock 1081 endBlock 1082 - startBlock 1083 endBlock 1084 - startBlock 1087 endBlock 1088 - startBlock 1089 endBlock 1090 - startBlock 1095 endBlock 1096 - startBlock 1097 endBlock 1098 - startBlock 1099 endBlock 1102 - startBlock 1103 endBlock 1106 - startBlock 1109 endBlock 1136 - startBlock 1141 endBlock 1156 - startBlock 1157 endBlock 1160

So audio/video sync is way off?

(In reply to Diego Wilson [:diego] from comment #58) > So audio/video sync is way off? comment #57 just says, audio track reads ahead more than video track. For video playback, video frames are not copied. But audio frames are copied and stored.

(In reply to Sotaro Ikeda [:sotaro] from comment #59) > comment #57 just says, audio track reads ahead more than video track Is this expected? I'm assuming some component is supposed to keep audio and video track times close together. I'm just not sure which one that is. nsBuiltinDecoder? nsOmxReader?

Seems like this is related to bug 867688

(In reply to Diego Wilson [:diego] from comment #60) > (In reply to Sotaro Ikeda [:sotaro] from comment #59) > > comment #57 just says, audio track reads ahead more than video track > > Is this expected? It is expected that audio track read more ahead than video track. Between android::AudioTrack and audio hw there are buffers for audio data. And to prevent audio underruns some amount of decoded audio data are necessary. But a difference of data read positions between the tracks becomes more than 8MB is an unexpected thing.

(In reply to Diego Wilson [:diego] from comment #60) > I'm assuming some component is supposed to keep audio and > video track times close together. I'm just not sure which one that is. > nsBuiltinDecoder? nsOmxReader? Video frame is rendered depends on audio time in nsBuiltinDecoderStateMachine::AdvanceFrame() http://mxr.mozilla.org/mozilla-b2g18/source/content/media/nsBuiltinDecoderStateMachine.cpp#2235

Solution to the MediaCache miss could be either following - [1] decrease read position difference between audio track and video track - [2] increase MediaCache size from 4MB. In android case, media cache is 20MB max for each video stream. But in Firefox OS case, 4MB in total video streams. http://androidxref.com/4.0.4/xref/frameworks/base/media/libstagefright/include/NuCachedSource2.h#75

During debugging, sometimes see crash of Bug 868073.

Found a good a/v sync video https://www.youtube.com/watch?v=szoOsG9137U It easy to see the prob if you start a timer on your watch at the same time as you start the playback. On my device the audio beeps steadily every second as expected, but the video increasingly lags. By the 3 minute mark the video is 10 seconds behind audio. By the end at 5 minutes it's already 20 seconds behind!

Found it! We're not buffering enough decoded video frames. If I set the decoded video frame limit to 3 here: https://mxr.mozilla.org/mozilla-b2g18/source/content/media/omx/MediaOmxStateMachine.h#24 In accordance with what is documented here: https://mxr.mozilla.org/mozilla-b2g18/source/content/media/nsBuiltinDecoderStateMachine.cpp#426 Video plays nice and smooth ;)

Diego, you rock! I also confirmed that the increase GetAmpleVideoFrames() value from 1 to 3 fix the problem. On buri device with v1.0.1 MozBuild, only the above change made other buffer starvation problem at hw codec, and then playback was jittering. Applying the patches in Bug 864230 fixed the jittering problem.

But Increase GetAmpleVideoFrames() value from 1 to 3 regress the Bug 837051 and Bug 864230 :-(

(In reply to Sotaro Ikeda [:sotaro] from comment #69) > But Increase GetAmpleVideoFrames() value from 1 to 3 regress the Bug 837051 > and Bug 864230 :-( That's ok. It just means we haven't found the proper fixes for these bugs yet. We'll definitely need 3 or more output video buffers, though. I'll see if I find more clues.

I feel that GetAmpleVideoFrames() could be still 1 in local playback case. Local playback videos could have longer reference video frames than videos via http. If it is OK, solutions of Bug 837051 and Bug 864230 still works.

(In reply to Sotaro Ikeda [:sotaro] from comment #71) > I feel that GetAmpleVideoFrames() could be still 1 in local playback case. > Local playback videos could have longer reference video frames than videos > via http. If it is OK, solutions of Bug 837051 and Bug 864230 still works. A GetAmpleVideoFrames() of 1 also messes the FPS of local and network video playback :( I'm pretty sure the real root of the problem for Bug 837051 and Bug 864230 is somewhere else.

(In reply to Diego Wilson [:diego] from comment #75) > I'm pretty sure the real root of the problem for Bug 837051 and Bug 864230 > is somewhere else. What is your idea? I thought, Bug 837051 and Bug 864230 are caused by buffer starvation in hw codec. hw codec need to have all reference video frames in hw codec. When client side grab more video frames, hw codec do not have all reference video frames need to decode next video frame.

(In reply to Diego Wilson [:diego] from comment #75) > > A GetAmpleVideoFrames() of 1 also messes the FPS of local and network video > playback :( Did you observe the problem in local playback case?

(In reply to Sotaro Ikeda [:sotaro] from comment #76) > (In reply to Diego Wilson [:diego] from comment #75) > > I'm pretty sure the real root of the problem for Bug 837051 and Bug 864230 > > is somewhere else. > > What is your idea? I thought, Bug 837051 and Bug 864230 are caused by buffer > starvation in hw codec. hw codec need to have all reference video frames in > hw codec. When client side grab more video frames, hw codec do not have all > reference video frames need to decode next video frame. OMX Codec allocates 12 output buffers. If GetAmpleVideoFrames() is 3 that means we would keep 3 buffers in the video queue and 1 currently rendered. That means the OMX Codec gets to keep 8 buffers. Wanting more than that sounds pretty greedy to me :P

But during playback 2 more buffers are in gecko side for rendering between compositor and media framework. And even in GetAmpleVideoFrames() 3 setting, almost all videos can be playback, only some videos that seems to have long frame reference can not be playback.

(In reply to Sotaro Ikeda [:sotaro] from comment #77) > Did you observe the problem in local playback case? Yep

With GetAmpleVideoFrames() of 1 a 30 FPS VGA local clip plays back at 20 FPS. With GetAmpleVideoFrames() of 3 the same 30 FPS VGA local clip plays at 30 FPS.

Sorry, comment #69 was incorrect. I created the incorrect ROM:-( I reconfirmed about it again. By applying attachment 744257 [details] [diff] [review] in Bug 864230, Bug 837051 and Bug 864230 is fixed. The path add 2 more buffers to hw codec.

So, GetAmpleVideoFrames() of 3 does not regress Bug 837051 and Bug 864230.

(In reply to Diego Wilson [:diego] from comment #81) > With GetAmpleVideoFrames() of 1 a 30 FPS VGA local clip plays back at 20 > FPS. > > With GetAmpleVideoFrames() of 3 the same 30 FPS VGA local clip plays at 30 > FPS. Thanks for the confirmation.

By the patch, youtube video can be played without freeze. The patch needs attachment 744257 [details] [diff] [review] in Bug 864230 for not to regress Bug 837051.

Comment on attachment 745702 [details] [diff] [review] patch - increment GetAmpleVideoFrames() to 3 doublec, can you review the patch?

FYI, I wrote a diagram around ChannelMediaResource. https://github.com/sotaroikeda/firefox-diagrams/blob/master/media/content_media_ChannelMediaResource_FirefoxOS_1_01.pdf?raw=true

Committable patch. Carry "chris.double: review+".

https://hg.mozilla.org/projects/birch/rev/a8593b74704a ==> if GetAmpleVideoFrames()returns 3 then we see video does not play at all for mpeg video. It works fine with h264 video. Please let me know if I need to upload media file here or if I should apply another patch with this patch.

(In reply to Tapas Kumar Kundu from comment #91) > https://hg.mozilla.org/projects/birch/rev/a8593b74704a ==> if > GetAmpleVideoFrames()returns 3 then we see video does not play at all for > mpeg video. It works fine with h264 video. > > Please let me know if I need to upload media file here or if I should apply > another patch with this patch. I observed this with local playback of mpeg file.

(In reply to Tapas Kumar Kundu from comment #91) > Please let me know if I need to upload media file here or if I should apply > another patch with this patch. Did you applied the patches in Bug 864230?

No. I will apply now.

It works with patches from Bug 864230 .

https://hg.mozilla.org/releases/mozilla-b2g18/rev/c73be780773d https://hg.mozilla.org/releases/mozilla-b2g18_v1_0_1/rev/c3a5041bfe5a

Issue still happens on Inari v1.1 Moz RIL Build ID: 20130509070205 Kernel Date: Feb 21 Gecko: http://hg.mozilla.org/releases/mozilla-b2g18/rev/84f4c17f1605 Gaia: 4e7d63a83508caa391c4db164c3f68422d9ca5b6 However we see improvements like Bug 867688 - "Youtube videos lose sync between audio and video" - no longer repro But every time you play YouTube video it stops after few min with green screen.

(In reply to Nick Dee from comment #98) > > But every time you play YouTube video it stops after few min with green > screen. Nick, which data channel do you use for data channel? WLAN?

To fix this problem, fix of bug 864230 is also necessary.

Sotaro, Nick, Regardless of bug 864230, I still see some problems with this video. I've seen the green screen when my wi-fi connection resets. Which I guess is more of a networking issue than a media issue. *And* I see that https://www.youtube.com/watch?v=V1IBH1bz5-c still eventually freezes. It takes a very long time (> 30 mins), though. I may just open a new bug for long youtube videos.

(In reply to Diego Wilson [:diego] from comment #101) > *And* I see that https://www.youtube.com/watch?v=V1IBH1bz5-c still > eventually freezes. It takes a very long time (> 30 mins), though. I may > just open a new bug for long youtube videos. Diego, Thanks for the info. Can you open a new bug?

Filed bug 870564. I think we're making progress :)

Yup, thanks.

refer to comment 98 for build info In reference to comment 101 I agree that green screen is something to do with Wi-Fi connection I tested YouTube via cellular data and have not experience green screen. I tested YouTube on Inari with cellular data with 2-3 Bars and video will freeze completely after 1-2 minutes and screen becomes unresponsive and the only way at this point to go back is to push home button and browser icon again. When I got 4 bars, direct visibility of cell site, YouTube videos were playing without any issues(5-7 min in length).

(In reply to Nick Davidson from comment #105) > refer to comment 98 for build info > > In reference to comment 101 I agree that green screen is something to do > with > Wi-Fi connection I tested YouTube via cellular data and have not experience > green screen. > I tested YouTube on Inari with cellular data with 2-3 Bars and video will > freeze completely after 1-2 minutes and screen becomes unresponsive and the > only way at this point to go back is to push home button and browser icon > again. I feel that it is better to be handled as different bug.

I feel the source of the problem of comment #105 could be same to Bug 870564.

Verified fixed. Videos will now play their full length with no errors on Inari 1.0.1 and 1.1 commercial RIL.

This patch will cause a new bug, Bug 873373. MP4 video will stuck.

(In reply to James Zhang from comment #109) > This patch will cause a new bug, Bug 873373. MP4 video will stuck. Did you apply Bug 864230?

I checked it on v1.0.1 buri device. I can not reproduce your problem without Bug 864230. -------------------------------------------------- commit 8792072743a6f9f918fcbaf75b52f5543bd54eec Author: Camilo Viecco <cviecco@mozilla.com> Date: Thu May 16 18:45:30 2013 -0700 Bug 841569 - Cannot install image-uploader (fix handling of large filanames on b2g apps). r=bsmith, a=tef+