The above is a WIP. I'm going to add a control script to automatically run through a bunch of frame rate/frame size variations.

Here is the output of this benchmark: Resolution Rate Backlog 176x144 5 1 176x144 10 1 176x144 15 1 176x144 20 1 176x144 25 1 176x144 30 2 176x144 40 482 320x240 5 1 320x240 10 1 320x240 15 1 320x240 20 2 320x240 25 2 320x240 30 3 320x240 40 501 640x480 5 1 640x480 10 2 640x480 15 2 640x480 20 2 640x480 25 4 640x480 30 48 1280x720 5 190

c4 refers to a peak.

You will also need a YUV video of around 30-60 seconds (the benchmark loops). I captured mine by using quicktime and then mplayer, i.e., mplayer -nosound -endpos 30 -vo yuv4mpeg /mnt/hgfs/ubuntu12-64-2/EKR-talking.mov yuvscaler can be found in the mjpegtools pkg.

Here is a different set of benchmarks run with the following Google clip: http://downloads.webmproject.org/ietf_tests/niklas_1280_720_30.yuv.xz Note the big difference at 640x480. Resolution Rate Backlog 176x144 5 1 176x144 10 1 176x144 15 1 176x144 20 1 176x144 25 2 176x144 30 2 176x144 40 483 320x240 5 1 320x240 10 1 320x240 15 1 320x240 20 2 320x240 25 2 320x240 30 2 320x240 40 504 640x480 5 163 1280x720 5 261

Just looking at top, this doesn't appear to be overloaded. More investigation needed.

Command to convert from YUVMPEG to YUV format (from Ekr): mplayer -demuxer rawvideo -rawvideo w=1280:h=720:format=i420 niklas_1280_720_30.yuv -vo yuv4mpeg Output will be ins stream.yuv

Comment on attachment 772439 [details] [diff] [review] Extend Conduit with CodecObserver to dump Incoming and Outgoing Framerates A patch to allow WebRTC Video Engine to report Incoming and Outgoing Framerate/Bitrate every second.

Based on the patch above, the following tests were ran with the benchmark. Test 1: Run http://downloads.webmproject.org/ietf_tests/niklas_1280_720_30.yuv.xz with VideoEngine Frmerate set to 60 Fps Observation: a) Outgoing FrameRate reported was at 29 Fps (This indicates setting maxFramerate on Engine didn't get reflected) b) Incoming framerate varied between 2 Fps to 6 Fps. c) WebRTC log reported Drop frames with a reason being frameRate. I don't have exact count on number of drop-frames due to log file wrap up. Test 2: Run http://downloads.webmproject.org/ietf_tests/niklas_1280_720_30.yuv.xz with VideoEngine Framerate set to 30 Fps Observation: a) Outgoing FrameRate reported was at 29 Fps (Again, this indicates setting maxFramerate on Engine didn't get reflected) b) Incoming framerate varied between 2 Fps to 27 Fps. c) WebRTC log reported Drop frames with a reason being bitRate. I don't have exact count on number of drop-frames due to log file wrap up. Test 3: Created a video using Quicktime worth 60 seconds and generated YUV format with resolution 176*244. Run the benchmark with this video file. This has VideoEngine Framerate set to 60 Fps Observation: a) Outgoing FrameRate reported was at 29 Fps (This indicates setting maxFramerate on Engine didn't get reflected) b) Incoming framerate varied between 2 Fps to 9 Fps. c) WebRTC log reported Drop frames with a reason being bitrate. I don't have exact count on number of drop-frames due to log file wrap up. Test 4: Same as Test 3 but with VideoEngine Framerate set to 30 Fps Observation: a) Outgoing FrameRate reported was at 18 - 29 Fps (This indicates setting maxFramerate on Engine didn't get reflected) b) Incoming framerate varied between 3 Fps to 11 Fps. c) WebRTC log reported Drop frames with a reason being bitrate. I don't have exact count on number of drop-frames due to log file wrap up. With this, more investigation is needed based on the clues from above.

The benchmark wants MPEG-wrapper YUV and not raw YUV. You can make one with mplayer like so: mplayer -demuxer rawvideo -rawvideo w=1280:h=720:format=i420 inputfile.yuv -vo yuv4mpeg Adjust the parameters as appropriate. This will stuff the output in stream.yuv

Hmm .. Interesting .. I had this flag turned off few days back and that didn't improve the backlogs observed on the Mac. I was seeing 2 places where the frames/packets were dropped. One that applies to encoder logic of dropping frame on onyxif.c and second one was with the frameDropper. In the earlier case dropping of the frame ends up in no encoded bitstream to be sent and hence framerate on the receiver was low. But the second one happens eventually where the leaky bucket adjusts its value so that the consecutive frames get dropped. Even this would affect the incoming framerate on the receiver.

Also the above were observed for 176 * 144 and 30 FPS. The results were worst for the 640 * 480 cases though. Probably my test setup is wrong or something on my side. Not sure either.

The prior two attachments are representative of my results with the Peak. How to reproduce, Use a fresh copy of M-C (tip is 141367:ad0ae007aa9e at time of comment) apply patch 'benchmark.patch' (this has everything needed with the exception of the video) build & flash B2G push video_benchmark, located in the dist/bin of your objdir run run-benchmark.py (matplotlib required for graph generation)

(In reply to Ben Brittain (:bbrittain) from comment #21) > Created attachment 786021 [details] [diff] [review] > benchmark.patch Can you please explain this attachment? Does this obsolete something else or add in some way?

Hi Ekr, The patch is derived from bbrittain's. I change the policy of benchmark. My version is try to estimate the maximum capability of the device. I change some code in VPMVideoDecimator and VideoConduit. This is because 1. the input fps may be more than 30 2. the encoded bitrate may exceed the default maximum Here is the output on unagi. Resolution@Rate FPS 176x144@60 55.411500 320x240@60 21.547800 640x480@60 6.538580 1280x720@60 2.595920

(In reply to StevenLee[:slee] from comment #25) > The patch is derived from bbrittain's. I change the policy of benchmark. My > version is try to estimate the maximum capability of the device. I change > some code in VPMVideoDecimator and VideoConduit. This is because > 1. the input fps may be more than 30 > 2. the encoded bitrate may exceed the default maximum Some more details about the patch. 1. The previous version controls the input fps and makes it as 5, 15, 30, etc. But my patch keeps inputting the frames whenever the previous frame is encoded. 2. When inputting a frame by SendVideoFrame, it just copies the frame, wakes up the encoder thread and returns. For making input-frame thread and encoder thread cooperate synchronously, I check marker bit in Transport::SendRtpPacket. When marker bit is true, it can notify the input-frame thread to insert the next frame. If not, some frames may be over-written because GIPS uses only 2 buffers.

Comment on attachment 8357025 [details] [diff] [review] video-benchmark.patch Review of attachment 8357025 [details] [diff] [review]: ----------------------------------------------------------------- Is this still relevant? If so, please re-request review.

Mass change P3->P4 to align with new Mozilla triage process.