Enable GStreamer in official builds for Linux.

I suspect we don't want to do this until we have a plan for getting support for all the platforms we ship desktop Firefox on. Just shipping h.264 support on Linux doesn't help us. (And it might hurt us, by having different codecs supported on different desktop Firefox builds.)

Does having different codecs in mobile and desktop already hurt? Case: User wants to view a video they have accessed on Firefox mobile and synced with Firefox Sync onto their desktop. One scenario they can play it, another they cannot, frustration ensues? In addition, Flash continues to be problematic for security, reliability and so on. Enabling genuine HTML5 video support (h.264) across all desktop and mobile builds would enable developers to take a genuine pro-HTML5 open web, plugin-free step they cannot easily take right now because Firefox is a substantial blocker for h.264 on a seemingly shrinking basis. Re XP lacking native h.264 support, what are the issues surrounding bundling or linking to the x264 codec to solve this issue?

I think we should enable GStreamer in Linux builds even if we don't support H.264 in the other platforms yet. Things to be resolved before we can do this though: 1) Get the mochitest media tests passing. There's a H.264 video file in the tests already. 2) Ensure Firefox still runs on Linux distributions that don't have GStreamer or have older versions. 3) Check that we have new enough GStreamer libraries on build machines.

(In reply to Chris Double (:doublec) from comment #3) > I think we should enable GStreamer in Linux builds even if we don't support > H.264 in the other platforms yet. Agreed. > Things to be resolved before we can do this though: > > 1) Get the mochitest media tests passing. There's a H.264 video file in the > tests already. > 2) Ensure Firefox still runs on Linux distributions that don't have > GStreamer or have older versions. > 3) Check that we have new enough GStreamer libraries on build machines. I'd be more than willing to help test at least every Debian-based distribution I can get my hands on as I have basic familiarity with the Advanced Packaging Tool (apt) but not really with rpm or other systems.

(In reply to Chris Double (:doublec) from comment #3) > I think we should enable GStreamer in Linux builds even if we don't support > H.264 in the other platforms yet. > > Things to be resolved before we can do this though: > > 1) Get the mochitest media tests passing. There's a H.264 video file in the > tests already. > 2) Ensure Firefox still runs on Linux distributions that don't have > GStreamer or have older versions. > 3) Check that we have new enough GStreamer libraries on build machines. 4) Check that when MP4, H.264 and AAC support is present, canPlayType() behaves as in other H.264 and AAC-supporting browsers. See bug 760140. 5) Test that when both H.264 and AAC decoders are absent, canPlayType() behaves like in Firefox today even if an MP4 demuxer is present. 6) Test that MPEG4 Visual, MPEG2 or other non-H.264, non-AAC encumbered codecs do not get exposed to the Web as a side effect. #4 and #5 are important for avoiding poisoning canPlayType(). #6 is important to take care of before enablement, because it’s hard to take features away even if the features were unintentional.

I doubt this is possible to do for official builds. Distro builds could do it, but official will likely have a hard time maintaining binary compatibility with different versions of the system libraries. Look at the ABI changes between 0.10 and 1.0. http://upstream-tracker.org/versions/gstreamer.html

and no, embedding gstreamer like we embed other third party libraries won't help. The only way it would work would be to also embed the mp4 decoder, and that's opening a legal can of worms.

(In reply to Mike Hommey [:glandium] from comment #7) > and no, embedding gstreamer like we embed other third party libraries won't > help. The only way it would work would be to also embed the mp4 decoder, and > that's opening a legal can of worms. Perhaps we're missing the point here. I realize this specific bug focuses on GStreamer but AFAIK the broader goal is (or perhaps should be?) enabling HTML5 <video> support for the most common format(s) on all platforms that Firefox supports. Is it possible that this goal would be more easily achieved if Firefox were to use the native media frameworks for each platform, i.e. DirectShow, GStreamer and Quicktime, as Firefox mobile does with Stagefright? Would utilizing these frameworks possibly be less work than targeting a single framework that perhaps changes so often because it is trying to be everything to everybody? On another note, has anybody engaged the GStreamer people and asked them if they are willing to work with Mozilla by maintaining a branch of their code that preserves binary compatibility for longer than they historically have to date? If not glacial pack ice frozen APIs, perhaps ice cube APIs capable of thawing every summer or so? :) There could be a ceremonial GStreamer/Mozilla ice cooler that API documents get thrown into, a bit like a time capsule :) Sorry, just a bit of mirth there! In addition, doesn't any code base change a lot more frequently whilst getting to version 1 than it is likely to after that point? Perhaps then Mr Hommey's example in comment 6 is not necessarily representative of future development trends? Apologies if these points are less technical and more general than is ideal. However this bug thus far has seemed to evolve in more of a discussion manner than other bugs might normally so I hope it's ok to continue that style for the moment at least.

(In reply to Mike Hommey [:glandium] from comment #6) > I doubt this is possible to do for official builds. Distro builds could do > it, but official will likely have a hard time maintaining binary > compatibility with different versions of the system libraries. Look at the > ABI changes between 0.10 and 1.0. > http://upstream-tracker.org/versions/gstreamer.html I took the topic to dev-planning since Bugzilla isn’t a discussion forum: https://groups.google.com/d/topic/mozilla.dev.planning/aYOuhdkKNXo/discussion

Just to note that we are IMHO getting back to bug 422540, bug 435298, and bug 435339, which IMHO was a mistake to WONTFIX.

(In reply to Matej Cepl from comment #10) > Just to note that we are IMHO getting back to bug 422540, bug 435298, and > bug 435339, which IMHO was a mistake to WONTFIX. It was WONTFIX according to the policies and direction at the time. How (and if) support of H.264 across platforms is eventually done is yet to be decided and it may be that those approaches are used. I'd rather leave general H.264 support out of this bug though and concentrate on what needs to be done to get GStreamer enabled - if it's even a good idea based on Mike Hommey's comments. Is it better just to leave it up to distro's to enable? Should we change the GStreamer backend to use the MPAPI plugin approach that Android's media support uses to enable dynamic loading of it?

gladium makes a good point. I think it's clear that if we enabled the gstreamer backend in official builds, we'd want to include our own build. But that doesn't help if it's not abi compatible with the elements we want to use on the system. I just tried loading the fluendo binary elements for gstreamer-0.10 with today's git (gstreamer 1.0.0) and it wasn't able to find the entry points: WARN GST_PLUGIN_LOADING gstplugin.c:383:gst_plugin_register_func: plugin "/home/giles/gstreamer/fluendo-bin/libgstflump3dec.so" has incompatible version, not loading So while I thought it was clear official biulds would include their own builds of gstreamer if we enabled this back end, that doesn't help if we can't load elements from the system gstreamer install. We could still pick a target version and only support that, which is easier to do now that gstreamer 1.0 is released. That might be ok if we could fall back to our built-in decoders when the requested elements aren't available.

http://www.collabora.com/about/gstreamer-interview/ Looks like simple usages of GStreamer shouldn't be broken (but http://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/random/porting-to-1.0.txt is pretty long...). Maybe we can load GStreamer dynamically and, if we use only a subset of the API, we could avoid running into issues (just like we're now doing with libnotify). AFAIK, there will be other API breakages in the future.

Keep in mind that calling system gstreamer libraries means feeding data from the network into code we have no way to issue security fixes for. Being able to protect our users independent of distro updates is why the official builds include there own copy of even ubiquitous libraries like zlib.

some related discussion in the distros, http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=682917 https://bugzilla.redhat.com/show_bug.cgi?id=843583

(In reply to Ralph Giles (:rillian) from comment #14) > Keep in mind that calling system gstreamer libraries means feeding data from > the network into code we have no way to issue security fixes for. We are already on track to doing that on Android, and the Android security update situation is worse than the security update situation on various desktop Linux distros. As in: It’s worse to the point of not having system updates—security or other—in most cases. (In reply to sam tygier from comment #15) > some related discussion in the distros, > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=682917 > https://bugzilla.redhat.com/show_bug.cgi?id=843583 I think it would be unwise for distros to turn GStreamer on before bug 760140 is fixed.

Additionally, the mochitests fail with the GStreamer backend enabled. Currently the GStreamer backend takes over playback of Ogg and WebM resources when it's enabled, and Firefox times out trying to run test_buffered because GStreamer can't handle playing Opus in Ogg. test_a4_tone also fails. If I change our code to use our existing Ogg/WebM backends for Ogg/WebM when the GStreamer backend is enabled (so GStreamer only plays H.264, not Ogg or WebM), the test run completes but with test failures and a shutdown hang with the decode thread stuck in nsGStreamerReader::WaitForDecodedData(). The unexpected test failures from the mochitest run's log are attached. Looks like mostly problems with duration calculation.

(In reply to Henri Sivonen (:hsivonen) from comment #16) > We are already on track to doing that on Android, and the Android security > update situation is worse than the security update situation on various > desktop Linux distros. As in: It’s worse to the point of not having system > updates—security or other—in most cases. This is a good point of perspective, although relinquishing control of code on one platform does not make it ok to do so on another. Perhaps we should add some infrastructure to let us blacklist particular gstreamer elements, the way we do with plugins? (In reply to Chris Pearce (:cpearce) from comment #17) > Additionally, the mochitests fail with the GStreamer backend enabled. > Currently the GStreamer backend takes over playback of Ogg and WebM > resources when it's enabled, and Firefox times out trying to run > test_buffered because GStreamer can't handle playing Opus in Ogg. Do the mochitests pass if you update your gstreamer to support Opus? I'm wondering if the canPlayType filters can be improved to handle the lack of Opus support, the way mp4 files pass outside the gstreamer build. Is this just because of bug 760140 or are there occasions were we need to verify probably and not just truthiness? I wrote the opus tests with the expectation that they should fail if opus support isn't available, but apparently I missed some more permissive tests. :)

We talked about this in Auckland, and we'd prefer to use our existing built-in backends for Ogg and WebM, since they've been fuzz tested and we know they're reliable. I'm going to change our decoder creation code in bug 799344 to not use GStreamer for Ogg and WebM. (And my GStreamer is whatever is the current up-to-date version in Ubuntu 12.04. I don't think we should rely on users having a newer version.)

(In reply to Chris Pearce (:cpearce) from comment #19) > I'm going to change our decoder creation code in bug 799344 to not use > GStreamer for Ogg and WebM. Works for me. Thanks! > (And my GStreamer is whatever is the current up-to-date version in Ubuntu > 12.04. I don't think we should rely on users having a newer version.) I don't think we should either. I was just trying to understand the origin of the failure you described.

(In reply to Ralph Giles (:rillian) from comment #18) > Perhaps we should add some infrastructure to let us blacklist particular > gstreamer elements, the way we do with plugins? I think it’s more important to make really, really sure that the only GStreamer things that are whitelisted are: * MP4 demuxer * H.264 decoder * (LC-)AAC decoder * MP3 decoder only in the context of .mp3 container I’d leave Firefox agnostic to which specific implementations of these live on the other side of GStreamer.

Maybe it was already discussed, but Mozilla was not interested in supporting only free codecs? Why the change to support also (software that can enable play of) patented codecs?

(In reply to Fabio from comment #22) > Maybe it was already discussed, but Mozilla was not interested in supporting > only free codecs? Why the change to support also (software that can enable > play of) patented codecs? https://brendaneich.com/2012/03/video-mobile-and-the-open-web/

(In reply to Chris Pearce (:cpearce) from comment #19) > We talked about this in Auckland, and we'd prefer to use our existing > built-in backends for Ogg and WebM, since they've been fuzz tested and we > know they're reliable. Maybe I'm misunderstanding this, but are there outstanding exploits that Mozilla found that haven't made it upstream yet to Ogg, WebM? If it is gStreamer specific, but not the codecs, then all someone would need to do is use H264 to exploit it. > I'm going to change our decoder creation code in bug 799344 to not use > GStreamer for Ogg and WebM. One of the reasons I am interested in this bug is due to what seems to be much better video acceleration code in gStreamer vs Firefox. At least could it be be made an about:config option, pass all to gstreamer?

Having a pref to toggle on/off built-in codecs in addition to external codecs could be really useful at least for troubleshooting.

As per comment 24, GStreamer 1.2 (planned for december 2012) should provide hardware acceleration http://gstreamer.freedesktop.org/wiki/ReleasePlanning/RoadMap

(In reply to Bryan Quigley from comment #24) > (In reply to Chris Pearce (:cpearce) from comment #19) > > We talked about this in Auckland, and we'd prefer to use our existing > > built-in backends for Ogg and WebM, since they've been fuzz tested and we > > know they're reliable. > > Maybe I'm misunderstanding this, but are there outstanding exploits that > Mozilla found that haven't made it upstream yet to Ogg, WebM? If it is > gStreamer specific, but not the codecs, then all someone would need to do is > use H264 to exploit it. We apply several patches on top of our libvpx, and a couple on top of the Xiph Ogg libraries. I don't know if these have been fixed upstream yet. Possibly some or all of these patches have been upstreamed and our libraries are behind current-stable versions. This decision also motivated by the fact that we know our built in decoders work reliably, and we know that the GStreamer backend has some problems; it fails our unit tests, so we know there are bugs either in GStreamer or in our use of it. > > I'm going to change our decoder creation code in bug 799344 to not use > > GStreamer for Ogg and WebM. > > One of the reasons I am interested in this bug is due to what seems to be > much better video acceleration code in gStreamer vs Firefox. At least could > it be be made an about:config option, pass all to gstreamer? Sure, I will do this, it's a good idea.

(In reply to Chris Pearce (:cpearce) from comment #27) > We apply several patches on top of our libvpx, and a couple on top of the > Xiph Ogg libraries. I don't know if these have been fixed upstream yet. > Possibly some or all of these patches have been upstreamed and our libraries > are behind current-stable versions. For libvpx, I believe the only patch that isn't upstream or a backport of a commit that is upstream is the one dealing with the stdint.h types. For libogg, we have one Solaris-specific patch for inttypes.h that is not upstream, and that's it. For libvorbis, we have a Solaris-specific patch for alloca, a fix for bug 719612 (CVE-2012-0444), which is upstream as r18151, and a fix for bug 722924 (no CVE), which is upstream as r18166. Both of these were released in libvorbis-1.3.3. For Tremor, we have a similar fix for bug 719612, which is upstream in r18152. Tremor was not affected by bug 722924. Xiph.Org does not publish releases for Tremor. For libtheora, we currently carry four patches (bug 468275, bug 625773, bug 752139, and bug 752668), all of which are upstream (in r18219, r17780, r18031, and r18268, respectively). None of these bugs affect the most recent stable release (libtheora-1.1.1). We are actually using a more recent (unreleased) snapshot.

(In reply to Timothy B. Terriberry (:derf) from comment #28) > For libvpx, I believe the only patch that isn't upstream or a backport of a > commit that is upstream is the one dealing with the stdint.h types. Is there a plan to update libvpx, currently at 0.9.5? Latest releases have some decoder speed improvements.

(In reply to Fabio from comment #29) > Is there a plan to update libvpx, currently at 0.9.5? Latest releases have > some decoder speed improvements. media/libvpx is based on 1.0.0 (see bug 730907), but README_MOZILLA and some other bits haven't been updated to reflect that.

(In reply to Chris Pearce (:cpearce) from comment #27) > This decision also motivated by the fact that we know our built in decoders > work reliably, and we know that the GStreamer backend has some problems; it > fails our unit tests, so we know there are bugs either in GStreamer or in > our use of it. Some of the test failures with GStreamer were due to the test file having the wrong duration listed in its manifest, I fixed that in bug 803427. > > > I'm going to change our decoder creation code in bug 799344 to not use > > > GStreamer for Ogg and WebM. > > > > One of the reasons I am interested in this bug is due to what seems to be > > much better video acceleration code in gStreamer vs Firefox. At least could > > it be be made an about:config option, pass all to gstreamer? > > Sure, I will do this, it's a good idea. I'm doing this in bug 803287.

(In reply to Chris Pearce (:cpearce) from comment #31) > > > One of the reasons I am interested in this bug is due to what seems to be > > > much better video acceleration code in gStreamer vs Firefox. At least could > > > it be be made an about:config option, pass all to gstreamer? > > > > Sure, I will do this, it's a good idea. > > I'm doing this in bug 803287. Just landed this change. I also renamed the pref to enable gstreamer, and nsHTMLMediaElement's gstreamer related functions, from .*H264.* to .*GStreamer.*, since we'll have other backends which also provide H.264 playback on other platforms.

I would like to pick up from https://bugzilla.mozilla.org/show_bug.cgi?id=799315#c11 "Will GStreamer support be added to aurora/beta? Don't know when. We need to figure out a binary compatibility strategy. Best to take up that cause in a post to mozilla.dev.media or in the bugs Paul mentioned." Any chance to make testing for early adapters easier? Maybe compile with --enable-gstreamer by default, but hide behind a pref? Are there any test builds, maybe on try-builds?

(In reply to Markus Popp from comment #33) > Any chance to make testing for early adapters easier? Maybe compile with > --enable-gstreamer by default, but hide behind a pref? Are there any test > builds, maybe on try-builds? The problem is library compatibility amongst different GStreamer versions and distros. If we turn it on then Firefox won't run on linux distro's that don't have GStreamer or have an incompatible version. We need to look at runtime loading of the shared libraries or some other approach. See the thread "Use cases for official generic “Linux” builds vs. ABI-unstable system libraries" in mozilla.dev.planning that occurred around the end of September.

This is probably going to be a royal pain to implement, but alas it is a necessary evil. I despise the fact that Firefox is going to have to implement proprietary codecs to stay relevant, but it's better than having 90% of the videos online not working. As a user and bug tester, this is indeed a sad day for us. This implementation is especially important since Adobe only supports Flash for Linux in Pepper form currently. Last thing I want is to have to reboot into Windows for yet another thing. I love Linux (any distro) but cannot sacrifice usability for ideology. Trust me, this will be worth it. We are getting a second rate experience, thanks Adobe! Oh well. Can't win all the battles out there but the war for freedom rages on. Just thought I'd deliver the non-typical power user opinion on this. Sorry if I got off topic. But the end user would wonder (on any platform) why videos wouldn't work on Firefox if we didn't support this, and they would likely dump Firefox for a proprietary browser such as Google Chrome. Best we stay relevant. Just imagine the consequences on Linux, where a lot of the brainpower is. Not to insult any other platform's users, but many Linux users are developers and users. Many Windows and Mac users are just users, with a few developers here and there. Any implementation to help us stay relevant is deeply appreciated. End rant.

Bug 836243 changed the minimum required gstreamer version from 0.10.33 (2011-5-10) to 0.10.25 (2009-9-25). For Ubuntu, this means the minimum supported LTS release was changed from 12.04 to 10.04. The previous LTS, 8.04, is no longer supported on desktop and server support will be dropped in a couple months. Debian 6.0 (stable) is also now supported. Fedora 11 also now is supported (if I'm reading their site right). Two questions: 1) How many people are on distros older than these and using the official build? 2) Does Mozilla actually want to support any distros older than these in the official build? It looks like it might be practical to just set a minimum gstreamer version requirement of 0.10.25 for the official build and drop support for old no longer developer supported OSes. It's may not be worth it to maintain support for Linux users who haven't updated in over 3 years. A separate build with disabled gstreamer support could be put up somewhere for them if needed. Of course, if runtime loading (comment 34) is easily doable then that would be great, but dropping support for particularly old distros and actually getting it turned on would be better than not doing anything. Windows is set to get h.264 support ahead of Linux, which is not feeling very Tier-1-y at the moment.

Something else to note is that this is using gstreamer 0.10 which is now out of date. This is preventing some distros from enabling it by default (see arch bug https://bugs.archlinux.org/task/33009). It would probably be wise to upgrade to using the gstreamer 1.0 release.

gstreamer 1.0 support is already being worked on in bug 806917.

Just to mention that I set the minimum gstreamer version requirement to 0.10.25 (as per bug 836243) as I didn't have older(than Fremantle :) ) distros around to test with. I am almost sure this could be relaxed even more, for example the minimum version could be set to the first gstreamer version playbin2 appears in.

Based on the existing system requirements Firefox already has a hard minimum requirement of GTK+ 2.10 which was released in 2006 (I think), and it looks like it has been this way since Firefox 3. (see bug 418885 complaining about this, which someone should probably close out at this point) It looks like Ubuntu 6.06 LTS doesn't meet this, so 8.04 LTS looks like our current minimum Ubuntu LTS version. (please do correct me if I'm wrong here) That had gstreamer 0.10.18, which looks to have playbin2 (I think). My guess is that if gstreamer support 0.10.18 were added, and you could successfully run Firefox under Ubuntu 8.04 LTS (and thus probably other distros of that time), then gstreamer could be turned on by default without practically increasing any minimum system requirements beyond requiring users of some KDE-based distros to install gstreamer (which is often needed in current distros anyway).

Now that I look at it more, though, it looks like the current route to gstreamer 1.0 support in bug 806917 is to be able to build to support 0.10 or 1.0, not both. So unless there's some way this could be changed to allow the same build to run in either, it looks like once gstreamer 1.0 support is in common use we'd be unable to support anything less unless two libraries were shipped and dynamically loaded based on detected gstreamer version.

Well according to https://www.mozilla.org/en-US/firefox/19.0/system-requirements/ the actual GTK requirements are at 2.18 or higher. Which does already push us to Ubuntu 10.04+, Debian Squeeze+, and to RHEL 6+.

It depends if you are talking about runtime requirements or build-time requirements.

(In reply to Bryan Quigley from comment #42) > Well according to > https://www.mozilla.org/en-US/firefox/19.0/system-requirements/ the actual > GTK requirements are at 2.18 or higher. Ah, I just looked at: https://www.mozilla.org/en-US/firefox/system-requirements.html Which looks like it's actually the Firefox 14 system requirements. That's odd. Someone should probably fix that. :/ > Which does already push us to Ubuntu 10.04+, Debian Squeeze+, and to RHEL 6+. Ok, forget about comment 40 then. If this is true then gstreamer could be enabled right now without practically impacting the stated minimum requirements. Might I suggest turning this on for Trunk builds for now and backing out for Aurora if no good solution for future distros that only have 1.0 available is found?

(In reply to Dave Garrett from comment #41) > Now that I look at it more, though, it looks like the current route to > gstreamer 1.0 support in bug 806917 is to be able to build to support 0.10 > or 1.0, not both. So unless there's some way this could be changed to allow > the same build to run in either, it looks like once gstreamer 1.0 support is > in common use we'd be unable to support anything less unless two libraries > were shipped and dynamically loaded based on detected gstreamer version. According to some people knowing more about GStreamer than I do it's pretty problematic to have 0.10 and 1.0 gstreamer mapped into one process as there are clashes in the implementation and you could expect crashes at runtime. On Linux distributions it's pretty easy to get different GStreamer in one process. For example libcanberra loads GStreamer itself usually and is also loaded into Firefox.

Sorry I'm late to the discussion. There's something that hasn't been said that I want to clarify: GStreamer does take ABI compatibility very seriously. 0.10 was released in 2005 and we broke ABI compatibility for the first time now with 1.0. When we do break ABI compatibility, we make sure that older and newer versions are parallel-installable. This was the case for 0.8 vs 0.10 and it's the case today with 0.10 vs 1.0. There's a patch in https://bugzilla.mozilla.org/show_bug.cgi?id=806917 to support GStreamer 1.0. I'm going to review it in the next few days and try to get it landed. Keeping compile time compatibility with 0.10 and 1.0 should be relatively easy. If people think that runtime compatibility with both 0.10 and 1.0 is essential, I can look into that too. We could dinamically probe and load GStreamer or even have two distinct backends using MPAPI. About the security concerns, the upstream gst community is fairly receptive to any kind of patches, and there's some overlap with debian/ubuntu/fedora developers, so those distro usually get the latest and greatest gst fairly quickly. In addition to that, we could potentially consider importing the ogg/webm/h264 gst plugins in m-c so there's even more control over those.

(In reply to Alessandro Decina from comment #46) > In addition to that, we could potentially consider importing the > ogg/webm/h264 gst plugins in m-c so there's even more control over those. I'd rather keep our existing backends/libraries for decoding Ogg and WebM unless there's a compelling reason to switch to using GStreamer for those formats. Our Ogg and WebM libraries have been fuzzed pretty hard over the years, and our backends using those libraries have been well tested and are known to be robust and work well. We don't really want to redo all that "robustification" work for Ogg and WebM, we won't gain much by doing that now. Keeping our existing libraries also ensures Firefox's media playback behaviour is more likely to remain consistent across all platforms.

(In reply to Chris Pearce (:cpearce) from comment #47) > > I'd rather keep our existing backends/libraries for decoding Ogg and WebM > unless there's a compelling reason to switch to using GStreamer for those > formats. I am not too familiar with the current backends but there is a good chance that gstreamer would be more performant and reliable. I know that gstreamer has support for accelerated video playback on many GPU drivers. I'm not saying that gstreamer should be the default but it would be nice to have it as an option.

You can already set the pref "media.prefer-gstreamer" to true if you want to play WebM and Ogg with GStreamer in Firefox.

(In reply to Chris Pearce (:cpearce) from comment #49) > You can already set the pref "media.prefer-gstreamer" to true if you want to > play WebM and Ogg with GStreamer in Firefox. Sorry, I misunderstood you. I thought you were considering getting rid of this. My mistake.

(In reply to Alessandro Decina from comment #46) > In addition to that, we could potentially consider importing the > ogg/webm/h264 gst plugins in m-c so there's even more control over those. If this means we'd be shipping an H.264 decoder, we can't do that.

(In reply to Chris Double (:doublec) from comment #51) > (In reply to Alessandro Decina from comment #46) > > In addition to that, we could potentially consider importing the > > ogg/webm/h264 gst plugins in m-c so there's even more control over those. > > If this means we'd be shipping an H.264 decoder, we can't do that. Yeah sorry for not being more explicit. I meant only the gst side of the plugin, which is the part that wraps the actual codec implementation and just does setup, makes sure input isn't grossly malformed etc.

(In reply to Chris Pearce (:cpearce) from comment #47) > (In reply to Alessandro Decina from comment #46) > > In addition to that, we could potentially consider importing the > > ogg/webm/h264 gst plugins in m-c so there's even more control over those. > > I'd rather keep our existing backends/libraries for decoding Ogg and WebM > unless there's a compelling reason to switch to using GStreamer for those > formats. The only possible advantage I can see in using GStreamer for webm is performance on mobile platforms. I have seen ARM vendors spend time optimizing webm with their own codecs and some have announced webm hw support. GStreamer has pretty good support for vendor provided codecs in android/linux platforms. I do see your point though and realize that what I said above can be handled on a case by case basis with the media.prefer-gstreamer pref. > Our Ogg and WebM libraries have been fuzzed pretty hard over the years, and > our backends using those libraries have been well tested and are known to be > robust and work well. We don't really want to redo all that > "robustification" work for Ogg and WebM, we won't gain much by doing that > now. Keeping our existing libraries also ensures Firefox's media playback > behaviour is more likely to remain consistent across all platforms. This makes sense. FWIW I saw that the test suite includes some half broken/forged files. Last time I checked the gst backend did pretty well with those, with the exception of one theora file that caused a crash for which I have a patch laying somewhere.

I built latest trunk on my Fedora box this morning with --enable-gstreamer, and I still get the following test failure in content/media/test/test_buffered.html: TEST-UNEXPECTED-FAIL| unknown test url | owl.mp3: First range end should be media end - got 3.368, expected 3.3698 (In reply to Henri Sivonen (:hsivonen) from comment #5) > (In reply to Chris Double (:doublec) from comment #3) > > I think we should enable GStreamer in Linux builds even if we don't support > > H.264 in the other platforms yet. > > > > Things to be resolved before we can do this though: > > > > 1) Get the mochitest media tests passing. There's a H.264 video file in the > > tests already. Almost there. > > 2) Ensure Firefox still runs on Linux distributions that don't have > > GStreamer or have older versions. I haven't done this. > > 3) Check that we have new enough GStreamer libraries on build machines. We don't have any GStreamer libs on our build machines. We should get our regression tests running, even if we're not shipping builds with GStreamer enabled ourselves. > 4) Check that when MP4, H.264 and AAC support is present, canPlayType() > behaves as in other H.264 and AAC-supporting browsers. See bug 760140. This is tested by our mochitest content/media/test/test_can_play_type_mpeg.html I checked and this test is passing, so we're reporting that we can at least all the things that we should. > 5) Test that when both H.264 and AAC decoders are absent, canPlayType() > behaves like in Firefox today even if an MP4 demuxer is present. I tested this, I removed by gstreamer plugins and test_can_play_type_mpeg.html fails across the board. > 6) Test that MPEG4 Visual, MPEG2 or other non-H.264, non-AAC encumbered > codecs do not get exposed to the Web as a side effect. I'm not sure that this is the case. I see that 3gpp is listed in GStreamerFormatHelper::mCodecs, we don't want that being exposed in canPlayType or for it to be playable. It's also doesn't look like the GStreamer backend is whitelisting only the codecs that we want to support though. We'd want to only support H264, AAC and MP3, so as to prevent further format proliferation. For example this video plays with GStreamer, but shouldn't: http://www.tools4movies.com/dvd_catalyst_profile_samples/Harold%20Kumar%203%20Christmas%20bionic%20fast.mp4 The video stream here is "MPEG-4 part 2". No other browsers play that file; we can but refuse to in the Windows Media Foudation backend (bug 839055). And we should also refuse to play the file if either the audio or video stream are present but of unsupported formats as well... Does our GStreamer backend already do this?

This patch makes the gst backend stop in ReadMetadata if the stream being decoded includes unsupported codecs. I'm not sure it's stopping in the right place, it makes playing http://www.tools4movies.com/dvd_catalyst_profile_samples/Harold%20Kumar%203%20Christmas%20bionic%20fast.mp4 fail with a "video has stopped playing since the file is corrupt" message.

(In reply to Chris Pearce (:cpearce) from comment #54) > I built latest trunk on my Fedora box this morning with --enable-gstreamer, > and I still get the following test failure in > content/media/test/test_buffered.html: > > TEST-UNEXPECTED-FAIL| unknown test url | owl.mp3: First range end should be > media end - got 3.368, expected 3.3698 I will look into this, but isn't the resolution we're checking a bit too high? Do all the backends report exactly that duration or is it just that the test written against a specific backend? > (In reply to Henri Sivonen (:hsivonen) from comment #5) > > (In reply to Chris Double (:doublec) from comment #3) > > > 5) Test that when both H.264 and AAC decoders are absent, canPlayType() > > behaves like in Firefox today even if an MP4 demuxer is present. > > I tested this, I removed by gstreamer plugins and > test_can_play_type_mpeg.html fails across the board. Is it failing in the expected way? > > 6) Test that MPEG4 Visual, MPEG2 or other non-H.264, non-AAC encumbered > > codecs do not get exposed to the Web as a side effect. > > I'm not sure that this is the case. I see that 3gpp is listed in > GStreamerFormatHelper::mCodecs, we don't want that being exposed in > canPlayType or for it to be playable. Yes 3gpp was copied there from the list of codecs supported by GONK i think. I'll remove it. > It's also doesn't look like the GStreamer backend is whitelisting only the > codecs that we want to support though. We'd want to only support H264, AAC > and MP3, so as to prevent further format proliferation. > > For example this video plays with GStreamer, but shouldn't: > http://www.tools4movies.com/dvd_catalyst_profile_samples/ > Harold%20Kumar%203%20Christmas%20bionic%20fast.mp4 > > The video stream here is "MPEG-4 part 2". No other browsers play that file; > we can but refuse to in the Windows Media Foudation backend (bug 839055). > > And we should also refuse to play the file if either the audio or video > stream are present but of unsupported formats as well... Does our GStreamer > backend already do this? Attachment #72711 [details] [diff] does this, although i'm not sure the behaviour is 100% right.

(In reply to Alessandro Decina from comment #56) > (In reply to Chris Pearce (:cpearce) from comment #54) > > I built latest trunk on my Fedora box this morning with --enable-gstreamer, > > and I still get the following test failure in > > content/media/test/test_buffered.html: > > > > TEST-UNEXPECTED-FAIL| unknown test url | owl.mp3: First range end should be > > media end - got 3.368, expected 3.3698 > > I will look into this, but isn't the resolution we're checking a bit too > high? Some consumers of the audio content may rely on the duration being accurate, like web apps using the decoded file via WebAudio or MediaStreams API for example. > Do all the backends report exactly that duration or is it just that > the test written against a specific backend? Hmm... The Windows Media Foundation backend is reporting a duration of 3.343673 or that file, and content/media/test/manifest.js lists its duration as 3.29. :{ Anyway, the test that's failing is: http://mxr.mozilla.org/mozilla-central/source/content/media/test/test_buffered.html?force=1#34 This is checking if video.buffered.end(0)==video.duration I think you can make this test pass by changing GStreamerReader::GetBuffered() to cap the range end time to the media duration here: http://mxr.mozilla.org/mozilla-central/source/content/media/gstreamer/GStreamerReader.cpp#601 I see that GStreamerReader::GetBuffered() is using gst_element_query_convert()... Does gst_element_query_convert() just do a byteoffset/length*duration estimation, or is it using some deeper understanding of the stream in order to get more accurate byte-to-timestamp mappings? If gst_element_query_convert() is just doing an estimation, you could just use GetEstimatedBufferedTimeRanges() [http://mxr.mozilla.org/mozilla-central/source/content/media/VideoUtils.h#142] instead, it's used in other backends already to estimate the buffered ranges, and caps the end time of the buffered ranges to the duration. > > I tested this, I removed by gstreamer plugins and > > test_can_play_type_mpeg.html fails across the board. > > Is it failing in the expected way? Yep. > Yes 3gpp was copied there from the list of codecs supported by GONK i think. > I'll remove it. Great, thanks! > > And we should also refuse to play the file if either the audio or video > > stream are present but of unsupported formats as well... Does our GStreamer > > backend already do this? > > Attachment #72711 [details] [diff] [diff] does this, although i'm not sure the > behaviour is 100% right. Thanks! I will test this patch's behaviour.

(In reply to Alessandro Decina from comment #55) > Created attachment 727111 [details] > gst codec whitelist > > This patch makes the gst backend stop in ReadMetadata if the stream being > decoded includes unsupported codecs. I'm not sure it's stopping in the right > place, it makes playing > http://www.tools4movies.com/dvd_catalyst_profile_samples/ > Harold%20Kumar%203%20Christmas%20bionic%20fast.mp4 fail with a "video has > stopped playing since the file is corrupt" message. This is the same behaviour that we have on Windows.

Comment on attachment 727111 [details] gst codec whitelist Punting to Chris Pearce since he seems to be following what behaviour should be here.

Comment on attachment 727111 [details] gst codec whitelist That looks fine, it behaves as expected and stops playback of the 3GPP container in particular. Mochitests timeout in test_buffered for me now though, and the fix I suggested of clamping the buffered ranges at duration didn't seem to fix the test_buffered failure either. :{

I created https://bugzilla.mozilla.org/show_bug.cgi?id=853306 to track and land the whitelist patch. Looking at the test_buffered failure next.

A fix for the test_buffered failure is in https://bugzilla.mozilla.org/show_bug.cgi?id=853325

Green on try: https://tbpl.mozilla.org/?tree=Try&rev=5dc42fa92b2d Some red due to buildbot config; just waiting on releng to get that fixed up and we're golden.

Comment on attachment 763014 [details] [diff] [review] Enable gstreamer by default, preffed off Review of attachment 763014 [details] [diff] [review]: ----------------------------------------------------------------- r=cpearce for the changes other than those in configure.in. I'll leave that to khuey. ::: configure.in @@ +5754,5 @@ > dnl ======================================================== > dnl = Enable GStreamer > dnl ======================================================== > + > +if test "$OS_TARGET" = "Linux"; then Doesn't this mean you're turning on building GStreamer on Linux, but not providing a way to disable it since you only have an --enable-gstreamer option? You should have way to disable gstreamer support, for Linux systems without GStreamer, or those who choose to not have it. ::: content/media/DecoderTraits.cpp @@ +326,5 @@ > result = CANPLAY_YES; > } > #endif > #ifdef MOZ_GSTREAMER > + if (MediaDecoder::IsGStreamerEnabled() && Put the MediaDecoder::IsGStreamerEnabled() check inside GStreamerDecoder::CanHandleMediaType(). It can't handle anything if it's disabled right? ::: content/media/test/manifest.js @@ +680,3 @@ > var oldOpus = undefined; > try { > + oldGStreamer = branch.getBoolPref("gstreamer.enabled"); IIRC getBoolPref throws if a pref isn't found. Meaning that we won't check the defaults for the other prefs, if say gstreamer.enabled isn't defined on the platform the test is running on (i.e. Windows). Please put each of these get*Pref calls inside their own try{}catch(){} block.

Addressed comment 64 and moved flag to build on desktop firefox only for now (i.e. not desktop b2g yet) since those build slaves don't yet have the right gstreamer packages.

All green: https://tbpl.mozilla.org/?tree=Try&rev=1539685e2272

Comment on attachment 763014 [details] [diff] [review] Enable gstreamer by default, preffed off Review of attachment 763014 [details] [diff] [review]: ----------------------------------------------------------------- ::: configure.in @@ +5762,4 @@ > MOZ_ARG_ENABLE_BOOL(gstreamer, > [ --enable-gstreamer Enable GStreamer support], > MOZ_GSTREAMER=1, > MOZ_GSTREAMER=) Chris, this should override the earlier Linux default if --disable-gstreamer is passed to configure. I suggest updating the help string to list the --disable variant instead since the default has changed for most users. If you wanted to be clever you should change the help string based on the current value of MOZ_STREAMER.

Excellent, thanks Ralph.

Backed out because of mochitest-1 timeouts on Linux: https://hg.mozilla.org/integration/mozilla-inbound/rev/2e1acd3b9ce8

A polite, friendly error message, with a clear call to action.

Comment on attachment 765205 [details] [diff] [review] Better error message Review of attachment 765205 [details] [diff] [review]: ----------------------------------------------------------------- A build peer should review this.

Comment on attachment 765640 [details] [diff] [review] Add gstreamer packages to mach bootstrap scripts Review of attachment 765640 [details] [diff] [review]: ----------------------------------------------------------------- Nice!

Comment on attachment 765205 [details] [diff] [review] Better error message Make it a complete sentence. "gstreamer and gstreamer-plugins-base development packages are needed to build gstreamer backend"

https://hg.mozilla.org/integration/mozilla-inbound/rev/0f5225fd5622 https://hg.mozilla.org/integration/mozilla-inbound/rev/7426038bcac2 https://hg.mozilla.org/integration/mozilla-inbound/rev/dda6f2aa5225

https://hg.mozilla.org/mozilla-central/rev/0f5225fd5622 https://hg.mozilla.org/mozilla-central/rev/7426038bcac2 https://hg.mozilla.org/mozilla-central/rev/dda6f2aa5225

Thanks, guys!

I discovered a Geolocation issue which appeared right after landing of GStreamer being enabled, reported it as bug 886138. Can't rule out a coincidence, but it coincides so closely with this that some connection might be likely.

VERIFIED FIXED Clicking on a link which URL goes directly to an MPEG4 file (mimetype video/mp4) plays the video. Clicking on an MPEG2 file (mimetype video/mpeg) does not work, it shows the download "Save as..." dialog. This is unfortunate, but unfortunately intended (comment 5).

Testcase (MP4 video): http://www.bucksch.org/xfer/montee.mpg

Correction: Testcase MPEG2 video, mimetype video/mpeg: http://www.bucksch.org/xfer/montee.mpg Testcase MPEG4 video, mimetype video/mp4: http://www.bucksch.org/xfer/walter-roehrl-short.mp4

(In reply to Ben Bucksch (:BenB) from comment #83) > Correction: > Testcase MPEG2 video, mimetype video/mpeg: > http://www.bucksch.org/xfer/montee.mpg > Testcase MPEG4 video, mimetype video/mp4: > http://www.bucksch.org/xfer/walter-roehrl-short.mp4 FWIW, I can not play these files. However, I have a very old video card - intel 915.

Multiplexing h264 streams in ts files is fairly common and recommended in many video streaming protocols. What is the rationale in not supporting it? Moreover is the decision final?

Agreed, MPEG TS is definitely needed.

In general we try to limit the number of supported formats. Web developer refusal to support webm compelled us to add support for h.264 in mp4. I haven't heard a similar argument for h.264 in mpeg-ts. Also, we're implementing the Media Source extensions (bug 778617) which make is straightforward to remux ts as mp4 in client-side script for playback in the <video> element. So this is a use case which can be addressed without native support. If you think native support for mpeg-ts is important, the following steps are the best way to further that goal: - Open a new bug specifically for it, rather than arguing either way in this bug. - Provide a factual argument why this is important. - Provide a patch implementing the new feature. I can't speak for the media peers, but I don't expect the support decision to change unless one of the rationales I mentioned above changes.

(In reply to Ralph Giles (:rillian) from comment #87) > - Open a new bug specifically for it, rather than arguing either way in this > bug. > - Provide a factual argument why this is important. > - Provide a patch implementing the new feature. Starting a thread on mozilla.dev.media would be good too, outlining why you think it's useful and important. For example, this thread was started to discuss VP9: https://groups.google.com/d/msg/mozilla.dev.media/iurA0-DJFoE/ObucKnJ3_ZQJ

> configure: error: gstreamer and gstreamer-plugins-base development packages are needed to build > gstreamer backend. Install them or disable gstreamer support with --disable-gstreamer Stupid question: Which Ubuntu packages do I need to install to get this to work? I flailed about and finally got it to build, but I'm not sure exactly what I needed to install. I think it's a subset of libgstreamer-plugins-base1.0-dev libgstreamer-plugins-base0.10-dev libgstreamer0.10-dev libgstreamer1.0-dev We should probably update the wiki with real package names for Ubuntu and other distros. https://developer.mozilla.org/en-US/docs/Developer_Guide/Build_Instructions/Linux_Prerequisites

I am using Ubuntu 12.10 but failed to build xpcshell test. I resolved it now and narrow down to two packages: libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev

@Chuck Lee, Justin Lebar: Ah, I think gstreamer1.0 was implemented so you should use gstreamer1.0 packages instead of the old gstreamer0.10 ones.

(In reply to nucrap from comment #91) > @Chuck Lee, Justin Lebar: Ah, I think gstreamer1.0 was implemented so you > should use gstreamer1.0 packages instead of the old gstreamer0.10 ones. I installed libgstreamer-plugins-base1.0-dev and libgstreamer1.0-dev, but failed to pass the check. Maybe its because the version check in configure.in is 0.10[1]. [1] https://mxr.mozilla.org/mozilla-central/source/configure.in#5780

Oh, sorry yeah it seems that gstreamer1.0 support is a work in progress: https://bugzilla.mozilla.org/show_bug.cgi?id=806917

Interestingly there is a bug common to GNU/Linux & Firefox OS concerning Vimeo. See Bug 884558 for the case and its workaround

To enable H.264 in Debian Firefox 24/25 (Iceweasel) build you must install apt-get install gstreamer0.10-plugins-good gstreamer0.10-ffmpeg and enable gstream support in about:config "media.gstreamer.enabled" according to http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=682917

Using gstreamer on 24/25 will exhibit bug 884651 which is not fixed until Firefox 26. This is probably the most annoying of the bugs that blocked the pref on by default.

Why is this marked as FIXED? Is it already the default in new firefox releases on Linux?

(In reply to nucrap from comment #100) > Why is this marked as FIXED? Is it already the default in new firefox > releases on Linux? Yes. This bug enabled it in builds (meaning it was capable of being turned on at all) and bug 886181 enabled it by default for Firefox 26 and later. It's been on by default for 4 major releases now. The current extended support release (ESR) is based on Firefox 24, so that branch doesn't have it on by default yet, though it is available to be turned on via about:config. Generally, you should be using the normal current release, however. If you have an issue with gstreamer in your installation, please file a new bug with specific information.