User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36

Steps to reproduce:

Updating Windows 10 to the current version 1809 and installing the official media feature pack.

Actual results:

Opening the following websites crash tabs reproducibly:

• Youtube videos (youtube.com/watch?v=*)
• the Mozilla support forum (support.mozilla.org/*)

Expected results:

The websites should have loaded without a tab crash.

All crashes have the signature Ndr64pClientInit with crash reason EXCEPTION_ACCESS_VIOLATION_WRITE.

I went backwards in Firefox versions from 65 all the way to 50:

• Versions 56 and newer are crashing
• Versions <=55 are working

Crash reports with current version 64.0.2:

• 1ebdd242-705a-415a-8216-ce2a70190120 (Youtube)
• b9ee82d6-b8de-4ec2-b6cb-9a3fb0190120 (Youtube)
• 2bb4e42c-e34c-4d08-b205-8dc590190120 (support.mozilla.org)
• 6537e1f7-21b7-4ce4-97ec-6a41e0190120 (support.mozilla.org)

Crash reports with current beta version 65.0b12:

• 14ea59a0-2b8e-4924-b5df-c58680190120 (Youtube)
• 3952f5ff-66a5-4e79-a5f4-c01a30190120 (Youtube)
• 9fe449fc-a234-4c44-b146-c35eb0190120 (support.mozilla.org)
• 7a24dfab-aa27-4e9d-b3c9-033c10190120 (support.mozilla.org)

The tabs also crash in safe mode (here on the current beta version 65.0b12):

• 84711d82-ce78-4808-82d6-61a600190120 (Youtube)
• 3d027682-c6bd-4af9-a640-b94f70190120 (Youtube)
• 74e27c9d-f77e-4e79-80a7-4b6270190120 (support.mozilla.org)
• 95799aab-b329-43e2-9dc2-652390190120 (support.mozilla.org

Things I already tried (roughly in order):

• update graphics driver (GTX 1060 6GB)
• Firefox save mode
• disable hardware acceleration
• reinstall Firefox (with and without deleting my profile)

It looks like the Microsoft vp9 decoder is causing the crash.

I think you might be right. Setting media.wmf.vp9.enabled to False "solves" the issue.

Jean-Yves, who's the best person to look into this?

similar to comment #0 there's also a crash comment at bp-35ed80ff-5afe-4db5-99fd-d1f8f0190120 saying they are using Windows 10 N and the crashes started appearing after updating to windows 10 version 1809 and installing the media feature pack.
should we reach out to microsoft about this?

Confirming this here: 1809 upgrade on Windows 10 Pro N, then installed media feature pack for that version from https://www.microsoft.com/en-us/software-download/mediafeaturepack, then reproducible crashes on YouTube and mozilla support pages, among others.

• 64.0.2: 12caa1f1-c382-49cc-afbf-143430190125
• 66.0a1: 541e0bfc-a2dd-47ae-938f-611060190125

media.wmf.vp9.enabled to false prevents the crash.

I also confirm the issue. 1809 upgrade and Media Feature Pack installed. I tried to update to latest Firefox (I was using ESR) to solve the issue, but ultimately disabling the vp9 decoder is what made it work again. From my viewpoint, this is Microsoft's fault, not Mozilla's. The latest Media Feature Pack have a broken vp9 decoder which is the root cause of this issue.

Can confirm.

downgraded to Firefox V63. same problem. disabled virus protection: same problem.

on this page the problem only seems to be with html5 and webm: http://techslides.com/sample-webm-ogg-and-mp4-video-files-for-html5

The problem occurs with both integrated HD 630 graphics and with an Nvidia card. Both with the latest drivers. The problem GOES AWAY when I boot into safe mode with networking.

Other things I have tried: reducing content process limit to 1, media.mediasource.enabled -> True, media.mediasource.webm.enabled -> True.

The youtube page for seeing which codecs are supported crashes :-( https://www.youtube.com/html5

I installed this plugin and now videos on youtube are working. https://addons.mozilla.org/en-US/firefox/addon/h264ify/ It seems that firefox and VP8/VP9 webm get broken with the media pack. Other web pages still crash of course, because the plugin is only for youtube. Ironically, the page for how to submit a crash report crashes: https://support.mozilla.org/en-US/kb/mozillacrashreporter

hi adam, could you reach out to microsoft about this issue on any open channels we're having with them? thank you

Reaching out to MS via partner mailing list.

Hi Korbinian, Can we have your permission to share the raw crash dump data from your crashes in comment 7 with Microsoft engineers? That may help get the issue fixed from their side. Thanks!

Hello Kosuke, do you have links to your particular crash reports? Would you be willing to give permission for us to share the raw crash dump data with Microsoft engineers? thanks!

Andrei, can the QA team try to reproduce this crash, so that we can have some crash dumps to share with Microsoft? Thanks!

Hey Liz -- sure, go ahead!

Thanks Korbinian - I passed it along. Maybe that will shed some light on the issue!

A note:

other browsers do not have any problems for Windows 10 Pro N 1809 with media pack upgrade. I tested Edge, chrome.

Hello,

Currently we don't have any environments with Windows 10 N setup. We will request a machine setup with that particular OS from our IT department, but considering it takes a bit of time for the setup and the update, the fastest we could provide an answer and crash dumps will be on Monday.
Taking this into account, could you let us know if the crash dumps are still needed that late or the ones provided by Korbinian will suffice?

I could provide some more dumps in the (CET) evening, if you need them.

Hi Cristian, thanks for the update. I think we have enough info from Korbinian's crash record.

Massimiliano, thank you! I'll ask if we need more!

Paul as jya gave you his Win 10 laptop, could you try to repro this?

Sorry forgot the NI

I updated the machine to 1809, and played back a few videos that are in VP9, checking via the devtools that we're using the hardware decoder, but no dice, it works nicely.

This is not an N edition with the codec pack though, this is a regular Windows.

can we programmatically disable VP9 playback through WMF if we detect we're in a Windows 10 N environment to work around the issue?

Crashes mentioned on French forum : https://forums.mozfr.org/viewtopic.php?f=5&t=140712

report: https://crash-stats.mozilla.com/report/index/57651beb-ae6a-4dbc-8e3a-0bdc80190207

Microsoft is suggesting to install the LCU (latest monthly OS update) and that should repair the torn state.

AFAIK I have the latest updates, but I can still reproduce the problem.

Countermand.
I noticed that I installed the Windows Feature Pack (forgive me if it's the wrong translation, I'm using Windows in Italian), KB4134255, in the same day of 2019-01 Windows 10 Version 1809 Cumulative Update for x64 (KB4476976), but after the latter.
I got a scruple and thought: what if I uninstalled the LCU and reinstalled it?

I did it and I just jumped on Youtube, opened the first video and... the tab doesn't crash and I can see it with this codec:

vp09.00.51.08.01.01.01.01 (247) / opus (251)

Is this Windows' one?

Massi

With media.wmf.vp9.enabled reset I can no longer reproduce the crash (using 67.0a1 (2019-02-09))
I did let windows check for updates yesterday, but only got Windows Office updates. In particular, KB4476976 shows 2019-01-24 as installation date, so before I had the crashes. I don't know what fixed it?

(In reply to Adam Stevenson [:adamopenweb] from comment #26)

Microsoft is suggesting to install the LCU (latest monthly OS update) and that should repair the torn state.

there is no sign that the crash volume is decreasing after this month's patch tuesday - can we look into a workaround from our side and disable WMF vp9 decoding once we detect crashes or know that we're in an affected environment (win10 1809 N-edition)?

What is the current status? Are we able to assign a resource to actively work this bug? Thanks in advance for your help!

the priority was set at a time where the crash rate was still small compared to where it is now - so P1 is a better fit after conferring with marissa.

Add me to the list. I had to downgrade to 1803 because of VP9.

1809 still caused Firefox to crash after all updates were installed for February 2019 and prior.

I can confirm the same behavior when using WebRTC as well if that information is of any help. 1809 Windows 10 Pro N with Media Feature Pack from October 2018. Korbinian's suggestion to change media.wmf.vp9.enabled to false has resolved the issue for that use case, thanks for that!

It is not trivial to reproduce as we need to find an engineer with a Windows machine with VP9 hardware decoding capability (which currently nobody in the media team does). And then Windows 10 on such a machine needs to be changed to Windows N.

Jean-Yves since this appears to affect more people I think we should consider turning off VP9 hardware acceleration until we are able to track down the issue and find a work around or convince MS to fix the issue on their end.

(In reply to Nils Ohlmeier [:drno] from comment #36)

It is not trivial to reproduce as we need to find an engineer with a Windows machine with VP9 hardware decoding capability (which currently nobody in the media team does). And then Windows 10 on such a machine needs to be changed to Windows N.

Jean-Yves since this appears to affect more people I think we should consider turning off VP9 hardware acceleration until we are able to track down the issue and find a work around or convince MS to fix the issue on their end.

How can this not affect Edge or Chrome user? they would have the same issue as we do.

It would be extremely unfortunate to disable this feature, making YouTube a painful experience for all the people that currently do not have any issues. And there's a much greater amount of those than those with a problem.

Currently, this would be handled like a GPU process crash, after it has crashed the GPU process and/or tab, HW decoding is automatically disabled after (2 I believe) crashes.

Can anyone confirm (or not) that this isn't the case?

We should have nothing to do here. Yes, it crashes, but does so once only until Firefox gets upgraded or the drivers are updated.

(In reply to [:philipp] from comment #30)

(In reply to Adam Stevenson [:adamopenweb] from comment #26)

Microsoft is suggesting to install the LCU (latest monthly OS update) and that should repair the torn state.

there is no sign that the crash volume is decreasing after this month's patch tuesday - can we look into a workaround from our side and disable WMF vp9 decoding once we detect crashes or know that we're in an affected environment (win10 1809 N-edition)?

this is what should happen already

Matt, shouldn't the GPU/HW decoder be disabled after such crash? it goes through the crash thingy.

(In reply to Jean-Yves Avenard [:jya] from comment #38)

(In reply to [:philipp] from comment #30)

(In reply to Adam Stevenson [:adamopenweb] from comment #26)

Microsoft is suggesting to install the LCU (latest monthly OS update) and that should repair the torn state.

there is no sign that the crash volume is decreasing after this month's patch tuesday - can we look into a workaround from our side and disable WMF vp9 decoding once we detect crashes or know that we're in an affected environment (win10 1809 N-edition)?

this is what should happen already

I wouldn't bet on this: in my case it stopped crashing when I manually removed the LCU and reinstalled it: i figure it that it crashed cause the LCU was installed before the feature pack.

(In reply to Massimiliano Caniparoli from comment #40)

I wouldn't bet on this: in my case it stopped crashing when I manually removed the LCU and reinstalled it: i figure it that it crashed cause the LCU was installed before the feature pack.

yes this crash earlier, and in the content process as just to determine if the machine can do VP9 we attempt to create a VP9 decoder.

There's not that many ways in which we can create a WMF VP9 decoder, and that's exactly how Chrome is doing it too. So it must crash for them just the same.

Can you test with Chrome?

Back when the crash was occurring for me, I could view the affected sites like support.mozilla.org just fine in Chrome.

Same for me: before the fix, Chrome worked (but I can't guarantee it was using VP9).

(In reply to Jean-Yves Avenard [:jya] from comment #39)

Matt, shouldn't the GPU/HW decoder be disabled after such crash? it goes through the crash thingy.

As you mentioned later, this crash is happening when we create a test MFT decoder when trying to answer IsTypeSupported.

We don't configure it with HW decoding or anything at that point, and we don't have any sort of crash protection on that code.

We have existing code within graphics that detects when a crash happens inside a given block, and prevents us from running it on future instances of the content process, we could look at adding that here?

Alternatively, if we move more/all decoding to be OOP, then we should make sure we're handling IsTypeSupported through the decoder process too. Avoiding calling into WMF (which includes driver provided binaries) from the content process would improve stability.

(In reply to Korbinian Abenthum from comment #42)

Back when the crash was occurring for me, I could view the affected sites like support.mozilla.org just fine in Chrome.

You need to be watching YouTube of course.

The crash may be due to us testing VP9 decoding speed at idle. By removing this it should at least prevent the startup crash. We no longer need to check with MediaCapabilities now

Can someone in QA test this and try to reproduce the crash?

Jean-Yves, is that something we could try quickly in beta to see if it brings down the crash rate?

Could the people experiencing that crash try with Firefox Beta 66 (available at https://www.mozilla.org/en-US/firefox/channel/desktop/)

It may be fixed such that it no longer crash the tabs and will immediately recover.

thank you

Hello Liz,

We have talked with our IT department and requested an environment setup with Windows 10 Pro N. Unfortunately the soonest they can have that ready is sometime next Tuesday.
We will investigate the crash as soon as we have that available, so please bear with us.

In the meantime, to double check we have a good understanding of this, could someone confirm these are the correct steps in reproducing the issue?

1. Have one of the affected Fx versions installed (Fx64.0.2, Fx65.0b12).
2. Upgrade OS to Win 10 Pro N 1809 and install the media feature pack, mentioned in the description.
3. Launch Firefox and access a youtube video -> tab should crash.
4. Access support.mozilla.org/ -> tab should crash

Steven, novalis, does this work now in Firefox beta 66? If you don't mind testing again, that will help out a lot.

I just updated to 66.0b11, enabled VP9 encoding and restarted the browser. Now it seems to work without crashes!

Fabulous! Jya, so, we can call this fixed/worksfor me? Can you clarify what actually fixed it (from comment 45) ?

Just tested in Windows 10 Enterprise N 1809 with the Beta channel release. Whatever the bug was, it's fixed in the latest beta.

ETA on version 66 hitting stable? I'm delaying my 1809 update until then.

This is a side effect of a bug introduced by bug 1513308.

I am fixing the side effect of this bug, and will be adding here something that makes in not possible to reproduce.

Depends on D21477

It is still crashing for me with the 66.0b11 beta. Did I receive the wrong version? Same for the Developer Edition and Nightly.

Steven - Firefox 66 is scheduled to release March 19.

What site(s) is it still crashing on? Make sure to check for updates manually by going to Help -> About Firefox.

Youtube main site is already enough. It seems to start loading and I can already see some video thumbnails, but before it finishes, it crashes. I assume it's the advertisement video playing. Changing the vp9.enabled flag in about:config still makes the issue disappear.

And yes, I always check the Help -> About Firefox, it says it's up to date. I also re-installed the version to be sure.

(In reply to Christoph E. from comment #60)

Youtube main site is already enough. It seems to start loading and I can already see some video thumbnails, but before it finishes, it crashes. I assume it's the advertisement video playing. Changing the vp9.enabled flag in about:config still makes the issue disappear.

And yes, I always check the Help -> About Firefox, it says it's up to date. I also re-installed the version to be sure.

Please try with Firefox Beta as mentioned in comment 48

(In reply to Christoph E. from comment #60)

Youtube main site is already enough. It seems to start loading and I can already see some video thumbnails, but before it finishes, it crashes. I assume it's the advertisement video playing. Changing the vp9.enabled flag in about:config still makes the issue disappear.

And yes, I always check the Help -> About Firefox, it says it's up to date. I also re-installed the version to be sure.

66.0b12 pushed out recently. Can you try with that version?

(In reply to Jean-Yves Avenard [:jya] from comment #61)

(In reply to Christoph E. from comment #60)

Youtube main site is already enough. It seems to start loading and I can already see some video thumbnails, but before it finishes, it crashes. I assume it's the advertisement video playing. Changing the vp9.enabled flag in about:config still makes the issue disappear.

And yes, I always check the Help -> About Firefox, it says it's up to date. I also re-installed the version to be sure.

Please try with Firefox Beta as mentioned in comment 48

As I mentioned, I am trying it with the beta and basically downloaded every single available version on that site, making sure it says it's up to date.

(In reply to Arthur K. from comment #62)

66.0b12 pushed out recently. Can you try with that version?

It claims it's still on the newest version with the 0b11 in the "About Firefox" - any trick to force it?

(In reply to Christoph E. from comment #63)

(In reply to Arthur K. from comment #62)

66.0b12 pushed out recently. Can you try with that version?

It claims it's still on the newest version with the 0b11 in the "About Firefox" - any trick to force it?

It's at https://ftp.mozilla.org/pub/firefox/candidates/66.0b12-candidates/build1/

(In reply to Arthur K. from comment #64)

(In reply to Christoph E. from comment #63)

(In reply to Arthur K. from comment #62)

66.0b12 pushed out recently. Can you try with that version?

It claims it's still on the newest version with the 0b11 in the "About Firefox" - any trick to force it?

It's at https://ftp.mozilla.org/pub/firefox/candidates/66.0b12-candidates/build1/

Installed 0b12 from there -> Youtube videos still crashing. I left the ticket name in the report tab before closing in case that is helping. Let me know if I should try anything else to find the cause of this issue.

Update: This will be fixed from the Windows side of things in mid May by an optional Windows update and then will go out to all users on the 2nd Tuesday in June. So, we will be looking to fix things in 66/67 on our end until the Windows fix rolls out.

https://hg.mozilla.org/mozilla-central/rev/ae274634278f

Is this something you are aiming to uplift for 66? I just noticed this patch.

I hope it's being uplifted for 66. Myself and several other users here are holding back on the 1809 upgrade because of this crash bug.

(In reply to Liz Henry (:lizzard) (use needinfo) from comment #69)

Is this something you are aiming to uplift for 66? I just noticed this patch.

I believe the patch that went in will prevent the crash in the most common case. And I'd like people to test with Firefox Nightly and confirm if that is indeed the case.

However, I have another patch on the way that I believe will allow more robust handling. The first version got r- but I'll have a revise version by the end of this week.

In the mean time, people having the issue, thank you for testing with nightly

Firefox Nightly 67.0a1 (2019-03-06) (64-Bit) is still crashing for me.

Hello everyone,

Sorry for the late response on the QA side. I started testing the crashes on the reported versions as well as the latest versions.
As mentioned above by Cristoph, the crash is still reproducible on the latest beta (in his case 66.0b12). I tested on Fx66.0b13 and could still reproduce the crash.

On the latest nightly (Fx67.0a1 - 20190306095759) the crash is still reproducible. Here is the crash report.

Cristoph beat me to it again.

(In reply to Cristian Baica [:cbaica], Release Desktop QA from comment #73)

Hello everyone,

Sorry for the late response on the QA side. I started testing the crashes on the reported versions as well as the latest versions.
As mentioned above by Cristoph, the crash is still reproducible on the latest beta (in his case 66.0b12). I tested on Fx66.0b13 and could still reproduce the crash.

On the latest nightly (Fx67.0a1 - 20190306095759) the crash is still reproducible. Here is the crash report.

Cristoph beat me to it again.

I should have mentioned:
It is expected that it will crash at least once, but once it has crashed it shouldn't crash again as HW decoding gets automatically disabled.

So the test should check that it crashes, but after reloading the tab or restarting Firefox then it no longer crashes.

@Jean-Yves I have just tried again, to restart Fx after the crash (CTRL+Alt+R on the browser console) as well as reloading the page or accessing the same video again, but Firefox crashes every time.

Please check the latest attached file for more details.

Should the configuration had changed at startup, a crash guard would have always been re-attempted even if a new crash occurred.

Depends on D21477

Depends on D22623

(In reply to Cristian Baica [:cbaica], Release Desktop QA from comment #75)

@Jean-Yves I have just tried again, to restart Fx after the crash (CTRL+Alt+R on the browser console) as well as reloading the page or accessing the same video again, but Firefox crashes every time.

Please check the latest attached file for more details.

Here is a beta 64 bits build with the proposed fix:
https://queue.taskcluster.net/v1/task/UJu54OliRP6ovcFi2QvD0Q/runs/0/artifacts/public/build/install/sea/target.installer.exe

Don't install it over the current beta, this version won't update itself.

With this version attempt to play a YouTube video. It should crash the tab and offer to reload it. Once reloaded it will no longer crash until a new version of Firefox gets installed or the video card or its drivers are changed.

Still no change with that version - crashes constantly, no matter if I reload or restart.

Hi guys and sorry for the non-formal redaction and jump-in.

Config :
W10 enterprise N 1809
Video : AMD Radeon 7500 HD series (latest drivers 15.200.1062.1004 / also tested with windows update 15.200.1045.0)

Issue : identical to OP's + many sites (not only YT)

Personal diag : Latest MM feature pack is involved, confirm that other browsers aren't impacted (tested chrome, Edge)

Personal (and probably dirty) fix :
about:config
search for media.wmf.vp9.enabled and set to disable.
No more crash, no perf issue noticed on Youtube, all modes up to 4K still avail.
(thinking out loud : does the 750HD even support VP9 ? Could it be the cause of the problem ?)

Latest report before the "hack" :
Beta 66.0b13 (64-bit) with 15.200.1045.0 Radeon Drivers (disabled accessibility option as mentioned here : https://support.mozilla.org/fr/questions/1209746 as you can guess, w/o success)
https://crash-stats.mozilla.org/report/index/bp-043882b3-d938-4dfc-9503-d8b670190308

Sorry for interrupting, @mod, feel free to trash this if misplaced or whatever.

(In reply to Antoine B. from comment #81)

(thinking out loud : does the 750HD even support VP9 ? Could it be the cause of the problem ?)

I obviously meant "Could it be part of the problem"

(In reply to Antoine B. from comment #82)

(In reply to Antoine B. from comment #81)

(thinking out loud : does the 750HD even support VP9 ? Could it be the cause of the problem ?)

I obviously meant "Could it be part of the problem"

It does not, but the issue here is that just loading the VP9 windows decoder (which will be either software or hardware) is causing the crash, before we even get the chance to query if it supports HW.

The crash is due to a mismatch of version that occurred when the windows media pack got upgraded prior windows version 1809. This left things in an unusable state.
The crash doesn't affect just Firefox FWIW

(In reply to Christoph E. from comment #80)

Still no change with that version - crashes constantly, no matter if I reload or restart.

OK, now that is weird.
Can you please attach a copy of the about:support output with that special build?

Thanks.

Or maybe it's because on how it was build, can you set the environment variable MOZ_FORCE_CRASH_GUARD_NIGHTLY

You can start Firefox from a command prompt:
Then type:
Set MOZ_FORCE_CRASH_GUARD_NIGHTLY=1
C:\path\to\build\firefox.exe

about:support is here https://pastebin.com/jnckXHz0

I can confirm it is working when setting the crash guard manually after the initial crash. about:support when it's working: https://pastebin.com/C3iv8Jx8

(In reply to Jean-Yves Avenard [:jya] from comment #83)

(In reply to Antoine B. from comment #82)

(In reply to Antoine B. from comment #81)

(thinking out loud : does the 750HD even support VP9 ? Could it be the cause of the problem ?)

I obviously meant "Could it be part of the problem"

It does not, but the issue here is that just loading the VP9 windows decoder (which will be either software or hardware) is causing the crash, before we even get the chance to query if it supports HW.

The crash is due to a mismatch of version that occurred when the windows media pack got upgraded prior windows version 1809. This left things in an unusable state.
The crash doesn't affect just Firefox FWIW

Thanks for clarifying this out.
So, may we think a fresh 1809 install + latest media pack (not update) won't be affected ?
If yes, this mean we could find a workaround with a real uninstall of media pack (a pain ...) and a fresh mp install. I believe I had a similar problem before, will check my bookmarks and keep you posted if I can confirm.

There's nothing to get around it unfortunately, making firefox unusable without it.

Depends on D22624

I can't reproduce the crash.
Here is what I've done so far:
Installed Windows 10 Pro N edition (original version 1504)
Updated to Windows 1809
Installed Windows Media Pack for Windows 1809
Installed the VP9 video extension (https://www.microsoft.com/en-us/p/vp9-video-extensions/9n4d0msmp0pt?activetab=pivot:overviewtab) using Microsoft store.

Going to YouTube, I can watch VP9 video using hardware decoder on either the Intel 630 or nvidia 1050Ti.

What am I missing?

:cbaica, what did you do to be able to reproduce it on your side?

Hi Jean,

Here is what I've done on my end:

• Installed Windows 10 Pro N edition (version 1803)
• Updated Windows to version 1809
• Installed Windows_MediaFeaturePack_x64_1809Oct media pack

Start Firefox with a fresh profile and access a random video on youtube --> tab will crash.

Please note that I'm using a VM to reproduce/confirm the fix. Do you think that would interfere with the result?

The most recent development for the issue that I noticed, is that sometimes, a video would start playing, but if I refresh the page or try to access the same video again from youtube main page, the tab will crash. This occurred on the latest nightly as well as the build you provided.
My best guess is that it has something to do with the video formats?

Resetting 67 to affected as the first fix didn't have an impact on crashes. Tracking for 67 given the volume.

Through the MainThreadVPXCrashGuard class, it is now possible to guard an operation that will not run in the main thread. This breaks the assumption that only one driver crash guard could be in use in the ContentParent.

It is also possible for the crash guard destruction to be pending (as MainThreadVPXCrashGuard does it asynchronously) while a new one is created. This operation is fine, as we only care about crashes, and a crash is final.

Depends on D22874

Question for the people experiencing the crash. Do you have the VP9 codec extension package installed? It's available via the Windows app store.

I never installed the VP9 Codec extension. The Media Feature Pack has always been enough in the past.

I didn't install it either - installing it actually didn't seem to change anything at all for me.

I did install it without notable change.
Despite the media.wmf.vp9 disabled fix.
Note : latest Windows update failed (I believe because of media pack).
Will report if any change.

Installing the codec made no significant improvement. Tabs are still crashing constantly.

Follow-up / Additional notes. FWIW.

I rolled back from FF beta to 65.0.2 and deactivated the media feature pack.
Since "we" found that it was the source of the problem, I thought :

1. I'd get rid of crashes = > No, occasional crashes on FB (probably ads ...)
2. (may be interesting) I'd get the FF notice that initially made me install the latest media pack fo 10N (like "You should install Windows media pack ... let us show you how") => No.

Now, I might try to run a CCCleaner to scan the registry or try to effectively uninstall Media Service Pack (disabling it does not uninstall it, I believe it's marked as permanent ...).

Another idea, are there any 3rd party options that could act as a stand in replacement for the Media Feature Pack? I know the VideoLAN folks used to have a plugin that acted like a MFP for Linux.

Has there been any communication from Microsoft about this? You would think that they'd care about one of the major web browsers crashing when calling their APIs. If they're silent about this, Mozilla should definitely consider a more trusted replacement considering the circumstances.

(In reply to Steven Bock from comment #99)

Another idea, are there any 3rd party options that could act as a stand in replacement for the Media Feature Pack? I know the VideoLAN folks used to have a plugin that acted like a MFP for Linux.

Of course: K-Lite Codec Pack Basic.

Has there been any communication from Microsoft about this? You would think that they'd care about one of the major web browsers crashing when calling their APIs. If they're silent about this, Mozilla should definitely consider a more trusted replacement considering the circumstances.

Curious if today's Monthly Rollup made any difference?

And only do it once.

Depends on D22874

:pascalc, I believe this should be a blocker for 66 too.

Right now, Microsoft has been pretty slow at enabling build 1809 to everyone (and on my test machine it wasn't being installed automatically, I had to download it manually), but they have resumed according to their blog.

As such, the number of crashes occurring when watching YouTube is going to greatly increase, and as everyone knows YouTube is the internet.

Forwarding the blocking request to Liz as she is the owner for 66. Liz, see comment #102.

Jean-Yves, could you prepare an uplift for the release branch in case Liz wants to take the patch in a RC2 please? Thanks

(In reply to Pascal Chevrel:pascalc from comment #104)

Forwarding the blocking request to Liz as she is the owner for 66. Liz, see comment #102.

Jean-Yves, could you prepare an uplift for the release branch in case Liz wants to take the patch in a RC2 please? Thanks

The changes were crafted to be as minimal as possible (with several iterations to make it as safe as possible).

I have thoroughly tested them in preparation for this uplift.

For people wanting to try now and not wait for the next nightly;
https://queue.taskcluster.net/v1/task/H24rBXg6Qtuvr1gvn7D4gw/runs/0/artifacts/public/build/install/sea/target.installer.exe

With this build, when loading the VP9 causes a crash (unfortunately we can't go around that one yet), the crash will be detected and once the tab is reloaded it will load as usual, but with HW decoding disabled.

https://hg.mozilla.org/mozilla-central/rev/a6724ca6cb91
https://hg.mozilla.org/mozilla-central/rev/6ee4b21eb0e0
https://hg.mozilla.org/mozilla-central/rev/15e52cb0872d
https://hg.mozilla.org/mozilla-central/rev/ee3013ee7a6d
https://hg.mozilla.org/mozilla-central/rev/303a1f4c20e8

We don't have time to verify this in nightly and get it into the RC2 build today.
I looked at the usage stats for 1809 vs. 1803. More users are still on 1803, so I agree that the problem will get worse over the next month or two as users continue to update their OS.

But, it isn't going to be a huge difference from this week till next week when we release.
I'll mark this as a blocker to indicate that it will be lined up for a dot release in 66.

About the K-Lite suggestion, would a codec pack like that really have that dramatic of an effect? Would it be a good idea to formally suggest that (or CCCP if it's still around) as an alternative to Microsoft's offering if the crashes are going to continue?

The VLC idea was also because of its open source history. I thought that would be more favorable considering the open source nature of Firefox. If K-Lite can meet that wishlist item, then I'm all for it (more for others needs, I'm personally willing to install something proprietary if it fixes the problem).

(In reply to Steven Bock from comment #108)

About the K-Lite suggestion, would a codec pack like that really have that dramatic of an effect? Would it be a good idea to formally suggest that (or CCCP if it's still around) as an alternative to Microsoft's offering if the crashes are going to continue?

I was looking at it from the perspective of de-coupling Microsoft's media feature pack. It seems proprietary and you can't really know what's going on behind the scenes.

The VLC idea was also because of its open source history. I thought that would be more favorable considering the open source nature of Firefox. If K-Lite can meet that wishlist item, then I'm all for it (more for others needs, I'm personally willing to install something proprietary if it fixes the problem).

But it introduces another app to the mix. K-Lite just installs and registers working codecs that are well tested. It's not clear to me how vetted what Microsoft is offering. Much if not all in K-Lite is based on open source and it has supported VP9 well. It's the main reason I suggested trying it as an alternative to Microsoft's solution.

(In reply to Arthur K. from comment #109)

Okay, now your choice is making more sense. Thanks for the explanation.

It seems like the Microsoft media pack was chosen because it was assumed to be simply reintroducing a Windows OS component that was removed for EU compliance. Based on the behavior here, it seems that it may not be as simple as that. Maybe the non-N versions have some sort of special integration sauce that make the codecs work properly. Maybe this is Microsoft's version of an "I told you so" to Europe. Maybe it really is Firefox's fault and the issue lies in our code. Who knows at this point.

As for VLC, I forgot my bias since I've been installing it for decades now in both personal and professional settings as a quick fix for most A/V related issues.

The questions at this point are:

1. Do we want to recommend another app or codec pack instead of Microsoft's own offering (since the Media Feature Pack behaves like it's not intended to be installed as a decoupled OS component)?

or

2. Do we want to wait for Microsoft to fix this?

Another thing to consider is that Build 1904 supposedly comes out next month if Microsoft keeps to their update schedule. 1904 could have a fix in it if this really is Microsoft's problem.

(In reply to Steven Bock from comment #110)

The questions at this point are:

1. Do we want to recommend another app or codec pack instead of Microsoft's own offering (since the Media Feature Pack behaves like it's not intended to be installed as a decoupled OS component)?

not sure why you're continuing on about this, we have a fix, it's now in Nightly and will be in Firefox 67.0.1

The issue affects Chrome too, they just crash more nicely than we do and recover automatically. Open YT with 1809 and the media pack installed, you'll see all the windows briefly disappearing and being reloaded.

or

2. Do we want to wait for Microsoft to fix this?

we have no choice.
The K-Lite is a poor one, it crashes often, causes plenty of compatibility issues and this is our #1 item in troubleshooting: how to uninstall it.

Another thing to consider is that Build 1904 supposedly comes out next month if Microsoft keeps to their update schedule. 1904 could have a fix in it if this really is Microsoft's problem.

They won't have a fix until May for this.

(In reply to Jean-Yves Avenard [:jya] from comment #111)

2. Do we want to wait for Microsoft to fix this?

we have no choice.
The K-Lite is a poor one, it crashes often, causes plenty of compatibility issues and this is our #1 item in troubleshooting: how to uninstall it.

I respectfully disagree. Been using it for nearly decade or so without crashes caused by a single codec in a 1000+ machine university environ. You have evidence and Bugzilla reports for this? It is the defacto codec pack that millions of people use when they're not resorting to VLC player (which, if I am not mistaken, has most of K-Lite's codecs in-built as well).

If it is a Microsoft issue, as it seems to be, is there harm is seeing how it behaves as an alternative to Microsoft's offering? At the risk of sounding obstinate: Do you have any factual evidence to the contrary.

Maybe 1904 will fix it. Maybe not. What harm is there in gathering more data?

(In reply to Jean-Yves Avenard from comment #111)

So we're just going to sit here and wait until May for Microsoft to take their time with this?

If K-Lite crashes Firefox and Microsft's pack is bugged, is there a third option? What about taking the mechanism that Linux uses and running it on Windows?

Regardless, it seems like depending on Microsoft for these codecs might be harmful.

(In reply to Steven Bock from comment #113)

(In reply to Jean-Yves Avenard from comment #111)

So we're just going to sit here and wait until May for Microsoft to take their time with this?

the work-around is in Nightly 20190313215409, you can test it today.
Once the fix is confirmed to work we can uplift it to beta 66 which will be released in the next few days.

If K-Lite crashes Firefox and Microsft's pack is bugged, is there a third option? What about taking the mechanism that Linux uses and running it on Windows?

We can't ship H264 or AAC decoders for licensing reasons, regardless of how it's packaged. We must rely on what the OS offers us, that includes the system FFmpeg installed by default.

Regardless, it seems like depending on Microsoft for these codecs might be harmful.

Blame patent encumbered codecs.

To watch YouTube, you don't need to install the media pack at all, you'll be using VP9 just fine. Netflix will play too as they use Widevine encryption. Most other sites won't work.
The only use for the media pack for YouTube is getting support for hardware accelerated decoding. It is not required.

You should not have to wait until May; while it's too late to get this fix into next week's 66 release, we can likely get it into a point release in a couple of weeks. It won't necessarily have to wait till 67. I hope the workarounds will help; maybe even using beta 67 in the meantime.

(In reply to Liz Henry from comment #115)

The only use for the media pack for YouTube is getting support for hardware accelerated decoding. It is not required.

Blame patent encumbered codecs.

I just looked up some of the licensing for H.264 and AAC. Assuming we're talking about the MPEG patents, a possible workaround would be to ship a tamper-resistant version of the source and compile it on install. Mozilla could even put a subtle middle finger in the installer explaining this:

Due to unusually strict licensing restrictions in your country, we are forced to download and compile certain core Firefox components on your machine. We are sorry for any issues caused.

If you are as upset as we are, consider contacting your congressional representative. Click here to take action.

By clicking "Next", you agree to have the Firefox installer download the source code of these components onto your system and use considerable computer resources to build them in order to fit a loophole.

I mean, let's not beat around the bush here. We're depending on Microsoft's codec for the political reasons of

1. Not willing to ship raw code to a user's machine and have the installer compile it to avoid the distribution clauses in the licenses and patents

and

2. Not willing to instigate public pressure to either get the patents changed or have the codecs released as MPL compatible OSS.

I just find it, frankly, ridiculous that, at the very least, we can't instruct users to install an alternative for both usability purposes as well as sending a message of self sufficiency to Microsoft. Software like K-Lite or VLC may not be perfect, but it's at least better than being at the mercy of whoever is in charge of OS development.

Could someone verify that the workaround currently available in Nightly is fixing the issue for them?

Until the fix is verified, it can't be uplifted to beta.

(In reply to Jean-Yves Avenard [:jya] from comment #117)

Could someone verify that the workaround currently available in Nightly is fixing the issue for them?

Until the fix is verified, it can't be uplifted to beta.

I did verify that with yesterdays Nightly Firefox no longer crashes.

As mentioned earlier, the reason I installed Media Pack (and that's a bad idea with 10N, next Windows Update will very likely fail) is that Firefox displayed a message about it, suggesting to proceed. I believe that most 10N users don't have media pack installed and, TBH, it is a bad idea to do so. Reverting is not that easy and you could get in a never ending suite of problems.

Long story short : I suggest you exclude 10N users from the recipients of the "advice" mentioned before. Whatever the versions, distribution or licensing models, there will be problems.

10N is not designed for multimedia an most users know that.

My 2 cents FWIW.

Tested the issue again on the latest nightly Fx67.0a1 buildID: 20190314214219.

The tabs no longer crash when playing youtube videos. Tested with about 20 random videos, no tab crashes. I haven't done any changes to the OS or the installed media packs.
My guess is we can move forward with this fix for beta as well.

(In reply to Antoine B. from comment #119)

I suggest you exclude 10N users from the recipients of the "advice" mentioned before. Whatever the versions, distribution or licensing models, there will be problems.

Agreed. Having the Microsoft option as the absolute solution is absurd given that their codec pack can have bugs as well (proven by this issue).

An alternative to this would be to direct users to a help page with an explanation of why they're receiving the message, as well as several possible solutions with some testing data (Microsoft's codec pack, K-Lite, VLC, etc).

10N is not designed for multimedia an most users know that.

I wouldn't say that. It's designed to get around an EU court ruling.

Beta is 67 at the moment, synced with m-c (also 67). So you don't need to request uplift for this to be in beta. 66 is on mozilla-release already.

Comment on attachment 9047263 [details]
Bug 1521370 - Add crash guard around VPX decoder creation. r?mattwoodrow

Beta/Release Uplift Approval Request

• Feature/Bug causing the regression: None
• User impact if declined: Crashes for all users located in Europe and Korea attempting to watch YouTubr
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Install windows N edition 1809, Install media feature pack. Attempt to play youtube.
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Fixing this bug revealed other hidden bugs in the crash guard infrastructure. While we would properly record a crash occurred, it would continue to crash.
  However that patches were crafted to minimise impacts and make sure the potential for regressions was nil
• String changes made/needed:
• Do you want to request approval of these patches as well?: on, on, on, on

Comment on attachment 9047263 [details]
Bug 1521370 - Add crash guard around VPX decoder creation. r?mattwoodrow

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration:
• User impact if declined: Crash for European and Korean users when visiting YouTube.
• Fix Landed on Version: 67
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): See earlier explanation for release request.
• String or UUID changes made by this patch:
• Do you want to request approval of these patches as well?: on, on, on, on, on

Just to clarify what we intend here as well as in email: Please don't check this in on m-r yet, but we'll aim to do that early next week so we can get it into a 66 dot release. Probably Tuesday or Wednesday.

Comment on attachment 9047263 [details]
Bug 1521370 - Add crash guard around VPX decoder creation. r?mattwoodrow

Crash fix in Audio/Video playback on Youtube with Windows 1809, fix already on 67 where crashes for this signature became anecdotal. uplift approved for our next 66 dot release, thanks!

https://hg.mozilla.org/releases/mozilla-release/rev/41f52aefb0c7
https://hg.mozilla.org/releases/mozilla-release/rev/a10b0362fd55
https://hg.mozilla.org/releases/mozilla-release/rev/2c61fe7ae01b
https://hg.mozilla.org/releases/mozilla-release/rev/15d61033a7ee
https://hg.mozilla.org/releases/mozilla-release/rev/fa9b70ce49a8

Conflict fixes: https://hg.mozilla.org/releases/mozilla-release/rev/59c448b943e7ab1e9de02266e6b23e8eddc71699

Hi, I retested this issue in Firefox 66.0.1 and I was able to reproduce it a few times, after which I retested on 66.0.2 as well as 67.0b5 and the issue no longer occurs, I tried with multiple youtube videos as well as 4k videos and I was unable to crash any tabs, I will mark this issue accordingly.

Forgive for my English (machine translation) I specially was registered to write about a problem which appeared after an exit 66.0.2 Probably correcting a problem for 1809 N users you broke hardware acceleration and for x64 Pro of the version. I perfectly watched 4k@60 video on gtx 1060 to the video card, now freeze. In about: support these records:
WMF VPX video decoding is disabled due to a previous crash.
CP [GFX1-]: WMF VPX video decoding is disabled due to a previous crash.

On my other PC Windows of 10 pro with gtx 1080 on which video too all lagging was perfectly played even 8k. The same errors there is also no hardware acceleration.
Firefox at me never fell pages too.
In chrome everything still reproduces with hardware acceleration and does not fall.

I do not want to pass to chrome disconnect please wmfvpxvideoCrashGuard on not N versions of Windows 10.

(In reply to Andrew from comment #130)

Forgive for my English (machine translation) I specially was registered to write about a problem which appeared after an exit 66.0.2 Probably correcting a problem for 1809 N users you broke hardware acceleration and for x64 Pro of the version. I perfectly watched 4k@60 video on gtx 1060 to the video card, now freeze. In about: support these records:
WMF VPX video decoding is disabled due to a previous crash.
CP [GFX1-]: WMF VPX video decoding is disabled due to a previous crash.

On my other PC Windows of 10 pro with gtx 1080 on which video too all lagging was perfectly played even 8k. The same errors there is also no hardware acceleration.
Firefox at me never fell pages too.
In chrome everything still reproduces with hardware acceleration and does not fall.

I do not want to pass to chrome disconnect please wmfvpxvideoCrashGuard on not N versions of Windows 10.

Thank you for your report.

The new code only track if a crash occurs when the Windows VP9 decoder is loaded. For you to see HW acceleration to be disabled, it can only due to Windows crashed to start with.

Please open a new bug, providing your about:support output so we can investigate.
Thank you

Comment on attachment 9047263 [details]
Bug 1521370 - Add crash guard around VPX decoder creation. r?mattwoodrow

Crash fix, OK for uplift for 60.7esr.

This will need rebased patches for the ESR60 uplift.

Nils, if jya is out, can you find someone to do the rebasing for esr60? Thanks.

I'll do this first thing Monday

Hi there, it's still not too late to get this into ESR 60.7. Just a reminder! Thanks.

I'm on it.

Here is a try push with the uplifted patch:
https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=244167987&revision=b54b9e6680db186ff6f8aff9b1df575ce45105c4

easiest is to pull from that.

Note that there are compilation errors on mingw which I've been told for esr60 is based on clang, while central use a clang-bases mingw.

I don't understand why it would error, the C++ is valid.

If you want something that compiles under such old mingw the patch to apply on the last commit would be to do something like:

diff --git a/dom/media/platforms/PDMFactory.h b/dom/media/platforms/PDMFactory.h
index e96f023f55f3..051382580aae 100644
--- a/dom/media/platforms/PDMFactory.h
+++ b/dom/media/platforms/PDMFactory.h
@@ -50,6 +50,8 @@ class PDMFactory final {
   static constexpr int kYUV422 = 2;
   static constexpr int kYUV444 = 3;

+  static void EnsureInit();
+
  private:
   virtual ~PDMFactory();
   void CreatePDMs();
@@ -72,8 +74,6 @@ class PDMFactory final {
   bool mFFmpegFailedToLoad = false;
   bool mGMPPDMFailedToStartup = false;

-  friend class VideoDecoderParent;
-  static void EnsureInit();
   template <class T>
   friend class StaticAutoPtr;
   static StaticAutoPtr<PDMFactoryImpl> sInstance;

right now it's late and will look at the try result tomorrow morning if needed.

There were some errors that just appeared, I've fixed those and put the workaround msvc/mingw error.

try push:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5ad2ac89c50ffc4d8d0419711863d1eac3ed7b35

There was a conflict that was badly resolved which caused an occasional crash.

this should be fixed now:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=08b00576bba5dbec9e950f62951cbc63a005178a

This one should be good (crossing-finger)
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8edd028167e9b9cd4be67e979b7c8daab4963d1c

https://hg.mozilla.org/releases/mozilla-esr60/rev/106ef7289039
https://hg.mozilla.org/releases/mozilla-esr60/rev/8ab00f88875e
https://hg.mozilla.org/releases/mozilla-esr60/rev/1284882e1b3c
https://hg.mozilla.org/releases/mozilla-esr60/rev/c051c5c4282c

Hi, This issue is Verified as Fixed in the latest Firefox ESR, no crash tabs, no issues occur even with 4k videos. I will mark this issue accordingly.

Hi,
there is nothing I'm not sure I understood, and I need to clear up.
Some days ago I updated Windows 10 to 1903 and Firefox restarted telling me to download the Feature Pack in order to watch videos.
I did it and apparently nothing went wrong.
I have some questions:

1. was it the May Update cited in https://bugzilla.mozilla.org/show_bug.cgi?id=1521370#c66 ?
2. the patch which was created among these comments is intended to leave the tab crash and then disable VP9, or does it prevent the user to notice the tab has crashed? I wonder if my current configuration crashes or not :)
3. is the patch temporary and intended to be removed or will it stay? Does it re-test the hardware acceleration. once a tab crashes and it's disabled?

(In reply to Massimiliano Caniparoli from comment #144)

Hi,
there is nothing I'm not sure I understood, and I need to clear up.
Some days ago I updated Windows 10 to 1903 and Firefox restarted telling me to download the Feature Pack in order to watch videos.
I did it and apparently nothing went wrong.
I have some questions:

1. was it the May Update cited in https://bugzilla.mozilla.org/show_bug.cgi?id=1521370#c66 ?
2. the patch which was created among these comments is intended to leave the tab crash and then disable VP9, or does it prevent the user to notice the tab has crashed? I wonder if my current configuration crashes or not :)
3. is the patch temporary and intended to be removed or will it stay? Does it re-test the hardware acceleration. once a tab crashes and it's disabled?

The workaround is done by checking if when we created a system decoder things crashed. It will not re-run such test (e.g. will disable the system decoder) until one of the following condition is true:
1- New version of firefox since we last crashed
2- New version of the drivers since we last crashed.

When one of this condition is true, it will attempt again to create a system decoder and attempt to use a hardware decoder if successful.

To answer your 2), most of the time, the crash will not be visible to the user as the recovery is pretty quick. Typically the page will quickly refresh. However, for some it may show as "Gah, your tab has crashed" with a button to reload.

The patch as it is, is permanent. It is only a protection against a crash. If things no longer crash, then it won't do anything

Thanks for your thorough response!
My main concern was about the potential permanent deactivation of the feature.
Now, since every new version of FF of driver (how is it checked?) reenables the hardware acceleration, I'm not concerned anymore ;)
Win 10 N users will likely all "return" to HW acceleration with the new update :)

Please specify a root cause for this bug. See :tmaity for more information.

Looks like we are still getting some crashes with this signature, there was a small bump around Jan 18 and 19.

jya, do you think there are things that we could do here?

(In reply to Sean Feng [:sefeng] from comment #148)

Looks like we are still getting some crashes with this signature, there was a small bump around Jan 18 and 19.

jya, do you think there are things that we could do here?

this workaround was recently removed ; maybe it should be re-added

I had an offline discussion with jya about that, removing WMF in content process can also fix this problem. I will do that in bug1685321.

Super high crash count across 3 installations in yesterday's Nightly: https://crash-stats.mozilla.org/signature/?build_id=20210218092411&build_id=20210218214637&release_channel=nightly&product=Firefox&platform=Windows&signature=Ndr64pClientInit&date=%3E%3D2021-02-18T00%3A00%3A00.000Z&date=%3C2021-02-19T18%3A18%3A00.000Z

Last time when I discussed with Jya, the solution for solving this issue is to completely remove WMF usage in the content process. In current situation, on Windows, WMF is only being used in either RDD or GPU process by default, per this premise, I think most users won't encouter this crash again. That is why I didn't remove the WMF in content process immediately.

In addition, after discussing that with team, they think we shuld still keep them as a fallback method if the RDD/GPU process is failed to start, which allows user still to be able to decode H264/AAC media. So I added a Telemetry in bug1690372 to measure how many users would still use WMF in non-RDD/GPU process. If those amount is really small enough, then we can completedly remove WMF in content process and don't worry about the fail creation of RDD/GPU.

If we still get a high crash rate in next couple of days, then I will check jya's old patches and see what workaround we need to add back.