User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:55.0) Gecko/20100101 Firefox/55.0 Build ID: 20170414030225 Steps to reproduce: I upgraded the Nahimic 2 software (audio effects) to the new version v2.3.15. Tested on Windows 10 x64 Creative Update on my MSI laptop GE72 6QD Apache Pro. Now the firefox nightly 55 updates won't work anymore. Actual results: check for new nightly version, download and click restart to install updates. if nahimic is running then the firefox.exe process restarts and is blocked: no window is shown and I have to kill the process. If I shut down the nahimic process before the update then the update is installed correctly. After the update I can use firefox as usual, with nahimic running. Expected results: firefox should update as usual when nahimic is running.

This actually sounds like Firefox is not actually restarting when nahimic is running. Ryan, can you have QA try to reproduce and check if this affects restarting in general? Thanks!

Unfortunately we're not having any MSI hardware and Nahimic seems dependent of it. I'm going to ask around on IRC for this.

I reported the issue to Nahimic too: http://nahimic.helprace.com/i226-firefox-updates-stopped-working-after-2-3-15

I've got a machine with an MSI motherboard, so it can run Nahimic, and I can reproduce this bug on it. After launching the updater, firefox.exe hangs inside the exit() call at [1]. The updater waits for the browser process to exit (at [2]) before it will replace the application with the staged copy, so the fact that firefox.exe was unable to exit causes that wait to time out, meaning the updater gives up and quits without doing anything. I'll attach a minidump I collected from the hung firefox.exe process. It shows the exit() procedure is interrupted by Nahimic2DevProps.dll, which has been injected into our process and, if I'm reading the stack correctly, decided it needed to do a bunch of sleeping in its DLL_PROCESS_DETACH handler. [1] https://dxr.mozilla.org/mozilla-central/rev/c0ea5ed7f91a6be996a4a3c5ab25e2cdf6b4377e/toolkit/xre/nsUpdateDriver.cpp#727 [2] https://dxr.mozilla.org/mozilla-central/rev/c0ea5ed7f91a6be996a4a3c5ab25e2cdf6b4377e/toolkit/mozapps/update/updater/updater.cpp#2886

Interesting https://github.com/Kaldaien/SpecialK/blob/master/src/diagnostics/compatibility.cpp

Matt, were you going to take this or should I get one of the people that work on the DLL blocklist to?

I didn't intend to take this, it needs to be passed on for DLL blocklisting I think.

Matt, since you have reproduced, it would be best if you tried a blocklist entry to see if it actually worked.

Good point. I'll make that attempt later today.

Nahimic injects two DLL's into firefox.exe, Nahimic2OSD.dll and Nahimic2DevProps.dll. Adding Nahimic2DevProps.dll to the blocklist makes this problem go away. Unfortunately that one has no version info. Nahimic2OSD.dll does, but it doesn't seem relevant to this bug. I don't know if the contacts we had from bug 1233556 are still good, but we need to reach out to somebody at the vendor.

Requested tracking 53 in case we have a point release and there is a patch that is ready. [Tracking Requested - why for this release]: We already have two bug reports of this happening and it has also happened to an employee's system. I am concerned this that this product which prevents Firefox from exiting properly and in turn prevents updating Firefox is going to affect a significant number of clients.

(In reply to Matt Howell [:mhowell] from comment #11) > Nahimic injects two DLL's into firefox.exe, Nahimic2OSD.dll and > Nahimic2DevProps.dll. Adding Nahimic2DevProps.dll to the blocklist makes > this problem go away. Unfortunately that one has no version info. > Nahimic2OSD.dll does, but it doesn't seem relevant to this bug. > > I don't know if the contacts we had from bug 1233556 are still good, but we > need to reach out to somebody at the vendor. Note that the policy is that third-party DLLs are not allowed to inject into Firefox: https://blog.mozilla.org/addons/2017/01/24/preventing-add-ons-third-party-software-from-loading-dlls-into-firefox/. This means we can blocklist without reaching out first.

You can find contacts for some MSI/Nahimic people on the Rolodex: https://docs.google.com/spreadsheets/d/1H38b9RgunWtuUjuv0jc3wiSwcAbpc4Ee2T5egmEojQw/edit#gid=0&range=85:85. mhowell, can you take care of the patch and of contacting them?

I'll submit the patch. I don't have access to that Rolodex document, so I'm requesting it. But I would feel more comfortable if someone more familiar with this process and the relevant policies were responsible for the contact.

Until we know more, I will tag this as blocking 53 updates.

Comment on attachment 8860445 [details] Bug 1356637 - Blocklist a Nahimic DLL which is causing app update problems. https://reviewboard.mozilla.org/r/132436/#review135318

Pushed by mhowell@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/b0ccd97b40f4 Blocklist a Nahimic DLL which is causing app update problems. r=bsmedberg

I've contacted them.

Is Firefox 52 affected? If 52 is affected too, perhaps we should unblock updates (as updates would be blocked anyway by Nahimic).

Yes, I can reproduce the bug on 52.

(In reply to Marco Castelluccio [:marco] from comment #22) > Is Firefox 52 affected? If 52 is affected too, perhaps we should unblock > updates (as updates would be blocked anyway by Nahimic). Yes, according to the duplicate bug 1358492.

Hello, I have a question about the fix: from what I understand the fix blocks the nahimic dll from interacting with firefox. Does this mean that the audio enhancements of nahimic won't work in firefox? (ex while playing videos) thanks

Can anyone confirm that just the blacklist fixes the issue for them? i.e. if they had their nightly not updating since the install of Nahimic version 2.3.15, after yesterday's update, the updates actually work? Here is what I am observing, In the AppData/Local/Mozilla/updates/<some_alpha_num_folder>/active-update.xml, I see the to be installed build number as 20170422030205 , which is well after the patch for this bug landed in central, so the to-be installed version should contain the fix. Even then I see that the updater is not working and the same behavior (nightly process not shutting down) is happening. Its just a guess, but I think that the nightly.exe which actually applies the update is still corrupt with the nahimic dll and thus the updates wont work as only the future nightly.exe has the issue fixed. Maybe we need a special forced removal of some sorts? I believe the same behavior would be observed with folks stuck on Firefox 52 (release version) and they would also not be able to get automatic updates.

(In reply to Tortino from comment #27) > Does this mean that the audio enhancements of nahimic won't work in firefox? > (ex while playing videos) I do not know the answer to this question. I'd like to get the answer from Nahimic, but I haven't seen any reply to our attempts to contact them yet. (In reply to Girish Sharma [:Optimizer] from comment #28) > Its just a guess, but I think that the nightly.exe which actually applies > the update is still corrupt with the nahimic dll and thus the updates wont > work as only the future nightly.exe has the issue fixed. Your guess is correct. The fixed version has to be installed already before updates will start working.

(In reply to Matt Howell [:mhowell] from comment #29) > (In reply to Girish Sharma [:Optimizer] from comment #28) > > Its just a guess, but I think that the nightly.exe which actually applies > > the update is still corrupt with the nahimic dll and thus the updates wont > > work as only the future nightly.exe has the issue fixed. > > Your guess is correct. The fixed version has to be installed already before > updates will start working. So regular non-techy users are just stuck with the same version of firefox? (unless, nahimic fixes the core issue that the exe doesn't quit). Or are there plans to do something about it?

I can confirm that this patch effects the sound quality and levels of firefox when nahimic 2.3.15 is installed. Nahimic basically reduces the actual sound output to 50% and then applies all its effects on top of that reduces sound (in order to get smooth upper frequencies post effects). This also includes the smart loudness effect. Without all of such effects, the end result is that firefox's sound levels are reduced to 50% of original raw output levels. Effectively, this is around 30% of actual sound that was observed before this fix. I would try getting my hands on version 2.3.14 or 2.3.12 of nahimic to see if that contains the same nightly update issue. But I am not sure whether even 2.3.14 or below will work, since the patch had to block all versions of the nahimic dll, which might be present in 2.3.14 and below as well.

Correction: >> ... on top of that reduces sound (in order ... reduced

Unfortunately, Nahimic 2 doesn't support downgrade and I can't uninstall and install lower version right now..

Track 54+ as nahimic may prevent Firefox from updating to newer version, which will impact a lot of users.

I'm using the blocking flag here to indicate that we should uplift this blocklist entry for a 53 dot release. Can you request uplift to m-r and other branches? Or, does this need more testing? Thanks!

Benjamin, what do you think here? We can uplift this blocklist patch, but it may not get to the users already affected. They would have to wait for nahimic's update. If we put this into a 53 dot release, then anyone newly installing either 53 or nahimic will avoid the problem. I don't have a good idea of the user impact here (number of users potentially affected).

Given my comment #30 and #31, are we still looking at uplifting this fix as is? Its not much of help. Moreover, the overall sound experience of Firefox will deteriorate after the fix.

For reference: The workaround for this issue is to exit Nahimic from its notification area icon. Right-click the Nahimic logo icon in your notification area (AKA the system tray) and click Exit. You can do this while Firefox is running, and you can run Nahimic again after the update is installed.

Liz I would recommend taking this as a ridealong but not a driver for a 53 dot release.

hey joni, could we write-up a sumo article about this problem and the workaround in comment #38?

Hi :bsmedberg, Is this worth uplifting to Beta54?

Kev, Nahimic is asking us to remove the block once they fix the issue and push an updated version to their users. What should we do? The blocklist here is ALL_VERSIONS, as they are not versioning their DLL.

Please have a look here: https://blog.mozilla.org/addons/2017/01/24/preventing-add-ons-third-party-software-from-loading-dlls-into-firefox/

re comment 42, our policy is that 3rd-party DLLs should not be in the Firefox process, so my recommendation is to keep this block in place permanently. Gerry, did you read comment 39?

Looking at crash-stats, I don't think this can be affecting very many users. Let's go ahead and uplift to 54, since we want users to be able to update and we don't want DLL injection at all. Because of the low amount of users in this situation, I don't think this warrants uplift to 53.

Given comment 45, could you please request beta uplift?

I agree with blocking permanently, but I'd like to be 100% certain that we're going to do it so I can inform Nahimic.

Comment on attachment 8860445 [details] Bug 1356637 - Blocklist a Nahimic DLL which is causing app update problems. Approval Request Comment [Feature/Bug causing the regression]: N/A [User impact if declined]: Unable to install updates while Nahimic software is running [Is this code covered by automated tests?]: No [Has the fix been verified in Nightly?]: Yes [Needs manual test from QE? If yes, steps to reproduce]: No [List of other uplifts needed for the feature/fix]: None [Is the change risky?]: No [Why is the change risky/not risky?]: We know this change will cause loss of Nahimic audio effects on audio played in Firefox. That's the only functionality that could be affected. [String changes made/needed]: None

(In reply to Matt Howell [:mhowell] from comment #48) > [Why is the change risky/not risky?]: > We know this change will cause loss of Nahimic audio effects on audio played > in Firefox. That's the only functionality that could be affected. According to comment 32, Firefox will not only lose audio effects, but its maximum output volume will be reduced to 50% the actual maximum available on the system, can you confirm if that's actually the case?

Not right now, but later today I can. The machine I have that's running an MSI motherboard is at my home, and I'm in the office today.

I can't reproduce any volume drop while playing YouTube videos. At least in my configuration, Nahimic doesn't seem to be providing its own volume control, the one in its interface just mirrors the system control.

Comment on attachment 8860445 [details] Bug 1356637 - Blocklist a Nahimic DLL which is causing app update problems. Nahimic dlls will prevent Firefox from updating to newer version. This is a huge impact for users. We need to block the dlls. Beta54+. Should be in 54 beta 4.

The issue should be fixed in nahimic 2.5.16, released last Friday.

Kev replied to me on IRC and said we should keep the blocklisting in place permanently.