User Agent: Mozilla/5.0 (Windows NT 5.1; rv:52.0) Gecko/20100101 Firefox/52.0

Steps to reproduce:

On a clean Windows 7 installation (with all MS updates) and with Comodo Firewall 5.12.256249.2599 and Avira Antivirus installed (so far no other software installed) Firefox 74.0 crashes on startup (a message window pops-up saying "Firefox has stopped working"). Firefox crashes on Windows 7 since versions greater then 70.0.1. Versions 70.0.1 and below run fine.
After I installed and tried to start 74.0 (which failed) I re-installed 70.0.1 again and that version works normally.

I've been using Firefox and Comodo Firewall 5.12 together for many many years (since Windows 2000, Windows XP) without problems.
I have read about some other Firefox startup crash bugs related to Comodo software so I think my problem could be in the same category but I'm not sure.

Actual results:

Firefox crashes on startup on Windows 7. Firefox window does not show up at all.

Expected results:

Normal startup, showing Firefox window on Windows 7.

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Bugbug, I'm not sure whether your component change is correct or not. But I think the Product should be Firefox so I made the change.

hi, this is likely the same issue or similar to bug 1590430. please try to update or else ditch comodo.

Hello,

I don't think this is a duplicate bug because there is no "iseguard64.dll" on the whole Windows 7 partition (not even a match for "ise*.*") . However, there is a "guard64.dll" file located in the Windows 7 "c:\Windows\System32" directory and one in "c:\Windows\SysWOW64\guard32.dll".
Could it be that these dll files are causing the same issue as the "iseguard64.dll" file?

I was hoping that Mozilla could fix this issue so that I still could continue to use Comodo Firewall 5.12 (I know there are newer versions but they come with a lot of stuff that I don't need.)

Is there any chance from Mozilla to fix the Firefox crash for this Comodo version 5.12?

Thanks.

Hello,

Also referring to my previous comment...
Please note that no Comodo Antivirus has been installed on Windows 7 only the Firewall part is installed which might cause this issue (still not sure though).

Please feel free to close this bug for good if Mozillas final decision is to not fix this issue for Comodo Firewall 5.12.

Thank you very much.

Additional information:

To prove that Comodo Firewall 5.12 is causing the Firefox 74.0 startup crash I did the following.

1. Created an image of the Windows 7 HD partition for backup purpose.
2. Booted into Windows 7 (with Comodo Firewall 5.12 and Firefox 70.0.1 installed, both working normally).
3. Uninstalled Comodo Firewall 5.12 and as instructed by the uninstall software rebooted Windows 7.
4. Installed Firefox 74.0 by upgrading version 70.0.1 to 74.0.
5. Started Firefox 74.0 successfully (no crash).
6. Installed Comodo Firewall 5.12 again and as instructed by the install software rebooted Windows 7.
7. Started Firefox 74.0 again, this time unsuccessfully (crash).
8. Uninstalled Comodo Firewall 5.12 once again and as instructed by the uninstall software rebooted Windows 7.
9. Started Firefox 74.0 again successfully (no crash).
10. Backup image of step 1 put back on Windows 7 HD partition.

Above steps show clearly that Firefox 74.0 conflicts with Comodo Firewall 5.12.
Could you please clarify what the root cause of this conflict is and if this issue could be solved by Mozilla?

Thanks you.

(In reply to nototheleft from comment #6)

Could you please clarify what the root cause of this conflict is and if this issue could be solved by Mozilla?

It's almost certainly due to dlls injected by comodo into the firefox process, which are not dealing correctly with changes made in Firefox since 70.0.1. But we have no visibility into what those dlls do so there is very little we can do, apart from attempting to block the dlls if/when we know what they are -- this will likely break whatever functionality Comodo tries to offer within Firefox, but would also stop the crashes.

I have a few questions to try to help narrow down what's happening here:

1. when Firefox 74 crashes due to Comodo, does the Firefox crash reporter come up?
2. if you start Firefox in safe mode (pass --safe-mode on the commandline), does it work?
3. if the answer in (1) was "yes", if you start Firefox 70.0.1, and visit about:crashes, do you see crash reports from (1) (judging by the date/time), and if so, can you open them and then copy/paste the URLs in a comment here?
4. if the answer for (1) was "no", can you visit about:crashparent in Firefox 70.0.1 -this will crash Firefox - and then do the same as in (3) ? Those crash reports will tell us the names of all the DLLs in the process, and might help figure out what dll is causing the crashes.

Thank you!

Hello,

Thank you for getting back on this issue!

Here is already one answer to your questions:

1. No, I don't see any Firefox window appearing after the crash. So no, I don't see a crash reporter come up. I attached the picture of the Windows 7 message window which pops-up immediately after the crash.

I get back to you for the answers on questions 2, 3 and 4 (need some time for that to check)

Thank you.

Here are all the answers in sequence of execution.

1. Answer to this question (as already given) is no.
2. When starting Firefox 74.0 from the commandline with "E:\Mozilla Firefox>firefox --safe-mode" it still crashes immediately, no Crash Reporter window comes up.
3. Answer in (1) is no, so no further action here at the moment.
4. Downgraded Firefox 74.0 to 70.0.1 first (I guess that is the intention) and started Firefox 70.0.1 and visted "about:crashparent". As expected Firefox 70.0.1 crashed and comes up with Crash Reporter window. In the Crash Reporter window (see attached picture) I clicked on "Restart Firefox" and visited "about:crashes" as per instruction in question (3), Firefox window says "No crash reports have been submitted."

I repeated the whole sequence once more:

• Installed 74.0 again.
• Started 74.0 and let it crash.
• Installed 70.0.1 again
• Started 70.0.1 and visited "about:crashparent" and let it crash.
• Crash Reporter window comes up (see attached picture).
• Restarted Firefox 70.0.1 and visited "about:crashes".
• Firefox 70.0.1 showing a empty window saying "No crash reports have been submitted."

If all the above steps were executed correctly then, unfortunately, it seems that there is no crash report generated by Firefox 74.0 when it crashes.

Thank you.

We can solve this issue but we'll need you to collect a minidump for us to be able to identify the offending DLL. Since the crash is happening too early for our crash reporter to catch it you'll have to collect a user-mode dump. The instructions to do so are here but here's quick run-down:

• Using regedit.exe create the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps registry key. The key can be left empty as the default settings are fine.
• Launch Firefox and wait for it to crash
• Go into C:\Users\<username>\AppData\Local\CrashDumps and see if there's a minidump in there (.dmp extension)
• Send the minidump to me via e-mail so that I can analyze it

Important note: do not attach the minidump to this bug. It may contain sensitive information so it should not be public.

Ok, will do.
Need some time again.

Thanks

Done, I have sent the minidump via email to you.

Thanks

I've identified the DLL causing the crash. It's guard64.dll version 5.12.59641.2599. I'll make a build that blocks it on try so that we can test if blocking fixes the problem.

Great!
Looking forward to test it!

This is a test build of Firefox nightly that blocks the DLL causing the crash:

https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/C7PggDMwQzq6pnCGxIefUw/runs/0/artifacts/public/build/install/sea/target.installer.exe

Can you try it and report if it works?If you don't have nightly installed it will create a new profile so it won't touch your main Firefox profile. You can simply uninstall it once you're done testing it.

I'll download the file and test it and let you know if it works.

Looks like it worked? Yay!

As for pingsender, it is a small executable that comes with Firefox that is used to send telemetry (that otherwise is sent by Firefox itself). You can turn off sending telemetry in the Firefox options.

I'm pretty sure it doesn't itself try to modify any registry keys (the full code for Windows is at https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/pingsender/pingsender.cpp and https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/pingsender/pingsender_win.cpp ) - it just calls standard Windows APIs to say "send this http request" (Firefox normally doesn't do so; it has its own http stack. pingsender doesn't use that, it's deliberately designed to be very lightweight). If those end up touching windows registry settings internally, that's a windows issue...

I've installed the Firefox Nightly 76.0a1 build and it starts without crashing!!!
Great!!
I've attached a screenshot of it!

After closing down Firefox Nightly I got Comodo Firewall pop-up Alerts of Firefox Nightly trying to execute the unknown "PingSender.exe" executable in trying to access system resources and an Alert on "PingSender.exe" trying to access the internet. As a test case I kept on blocking all Alerts generated "PingSender.exe" to check if "PingSender.exe" would finally give up trying to access the resources and trying to access the internet. It did give up after many blocking Alerts, great. I've attached screenshots of those Alerts too.

So to me your bug fix seems to work for Comodo Firewall 5.12.
Maybe it needs some more testing from my side but first results are looking good!

I do have a question regarding the fix you've implemented in Firefox Nightly.
Is this fix common for all Comodo Firewall versions including the most recent one (version 12)? In other words, does the most recent version of Comodo Firewall also inject dlls into the Firefox process(es) and is it blocked (by the fix you made) more or less in the same way or does the most recent version of Comodo Firewall inject dlls in a more compatible way so that it not conflicts with the Firefox process(es) or maybe the most recent version does not inject dlls at all (using another protecting scheme).
Also can you elaborate on, from the feedback given in comment #7 "this will likely break whatever functionality Comodo tries to offer within Firefox", what the impact of the fix is regarding the Comodo protecting level given to Firefox (i.e. what kind of protection still works and what not)

Thank you very much!

@Gijs your reply was much quicker then mine, I didn't see your reply because I was in the middle writing my reply, I even got a collision error while saving changes :)
Thanks for the feedback!

(In reply to nototheleft from comment #25)

After closing down Firefox Nightly I got Comodo Firewall pop-up Alerts of Firefox Nightly trying to execute the unknown "PingSender.exe" executable in trying to access system resources and an Alert on "PingSender.exe" trying to access the internet. As a test case I kept on blocking all Alerts generated "PingSender.exe" to check if "PingSender.exe" would finally give up trying to access the resources and trying to access the internet. It did give up after many blocking Alerts, great. I've attached screenshots of those Alerts too.

That's normal because none of the executables are recognized, they're all freshly built.

So to me your bug fix seems to work for Comodo Firewall 5.12.
Maybe it needs some more testing from my side but first results are looking good!

I do have a question regarding the fix you've implemented in Firefox Nightly.
Is this fix common for all Comodo Firewall versions including the most recent one (version 12)? In other words, does the most recent version of Comodo Firewall also inject dlls into the Firefox process(es) and is it blocked (by the fix you made) more or less in the same way or does the most recent version of Comodo Firewall inject dlls in a more compatible way so that it not conflicts with the Firefox process(es) or maybe the most recent version does not inject dlls at all (using another protecting scheme).

This patch will only stop the version that was crashing Firefox and older versions. New versions of the software will be unaffected.

Also can you elaborate on, from the feedback given in comment #7 "this will likely break whatever functionality Comodo tries to offer within Firefox", what the impact of the fix is regarding the Comodo protecting level given to Firefox (i.e. what kind of protection still works and what not)

The DLL was probably modifying Firefox behavior in some way - possibly to block URLs and the like. That functionality won't be available with the patch in place because the DLL is blocked from loading.

I have done some more testing of the Nightly build. Till now everything seems fine.

For your information, I started ProcessExplorer to check the loading of "guard64.dll" after starting Firefox 70.0.1 and after starting FirefoxNightly 76.0a1 (both were run separately, so not at the same time). I've attached some screenshots of ProcessExplorer.
As can be seen from the screenshots:

• There are two instances of "guard64.dll" loaded by Firefox 70.0.1 (one for the main process and one for a sub-process)
• There is only one instance of "guard64.dll" loaded by FirefoxNightly 76.0a1 in a sub process (by the fix the main process doesn't load it anymore)

I was wondering why the sub-process of FirefoxNightly 76.0a1 doesn't crash on the loading of "guard64.dll" and if FirefoxNightly as a whole is still nicely protected by Comodo Firewall even though the main process doesn't load it?

Yes, we can block DLL loading both in the main process and in child processes. I think that our default policy is to block loading DLLs only where it prevents the crash, not everywhere by default.

I understand, blocking it everywhere would cause a serious security risk as there would be no or less protection from Comodo Firewall.
The ultimate solution would be to not block the DLL from loading anywhere in the process but I guess that can't be solved I'm afraid (or can it?).

A small correction to the number of loaded instances of "guard64.dll" by Firefox 70.0.1 as reported earlier.
All processes (main and all child processes) of Firefox 70.0.1 do load "guard64.dll" whereas FirefoxNightly 76.0a1 only loads one instance in a child process.
Please see newly attached screenshot showing both Firefox 70.0.1 and FirefoxNightly 76.0a1 running in ProcessExplorer.
I my earlier screenshot I didn't sort ProcessExplorer Search window on Processes (but on PID) so I didn't see all running processes of Firefox 70.0.1 in the first place.

To sum it up:

• Firefox 70.0.1 has in total 5 running processes (main and childs) and has also 5 loaded instances of "guard64.dll" for each process one.
• FirefoxNightly 76.0a1 has in total 6 running processes (main and childs) but has only one loaded instance of "guard64.dll" for a child process.

I get the feeling or the impression that by the fix FirefoxNightly is very much less protected by Comodo Firewall then Firefox 70.0.1.
This lower protection is undesirable and makes FirefoxNightly more prone to potential attacks and higher serious security risks.

Is there an another way (a more clean way) to fix the crash so that Comodo Firewall 5.12 still can give the full protection to Firefox(Nightly).

Thank you.

Unfortunately not, you'll have to update that software to a newer version. That particular program is changing how Firefox behaves and it's simply incompatible with the newer versions of Firefox. That being said a firewall should work even without a browser extension so I doubt that disabling its injection process changes much with regard to that.

I understand your point of view and you're correct about that the firewall will still work.
However, Comodo Firewall is not only a plain firewall but also has a feature called Defense+ which monitors each executable (and its child processes) and other running processes what they are doing on a system (i.e. is an executable or running process allowed to access or modify certain registry keys or allowed to access certain system directories or allowed to make specific modification to files and what more...) that feature makes Comodo Firewall so powerful in protecting the whole system.
If you are interested you can read more about Defense+ here: https://help.comodo.com/topic-72-1-284-3036-General-Settings.html

I would love to upgrade the software but it comes which such an amount of other software and other features that I really don't need so I really don't like to start using it.

I don't know the ins and outs of Firefox and why suddenly after version 70.0.1 things have had to change so drastically in Firefox.
Comodo Firewall 5.12 was designed and released to run on Windows 7 and 8, unfortunately Firefox now breaks its functionality.

Anyhow, either I'll stay with 70.0.1 or upgrade or use the fix as is I don't know. There is obviously no other solution.

Thank you.

In addition to my previous comment:

• It is the Comodo Defense+ feature that is broken by the fix, not the firewall feature (maybe you already got that point). Comodo Defense+ can't monitor the main Firefox process or it's child processes (executed by itself or by Add-ons or by Extensions) any longer and as such all Firefox processes are free to make whatever changes to the system they want to. This is highly unwanted.
• Upgrading Comodo to the latest version would introduce many new (and yet unknown) bugs as it comes with plenty of other software and features. I did have a look at the Firefox bug-list regarding Comodo V8 and v10 and that doesn't make me happy (I'm not saying that Firefox alone is causing all the bugs, Comodo creates bugs too). This is also unwanted.

What if I would upgrade Comodo to v12 anyway on Windows 7?
Would I then get the full Comodo protection towards Firefox that Comodo is designed to offer to Firefox?
Would the Defense+ feature (in the latest version v12 they call the feature HIPS) fully work on Firefox or does Firefox above version 70.0.1 keep Comodo v12 HIPS out of the door (block it) as well (not by the Comodo Firewall 5.12 fix but generally speaking)?
In other words, do any Firefox versions (in particular the versions above 70.0.1) cripple any feature of the latest Comodo version v12 on Windows 7 or Windows 10?

Thanks.

Bugbug thinks this bug is a regression, but please revert this change in case of error.

(In reply to nototheleft from comment #39)

What if I would upgrade Comodo to v12 anyway on Windows 7?
Would I then get the full Comodo protection towards Firefox that Comodo is designed to offer to Firefox?
Would the Defense+ feature (in the latest version v12 they call the feature HIPS) fully work on Firefox or does Firefox above version 70.0.1 keep Comodo v12 HIPS out of the door (block it) as well (not by the Comodo Firewall 5.12 fix but generally speaking)?
In other words, do any Firefox versions (in particular the versions above 70.0.1) cripple any feature of the latest Comodo version v12 on Windows 7 or Windows 10?

Firefox doesn't purposefully "cripple" any features from any external products. The history in bug 1590430 suggests newer versions of Comodo will have a higher chance of working correctly, but that's not a guarantee. Whether Comodo's interference will cause crashes in Firefox again, or how much of its featureset is supported in Firefox or any other browsers is pretty much up to Comodo.

I don't expect Firefox to purposefully "cripple" any third party features but blocking (any) third party features to prevent Firefox from crashing is in my opinion not the right way to go.

Please can you tell if currently all features (specifically HIPS) of Comodo v12 will/do work with Firefox above 70.0.1?
Else, what sense does it make to upgrade from 5.12 to 12 when the HIPS feature is still being disabled by Firefox.

Furthermore, do Firefox and Comodo work together to solve bugs like these?
Comodo's browser Dragon uses Firefox code so I expect you work together somehow.

Any comments on attached screenshots perhaps???

At this point we are only blocking very old versions of COMODO software that we know cause Firefox to crash thus harming our users. Recent versions (including 12) are likely to run but nobody here can guarantee that. That's entirely up to COMODO. We have absolutely no control over that.

To summarize this bug report:

• Firefox versions 70.0.1 and below worked perfectly well together with Comodo Firewall/Defense+ 5.12 and there were absolutely no crashes in the past. Firefox was all the time very well protected by Comodo Firewall and its Defense+ feature.
• All Firefox versions above 70.0.1 crashed immediately after startup when Comodo Firewall/Defense+ 5.12 was installed and running.
• The initial, at first sight, root cause of the startup crash was related to a Comodo DLL.
• Firefox stated that Comodo Firewall is not compatible with Firefox versions above 70.0.1 and as such causes the startup crash of those Firefox versions.
• A Firefox Nightly build 76.0a1 was made to block the offending Comodo DLL from loading into the Firefox process as a workaround to stop the startup crashes.
• The Firefox Night build 76.0a1 didn't crash at startup anymore, so the blocking seemed to work.
• However, the blocking of the offending Comodo DLL in Firefox Nightly 76.0a1 completely disabled Comodos Defense+ feature offered towards Firefox and as such Firefox and all its Add-ons and Extensions can now make whatever changes to the underlying system they want to. The Comodo Defense+ feature is crippled by blocking the offending DLL and not any longer able to Alert the user about suspicious system changes. This poses a serious security threat.
• Firefox version 74.0 was installed again together with Comodo Firewall/Defense+ 5.12 running.
• As expected Firefox 74.0 did again crash immediately at startup.
• A configuration setting change was made in Firefox 74.0 to disable the Firefox LauncherProcess during startup.
• Very very surprisingly Firefox 74.0 now started without crashing and runs normally together with Comodo Firewall/Defense+ 5.12 running !!!!!! See the newly attached screenshots !!!!!!
• Firefox 74.0 now is, like Firefox 70.0.1, again very well protected by Comodos Defense+ feature because it now loads the offending DLL in all its processes (main and childs) without crashing.
• In Firefox versions 70.0.1 and below the LauncherProcess was all the time enabled by default but those Firefox versions did not crash !!!!!!
• It is not, as initially thought, the offending Comodo DLL that is causing the startup crash of Firefox versions 70.0.1 and above but a bug in Firefox LauncherProcess is causing the Firefox startup crash when Comodo Firewall/Defense+ 5.12 is running.

Firefox made a statement that Comodo Firewall/Defense+ 5.12 is simply not compatible with Firefox versions above 70.0.1.
Above I have proven that this statement is incorrect and just not true, it is Firefox itself that makes Comodo Firewall/Defense+ incompatible by blocking the loading of the "offending" DLL in the Firefox LauncherProcess executed at startup and then just let Firefox crash to make the user think it is incompatible.

Firefox policy is to block all DLL injections in an effort to improve stability, security, and privacy.
And as such cripples any third party features that injects DLLs for their needs.
In my opinion this policy is just a cover and an opportunity to build a base for a Trojan Horse.

(In reply to nototheleft from comment #47)

In my opinion this policy is just a cover and an opportunity to build a base for a Trojan Horse.

It's really not -- though if you already believe this, I'm not sure anything I'm going to say is going to make you change your mind. You know all the source code for Firefox is public, right? If it was a trojan horse, people would notice pretty quickly.

It's useful to know that things work if you disable the launcher process. To be clear, it's Comodo's dll that causes the crash here, not Firefox "deciding" to crash in that case. If the launcher process breaks when this old version of Comodo injects into it, perhaps we can avoid that somehow. Though frankly, it's pretty unreasonable for applications doing perfectly legitimate things to have to make changes to make other, unmaintained, external applications that inject random executable code into their process, and for which no source code is available, happy -- and it's not clear that would even be possible.

Gabriele, is there anything in the minidump that'd allow you to see what part of the launcher process upsets Comodo and how we could make it less upset?

We're crashing deep into Microsoft DLLs. This seems to happen here when we're querying for GetSystemTimePreciseAsFileTime() in kernel32.dll. That's a perfectly valid call and it should simply yield NULL on the target machine because it's a function that's not available on Windows 7. This particular instance of COMODO injected a hook in Microsoft's KERNELBASE.dll and my guess is that since it's older than the method we're requesting it's breaking because it doesn't know what to do with that function. That idea is reinforced by the fact that the crash is a NULL pointer access.

Is it not common practice to first check what OS is running so one can make existing system calls?
Making non-existing system calls on any OS is asking for trouble in any case.

What's the minimum OS system requirement for Firefox 74.0? I guess it should be Windows 8 then.

(In reply to nototheleft from comment #50)

Is it not common practice to first check what OS is running so one can make existing system calls?

We're not making a system call, we're requesting the pointer to that function in a DLL. If it's not there we'll get NULL and we know we can't use it. That's the standard way of checking for optional functionality in Windows DLLs.

What's the minimum OS system requirement for Firefox 74.0? I guess it should be Windows 8 then.

Still Windows 7. That particular function is only used if it is found.

Do you check for optional (non Windows 7) functionality in Windows DLLs only in the Launcher Process code?

Do you check for optional (non Windows 7) functionality in Windows DLLs only in the Launcher Process code?

No, we definitely do this elsewhere as well. This same call for example is also being checked in the JS engine, so I guess that might crash as well due to the Comodo bug: https://searchfox.org/mozilla-central/search?q=GetSystemTimePreciseAsFileTime&path=

Mr. Toshihito

Can you please explain and clarify for what reasons and for what purposes Firefox detours ntdll!LdrLoadDll ?

https://hg.mozilla.org/mozilla-central/rev/5a8b721ced71

(In reply to nototheleft from comment #54)

Can you please explain and clarify for what reasons and for what purposes Firefox detours ntdll!LdrLoadDll ?

Our purpose is to collect information about DLLs which are loaded into our processes. In my understanding, the situation where two parties try to hook the same function is not a problem. It's even technically possible that multiple hooks co-exist.

(In reply to Gabriele Svelto [:gsvelto] from comment #49)

We're crashing deep into Microsoft DLLs. This seems to happen here when we're querying for GetSystemTimePreciseAsFileTime() in kernel32.dll. That's a perfectly valid call and it should simply yield NULL on the target machine because it's a function that's not available on Windows 7. This particular instance of COMODO injected a hook in Microsoft's KERNELBASE.dll and my guess is that since it's older than the method we're requesting it's breaking because it doesn't know what to do with that function. That idea is reinforced by the fact that the crash is a NULL pointer access.

Gabriele, could you share the dump file with me? I'd like to analyze it, too. I know the chance is really slim, but there might be something we can implement to mitigate the issue, or something we can improve for similar cases.

Thank you for your feedback, really appreciate it.

To help you finding a good solution and a better fix for this bug I did some debugging and things crash at the point when Firefox detours LdrLoadDll, guard64.dll is loaded into a child process and also detours LdrLoadDll and then when a call is made to LoadLibraryW the whole thing crashes (everything happens in the Launcher Process code part).

If I have read the Firefox source code well then detouring LdrLoadDll is not thread safe (as stated by a comment in the Firefox code) and in my opinion this is what's causing the crash.

Hopefully now you are able too come up with a real good fix for this bug (not by blocking the loading of "guard64.dll" I mean) .

If you need more help then ask me.

Thank you.

Management,

I would like to know how you follow up on this bug.
Should this bug be reopened or a new bug be created?
Or do you leave everything as it is?

Thank you.

This bug is fixed, so there's no point in re-opening it.

Mr. Toshihito was interested in finding out what Comodo did wrong that they later fixed, so if this leads to something actionable that removes the need for the block, he'd either follow up here or make a new bug and link it to this one. We're reaching out to Comodo to get the exact version that you observed the problem with.

Thank you for your answer.

If you like I can send you the exact version as well, I still have the official and original setup file.
You can have it verified by Comodo if you don't trust.

I would like to follow Mr. Toshihito findings or analysis. Also when a new bug is made I would like to follow that as well. Can you arrange that please? It would be interesting to know why Firefox crashes in the way I explained earlier in comment#58.

Thanks.

Sorry for not responding promptly. I spent some time on analyzing the dump Gabriele shared with me, and debugging the latest Comodo F/W on Win10 which works fine with the latest Firefox.

(In reply to nototheleft from comment #58)

To help you finding a good solution and a better fix for this bug I did some debugging and things crash at the point when Firefox detours LdrLoadDll, guard64.dll is loaded into a child process and also detours LdrLoadDll and then when a call is made to LoadLibraryW the whole thing crashes (everything happens in the Launcher Process code part).

Yes, and as Gabriele mentioned, the crash happened here when Firefox called LdrLoadDll to load kernel32.dll to query GetSystemTimePreciseAsFileTime. GetSystemTimePreciseAsFileTime does not exist in Win7, but it can't be a problem.

Unfortunately the hook planted on LdrLoadDll is not captured in the minidump, so it's impossible to tell what exactly happened.

Looking at the code adjacent to the crashing point, mov r11, imm64 is found just before the crashing instruction as below. This means something to us because the pattern of our hook is mov r11, imm64; jmp r11. Plus, the crashing instruction add is syntactically ok, but I think it does not mean anything but just two zero bytes.

00000000`6fff0d1f cc              int     3
00000000`6fff0d20 49bbc0245c3fff251400 mov r11,1425FF3F5C24C0h <---- probably jumped here from `KERNELBASE!LoadLibraryExW` via `KERNELBASE!_imp_LdrLoadDll`
00000000`6fff0d2a 0000            add     byte ptr [rax],al  <---- crash here (invalid opcode)
00000000`6fff0d2c cc              int     3

The latest Comodo F/W coexists with our hook. In that case, our hook comes first, and then their hook is planted on top of ours. Given those, my guess is in the crash case, they failed to fully hook LdrLoadDll because their code could not fully recognize our hook's pattern. To prove this, we need a local repro or a full dump of the crash.

If I have read the Firefox source code well then detouring LdrLoadDll is not thread safe (as stated by a comment in the Firefox code) and in my opinion this is what's causing the crash.

In the minidump, there was only one thread causing AV. So at least this crash is not a threading issue.

(In reply to nototheleft from comment #61)

Thank you for your answer.

If you like I can send you the exact version as well, I still have the official and original setup file.
You can have it verified by Comodo if you don't trust.

Thank you for offering that. Yes, could you please share the setup files to get a repro on our end? I cannot guarantee that I'll find the root cause, but it would be helpful.

I know that analyzing and debugging code can take a lot of time so your late reply is no problem at all. I'm glad that you did some analyzing and debugging so far so thank you for that.

If I have some time I will try to do some more in-depth debugging and collect more info around the crash point and share it with you.

The Comodo FW 5.12 setup file size is about 70 Mb. This is too large to send it as a whole via e-mail I believe. I could split it up in ten smaller zip-parts and then send those zip-parts via e-mail to you.
I don't know whether attaching to this bug is allowed or not by you or by Comodo as this bug is publicly.
Please let me know what way you prefer to get the setup file to you.

Thanks.

(In reply to nototheleft from comment #63)

I don't know whether attaching to this bug is allowed or not by you or by Comodo as this bug is publicly.
Please let me know what way you prefer to get the setup file to you.

I talked with my manager. To avoid the risk to violate a license, let me discuss about our option within the company first. Sorry that I was in a rush.

For your information:

The Comodo FW 5.12 software was free software back in the days. Same as it is now for the latest version 12 of Comodo software.
You don't need a license key to install or run the software.

Anyhow, when you reach agreement about sending the software to you then just let me know.

Hello and to all,

I was wondering if there is any news regarding getting the Comodo FW 5.12 software for your local repro?
Will there be any follow up on this bug so that this "guard64.dll" gets unblocked?
I can do more debugging and share the info with you if you want but that only makes sense when Firefox is willing to work on this too.

I see in the bug list that there are daily reported crashes related to "guard64.dll" even for the latest version of Firefox 75.
See this link:

https://crash-stats.mozilla.org/search/?release_channel=release&signature=%3Dguard64.dll&product=Firefox&version=73.0.1&version=74.0&version=74.0.1&version=75.0&date=%3E%3D2020-02-17T00%3A00%3A00.000Z&date=%3C2020-04-19T12%3A06%3A00.000Z&_facets=install_time&_facets=version&_facets=address&_facets=moz_crash_reason&_facets=reason&_facets=build_id&_facets=platform_pretty_version&_facets=signature&_facets=useragent_locale&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports

How does Firefox solve these crashes? Also by just blocking the "offending" dll?

If there is no interest from your side to fix this crash issue (or these crash issues from the link above) in a proper way (meaning not dll blocking) then I'll use Firefox 74 and above with a disabled Launcher Process setting as long as Firefox permits to disable it (I guess it won't take long before you block this disabling as well) as this gets back the FW 5.12 Defense+ functionality towards the Firefox browser.
Maybe disabling the Launcher Process is also a recommendation for other users who totally don't understand why certain functionality - which was working correctly in older Firefox versions - suddenly doesn't work anymore by crashing Firefox.

Maybe you should start thinking of solving all other crashes related to injected dlls in a correct and proper way instead of just blocking them or building a "wall" around Firefox. Building this wall decreases the user experience and confidence in Firefox and as such taking away users from using Firefox...

(In reply to nototheleft from comment #66)

I was wondering if there is any news regarding getting the Comodo FW 5.12 software for your local repro?

It's still under discussion. Probably there is no problem that we get the setup files from you, but we need to be very careful.

How does Firefox solve these crashes? Also by just blocking the "offending" dll?

It depends on root causes, and it's even not guaranteed that we can know root causes. If we can understand a root cause and we can fix or bypass it on our side, we're really willing to implement it, but please understand it's possible that we technically cannot bypass it. In such a case, blocking a module is the most reasonable mitigation for us and users. Keeping Firefox's browsing functionality is more important than Firefox's crash even though a third-party application's functionality is lost.

If there is no interest from your side to fix this crash issue (or these crash issues from the link above) in a proper way (meaning not dll blocking) then I'll use Firefox 74 and above with a disabled Launcher Process setting as long as Firefox permits to disable it (I guess it won't take long before you block this disabling as well) as this gets back the FW 5.12 Defense+ functionality towards the Firefox browser.

Maybe disabling the Launcher Process is also a recommendation for other users who totally don't understand why certain functionality - which was working correctly in older Firefox versions - suddenly doesn't work anymore by crashing Firefox.

It's hard to say what's a proper way. From Firefox's perspective, injecting code into our processes is not a proper way to interact with us. It's technically possible, but we basically don't expect such interaction because we think our process is our terriotory.

If a third-party application decides to inject code in our process, we want them to very careful because we can cover only a tiny part of their fault on our end. In this Comodo's case, it's still possible that we cannot bypass their crash at all. If so, a proper way is that Comodo provides a fix in their product. Actually they did in a newer version because we confirmed their latest product works good with our latest product.

Therefore, our recommendation is always to upgrade Comodo. Probably Comodo also recommends it. If you want to keep an old version and Comodo does not provide a patch for that version, maybe disabling the launcher process is the only solution but please understand it's not our recommendation.

we also have a policy against any such dll injections in place: https://blog.mozilla.org/addons/2017/01/24/preventing-add-ons-third-party-software-from-loading-dlls-into-firefox/

(In reply to nototheleft from comment #69)

According to this policy the latest Comodo product is going to be blocked too.

That's not correct. This change blocks guard64.dll equals to or older than 5.12.59641.2599. As of now, the latest Comodo FW contains gurad64.dll v12.2.2.7032, which coexists in Firefox keeping its functionality.

Have you tried the latest Comodo FW recently? I confirmed it worked, but if it's blocked on your end, there's a different problem we need to look into.

And not only Comodo, all other security suits that use DLL injection will undergo the same fate.

Why do you think so? We block injected DLLs only when they affect our functionality. As described here, we respect third-party apps as much as possible.