There are crash reports at least from 3 different machines, minimum affect OS version is 10.16.0 20A5364e

Crash report: https://crash-stats.mozilla.org/report/index/46d78d91-a289-46bc-bf6a-093580200930

Reason: EXC_BAD_ACCESS / KERN_INVALID_ADDRESS

Top 8 frames of crashing thread:

0 CoreFoundation _CFGetNonObjCTypeID 
1 CoreFoundation CFRunLoopSourceSignal 
2 libeToken.dylib libeToken.dylib@0x27c18 
3 libeToken.dylib libeToken.dylib@0x2802d 
4 libeToken.dylib libeToken.dylib@0x6c0c0 
5 libsystem_pthread.dylib _pthread_start 
6 libsystem_pthread.dylib thread_start 
7 libeToken.dylib libeToken.dylib@0x6bffa 

This appears to concern a third party program. libeToken.dylib seems to be part of the Aladdin Etoken Pro Driver.

And there are lots of crashes with libeToken.dylib on the stack, in many different versions of macOS:

https://crash-stats.mozilla.org/search/?proto_signature=~libeToken.dylib&date=%3E%3D2020-10-02T16%3A21%3A00.000Z&date=%3C2020-10-09T16%3A21%3A00.000Z&_facets=signature&_facets=platform_version&_facets=version&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-platform_version

The disproportionate number of crashes on the ESR branch is due to the fact that these crashes aren't throttled (unlike those on release branch).

A crash report with this signature has appeared that contains interesting mac_crash_info:

bp-6da20163-add2-479d-9cfe-a2f980210603

    {
      "num_records": 1,
      "records": [
        {
          "message": "Invalid dylib load. Clients should not load the unversioned libcrypto dylib as it does not have a stable ABI.",
          "module": "/usr/lib/libcrypto.dylib"
        }
      ]
    }

14 crashes for 11 macOS installations of Firefox 102.0. Dennis, could you take a look at this?

This doesn't look like its in code we control and its not code we've touched recently as far as I can see. Any thoughts Dana?

The bug is marked as tracked for firefox102 (release). However, the bug still isn't assigned.

:haik, could you please find an assignee for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit auto_nag documentation.

These crashes are presumably a bug in Aladdin's driver for its eToken Pro, or in whatever's succeeded it. My link from comment #1 no longer works. The eToken product range seems to have been taken over by a company called Thales. From what I've seen elsewhere, it's likely that the driver software has its own libcrypto.dylib, but isn't loading it -- it's loading Apple's libcrypto.dylib instead. This would be the bug that triggers the crashes.

Yeah, this is third-party code. We could block that library, maybe? osclientcerts should be providing this functionality in its stead anyway.

hey folks, the crash is happening on my machine right now. Let me know if there is anything I can do to help with the troubleshooting.

(In reply to pedroreys from comment #10)

hey folks, the crash is happening on my machine right now. Let me know if there is anything I can do to help with the troubleshooting.

Do you have (and use) an "eToken" device? (It would look like a USB drive.) Where is libetoken.dylib located in your computer's file system?

Edit: To find libEtoken.dylib, open Terminal and do the following:

1. cd /
2. find . -name libEtoken.dylib

This may take a while (possibly half an hour or more). You'll see lots of error messages. You may find more than one match. If so, please list them all.

(In reply to pedroreys from comment #10)

hey folks, the crash is happening on my machine right now. Let me know if there is anything I can do to help with the troubleshooting.

Please also give us links to one or more of your crash reports. You should be able to find them in about:crashes.

Hey I do have and use sporadically an eToken device.

In the pkcs11.txt file in my profile (under ~/ApplicationSupport/Firefox/Profile/{profile_name}/pkcs11.txt) I had the following entry:

library=/usr/local/lib/libeTPkcs11.dylib
name=eCPF

In /usr/local/lib I didn't find that file. But I found it in /usr/local/lib/pkcs11 and that file was a symlink to libetoken.dylib:

ls -l /usr/local/lib/pkcs11
total 0
lrwxr-xr-x  1 root  admin  76 Jan 15  2021 libIDPrimePKCS11.dylib -> /Library/Frameworks/eToken.framework/Versions/Current/libIDPrimePKCS11.dylib
lrwxr-xr-x  1 root  admin  69 Jan 15  2021 libeTpkcs11.dylib -> /Library/Frameworks/eToken.framework/Versions/Current/libeToken.dylib

To test if this was what was causing the crash, I removed that libeTPkcs11.dylib library entry from pkcs11.txt and after doing so I was able to open Firefox without it crashing.

Just out of curiosity, I checked what was in that /Library/Frameworks/eToken.framework/Versions/Current/ directory and there are several files related to the "SafeNet Authentication Client" from Gemalto:

total 28192
ls -l
total 28176
drwxr-xr-x  4 pedroreys  staff      128 Oct 22  2019 Resources
-rw-r--r--@ 1 pedroreys  staff  5694204 Oct 22  2019 SACHelp.pdf
-rwxr-xr-x  1 pedroreys  staff   811056 Oct 22  2019 SACMonitor
-rwxr-xr-x  1 pedroreys  staff    73040 Oct 22  2019 SACSrv
-rwxr-xr-x  1 pedroreys  staff  1452624 Oct 22  2019 SACTools
drwxr-xr-x  3 pedroreys  staff       96 Jan 15  2021 SafeNet Authentication Client Tools.app
drwxr-xr-x  3 pedroreys  staff       96 Jan 15  2021 SafeNet Authentication Client.app
drwxr-xr-x  3 pedroreys  staff       96 Oct 22  2019 aks-ifdh.bundle
drwxr-xr-x  3 pedroreys  staff       96 Oct 22  2019 aks-vr.bundle
-rw-r--r--  1 root       wheel      487 Oct 22  2019 com.SafeNet.SACMonitor.plist
-rw-------  1 root       wheel      433 Oct 22  2019 com.SafeNet.SACSrv.plist
-rwxr-xr-x  1 pedroreys  staff   169904 Oct 22  2019 eTokend
drwxr-xr-x  3 pedroreys  staff       96 Oct 22  2019 eTokend.tokend
drwxr-xr-x  3 pedroreys  staff       96 Oct 22  2019 ikey-ifdh.bundle
-rwxr-xr-x  1 pedroreys  staff    32704 Oct 22  2019 libClassicClientPKCS11.dylib
-rwxr-xr-x  1 pedroreys  staff   283652 Oct 22  2019 libClassicClientTokenEngine.dylib
-rwxr-xr-x  1 pedroreys  staff    32688 Oct 22  2019 libIDPrimePKCS11.dylib
-rwxr-xr-x  1 pedroreys  staff   583856 Oct 22  2019 libIDPrimeTokenEngine.dylib
-rwxr-xr-x  1 pedroreys  staff    71264 Oct 22  2019 libSACLog.dylib
-rwxr-xr-x  1 pedroreys  staff   796416 Oct 22  2019 libSACUI.dylib
-rwxr-xr-x  1 pedroreys  staff   284656 Oct 22  2019 libcardosTokenEngine.dylib
-rwxr-xr-x@ 1 pedroreys  staff  1032400 Oct 22  2019 libeToken.dylib
-rwxr-xr-x  1 pedroreys  staff   120480 Oct 22  2019 libeTokenHID.dylib
-rwxr-xr-x  1 pedroreys  staff    57328 Oct 22  2019 libeTokenIfdh.dylib
-rwxr-xr-x  1 pedroreys  staff    29552 Oct 22  2019 libeTokenVR.dylib
-rwxr-xr-x  1 pedroreys  staff   207168 Oct 22  2019 libetvTokenEngine.dylib
-rwxr-xr-x  1 pedroreys  staff  2199604 Oct 22  2019 libgclib.dylib
-rwxr-xr-x  1 pedroreys  staff    55168 Oct 22  2019 libiKeyIfdh.dylib
-rwxr-xr-x  1 pedroreys  staff   348960 Oct 22  2019 libikeyTokenEngine.dylib

BTW, here is the link to my latest crash report: bp-7e9c4519-4aec-489f-8fec-ff1ac0220629

Incidentally, presumably you use that library to access client authentication certificates. Firefox now has the ability to access client certificates in the keychain by default - if you disable that library, are you able to use your client authentication certificates as expected? You may not even need that library.

As I understand it (and please tell me if I'm wrong), an eToken device can be prompted to display a passcode that you then use for multi-factor authentication. I doubt this can be handled by Firefox's osclient implementation ... but please let me know if I'm wrong.

As best I can tell, documentation for Mozilla's osclient stuff is here. You presumably no longer need to change any settings to turn it on.

pedroreys, can you report these crashes to Gemalto, Thales, or whoever currently maintains the "SafeNet Authentication Client"? This is really their bug.

Edit: I assume the contents of /usr/local/lib were installed at the same time as /Library/Frameworks/eToken.framework/ (or ~/Library/Frameworks/eToken.framework/). Let me know if they came from somewhere else. They're not system files.

(In reply to pedroreys from comment #13)

library=/usr/local/lib/libeTPkcs11.dylib
name=eCPF

None of my Firefox profiles' pkcs11.txt files have this block. I assume it was added by the "SafeNet Authentication Client"'s installer.

Closing as invalid. Leaving the needinfo for myself to try to contact Thales and report back.

There are 155 of these crashes over the last week in FF 102.0.1, which together makes them the #1 Mac topcrasher. Someone should try to contact Thales.

I assume "someone" means you, Haik :-)

https://crash-stats.mozilla.org/search/?mac_crash_info=~should%20not%20load%20the%20unversioned%20libcrypto%20dylib&platform=Mac%20OS%20X&date=%3E%3D2022-07-07T14%3A42%3A00.000Z&date=%3C2022-07-14T14%3A42%3A00.000Z&_facets=signature&_facets=proto_signature&_facets=platform_version&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports

(In reply to Steven Michaud [:smichaud] (Retired) from comment #19)

There are 155 of these crashes over the last week in FF 102.0.1, which together makes them the #1 Mac topcrasher. Someone should try to contact Thales.

I assume "someone" means you, Haik :-)

Thanks, Steven. We have some contacts and are working on it.

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #9)

Yeah, this is third-party code. We could block that library, maybe? osclientcerts should be providing this functionality in its stead anyway.

We don't have any functionality to block this dylib specifically, but our macOS entitlement hardening work (for bug 1606778 ) might be able to accomplish this. That should allow us to turn remove the macOS entitlement com.apple.security.cs.disable-library-validation for the parent process blocking loading of any modules not signed by us or Apple. However, Firefox does use the com.apple.security.smartcard entitlement, described here, and added in bug 1593041. I don't know if the smartcard entitlement includes allowing third party dylibs to be loaded.

@Dana, are you familiar with how the smartcard entitlement and loading of these dylibs might be related? Could you confirm Firefox no longer needs to load any of these type of libraries for smart card or crypto devices?

I don't think the smartcard entitlement is related to loading dylibs.

The goal of osclientcerts is to make it so it isn't necessary to load these kinds of libraries, but there are still some remaining issues that may mean for some users that it isn't yet a suitable replacement.

This bug is very complex (involving, as it does, third party applications/utilities, possibly several of them at once). And there doesn't (yet) seem to be a clear path to dealing with it by unilaterally disabling one or more of those third party apps. But there is one clear, simple rule that must always be followed -- never allow /usr/lib/libcrypto.dylib to be loaded on macOS and OS X systems. Going back many years, the code that initializes this dylib also crashes it -- the ___report_load function, to which the first pointer in its __mod_init_func section points.

Apple's justification for this is that, if you use Apple's "crypto" functions, you should really be using versioned copies, like those found in libcrypto.0.9.7.dylib, libcrypto.0.9.8.dylib, libcrypto.35.dylib, libcrypto.41.dylib, libcrypto.42.dylib, libcrypto.44.dylib and so forth.

This rule does not apply to libcrypto.dylib from any other vendor than Apple. It's almost certain that the third party apps that trigger these crashes are trying (and failing) to use a non-Apple libcrypto.dylib located somewhere else than /usr/lib .

But the files in /usr/lib are system files -- which means they're Apple files. It's always been difficult to replace these with non-Apple files. And since macOS 11 this has basically been impossible. So it's safe to assume that /usr/lib/libcrypto.dylib is always an Apple system file, and must never be allowed to load.

(Following up comment #22)

I just double-checked, and found that the self-crashing /usr/lib/libcrypto.dylib dates back only to macOS 10.15 (Catalina). On earlier versions of macOS and OS X it's a soft link to one of the versioned dylibs. So, strictly speaking, we'd only need to prevent loading /usr/lib/libcrypto.dylib on macOS 10.15 (Catalina) and up. But it probably wouldn't hurt to prevent it from loading on all supported versions of macOS. Non-Apple apps that try to load /usr/lib/libcrypto.dylib are almost certainly trying to load a non-Apple libcrypto.dylib, located elsewhere. Apple apps will (presumably) always load one of the versioned dylibs.

As I understand it, Firefox currently supports back to macOS 10.12 (Sierra).

(Following up comment #21 and comment #22)

Now I realize that this information might not help us. A lot of the recent crash stacks are badly corrupt. But as best I can tell, it's always libeToken.dylib that (mistakenly) tries to load /usr/lib/libcrypto.dylib, after Mozilla code uses PR_LoadLibraryWithFlags() to load libeToken.dylib.

So, as a last resort (if Thales refuses to deal with this bug), Mozilla could refuse to load libeToken.dylib. But it probably couldn't stop libeToken.dylib from loading /usr/lib/libcrypto.dylib (or anything else for that matter). Doing that would require hooking one or more system calls, at least temporarily.

I just reached out to Thales about this.

The libeToken.dylib included with the SafeNet Authentication Client tries to load "libcrypto.dylib". Thales could probably fix this bug by instead explicitly loading one of Apple's version-specific copies of libcrypto.dylib, using its full path -- for example "/usr/lib/libcrypto.0.9.8.dylib".

I just checked, and found that /usr/lib/libcrypto.0.9.8.dylib is available back to OS X 10.9, and is still available in macOS 13 beta.

(In reply to Steven Michaud [:smichaud] (Retired) from comment #26)

The libeToken.dylib included with the SafeNet Authentication Client tries to load "libcrypto.dylib". Thales could probably fix this bug by instead explicitly loading one of Apple's version-specific copies of libcrypto.dylib, using its full path -- for example "/usr/lib/libcrypto.0.9.8.dylib".

I just checked, and found that /usr/lib/libcrypto.0.9.8.dylib is available back to OS X 10.9, and is still available in macOS 13 beta.

I just confirmed this with a HookCase hook library. I also found out that Thales will need to change the path parameter to dlopen() (from "libcrypto.dylib" to "/usr/lib/libcrypto.0.9.8.dylib") in two different modules -- libeToken.dylib and libSACLog.dylib.

Copying crash signatures from duplicate bugs.

Here's the hook library I used to test the fix I suggested in comment #27. I've uploaded it as a diff on https://github.com/steven-michaud/HookCase/blob/master/HookLibraryTemplate/hook.mm.

I loaded it into Firefox, then used the STR from bug 1781753 comment #6. I didn't crash.

HC_INSERT_LIBRARY=/full/path/to/hook.dylib /Applications/Firefox.app/Contents/MacOS/firefox

Edit: I tested on macOS 12.5 and macOS 10.14.6 (build 18G9323).

Has Thales made any progress on this bug?