This is a necessary task for bug 22687.
We should reuse an existing open source library that provides processing of OpenPGP keys and data.

The library's license must allow linking with Thunderbird code, and allow distributing of this "larger work" as MPL.

There are a few candidates, each with their own advantages/disadvantages.
Also, it's difficult to make a decision without having tried to use the library, so we might start experimenting with one library, and potentially might run into problems at a later time that might cause us to change plans.

I'd like to use this bug to track the general discussion, and as a tracker for individual related work items.

I'd like to explore if the RNP library can be used: https://github.com/rnpgp/rnp
It depends on the Botan crypto library: https://github.com/randombit/botan/

While RNP exports a pure C API, the Botan library heavily uses C++ exceptions internally. Mozilla's codebase doesn't allow the use of C++ exceptions, so linking the Botan in to library the Thunderbird build seems problematic or even impossible.

I've experimented linking only to the RNP library, and have Botan only as an indirect dependency, and that seemed to work.

However, direct linking might not be necessary at all. Because we consider to reuse a large part of the Enigmail extension, which is all JavaScript (including MIME processing), it might be easiest to use the js-types interface to work with the RNP library.

Please use no new encryption implementation, because the existant gnupg binary and library components are approved!

This is really a sensitive security point and up to now there seems to be no really problem to use the existant gnupg binary with the Enigmail-Plugin.
When you want or must use another encryption, please implement the standard way to use GnuPG too.

OpenPGP is proprietary and can be seen as history!
https://www.thesecuritybuddy.com/pgp-and-gpg/pgp-vs-openpgp-vs-gnupg/

GnuPG is distributed under the terms of the GNU General Public License.
GnuPG is active under developement closing new security problems.
(Like https://www.heise.de/security/meldung/Verschluesselung-GnuPG-verschaerft-Integritaets-Checks-4075908.html)

GnuPG is standard installed in most of the Linux distributions, especially when they are based on Debian.
Please ask Ladar Levison and think about implementing of darkmail https://darkmail.info/

Let's use this bug as a general tracker related to Thunderbird's use of OpenPGP engines.

It might be useful to have a small abstraction layer between the TB integration code, and the OpenPGP engine that provides processing of raw data packets. Enigmail already implements an JS interface in content/modules/cryptoAPI/interface.js

Enigmail used two engines to provide parts of that interface. It bundled the "openpgpjs" library. However, TB should avoid bundling "openpgpjs" because of its many dependencies on nodejs module.

Most of the interface was implemented by calling out to GnuPG. We don't want to use GnuPG by default, because we want to avoid the complexity of bundling GnuPG, and because of the license situation.

However, let's keep the existing glue code to call GnuPG for another while.

I've created RNP/Botan binaries for initial testing, no guarantees given for those binaries:
https://kuix.de/mozilla/openpgp/libs-20191115/

FYI, here are links to mailing list posts with some explanations and thoughts. Feel free to join the discussion on that list.

Why we intend to not use GnuPG by default:
https://mail.mozilla.org/pipermail/tb-planning/2019-December/007287.html

Regarding smartcard support, see this discussion:
https://mail.mozilla.org/pipermail/tb-planning/2019-December/007288.html

updated test libraries:
https://kuix.de/mozilla/openpgp/libs-20200217/

My experiences with the RNP library are good so far. We should probably use it for TB 78.