Because it's tricky to build these libraries natively on Windows, Rob suggested we could use a combination of docker/taskcluster to cross compile the binary shared libraries for all platforms on Linux.

If we use this approach, we could avoid importing the sources into the mercurial repository (bug 1518164). On the other hand, should the Thunderbird source distribution include the sources for those libraries? What happens if the original distribution site cannot be reached? Maybe it would be wise to mirror the source archives we're using somewhere on Mozilla infrastructure?

Detailed information about the libraries we want to use, download locations, and the verification I made.

File 1:

https://gnupg.org/ftp/gcrypt/libgpg-error/libgpg-error-1.36.tar.bz2
Size: 920542
sha256: babd98437208c163175c29453f8681094bcaf92968a15cafb1a276076b33c97c libgpg-error-1.36.tar.bz2
sha512: 6e5f853f77dc04f0091d94b224cab8e669042450f271b78d0ea0219658d059c9cab1ab0eaa020a4227f451520b417fc340b85971a6f5e144fa69be57e15df346 libgpg-error-1.36.tar.bz2

File 2:

https://gnupg.org/ftp/gcrypt/libgcrypt/libgcrypt-1.8.4.tar.bz2
Size: 2990108
sha256: f638143a0672628fde0cad745e9b14deb85dffb175709cacc1f4fe24b93f2227 libgcrypt-1.8.4.tar.bz2
sha512: b831fc337eb14806897e224b0d1e78d1f8e9db91dffa818a015a4aa104144e2d971e5a855904907ee2bb9990a9d526de32d6787e1cae98e28a65c3258b2b1ea1 libgcrypt-1.8.4.tar.bz2

For files 1 and 2, I also checked the distributed signatures, and saw the following:
gpg: Good signature from "Werner Koch (dist sig)" [unknown]
Primary key fingerprint: D869 2123 C406 5DEA 5E0F 3AB5 249B 39D2 4F25 E3B6
gpg: Good signature from "NIIBE Yutaka (GnuPG Release Key) <gniibe@fsij.org>" [unknown]
Primary key fingerprint: 031E C253 6E58 0D8E A286 A9F2 2071 B08A 33BD 3F06

File 3:

https://otr.cypherpunks.ca/libotr-4.1.1.tar.gz
Size: 655791
sha256: 8b3b182424251067a952fb4e6c7b95a21e644fbb27fbd5f8af2b2ed87ca419f5 libotr-4.1.1.tar.gz
sha512: c957206235b9f627542f23a645c822ea1b318d83fa655f41ed8d9a3c0dddf30b88f0ca82060026af155e48ecb13969dc9270831f20304669794151f666ae5cb0 libotr-4.1.1.tar.gz

For file 3, I also checked the distributed signature, and saw the following:
gpg: Good signature from "OTR Dev Team (Signing Key) <otr@cypherpunks.ca>" [unknown]
Primary key fingerprint: 22DF 3305 DF56 667C E157 84FC F24D E08F 42C2 ABAD

When I compiled the OTR library with mingw, I ran into build failures, no .dll file was produced.
I will attach two platform patches to fix the build. Only one of patches can be applied at a time.

Apply this patch to libotr when building for --host=i686-w64-mingw32

Apply this patch to libotr when building for --host=x86_64-w64-mingw32

To build, download the three source archives and both patches, and copy them to $HOME, then execute the following steps.

Set environment variables for the 32-bit build:

export PATCH=otr411-mingw-i686.patch
export HOST=i686-w64-mingw32
export PREFIX=$HOME/local-32

Set environment variables for the 64-bit build:

export PATCH=otr411-mingw-x86_64.patch
export HOST=x86_64-w64-mingw32
export PREFIX=$HOME/local-64

Common build steps:

mkdir $PREFIX
cd $PREFIX
tar xjf $HOME/libgpg-error-*.tar.bz2
tar xjf $HOME/libgcrypt-*.tar.bz2
tar xzf $HOME/libotr-*.tar.gz
cd libgpg-error*
./configure --host=$HOST --prefix=$PREFIX
make
make install
cd ..
cd libgcrypt*
./configure --with-pic --host=$HOST --prefix=$PREFIX --with-libgpg-error-prefix=$PREFIX
make
make install
cd ..
cd libotr*
cat $HOME/$PATCH | patch -p1
./configure --with-pic --disable-silent-rules --host=$HOST --prefix=$PREFIX --with-libgcrypt-prefix=$PREFIX
make
make install
cd ..

(EDIT: fixed 64-bit patch filename)

After building, copy the following four files to the Thunderbird distribution. They should be copied to the same directory that contains the other shared libraries we distribute with Thunderbird.

32-bit build:

$HOME/local-32/bin/libgpg-error-0.dll
$HOME/local-32/bin/libotr-5.dll
$HOME/local-32/bin/libgcrypt-20.dll
/usr/lib/gcc/i686-w64-mingw32/8.3-win32/libssp-0.dll

64-bit build:

$HOME/local-64/bin/libgpg-error-0.dll
$HOME/local-64/bin/libotr-5.dll
$HOME/local-64/bin/libgcrypt-20.dll
/usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/libssp-0.dll

The build instructions from comment 3 and following apply to the mingw build for the Windows platforms. I haven't yet tried to cross-compile for OSX.

(In reply to Kai Engert (:kaie:) from comment #1)

Because it's tricky to build these libraries natively on Windows,
What's tricky about building them? I don't believe I had issues compiling these with MSVC previously.

Rob suggested we could use a combination of docker/taskcluster to cross compile the binary shared libraries for all platforms on Linux.
This sounds like a terrible idea. It seems extremely complicated and custom.

If we use this approach, we could avoid importing the sources into the mercurial repository (bug 1518164). On the other hand, should the Thunderbird source distribution include the sources for those libraries?

Why would we avoid importing the source? I'm fairly certain we want to import it.

(In reply to Patrick Cloke [:clokep] from comment #8)

Why would we avoid importing the source? I'm fairly certain we want to import it.

Do you think we should import it, even if we don't use the regular build system to build that code?

(I don't understand yet how local builds will work, if distribution binaries are created using taskcluster/docker.)

Rob, FYI, in the past, the Torproject worked on a fork of Instantbird named Tor Messenger, which had also included the OTR libraries. Maybe it can help you in some way, it seems the old build scripts seem to be at https://gitweb.torproject.org/tor-messenger-build.git/tree/projects

(In reply to Kai Engert (:kaie:) from comment #9)

Do you think we should import it, even if we don't use the regular build system to build that code?

I'm confused at why we can't use the build system to build it? What's special about it that won't work?

(In reply to Patrick Cloke [:clokep] from comment #11)

I'm confused at why we can't use the build system to build it? What's special about it that won't work?

I think we don't want to roll our own buildsystem for those libraries.

The libgrypt page https://www.gnupg.org/software/libgcrypt/index.html says:
"Libgcrypt works on most POSIX systems and many pre-POSIX systems. It can also be built using a cross-compiler system for Microsoft Windows."

Based on that, I had concluded that cross-compiling is the only supported way to build those libraries. However, I gave it another attempt today. Below are the results.

I found a page that lists experiences with building on Windows:
http://techtootech.blogspot.com/2013/03/compiling-libgcrypt-win64.html

Building on Windows seems to require the MINGW toolchain. Does the mozilla-build environment for Windows support MINGW? I looked, but it doesn't seem to be included. If it's not, we'd have to get that added, our install it ourselves.

I manually installed MINGW in addition to mozilla-build on Windows. I manually tweaked PATH to also point to the gcc binaries from MINGW. The build ran, but there were a few issues.

The install target of the libgpg-error library fails because of a missing tool "msgfmt". This error doesn't occur when cross-compiling on Linux. However, the most important parts of the install target ran, the libs and headers were installed in the distribution area. So I could continue to build with the other targets.
Building and installing libgcrypt succeeded.
The libotr Makefile required a similar tweak as seen in the attached patches.
The libotr install target failed, because it attempts to set symbolic links. However, the libraries already built.

So, building on Windows is possible, but requires installation of additional tools on top of the regular mozilla-build tools, and it requires more manual tweaks than cross-compiling on Linux.

I think the primary question now is: Can we make the MINGW toolchain available on hosts that perform Windows builds in taskcluster? Are we OK to add this as an additional requirement for developers who build Thunderbird on Windows?

MINGW is not on the Windows build hosts. Getting it there would involve setting up a toolchain build and including it in the Taskcluster toolchains.

There are MINGW Linux -> Windows cross-compile Docker images that the build team maintains for Firefox. We can use one of those to build these libraries.

I'm going to try a few things and read up on the build system a bit more. There's some recommendations for how to deal with vendored libraries that apply here. I'll check with the CI/Build teams as well and see if they have suggestions.

We can also make it so if Windows builds are cross-compiled in a separate step that those build results are available to local builds with some "mach bootstrap" magic.

I did a little bit with this the other night, just trying things out to see what works, tackling Windows first since that seems to be the most problematic platform.

I've tried to build libgpg-error on my Windows dev VM, which is set up close to what we have in CI. I tried CLANG (both gcc and msvc-emulation modes) and MSVC. Problems manifest early on when generating the various bits that eventually create gpg-error.h. If we can generate that file another way, say with a mingw cross compiler on a local dev box, we could check in the resulting file and skip generating. I don't know how viable of an idea that is, it's just a possibility I ran across during my attempts. I did get past that in one incarnation of my crazy autogen.sh/configure commands only to be thwarted a missing unistd.h header file. (I guess that's a gcc-based compiler thing, and not part of the Windows headers?) What might work is to use the already present clang compiler with mingw header files (yes, that's a thing). I will try that next.

In contrast, I downloaded the Docker image and the mingw compiler build that Taskcluster has access to and it built on the first try without any issues. One problem though, is that the Firefox team only maintains mingw cross compiler that targets win32. The win64 build seems to have been removed. I plan to try this again using the clang cross compiler with the mingw header files as the build team does maintain this for both win32 and win64 targets. I expect this to work.

To sum up at the moment (libgpg-error):

• Cannot be built on Windows in CI. Either need to bring in a mingw compiler as a toolchain, or possibly just header files and use clang.
• Cross compiling works great, but can only target win32 so far.
• There's going to be complications with the cross-compiler/toolchain approach as Taskcluster doesn't know how to build toolchain artifacts with scripts that live outside of M-C. I'd have to add comm-central support.

I still prefer the idea of building these in a separate build step and making the results available as an artifact. Just seems unnecessary to build these libraries on every build if we're not going to be changing the code.

Do you think we should use the same approach for all target platforms?

I did consider a hybrid approach where we build in-tree for Linux/MacOSX and then do Windows in a separate build step, but that's rather confusing and I'd like to keep the different platforms as close to doing the same as we can.

The above try run has a "toolchain-linux64-gpgerror TL(gpgerror)" job that produces an artifact named gpgerror.tar.xz.

That tar file is the contents of a directory created with "make install" after building libgpg-error.

Some things to think about and will likely change:

• What do we really need in that file? "make install" also installs man-pages, translation files, pkgconfig, info docs .... I suspect that the contents of gpgerror/lib, gpgerror/include, and maybe gpgerror/bin will be sufficient.
• Translation files? yes or no?

The direction this is heading will wind up with 3-toolchains x 4 architectures, for a total of 12 jobs. That's perfectly fine. It makes it more complicated than necessary I think.

Unless there are objections, I am going to build all three of these libraries for each OS together and produce a single "otr.tar.xz" artifact. If for some reason we need to split them up later we can do that, but I kind of doubt it.

The downside is if one of those libraries gets an update, you're rebuilding all three, but we would probably do that anyway just to ensure compatibility.

(In reply to Rob Lemley [:rjl] from comment #18)

Some things to think about and will likely change:

• What do we really need in that file? "make install" also installs man-pages, translation files, pkgconfig, info docs .... I suspect that the contents of gpgerror/lib, gpgerror/include, and maybe gpgerror/bin will be sufficient.
• Translation files? yes or no?

We only want the shared libraries, .so for Linux, .dylib for OSX, .dll for Windows.

On Linux and OSX, we should also get the symbolic links for the shared library.

On Windows we need an additional library from the build system, see comment 7.

Unless there are objections, I am going to build all three of these libraries for each OS together and produce a single "otr.tar.xz" artifact.

Agreed.

The downside is if one of those libraries gets an update, you're rebuilding all three, but we would probably do that anyway just to ensure compatibility.

That's fine.

(In reply to Kai Engert (:kaie:) from comment #19)

(In reply to Rob Lemley [:rjl] from comment #18)

We only want the shared libraries, .so for Linux, .dylib for OSX, .dll for Windows.

I should be more clear because of the multiple steps involved. The artifact file I'm referring is only for use by Taskcluster builds and anyone that wants to use them for local builds. So, right now I'm looking at what will be required to build the code thats uses these libraries. So, include files like gpg-error.h?

There's going to be some work to do in the packaging step of the build to make sure files that need to be distributed to end-users get included, and there we will just include .so .dll and probably a text file or something for license compliance.

On Linux and OSX, we should also get the symbolic links for the shared library.

On Windows we need an additional library from the build system, see comment 7.

(In reply to Rob Lemley [:rjl] from comment #20)

We only want the shared libraries, .so for Linux, .dylib for OSX, .dll for Windows.

I should be more clear because of the multiple steps involved. The artifact file I'm referring is only for use by Taskcluster builds and anyone that wants to use them for local builds. So, right now I'm looking at what will be required to build the code thats uses these libraries. So, include files like gpg-error.h?

No, we really don't need anything else. We're using js-ctypes which directly opens the shared library and accesses the well known symbols. We don't require any header files for that.

Resulting artifact for linux64, contains only the lib*.so.* files for the three libraries and the symlinks.

To test, I uninstalled libotr5 from my debian system.

When running a plain local build of comm-beta, the OTR functionality was disabled (missing UI elements), as expected.

Then I copied all files from the attached artifact file to my dist/bin directory (ignoring the subfolder form the .tar.xz).
This worked, using strace I saw that the libotr.so.5 from dist/bin was loaded.

I couldn't test the gcrypt and gpg-error modules. Those are difficult to uninstall on a Linux system, because much of the base environment depends on it. When using strace to follow the .so files loaded by the Thunderbird process, those two libraries are loaded very early. The libotr.so.5 modules is loaded much later.

For Linux, the gcrypt and gpg-error modules we'll ship will likely be only used in fallback scenarios, if the system libraries aren't available for any reason.

Testing the Windows platform will be more interesting.

Rob, do you have an update? How much work is left from your side? Any realistic ETA?

I don't have much of an update, but now that the code is in tree I can get back to it hopefully.
There were some recent MacOSX toolchain updates that landed in version 70 that might prove helpful with that linking error I was running into. I can't really give an ETA unfortunately.

FYI, https://wiki.mozilla.org/Thunderbird:OTR has a download link to manually built shared libraries for Win32, Win64, Linux32, Linux64, OSX.

While testing the binaries for Mac OS X, I learned that we need to tweak the libraries, to ensure they can be loaded by Thunderbird.

When building a shared library that depends on other shared libraries, each dylib file refers to dependent libraries by embedding the full path of the dependent libraries. If the OTR libraries aren't installed in the referenced path, our Thunderbird js-ctypes code fails to load the OTR library. Because we want to allow installing the Thunderbird app anywhere on user systems, embedding a full path doesn't work.

I found a way to manually modify the dylib files, after they have been built.
(Inspired by information from http://osiris.laya.com/coding/dylib_linking.html )

(The manual commands to fix the dylib require the use of a OS X tool "install_name_tool". I haven't researched how we could change the build rules of the libraries to use different paths. Also, I haven't yet researched if this tool is available in cross compilation environments, maybe it will be required to set the correct rpath at cross compilation time.)

Details and manual fix:

On OSX, when using "otool -L libsomething.dylib", it will print a list of dependencies. The first listed entry is the "ID" of the file. The additional entries are dependent libraries.

In the files produced using the native library build on OSX, I see full path included for both ID and dependent libs.

I suggest to change the ID of the library by removing the path, and using the library filename, only.

The entries for dependent libs must be changed from "/Users/username/local-path/libsomething.dylib" to "@executable_path/libsomething.dylib".

The commands must be parameterized. The change parameter must specify the existing path, which can be obtained using "otool -L".

There's another potential issue, the space inside a library for storing the path is fixed at build time, so we can only shorten, but not increase the size of the specified path. To ensure this manual fix can work, we need to ensure the build --prefix (e.g. "/Users/username/local-path") is a longer string than the string "@executable_path".

First, obtain the path prefix that is contained in the dylib files. Execute "otool -L libotr.5.dylib", "otool -L libgcrypt.20.dylib", "otool -L libgpg-error.0.dylib". Check the prefixes printed before these library names.

I'll assume they are all identical. In the following I'll use "/Users/username/local-path", be sure to replace that with the prefix you see listed by the above commands.

install_name_tool -id libotr.5.dylib libotr.5.dylib
install_name_tool -change /Users/username/local-path/libgcrypt.20.dylib @executable_path/libgcrypt.20.dylib libotr.5.dylib
install_name_tool -change /Users/username/local-path/libgpg-error.0.dylib @executable_path/libgpg-error.0.dylib libotr.5.dylib

install_name_tool -id libgcrypt.20.dylib libgcrypt.20.dylib
install_name_tool -change /Users/username/local-path/libgpg-error.0.dylib @executable_path/libgpg-error.0.dylib libgcrypt.20.dylib

install_name_tool -id libgpg-error.0.dylib libgpg-error.0.dylib

With these changes, if the files are placed into the Thunderbird.app/Contents/MacOS/ directory, they will be successfully loaded at runtime.

We need to ensure that the libraries we distribute are compatible on the oldest system that the respective Thunderbird version still supports.

Thunderbird 68.x supports OS X version 10.9 and newer. The libraries I built on OS X 10.14 didn't work on 10.9,

The OTR libraries depends on the OSX system library /usr/lib/libSystem.B.dylib and when using otool -L libotr.5.dylib a version number is printed. This version number must not be higher than the number listed by other dylib files we distribute (e.g. libnss3.dylib).

(To produce backwards compatible libraries, I had to use an older version of Xcode, and I had to join the paid Apple Developer Membership in order to obtain an older version of Xcode.)

I downloaded TW64 libotr.tar.xz from https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=a73677dc6ba4a3514f9f35abc11877782750608a but the file doesn't contain gpg-error*.dll

libgpg-error6-0.dll is the file that is referenced by the provided libgcrypt-20.dll

Why are the dll's named differently for win32 vs win64? :)

https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&selectedJob=281822104

Note that linux and mac have the gcrypt and gpg-error libs statically linked in, only the windows builds have separate dlls.

(In reply to Rob Lemley [:rjl] from comment #39)

https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&selectedJob=281822104

Based on initial testing, these binaries appear to work with TB 68.x on the following Windows and Mac platforms:

• Win 10 64 bit with TB 64 bit in a VM
• Win 7 32 bit with TB 32 bit on an older netbook
• Mac OS 10.14
• Mac OS 10.9.5

Thanks! I tested Linux64 on TB 68.3.1 and that seems to be working well.

(In reply to Rob Lemley [:rjl] from comment #39)

https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&selectedJob=281822104

Based on initial testing, these binaries appear to work with TB 68.x on the following Windows and Mac platforms:

• Win 10 64 bit with TB 64 bit in a VM
• Win 7 32 bit with TB 32 bit on an older netbook
• Mac OS 10.14
• Mac OS 10.9.5

We have a successful build!
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=c92b6e59cdddf5c5d16da9d9dcdade90067953fd

The only thing not quite working is the artifact caching.
I tested the linux x64 build, and aside from a UI bug that's new in Daily it's working.

I need to go through all of the files one last time and add some comments and such before submitting for final review. And we still need the rest of bug 1609987 in M-C.

Pushed by thunderbird@calypsoblue.org:
https://hg.mozilla.org/comm-central/rev/05deee4d9421
Add build script for libotr and dependencies.
https://hg.mozilla.org/comm-central/rev/11a79da52f31
Add thirdparty kind for building libotr libraries.
https://hg.mozilla.org/comm-central/rev/32b795794bfa
Add thirdparty libotr as build dependency.
https://hg.mozilla.org/comm-central/rev/e0e05018f705
Include libotr and dependencies from build artifacts.