While working on the key migration wizard in Enigmail, I noticed that Thunderbird is getting very slow if the number of keys grows.

I imported 2 secret keys and 150 public keys.

1. After importing these keys, starting Thunderbird took almost 30 seconds (until the GUI was shown on the screen).

2. Importing the keys got slower and slower, as more keys are stored.

This is the log at the start of the process, with no keys in RNP:

2020-04-26 12:46:43.655 [DEBUG] setupWizard2.importKeys: pubFile: /tmp/enig-exp/0x3AE11208126222D1EFC1A856FB30385AB3E48E06.asc
2020-04-26 12:46:43.864 [DEBUG] setupWizard2.importKeys: pubFile: /tmp/enig-exp/0xF9273C0F0798863836FBF1D540859460C20ED58D.asc
2020-04-26 12:46:44.065 [DEBUG] setupWizard2.importKeys: pubFile: /tmp/enig-exp/0x6B1385827C0C7DD04BA5E29996B7E7F20CDD03C5.asc
2020-04-26 12:46:44.097 [DEBUG] setupWizard2.importKeys: pubFile: /tmp/enig-exp/0xD9E322378AE9F4157EB4EB3BBFA731E5CEDCDE8C.asc
2020-04-26 12:46:44.301 [DEBUG] setupWizard2.importKeys: pubFile: /tmp/enig-exp/0x38E3C2E8C5536623245CCEA7FA232783A323C417.asc

And this is the log file for the last few keys:

2020-04-26 12:49:16.841 [DEBUG] setupWizard2.importKeys: pubFile: /tmp/enig-exp/0xFE5DFAB8AC8E05CBABBD5BEEA5F6ADEC44F69881.asc
2020-04-26 12:49:20.453 [DEBUG] setupWizard2.importKeys: pubFile: /tmp/enig-exp/0x32B975E24AB29305614BE1CB7CD496F81F1873B0.asc
2020-04-26 12:49:24.017 [DEBUG] setupWizard2.importKeys: pubFile: /tmp/enig-exp/0xE0A063BE274BE5E7446A86890095B03D659ADB2A.asc
2020-04-26 12:49:27.537 [DEBUG] setupWizard2.importKeys: pubFile: /tmp/enig-exp/0x005BBB1F717E511B78F3946EAD1BDE567F73D9CC.asc
2020-04-26 12:49:31.483 [DEBUG] setupWizard2.importKeys: pubFile: /tmp/enig-exp/0xB79B0A23EB24809BBAEE5DB6F7F0E70F307D56ED.asc

One of the things I noticed during my tests is that Thunderbird is quite unresponsive while RNP is doing something.

Maybe it would be a good idea move accessing cytpes function-calls to a ChromeWorker or a PromiseWorker, which runs on a separate native thread. This would have the advantages that:

1. the main thread would not be blocked by RNP (and GPGME) function-calls, which would at least give a feeling of better performance to the user
2. improve the speed, as RNP operations and Thunderbird stuff could run in parallel.

Analyzed this a bit more. Saving the keyring is extremely slow. It takes about 3 seconds for 150 keys, and a lot more if the keyring grows. I suggest that we address this with the RNP developers.

Apart from this, as all RNP operations are running on the main thread, any other operations in parallel are blocked. I'll try to work on this, and move RNP-operations to a PromiseWorker.

A quick update: I implemented a prototype that uses a PromiseWorker and moved OpenPGP initialization to the Worker. Witthout my change, it took ca. 10 seconds from launching TB until the GUI was visible. Using the Worker, the GUI was displayed in 3-4 seconds.

On 77.0b2 (64-bit) On Ubuntu 18.04; I am running into problems importing any keys beyond a handful; console logs are not showing many errors, buy I am seeing these errors:

"Cannot load required GPGME library"

As well as

CryptoAPI.sync() failed result: Error: "RNP.importToFFI failed"
importKeyBlock chrome://openpgp/content/modules/RNP.jsm:1216
importKeyFromFile chrome://openpgp/content/modules/cryptoAPI/RNPCryptoAPI.jsm:111
importKeyFromFile chrome://openpgp/content/modules/keyRing.jsm:350
EnigmailCommon_importKeysFromFile chrome://openpgp/content/ui/commonWorkflows.js:107
oncommand chrome://openpgp/content/ui/enigmailKeyManager.xhtml:1

CryptoAPI.sync() leaving interface.js:55:13
TypeError: res.importedKeys is undefinedkeyRing.jsm:360:9
importKeyFromFile chrome://openpgp/content/modules/keyRing.jsm:360
EnigmailCommon_importKeysFromFile chrome://openpgp/content/ui/commonWorkflows.js:107
oncommand chrome://openpgp/content/ui/enigmailKeyManager.xhtml:1

Jon, thanks for testing and giving feedback. However, it seems your issue isn't the same as the slowness reported here, so could you please file a separate bug for your issue? Please also be more detailed than saying "running into problems", please describe what hapened, and what you expected. Thank you.

Patrick, will the use of a PromiseWorker spawn real operating system threads? This wouldn't be safe currently, RNP is explicitly NOT thread safe.

(In reply to Kai Engert (:KaiE:) from comment #5)

Jon, thanks for testing and giving feedback. However, it seems your issue isn't the same as the slowness reported here, so could you please file a separate bug for your issue? Please also be more detailed than saying "running into problems", please describe what hapened, and what you expected. Thank you.

Submitted as bug 1638194 - and reiterating here, very happy to test more; I work with a lot of human rights defenders who rely on TB+PGP in various forms around the world, so I definitely want to help this transition go smoothly once it goes out of beta.

(In reply to Kai Engert (:KaiE:) from comment #6)

Patrick, will the use of a PromiseWorker spawn real operating system threads? This wouldn't be safe currently, RNP is explicitly NOT thread safe.

Yes, I think so. The model I have in mind would mean that all operations involving RNP would work in one thread (but not the main thread). I believe that this could work safely. However, I noticed that the code in TB would require quite some substantial changes. And I would currently not recommend to do that. I'd first wait for an improved version of RNP with faster loading and saving of the keyring.

Patrick, any progress? If you had a WIP it might be worth attaching here so it's not lost

At the end, it was not working properly. It would require deep changes to RNPCryptoAPI.jsm and RNP.jsm.

(In reply to Patrick Brunschwig from comment #10)

At the end, it was not working properly. It would require deep changes to RNPCryptoAPI.jsm and RNP.jsm.

Are there bugs filed for that?
Or, is a different approach required?

There are no bugs filed for it. My approach would have required an almost complete rewrite of RNPCryptoAPI.jsm and RNP.jsm. But I don't have enough time to work on it.