User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.122 Safari/537.36

Steps to reproduce:

0. Running on Windows 10 v1909 64-bit, thunderbird installed using chocolatey with Russian language pack. Thunderbird version is 78.0.1 (64-bit).
1. Open thunderbird.
2. Go to menu (≡ near top right corner).
3. Select entry “Параметры учётной записи” (“account preferences”?). It has icon which looks like rounded rectangle separated into two regions using vertical split, with two long and one short line in the left region and nothing in the right.
4. In the newly opened tab in the left navigational bar select subtab “Сквозное шифрование” (“end-to-end encryption”?) belonging to the only account I have configured.
5. In the subtab which gets opened there is paragraph header “OpenPGP” with line “{Key drawing} Thunderbird doesn’t have a personal OpenPGP key for napav@spels.ru [{Key drawing with +}Add Key…]”. Press [{Key drawing with +}Add Key…] button.
6. Export secret key somewhere by running gpg --export-secret-keys {hex} > key.gpg in cygwin (powershell adds some crap to the start of the exported key which renders it unreadable by Thunderbird if you do the same thing there, -o key.gpg is not helpful). Using GnuPG 2.2.21 with libgcrypt 1.8.6, GnuPG installed using chocolatey as well.
7. After step 5 dialog “Add a Personal OpenPGP Key for napav@spels.ru” appears. Select option “Import an existing OpenPGP Key” there.
8. Press [Continue] button.
9. Press [Select File to Import…] button.
10. Choose key exported in step 6 in the file open dialog.
11. After choosing file with key Thunderbird lists keys found in that file, one (and the only) of those keys appears in the blue rounded rectangle. Select or leave unselected “Treat this key as a Personal Key” button, neither option appears to help.
12. Press [Continue] button.

Actual results:

After following those steps the “Add a Personal OpenPGP Key for napav@spels.ru” dialog displays “Importing your OpenPGP Keys…” with a “busy” circle below (small rotating blue ring with varying color intensity). It never actually finishes importing key and Process Monitor does not report any seeming relevant activity from Thunderbird process when it does that.

While “importing” happens other tabs (like Inbox) work just fine, Process Monitor also reports that Thunderbird is attempting to receive new mail every once in a while. You may also close account preferences tab.

Key in question is both password-protected (secret key) and not password-protected (public key). The same thing happens when trying to use non-password-protected key I generated for testing purposes and included in the attachement. Note though that actual key appears in gpg --list-secret-keys as

sec   rsa4096 2017-10-12 [SCA] [expires: 2022-10-11]
      XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
uid           [ultimate] napav <napav@spels.ru>
ssb   rsa4096 2017-10-12 [E] [expires: 2022-10-11]

while test key appears as

sec   rsa4096 2020-07-30 [SC] [expires: 2025-07-29]
      92D9A968249979F878925F786DA81535D6A233B0
uid           [ultimate] TESTS <napav@spels.ru>
ssb   rsa4096 2020-07-30 [E] [expires: 2025-07-29]

I do not know what does A mean there.

Key in question worked just fine with Enigmail.

Expected results:

Key imported successfully.

Some things which do not fix the problem:

1. Removing Enigmail extension and disabling the only other one present (Thunderbird Conversations).
2. Uninstalling gnupg-modern chocolatey package. (Did not try to uninstall gnupg though.)
3. Running Thunderbird with env PATH='C:\Program Files\Mozilla Thunderbird' thunderbird from Cygwin (that ended up having entries C and C:\Program Files\Mozilla Thunderbird in PATH).
4. Running Thunderbird with env -i PATH='/cygdrive/c/Program Files/Mozilla Thunderbird' thunderbird from Cygwin (that ended up having exactly one entry in PATH: C:\Program Files\Mozilla Thunderbird, with also SYSTEMROOT and WINDIR environment variables set to C:\WINDOWS and some MOZ_CRASHREPORTER* environment variables according to Process Explorer).

5. Moving key to C:\pp2 and setting permissions on the whole folder (and key) to essentially an equivalent for u=rx,g=,o= (owner may read and execute/list folder contents, others are allowed to do nothing).
6. Running Thunderbird as admin.
7. Attempting to import keys into fresh profile after minimal configuration (one account + enabling openpgp).

I see the same behavior, getting the following 2 errors:

CryptoAPI.sync() failed result:  Error: rnp_key_protect failed
    importKeyBlockImpl chrome://openpgp/content/modules/RNP.jsm:1676
    sync chrome://openpgp/content/modules/cryptoAPI/interface.js:51
    importKeyFromFile chrome://openpgp/content/modules/keyRing.jsm:361
    openPgpImportStart chrome://openpgp/content/ui/keyWizard.js:761
    wizardNextStep chrome://openpgp/content/ui/keyWizard.js:194
    wizardContinue chrome://openpgp/content/ui/keyWizard.js:143
    _fireButtonEvent chrome://global/content/elements/dialog.js:487
    _doButtonCommand chrome://global/content/elements/dialog.js:466
    _handleButtonCommand chrome://global/content/elements/dialog.js:460
interface.js:46:17


Uncaught (in promise) TypeError: res.importedKeys is undefined
    importKeyFromFile chrome://openpgp/content/modules/keyRing.jsm:373
    openPgpImportStart chrome://openpgp/content/ui/keyWizard.js:761
    wizardNextStep chrome://openpgp/content/ui/keyWizard.js:194
    wizardContinue chrome://openpgp/content/ui/keyWizard.js:143
    _fireButtonEvent chrome://global/content/elements/dialog.js:487
    _doButtonCommand chrome://global/content/elements/dialog.js:466
    _handleButtonCommand chrome://global/content/elements/dialog.js:460
keyRing.jsm:373:36

(In reply to Kim A. Brandt from comment #3)

I see the same behavior, ...

It might have something to do with the keys already being available in the `OpenPGP Key Manager' and having to set the following?!

OpenPGP Key Manager -> Key Properties -> Your Acceptance -> Yes, ...

It now works for me.

User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0

(In reply to Kim A. Brandt from comment #4)

...
It now works for me.
...
Encryption and decryption now works. The hanging issue persists.

(In reply to Kim A. Brandt from comment #5)

...

It now works for me.
...
Encryption and decryption now works. The hanging issue persists.

I'm not sure how helpful this will be for fixing the issue, but it seems to be a workaround!?

After it seemed that my secret key was forgotten by Thunderbird - of one of my accounts - I removed the remaining public key with the OpenPGP Key Manager and re-imported the public key and then the secret key. After entering my passphrase, the dialog hang again. Pressing space seems to allow cancelling the dialog. So, I entered the passphrase two more times - for good measure - and then canceled the dialog and exited the OpenPGP Key Manager. After this it seems Thunderbird has imported both public and secret keys and is able to remember the keys across restarts.

Not sure if this matters, but I'm using my existing GPG-keys for the import.

So far I have been unable to reproduce the issue. Bug 1654894 seems to be a duplicate.

Is there anything "special" about the key you are importing?

Just an idea, do you use different passphrases for the sub keys?

Do your keys contain many signatures (certifications) from other people?

Maybe the RNP library attempts to verify all the signatures at import time, and the import process is very slow?

(In reply to Nikolay Aleksandrovich Pavlov from comment #0)

I do not know what does A mean there.

It means Authentication is usable with the key, in addition to Sign and Encrypt.

You said you also had the import problem with a key without A, so it shouldn't make a difference.

Nickolay A P, can you please try again with the newer version 78.1.1 ?

Kim, do you see the problem with 78.1.1 ?

Nickolay A P, only now I discovered that you have attached a test key, thank you.

I tested with TB 78.0.1 on Linux 64 bit, and it immediately imported correctly.

(In reply to Kai Engert (:KaiE:) from comment #7)

...
Is there anything "special" about the key you are importing?

Just an idea, do you use different passphrases for the sub keys?

I don't think there's anything special about my key!? But, I haven't done any thorough examination thereof. I only have one passphrase.

I just successfully imported the key - with a clean profile - on a different machine, which has the same versions of everything. I'm not using any add-ons. So, I guess there could be something wrong with my profile!? It's an old profile, which has undergone many upgrades. If I can't figure out what causes the issue, I will probably just use a clean profile!?

Is there anything obvious I can try, or should look into?!

(In reply to Kai Engert (:KaiE:) from comment #8)

Do your keys contain many signatures (certifications) from other people?

Maybe the RNP library attempts to verify all the signatures at import time, and the import process is very slow?

Not sure this question is for me, but the key I'm importing hasn't been signed by others.

(In reply to Kai Engert (:KaiE:) from comment #10)

...
Kim, do you see the problem with 78.1.1 ?

Yes, I am using Thunderbird 78.1.1 (64-bit).

I didn't mention it - but maybe it's obvious?! - that I also used the Enigmail migration, such as Nikolay and Andreas (from bug 1654894). So, I was using Enigmail before, but now I don't have any add-ons left.

Thanks, this helped. I'm finally able to reproduce.
Used TB 68 with Enigmail to create a key.
Installed TB 78.0.1 and ran though key migration.
It reported failure to import the secret key (because of the known Enigmail bug 1656401).
Restarted, now at TB 78.1.1
Used gpg --export-secret-keys to create a backup file (no password protection).
User TB account settings to import the exported file from gpg.
UI keeps spinning the wheel, and I get the exceptions:

CryptoAPI.sync() failed result:  Error: rnp_key_protect failed
    importKeyBlockImpl chrome://openpgp/content/modules/RNP.jsm:1676
    sync chrome://openpgp/content/modules/cryptoAPI/interface.js:51
    importKeyFromFile chrome://openpgp/content/modules/keyRing.jsm:361
    openPgpImportStart chrome://openpgp/content/ui/keyWizard.js:761
    wizardNextStep chrome://openpgp/content/ui/keyWizard.js:194
    wizardContinue chrome://openpgp/content/ui/keyWizard.js:143
    _fireButtonEvent chrome://global/content/elements/dialog.js:487
    _doButtonCommand chrome://global/content/elements/dialog.js:466
    _handleButtonCommand chrome://global/content/elements/dialog.js:460
interface.js:46:17

Uncaught (in promise) TypeError: res.importedKeys is undefined
    importKeyFromFile chrome://openpgp/content/modules/keyRing.jsm:373
    openPgpImportStart chrome://openpgp/content/ui/keyWizard.js:761
    wizardNextStep chrome://openpgp/content/ui/keyWizard.js:194
    wizardContinue chrome://openpgp/content/ui/keyWizard.js:143
    _fireButtonEvent chrome://global/content/elements/dialog.js:487
    _doButtonCommand chrome://global/content/elements/dialog.js:466
    _handleButtonCommand chrome://global/content/elements/dialog.js:460
keyRing.jsm:373:36

This will allow me to investigate what happens.

I understand what's happening.

The Enigmail migration version 2.2 performs an incomplete migration, as reported in bug 1656401.
The "automatic passphrase for protecting secret keys" isn't created. File encrypted-openpgp-passphrase.txt is missing.
Secret keys are imported.
Because of the missing automatic passphrase, the keys are stored without protection on disk.
Because of the incomplete init, the imported secret key cannot be marked as a personal key (based on gpg ownertrust ultimate).

The next time Thunderbird is started, we run into an automatic check, that detects an inconsistency.
We should never have secret keys stored, without having a file encrypted-openpgp-passphrase.txt
The code runs into an exception, and subsequent actions are aborted.

When importing a key, rnp_key_protect fails, because the automatic passphrase cannot be read from encrypted-openpgp-passphrase.txt and again we run into an exception, which causes the import procedure to get stuck as reported.

I suggest the following remedy:

On start, we check if we have secret keys, but no passphrase file. If not (either no keys and no passphrase, or both), then everything is fine.

If we see the error condition, we do the following:

We check, if all secret keys are unprotected (not protected by a passphrase). This is the scenario that the failed migration produced. If all keys are unprotected, we can suggest an automatic repair. We can show an error message explaining. We should offer the user a chance to make a backup. After all, we're going to change their secret key file, and we should be on the safe side. We show the file that should be backed up. We offer the user the choice to repair now, or cancel. If the user cancels, we will start up with OpenPGP degraded/disabled. If the user confirms to repair, then we will create the automatic passphrase file, and then change all unprotected keys to be protected by it. This creates the intended configuration and the required consistency.

However, our check might discover that there are some protected key, that is, secret keys that are protected by a passphrase. This situation isn't produced by the Enigmail import. I suggest to handle this scenario nevertheless, to cover future possible failures that some users might experience (for example file corruption, or manually deleted files, or other future bugs). I'd like to inform the user if this scenario is detected. We cannot automatically repair it. If keys are passphrase protected, and we don't have the file with the passphrase, there's nothing we can do. (We don't import protected keys as they are, we always attempt to unprotect them, before considering to import them.)

I suggest additional changes to make our code more robust.

The bug was made possible, by importing secret keys into temp storage, unprotecting them, then copying them to the permanent storage, then protecting them (which failed).

I'd like to change the logic in the following way, so that we won't import unprotected keys.
New logic: Import secret keys into temp storage, unprotecting them, protect them with the automatic passphrase, then copying them to the permanent storage.
This way, if protecting fails, our import will fail, and we avoid having unprotected keys.

In addition, I want to change the init code related to preference mail.openpgp.enable
Currently, changing the pref from false to true doesn't dynamically trigger required initialization to use the OpenPGP feature. Currently a restart is required.

We should add code that discovers the pref change from false to true, and perform all init automatically.
(But I don't want to dynamically disable if the pref is set to false at runtime, the feature will remain enabled until the end of the process lifetime.)

To everyone affected by this bug:

Thank for you for testing Thunderbird OpenPGP code, and helping us to identify this bug.

At this time, to work around this issue, you could delete some files from your Thunderbird profile, and start from scratch. You would have to ensure that you have good backups of all your secret keys. Then you could quit Thunderbird, and move the secring.gpg file away. This should allow Thunderbird to create a fresh, empty secring.gpg file and also create a fresh encrypted-openpgp-passphrase.txt file. Then you could try to migrate again using Enigmail 2.2. To ensure you don't run into this bug again, verify that pref mail.openpgp.enable it set to true, and restart Thunderbird before allowing the key migration.

We're attempting to provide a repair mechanism for the following 78.x version of Thunderbird. Also, the Enigmail 2.2 migrator will get an update soon, which will avoid this scenario.

(In reply to Kai Engert (:KaiE:) from comment #15)

I'd like to change the logic in the following way, so that we won't import unprotected keys.
New logic: Import secret keys into temp storage, unprotecting them, protect them with the automatic passphrase, then copying them to the permanent storage.
This way, if protecting fails, our import will fail, and we avoid having unprotected keys.

Unfortunately this didn't work. With this change, our stored keys are unprotected. I don't fully understand yet why. Probably we're allowed to export unprotected key material from the temporary space, so the protection that as added in the protected space gets lost. I'll revert this part.

After enabling the pref, the account settings e2ee pane throws an exception. I think I need to change the am-e2e.js code to also listen for the pref change, and enable items accordingly.

(In reply to Kai Engert (:KaiE:) from comment #22)

After enabling the pref, the account settings e2ee pane throws an exception. I think I need to change the am-e2e.js code to also listen for the pref change, and enable items accordingly.

There are a lot of dynamic actions in that code. I'd prefer to do that in a separate step. It might be helpful if the migrator suggests to restart Thunderbird, after the migration was completed.

Pushed by kaie@kuix.de:
https://hg.mozilla.org/comm-central/rev/9f8203aa87e7
Handle missing file encrypted-openpgp-passphrase.txt and unprotected keys. r=PatrickBrunschwig

Comment on attachment 9169146 [details]
Bug 1656287 - Handle missing file encrypted-openpgp-passphrase.txt and unprotected keys. r=PatrickBrunschwig

Approval request:

As described in the bug, users of the Enigmail migration have corrupted profiles and keys stored without protection. It is very important to get that repaired.

There is a small amount of risk, because the backend initialization logic on startup had to be reordered.

For existing users of OpenPGP in TB 78 who haven't a corrupted profile, the risk is minimal. We introduce new checks at startup. Only users who have corruption will get the repair code executed. And we prompt and ask for confirmation, prior to repairing (in english only).

Nevertheless, it seems important to push this to users soon. It's a difficult call if this should be pushed to esr78 immediately, as the bug prevents the ability to use OpenPGP for affected users. On the other hand, it might be sensible to start with c-c and beta for a few days, to confirm it's not introducing any new major issue.

Comment on attachment 9169146 [details]
Bug 1656287 - Handle missing file encrypted-openpgp-passphrase.txt and unprotected keys. r=PatrickBrunschwig

[Triage Comment]
Approved for beta
Approved for esr78

https://hg.mozilla.org/releases/comm-beta/rev/95e70007e95e6ceb2febd99d5115893a765b9160

https://hg.mozilla.org/releases/comm-esr78/rev/8bbe9af4c6f2726db0fbf20e43ea9fe0e1a07bf1

(In reply to Kai Engert (:KaiE:) from comment #20)

At this time, to work around this issue, you could delete some files from your Thunderbird profile, and start from scratch. You would have to ensure that you have good backups of all your secret keys. Then you could quit Thunderbird, and move the secring.gpg file away. This should allow Thunderbird to create a fresh, empty secring.gpg file and also create a fresh encrypted-openpgp-passphrase.txt file. Then you could try to migrate again using Enigmail 2.2. To ensure you don't run into this bug again, verify that pref mail.openpgp.enable it set to true, and restart Thunderbird before allowing the key migration.

KaiE, which are these “some files”?

Anyway, as described in bug 1658187, we still can’t import PGP keys even with a new Thunderbird 78.2.0 profile.