I've been using the integrated OpenPGP with TB Daily (on MacOS) since it first appeared. I update my Daily build most days.

For some time, I've noticed that occasional signed messages I've sent, in my Cc copy or in my Sent mail get marked with a red glyph as "invalid digital signature"; see attached for an example.

It's not consistent; I see the red glyph for only 6 of the last 19 signed messages I have sent, in my Sent mail folder (and also on my Cc copies). The other 13 signed emails are fine in both Sent and Cc.

I can send two emails to the same recipient, within an hour, and find one with a red glyph, and one fine. Neither email having any (other) attachments. I've not yet seen any pattern to it.

I had assumed that this was some oddity relating to viewing a Cc copy, or viewing it in my Sent mail folder. But today a recipient of one of those emails happened to mentioned that they see the red glyph too (using TB). They see the red & green glyphs on the same emails that I do. So it's not local to my system: either the sent email really has a problem, or the recipient using TB is seeing the same false alarm as me.

Since I also see this issue on emails in my Sent folder, which TB is writing directly via IMAP, I think we can rule out any SMTP MTA messing up the email content.

This means I have now had to disable signing entirely, since it's very misleading for recipients to receive emails that are allegedly invalid, and may result in emails going to spam folders, etc.

I don't see anything different in the Error Console when first reading such a message (with the red glyph).

I'm currently using 101.0a1 (2022-04-11) (64-bit), and seeing the issue, but have been seeing it for a while.

I've been hearing similar reports from Magnus, but I've never been able to reproduce this yet.

Are there any notable differences between the emails?
For example, regarding message complexity, like, simple short emails fine, big emails with attachments bad, something like that?

thanks. No, I've not noticed any pattern.

As you can see from my attached image, that's a simple email of three words, to a single recipient, which fails. Other simple emails work fine.

Emails to the same recipient: some work, some don't, even an hour apart.

Emails with attachments: some work, some don't.

If there is a pattern there, I'm not seeing it.

Is there any way to get more diagnostics out of the code that detects the "technical error"?

For amusement, see my second attachment: same Subject and email body as the first, but this one has a good signature.

Comparing the two via ctrl-u message source, shows that the bad case doesn't have the signature at all:

=== good ===
--------------0DNithhyanhdFte9ruDJsQws
Content-Type: application/pgp-signature; name="OpenPGP_signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="OpenPGP_signature"

-----BEGIN PGP SIGNATURE-----

wsF5BAABCAAjFiEE4oRc7xbt3WabCv4Kg1FzQH+ZYy8FAmJV6x4FAwAAAAAACgkQg1FzQH+ZYy/w
OQ/7BgXPOojQN1hzSSD8ou0yFY8NUbYB9tHzv6oyx7eMBp5cgTrzdqI5qdGeXhv4JtWp5dv7Gf09
9PcTXZJDMyCB20/S7j0HsmoU+o6ynKnEgGGzN2vtOFxZU3ZlD1UKuf0SLiLd0Y92nEJpL8JGNf7x
SA2LFKzichgrfNp6gdSHcXto6TtKA+Zjs/lIoX8sJrF+2zdKG/liApJlVwQbYttzL50ttV17QLcG
TyYrzcny8uBE010toQIRiV/Bk6h0J9ateGgatBi5u1IIhP5r1gBOtheD67GleMZBm4xHEPpNaDPn
SKNJVhT4PaLm9o5RZti4prf8goPBBkhFkdKa7kwNDdNOzGQ0qFIrn7Ho/UjD4sO6oC6NSU7f0lBL
RRC/8GOTolU3cqDnklsgLI7Tcwgb2u+vAGcrw7rZaXazDoknK1vQYq7DqDiwLI9IcyE5JjsTtJ6z
YKhtdkZfi9GFAuQ7f6mcltrFp9iUbkeVC52RC3CyHeJAZR4ulYhcYTWcs93PfDW47Q0HJnk42/ET
KjOSUOcA0OxIGKJMpybTGl6vlE1e+TSVmpPBr0a5ZiZjVjLOCDJdpxdjZETbshSVRe7SQrfZMG02
9cPPNa+jybUXYdGLaXfnnYZ+/AzQEZO/AIxw02C8ukL2Rqhd2/e9QFHlj8VtzAT+C3fMS5bfruQn
5aY=
=iVgz
-----END PGP SIGNATURE-----

--------------0DNithhyanhdFte9ruDJsQws--

=== bad ===
--------------kBxlJlRwpuLSJlmYJVzSBbyM
Content-Type: application/pgp-signature; name="OpenPGP_signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="OpenPGP_signature"

-----BEGIN PGP SIGNATURE-----

--------------kBxlJlRwpuLSJlmYJVzSBbyM--

(In reply to Calum Mackay from comment #4)

Comparing the two via ctrl-u message source, shows that the bad case doesn't have the signature at all:

That's interesting, hopefully this finding will help us identify the cause.

Because there is the trailing boundary line of the MIME message, it means that the failure is specific to the signature creation code. And because the output isn't empty (it contains the BEGIN line), our code might not realize there was a failure.

It would be easiest, if you could get diagnostic information at the time such a broken message is created.
If you can, please check the message in your sent folder, each time after you have sent a message. If it's broken, then open the JavaScript console (press CTRL-SHIFT-J). Maybe on of the the most recent statements is a clear failure message, that sounds like it's related to RNP or to OpenPGP? If yes, please provide that failure.

In addition, it would be helpful to use a terminal window, inside it run the command export RNP_LOG_CONSOLE=1 once, then start thunderbird from without that terminal window. If you detect a broken sent message, check the terminal window for the most recent messages printed by RNP.

(RNP is the software library that we use for creating OpenPGP messages.)

I assume you aren't using the "external GnuPG" configuration.

Nickolay, it seems likely that this is a bug in RNP.

When creating a signed message, we call rnp_op_sign_detached_create and rnp_op_sign_set_armor.

We see that sometimes the output contains only "-----BEGIN PGP SIGNATURE-----", and nothing else.

Can you think of a reason when this might happen?

Magnus, you said that sometimes you get invalid signatures shown for your own messages.
Can you please check if you have any messages in your Sent folder that show invalid, each time you click them?
If you find such a message, do you see the same error, only the BEGIN line, but no additional signature contents?

I'll check the next time I see it.
I think it's somehow related to whether the message was fully available when opening it. But that's just a guess. Once it starts "working" I don't think I ever saw it go back to showing invalid.

Kai, this scenario should be definitely covered by our tests, so let's investigate it further. Could you please point me on code lines/file which calls rnp_op_sign_* functions? Also, debug output would be helpful as well.

thanks very much for the replies.

(In reply to Kai Engert (:KaiE:) from comment #5)

I assume you aren't using the "external GnuPG" configuration.

No, that's right. I am using the internal, integrated OpenPGP support. I'd intended to indicate that with the "native" word in the Subject, sorry if that wasn't clear.

I'll collect the data requested…

I haven't sent any messages yet, but got this RNP output at TB startup, if it's relevant:

[signature_validate() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/crypto/signatures.cpp:268] Insecure hash algorithm 2, marking signature as invalid.
[signature_validate() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/crypto/signatures.cpp:268] Insecure hash algorithm 2, marking signature as invalid.

First attempt to send showed the error.

The cmdline output showed:

[signed_fill_signature() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/librepgp/stream-write.cpp:1133] wrong secret key password
[signed_detached_dst_finish() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/librepgp/stream-write.cpp:1193] failed to calculate detached signature
[process_stream_sequence() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/librepgp/stream-write.cpp:1781] failed to finish stream
[armored_src_read() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/librepgp/stream-armor.cpp:281] premature end of armored input
[init_packet_sequence() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/librepgp/stream-parse.cpp:2388] cannot read packet tag

and the JS console showed:

13:23:25.728 getCryptParams parameters: from=0x835173407F99632F, to=calum.mackay@cdmnet.org, bcc=, hash=SHA256, flags=4289, ascii=0, errorObj= 
Object { value: "" }
 , logObj= 
Object {  }
encryption.jsm:73:13
13:23:25.729 getCryptParams, got: to=calum.mackay@cdmnet.org, bcc= encryption.jsm:113:13
13:23:25.729 getCryptParams returning: encryption.jsm:190:13
13:23:25.729
Object { sender: "0x835173407F99632F", sign: true, signatureHash: "SHA256", sigTypeClear: false, sigTypeDetached: true, encrypt: false, encryptToSender: false, armor: true, senderKeyIsExternal: false, to: (1) […], … }
encryption.jsm:191:13
13:23:25.738
Error: no cached password masterpass.jsm:195:13
13:23:25.757 sendFlags=000010c1 encryption.jsm:456:13

does that help?

Let me know if you need more…

Could it be that you are mistyping your password? As for me is that due to some reasons RNP gets invalid password for the secret key and fails to sign. It must return an error, but maybe it is somewhere (on RNP level or on upper level) mishandled.

thanks Nickolay, but which password? The only password I type is my master password, at startup. If I got that wrong, I would not see any new email, since my IMAP passwords would not be unlocked, right? So I think I would notice.

One possibly related issue: my TB startup involves two windows, and I get two master password prompts. I usually enter the password in one, and dismiss the other. This works fine for IMAP/SMTP, but perhaps it's relevant here?

TB has had a long history of not properly prompting for master pw at startup when there are multiple windows. It was fixed for a while (i.e. single prompt), but is now double-prompting again.

In addition, I have now sent further signed emails, in that same TB session, identical subject/body to the first one which failed to sign, and they have succeeded: good signatures.

No output from RNP for the successful ones, and JS console did not show "no cached password".

Yet I typed no password between the failing and successful messages. There was a gap of about two hours, though, but TB remained running throughout.

It certainly looks like it has something to do with the double Primary Password prompt:

• if I enter my primary password once, and dismiss the second PP prompt, then the first email I send fails to sign, with "no cached password", but the second email I send signs fine.

• if I enter my pp twice, once to each prompt (one per window), then the first email I send signs fine.

What's odd is that entering my pp only once, and dismissing the second prompt, doesn't cause a problem for anything else: I am still able to access all my IMAP & SMTP accounts, whose passwords are stored.

And of course it's my belief that TB should only prompt me once for the primary password, regardless of how many windows open at startup.

I will continue to test having entered the primary password twice, once for each prompt, and see if I get any failures at all.

See for example bug 177175, bug 1512083, bug 1685877

[Wayne: is it worth linking this to any of those, as it seems like the multiple prompt (if not all are responded to) is causing problems for e2ee signing?]

(In reply to Calum Mackay from comment #17)

See for example bug 177175, bug 1512083, bug 1685877

[Wayne: is it worth linking this to any of those, as it seems like the multiple prompt (if not all are responded to) is causing problems for e2ee signing?]

Perhaps. But I currently have little expertise in the pgp area

Thanks everyone.

I'm able to reproduce the issue.

If a master password dialog remains open, it is possible to compose and send a signed message. Signing will fail, but the message will be sent anyway.

(I really wish we could fix the master password issue, but it's very complicated, and we weren't yet able to focus on that.)

Comment on attachment 9283666 [details]
Bug 1764175 - Add error handling for failed OpenPGP signing. r=PatrickBrunschwig

[Approval Request Comment]
Regression caused by (bug #): 1629309
User impact if declined: cannot send signed/encrypted email with external gnupg config
Testing completed (on c-c, etc.): manually
Risk to taking this patch (and alternatives if risky): very low

Comment on attachment 9283666 [details]
Bug 1764175 - Add error handling for failed OpenPGP signing. r=PatrickBrunschwig

[Triage Comment]
Approved for beta

https://hg.mozilla.org/comm-central/rev/108cda059a95487a4d371c4e48c6ea51fc50e3b3

Thunderbird 103.0b3:
https://hg.mozilla.org/releases/comm-beta/rev/e3f11c26e1e8

Comment on attachment 9283666 [details]
Bug 1764175 - Add error handling for failed OpenPGP signing. r=PatrickBrunschwig

[Approval Request Comment]
Regression caused by (bug #): 1629309
User impact if declined: cannot send signed/encrypted email with external gnupg config
Testing completed (on c-c, etc.): 103.0b3
Risk to taking this patch (and alternatives if risky):

Comment on attachment 9283666 [details]
Bug 1764175 - Add error handling for failed OpenPGP signing. r=PatrickBrunschwig

[Triage Comment]
Approved for esr102

I think we wil also want this for next esr91

Thunderbird 102.0.2:
https://hg.mozilla.org/releases/comm-esr102/rev/9e01f9c1c040

Comment on attachment 9283666 [details]
Bug 1764175 - Add error handling for failed OpenPGP signing. r=PatrickBrunschwig

[Triage Comment]
approved for esr91

Hopefully Kai agrees

I agree with the suggestion to uplift to esr91

Thunderbird 91.12.0:
https://hg.mozilla.org/releases/comm-esr91/rev/81c14c64bbdb