Magnus, this has come back to haunt us. I'd better look at this real quick before this causes problems in TB 45.

Joshua, any comment on this? This is totally weird. I'm not familiar with nsMailboxProtocol.cpp and nsMailboxService.cpp where the message is shuffled around. Let's add Kent and Neil as well. Sadly this is going outside the area of my expertise. As far as composition is concerned, the two messages are treated exactly the same, but something deep in the bowels goes wrong with one of them. Note that replying works for both cases. This is an ancient old problem where an invalid fix was attempted in bug 494912 which caused regressions and was hence removed in bug 1239658 exposing the old problem again.

BTW, replying works since it's a very different code path. Also, for replies we set the charset here: https://dxr.mozilla.org/comm-central/source/mailnews/compose/src/nsMsgCompose.cpp#1988 Forwards seem to rely on the charset derived from the message content. Processing is here: https://dxr.mozilla.org/comm-central/source/mailnews/compose/src/nsMsgCompose.cpp#1922 and exits here: https://dxr.mozilla.org/comm-central/source/mailnews/compose/src/nsMsgCompose.cpp#1977 Setting the charset in this block makes no difference to the inlined content since the preceding MIME processing already got it wrong. So this is not a composition bug but a MIME bug.

I've done a little debugging, here is what I see: ===== Got UTF-8 from text/html; charset=UTF-8 ===== MimeMultipart_notify_emitter charset |UTF-8| ===== Getting charset from text/html ===== mimeCharset is null, using ISO-8859-1 So the charset was determined to be UTF-8, yet somewhere else it's not considered set, to we default to ISO-8859-1.

Slightly different: ===== Got UTF-8 from text/html; charset=UTF-8 ===== MimeMultipart_notify_emitter charset |UTF-8| ===== MimeMultipart_notify_emitter default is charset |ISO-8859-1| ===== Getting charset from text/html ===== mdd->mailcharset: ISO-8859-1 ===== mdd->options null ===== mimeCharset is null, using ISO-8859-1 Does anyone have a suggestion of how to get the charset information from point A to point B? Sorry about the NI-SPAM ;-)

This works. Who can review this? Let's see what it breaks: https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=da0e39fcaebb

Comment on attachment 8724514 [details] [diff] [review] Possible solution (v1). Damn, this does NOT work. I pressed "Reply" instead of "Forward" and "Reply" always works. Sorry. So back to comment #7: How can the information be transferred. This is funny, isn't it: ===== Got UTF-8 from text/html; charset=UTF-8 ... ===== Getting charset from text/html

More comprehensive debug: Setting: ===== Got UTF-8 from text/html; charset=UTF-8 <==== ===== MimeMultipart_notify_emitter charset |UTF-8| ===== MimeMultipart_notify_emitter default is charset |ISO-8859-1| ===== Setting obj->options->default_charset on obj = 07624660, obj->options = 0BBF8570 Retrieving: ===== mdd = 08EF4470, mdd->obj = 08EF4510 ===== mdd->options->default_charset = ISO-8859-1 ===== mdd->obj->options->default_charset = ISO-8859-1 ===== Getting charset from text/html <==== bad ???? ===== mdd->mailcharset: ISO-8859-1 ===== mdd->options is null ===== mimeCharset is null, using ISO-8859-1 So as we see, the part that knows the information operates on a different object from the part that retrieves the information. Let's compare with the "good" message where the parts are in the correct order: ===== Got UTF-8 from text/plain; charset=UTF-8; format=flowed ===== MimeMultipart_notify_emitter charset |UTF-8| ===== MimeMultipart_notify_emitter default is charset |ISO-8859-1| ===== Setting obj->options->default_charset on obj = 07810E00, obj->options = 09C85420 ===== Got UTF-8 from text/html; charset=UTF-8 <==== ===== MimeMultipart_notify_emitter charset |UTF-8| ===== MimeMultipart_notify_emitter default is charset |ISO-8859-1| ===== Setting obj->options->default_charset on obj = 09C29940, obj->options = 09C850E0 ===== mdd = 06F57B50, mdd->obj = 06F57F10 ===== mdd->options->default_charset = UTF-8 <=== how did that change to UTF-8?? ===== mdd->obj->options->default_charset = UTF-8 ===== Getting charset from text/html; charset=UTF-8 <==== good ===== mdd->mailcharset: UTF-8 ===== mdd->options is null As we can see, it works as designed, charset is retrieved from text/html; charset=UTF-8.

I had a chat with Joshua on IRC. He gave me some tips for debugging. He also said that no one really understands the mime code, so I'll clear the NIs. The problem appears to be that the inner alternative parts don't communicate the charset back to the outer related parts. Displaying and replying works since the charset is directly communicated to the window. Reply also works based on the window's charset.

OK, this is a bit hacky and adds more charset magic to the mime soup. Once detected, we store the charset in the mime object and move it up to the parent.

I should mention: I'm setting the default, so it can be retrieved here: https://dxr.mozilla.org/comm-central/source/mailnews/mime/src/mimedrft.cpp#1152 and then used here: https://dxr.mozilla.org/comm-central/source/mailnews/mime/src/mimedrft.cpp#1362 As we seen from the debugging, mimeCharset = MimeHeaders_get_parameter (mdd->messageBody->m_type.get(), "charset", nullptr, nullptr); comes back null (since mdd->messageBody->m_type.get() == "text/html") and we use mdd->mailcharset which was retrieved at line 1152 from the default.

Another remark: If mdd->messageBody->m_type.get() didn't come back with "text/html" but with "text/html; charset=UTF-8", of course I wouldn't need the default fiddle. Sadly I wasn't able to work out where this is set. I was searching for "messageBody->m_type" and I can't see it being set.

(In reply to Jorg K (GMT+1) from comment #14) > If mdd->messageBody->m_type.get() didn't come back with "text/html" but with > "text/html; charset=UTF-8", of course I wouldn't need the default fiddle. > Sadly I wasn't able to work out where this is set. I was searching for > "messageBody->m_type" and I can't see it being set. I've been debugging this for several hours now. I hope I can present a more elegant solution based on mdd->messageBody->m_type.get() returning the correct value in the next 24 hours.