Seems to be a regression from bug 1521007.

It works when using the UTF-8 system locale which is beta in Windows 10 1803.

Mike, can you take a look, please.

I'm sorry, but I am confused:

this bug is titled "Sending files with accented characters or Umlaut (äöü) via MAPI fails in TB 60.5.2 when system locale is set to (for example) English (United States)", which assumes that filenames contain characters absent in ACP. And that has always been the case IIUC; only Unicode API could change that (which we disabled for now). But bug 1530820 comment 27 (and 26 possibly) imply that it also doesn't work for non-ASCII characters in current ACP (which would e.g. mean Cyrillic in my case).

So this bug needs a clear statement what is the problem here.

For testing, on an English Windows 8.1, I set Language for non-Unicode programs to Russian, and restarted; and trying to send files named with Cyrillic characters in names, it fails - which supports the comments in bug 1530820.
Then I go to 'HKLM\SOFTWARE\Clients\Mail\Mozilla Thunderbird' and change 'SupportUTF8' to 0 - and it succeeds now. So yes, looks like that registry setting introduced in bug 1521007 was causing that - the fix should be changing suite/installer/windows/nsis/shared.nsh to explicitly set 'WriteRegDWORD HKLM "$0" "SupportUTF8" 0' (otherwise, the setting would stay 1 for updated installations where it was set to 1).

Mike, the bug summary is trying to express that sending file with non-ASCII names but where the characters are on the system code page (or ACP - Ansi code page) doesn't work when it worked before.

I can confirm that resetting HKLM\SOFTWARE\Clients\Mail\Mozilla Thunderbird - SupportUTF8 to 0 fixes the problem.

However, I fail to understand why this setting which is meant to advertise UTF-8 support has quite the opposite effect when an Ansi code page is used as system code page.

Can you enlighten me further?

My guess is that MS made different implementations of UTF-8-conversion code in its MSO and in Windows SDK (you might remember my references to my requests to them to clarify the documentation and inconsistencies in the SDK function). Well - I am almost sure that that inconsistency is what bites us here... so most possibly we need to not set the registry value to 1 - to disable the SDK implementation; knowing that MS Word ignores the setting anyway, and sends the UTF-8 wherever it is unable to send data using ACP. So that most of the fix for bug 1521007 is still relevant - for Word.

Pushed by mozilla@jorgk.com:
https://hg.mozilla.org/comm-central/rev/dbe0e592f60e
Revert SupportUTF8 registry setting from bug 1521007. r=me

TB 60.5.3:
https://hg.mozilla.org/releases/comm-esr60/rev/8626d03f14b3cc7b43a656f174c18cb984db3e88

Would this fix applies to both new install and installation via auto-update?

Only the latter since the former always worked.

(In reply to Mike Kaganski from comment #4)

My guess is that MS made different implementations of UTF-8-conversion code in its MSO and in Windows SDK (you might remember my references to my requests to them to clarify the documentation and inconsistencies in the SDK function). Well - I am almost sure that that inconsistency is what bites us here... so most possibly we need to not set the registry value to 1 - to disable the SDK implementation; knowing that MS Word ignores the setting anyway, and sends the UTF-8 wherever it is unable to send data using ACP. So that most of the fix for bug 1521007 is still relevant - for Word.

Could this help understand the difference?

https://docs.microsoft.com/en-us/windows/desktop/intl/unicode
https://docs.microsoft.com/en-us/windows/desktop/intl/conventions-for-function-prototypes
https://docs.microsoft.com/en-us/windows/desktop/intl/code-pages

Could it be an UTF-16 vs UTF-8 issue and conversion?

MS seems to recommand use of UTF-16 in application as their internal data representation for lossless conversion between encoding...

If I understand correctly... I thought maybe that can help...

(In reply to Richard Leger from comment #10)

Thanks Richard!

Unfortunately, the issue is different: see [1].

MS has published the special value (CP_UTF8) that might be passed to MapiMessage::ulReserved (i.e., to 8-bit API - which is handled in this specific case); and also they have implemented their MAPISendMailHelper as an inline function implemented in an SDK header - which gives full details on its handling that value; which also reveals the registry key in play here. But the documentation is incomplete (it gives no details on which strings are affected by the special value); and the implementation only converts some strings when uses it. But Word apparently uses a different set of affected strings when uses that value; so the result is that there are two implementations by MS that may generate 8-bit data to MAPI with CP_UTF8, and one of them (possibly) encodes all strings to utf-8 then (Word), while another only encodes some strings to utf-8, while the rest is still ACP-encoded. So removing the registry flag again forces the faulty (IMO) MAPISendMailHelper implementation to stop passing that flag with partial encoding.

[1] https://social.msdn.microsoft.com/Forums/en-US/4d029672-e1e6-4d79-9b13-d027e79bdf6e/what-is-the-true-scope-of-cputf8-in-mapimessageulreserved

Addicionally, we implemented the MAPISendMailW wide char/UTF-16 API in bug 1048658. Sadly we got crashes that we didn't understand, bug 1523818, bug 1527450. So we withdrew the API again on the beta channel, note that is was never shipped to the ESR channel.

I think I will re-enable it again on the beta channel now that the installer issues are fixed. Having two incompatible binary interfaces talk to each other is never a good idea as random havoc ensues :-(

TB 66 beta 3:
https://hg.mozilla.org/releases/comm-beta/rev/92296df5e4dc4608ecbd3694af659dd164871c5d

As discussion in bug 1530820 comment #122 and above shows, reverting the setting was not enough. That doesn't appear to be happening in and upgrade install.

Something like this?

(In reply to Robert Strong (Robert he/him) [:rstrong] (use needinfo to contact me) from comment #16)

This and the last patch seem like copy and paste errors.

Hmm, what do you mean be "copy/paste" errors in the "last" patch?
Originally we had:
https://hg.mozilla.org/comm-central/rev/1948a0ff606a#l1.12
And now:
https://hg.mozilla.org/comm-central/rev/dbe0e592f60e#l1.12

The value of "1" did indeed get set, so I don't see a copy/paste error. Can changing the 1 to a 0 should have been OK, no?

Do you have a windows system to check the patches?

In general, yes, but I'm not an expert on installers.

How about this then?

+  StrCpy $0 "Software\Clients\Mail\${ClientsRegName}"
+  WriteRegDWORD HKLM "$0" "SupportUTF8" 0

Both cases the code was placed in a new location.
New patch

   ; Upgrade the copies of the MAPI DLL's
   ${UpgradeMapiDLLs}
+  ; revert HKLM\SOFTWARE\Clients\Mail\Mozilla Thunderbird - SupportUTF8 from bug 1521007
+  StrCpy $0 "Software\Clients\Mail\${ClientsRegName}"
+  WriteRegDWORD ${RegKey} "$0" "SupportUTF8" 0

From one of the links

   WriteRegStr ${RegKey} "$0" "" "${ClientsRegName}"
   WriteRegStr ${RegKey} "$0\DefaultIcon" "" "$8,0"
   WriteRegStr ${RegKey} "$0" "DLLPath" "$6"
-  WriteRegDWORD ${RegKey} "$0" "SupportUTF8" 1
+  WriteRegDWORD ${RegKey} "$0" "SupportUTF8" 0

(In reply to Robert Strong (Robert he/him) [:rstrong] (use needinfo to contact me) from comment #19)

Both cases the code was placed in a new location.

Yes, in an attempt to get this code onto the patch that runs during automated update. Since we restored

  ; Upgrade the copies of the MAPI DLL's
  ${UpgradeMapiDLLs}

there, I thought that the right place to put writing the registry value.

It looks like the registry key is already being set in PostUpdate. If the Clients\Mozilla Thunderbird registry key is for the installation that is being updated in the section below then SetClientsMail is called which sets the value to 0.

    ; Only update the Clients\Mail registry key values if they don't exist or
    ; this installation is the same as the one set in those keys.

OK, so you're saying my patch is useless, that's OK :-)

As you've seen from the changeset I quoted, we've set HKLM\SOFTWARE\Clients\Mail\Mozilla Thunderbird - SupportUTF8 to 1 and then changed our mind and are setting it to 0 (instead of removing it).

We thought that would work until we got further complaints in bug 1530820 comment #109 and below. There someone was reporting that in HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Clients\Mail\Mozilla Thunderbird SupportUTF8 was still set to 1. But how could that be? If the original installer set 1, and the new installer set 0, then 0 should be set.

So I "invented" the story, that manual install ran the code to set the value, and automated update didn't. In this case, first a manual update and then an automated update would have led to the result we saw. But that fairy tale is wrong. As you pointed out, the registry should always be hit.

So then we're back to square one not understanding how the 1 was left in Wow6432Node (or, as ignorant as I am, how it got there in the first place).

I installed Thunderbird Daily 64 bit to check if the values get updated during PostUpdate

Steps:

1. Install Thunderbird Daily
2. Open regedit and change
   Key path: HKLM\SOFTWARE\WOW6432Node\Clients\Mail\Mozilla Thunderbird
   Key name: SupportUTF8
   Value: from 0 to 1
   Key path: HKLM\SOFTWARE\WOW6432Node\Clients\Mail\Mozilla Thunderbird
   Key name: SupportUTF8
   Value: from 0 to 1
3. Ran cmd.exe as admin
   Navigate in explorer to %SystemRoot%\System32
   Right click cmd.exe and select Run as administrator
   CD to "%ProgramFiles%\Thunderbird Daily\uninstall"
   Then ran in the console helper.exe /Postupdate
4. Checked registry entries in step 2 after select View -> Resfresh and both entries were set back to 0

It is possible that the affected client updated without elevation or the service

One of the key paths should have been
Key path: HKLM\SOFTWARE\Clients\Mail\Mozilla Thunderbird

Since this periodically comes up, app update intentionally doesn't always request elevatation or use the maintenance service which would have launch helper.exe /PostUpdate with the elevated privileges unless it needs to in order to update. The reason is to mitigate security exploits such as when a user already has write access to the installation directory they would be able to create hard or soft links pointing to pretty much any other file on the system including files in the Windows directory and thereby cause app update to manipulate them with the elevated privileges or the Local System account in the case of the maintenance service.

In this specific case, perhaps the client installed into a non-default location that they already have write access to. To handle this the nsis code would need to detect whether it has write access to the HKLM registry and if the value is not the same as the new value it should request elevation, etc.

Another reason this could happen is if the client didn't run the installer and instead used a zip build or similar.

Robert, thanks so much for testing and all the deliberations. I think, for now we're done here.

(In reply to Robert Strong (Robert he/him) [:rstrong] (use needinfo to contact me) from comment #23)

I installed Thunderbird Daily 64 bit to check if the values get updated during PostUpdate
(...)
2. Open regedit and change
Key path: HKLM\SOFTWARE\WOW6432Node\Clients\Mail\Mozilla Thunderbird
(...)
Key path: HKLM\SOFTWARE\WOW6432Node\Clients\Mail\Mozilla Thunderbird
(...)

Looks like the two paths are the same!

Did you meant...

Key path1: HKLM\SOFTWARE\Clients\Mail\Mozilla Thunderbird
and
Key path2: HKLM\SOFTWARE\WOW6432Node\Clients\Mail\Mozilla Thunderbird

As the issue raised with 60.5.2 version, wouldn't a better testing be to install version 60.5.1, change manually the .reg keys to 1 (both location) and auto-update to 60.5.3 to check what happens to the keys?

If both are not set correctly and mismatch in the end, you could then compare the differences between the 60.5.3 and daily version to check where something could have been missed in the post update configuration... unless the configuration is not the issue...

Also test with both win32 and win64 version of TB on 64bits system see if that make any difference in the results perhaps...

(In reply to Richard Leger from comment #28)

(In reply to Robert Strong (Robert he/him) [:rstrong] (use needinfo to contact me) from comment #23)

I installed Thunderbird Daily 64 bit to check if the values get updated during PostUpdate
(...)
2. Open regedit and change
Key path: HKLM\SOFTWARE\WOW6432Node\Clients\Mail\Mozilla Thunderbird
(...)
Key path: HKLM\SOFTWARE\WOW6432Node\Clients\Mail\Mozilla Thunderbird
(...)

Looks like the two paths are the same!

Did you meant...

Key path1: HKLM\SOFTWARE\Clients\Mail\Mozilla Thunderbird
and
Key path2: HKLM\SOFTWARE\WOW6432Node\Clients\Mail\Mozilla Thunderbird

See comment #25 where I corrected the misstatement.

I'm not a Thunderbird dev and I also haven't worked on the Firefox installer for quite some time. Feel free to take the time to perform the additional tests that you took the time to write out. It might very well help identify why it happened.

As I had a machine under hand with TB 60.3.3 (32bits) installed on win 7 pro (64bits) I did some quick check and testing...

0. No SupportUTF8 keys are present at all in both reg locations.

1. Run Sent to > Mail Recipient report error: There is no email program associated to perform the request action.
   No SupportUTF8 key present at all in both reg locations.

2. Go to Options > Advanced > General > System Integration > Press Check Now button

3. Set TB to be default client for Email, Newsgroup, Feeds (set by default)
   No SupportUTF8 key present at all in both reg locations.

4. Run Sent to > Mail Recipient again from a File Explorer .jpg photo
   No SupportUTF8 key present at all in both reg locations.

5. TB opened and updated itself to 60.5.0 (32bits)

6. Worked ok
   Attachement appear attached to email
   No SupportUTF8 key present at all in both reg locations (though I have a doubt if I had or not refreshed the registry view by pressing F5)

7. Went to Help > About which trigger auto-update and pressed 'Restart to update Thunderbird' when prompted

8. After restart TB version is showing as 60.5.3 (32bits)
   SupportUTF8 keys now appears in both location with value 0 (I did press F5 in registry view to refresh to see them)

Hope that may help...

For the record, on my Win 10 Pro (64bits) machine where I now run TB 66.0b2 (32-bit) I noticed that SupportUTF8 keys are set in both locations with the value 1 at the moment... all I have done month ago was to remove stable version of TB (en-GB originally downloaded from TB website and auto-updated for years) in order to install a beta version (en-GB from TB website beta channel) that have been auto-updated since then... so one of the TB version at some point did created those keys with value 1 somehow...

Follow up on test Comment 31 did the following additional tests on the same Win 7 machine:

0. Uninstalled TB version 60.5.3
   ==> it removed all the reg keys related to TB, including the SupportUTF8 ones in both locations

1. Installed TB Version 60.3.3 from https://ftp.mozilla.org/pub/thunderbird/releases/60.3.3/win32/en-GB/Thunderbird%20Setup%2060.3.3.exe
   ==> it installed all the reg keys related to TB in both locations, but none of the SupportUTF8 ones which remain missing completely in both locations.

2. Added the reg keys manually for testing purpose:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Clients\Mail\Mozilla Thunderbird]
"SupportUTF8"=dword:00000000

[HKEY_LOCAL_MACHINE\SOFTWARE\Clients\Mail\Mozilla Thunderbird]
"SupportUTF8"=dword:00000000

3. Changed value of 2nd reg key manually in registry to value 1, which automatically change value for the 1st key... so despite different locations in appearance, they are both the same key... (and vice-versa - changing the 1st key has the same effect on the 2nd)
   ==> Sent to > Mail Recipient feature still working as expected

4. Went to Help > About which trigger auto-update and pressed 'Restart to update Thunderbird' when prompted
   ==> SupportUTF8 keys now appears in both location with value 0

So this confirm that patch seems to works as expected to explicitly set the SupportUTF8 keys value to 0 in all possible cases, SupportUTF8 not present or present but with value 1 set.