Subject: Bug#566: X2Go Client for Windows 4.0.2.1 cannon create C:\Users\<username>\ssh\known_hosts file when the local Windows account username has non-English characters

From: George Trakatelis <[email-masked]>
Date: Mon, 11 Aug 2014 13:51:21 +0300

Package: x2goclient
Version: 4.0.2.1

When I try to connect to a remote server, I cannot get past the pop-up

'Host key verification failed' 

The server is unknown. Do you trust the host key?
Public key hash: xxx.xxx.xxx.xxx:22 -
xx.xx.xx.xx.xx.xx.xx.xx.xx.xx.xx.xx.xx.xx.xx.xx

no matter how many times I click Yes. The pop-up appears again and again.

It seems that x2goclient cannot create the folder C:\Users\<username>\ssh
and the known_hosts file when <username> has non-English characters.

Confirmed that it works fine when <username> is in English.
  
I'm using Greek Windows 8.1 Pro 64-bit and X2Go Client for Windows
4.0.2.1+hotfix1+build3.

---

From: George Trakatelis <[email-masked]>
Date: Mon, 1 Sep 2014 20:28:50 +0300

A suggested patch is to substitute the Qt4 toAscii() function with toLocal8Bit() 
in two specific places of SshMasterConnection::run().
A git diff is as follows:

$ diff /c/a/sshmasterconnection.cpp /c/b/sshmasterconnection.cpp
521c521
<     ssh_options_set ( my_ssh_session, SSH_OPTIONS_SSH_DIR, (mainWnd->getHomeDirectory()+"/ssh").toAscii());
---
>     ssh_options_set ( my_ssh_session, SSH_OPTIONS_SSH_DIR, (mainWnd->getHomeDirectory()+"/ssh").toLocal8Bit());
630c630
<     ssh_options_set ( my_ssh_session, SSH_OPTIONS_SSH_DIR, (mainWnd->getHomeDirectory()+"/ssh").toAscii());
---
>     ssh_options_set ( my_ssh_session, SSH_OPTIONS_SSH_DIR, (mainWnd->getHomeDirectory()+"/ssh").toLocal8Bit());

The change affects only Windows builds and was tested in Windows 8.1 Pro and Windows 7 Professional.

G. Trakatelis

-----Original Message-----
From: X2Go Bug Tracking System [mailto:[email-masked]] 
Sent: Monday, August 11, 2014 1:55 PM
To: [email-masked]
Subject: Bug#566: Acknowledgement (X2Go Client for Windows 4.0.2.1 cannon create C:\Users\<username>\ssh\known_hosts file 
when the local Windows account username has non-English characters)

...

--
566: http://bugs.x2go.org/cgi-bin/bugreport.cgi?bug=566
X2Go Bug Tracking System
Contact [email-masked] with problems

---

From: Mike Gabriel <[email-masked]>
Date: Mon, 01 Sep 2014 19:51:03 +0000

Hi Mike#2,

On  Mo 01 Sep 2014 19:28:50 CEST, George Trakatelis wrote:

> <     ssh_options_set ( my_ssh_session, SSH_OPTIONS_SSH_DIR,  
> (mainWnd->getHomeDirectory()+"/ssh").toAscii());
> ---
>>     ssh_options_set ( my_ssh_session, SSH_OPTIONS_SSH_DIR,  
>> (mainWnd->getHomeDirectory()+"/ssh").toLocal8Bit());
> 630c630
> <     ssh_options_set ( my_ssh_session, SSH_OPTIONS_SSH_DIR,  
> (mainWnd->getHomeDirectory()+"/ssh").toAscii());
> ---
>>     ssh_options_set ( my_ssh_session, SSH_OPTIONS_SSH_DIR,  
>> (mainWnd->getHomeDirectory()+"/ssh").toLocal8Bit());
>

Can you review these changes and if appropriate commit them to the  
master branch of x2goclient.git?

Thanks!
Mike

-- 

DAS-NETZWERKTEAM
mike gabriel, herweg 7, 24357 fleckeby
fon: +49 (1520) 1976 148

GnuPG Key ID 0x25771B31
mail: [email-masked], http://das-netzwerkteam.de

freeBusy:
https://mail.das-netzwerkteam.de/freebusy/m.gabriel%40das-netzwerkteam.de.xfb

---

From: Michael DePaulo <[email-masked]>
Date: Mon, 1 Sep 2014 19:23:12 -0400

On Mon, Sep 1, 2014 at 3:51 PM, Mike Gabriel
<[email-masked]> wrote:
> Can you review these changes and if appropriate commit them to the master
> branch of x2goclient.git?
>
> Thanks!
> Mike
>
> [...]

Sure. I'm busy today, but I'll do so within the next few/several days.

-Mike

---

From: Michael DePaulo <[email-masked]>
Date: Sun, 7 Sep 2014 00:44:25 -0400

Hi George,

1st, please use either "diff -c" or "git diff" to prepare the patch.
You can even use "git diff" on a files/folders that are outisde of a
git source tree.

2nd, I tried what I think was your patch. I've attached the change I
used, it is outputted with "git diff". I used the x2goclient master
branch (4.0.3.0.)

Unfortunately, it failed to fix the bug. I did not experience any
regressions though.

I tested it on both a Windows XP 32-bit SP3 machine and a Windows 8.1
64-bit machine (with the latest required updates & optional updates
from MS.)

3rd, the username I used was "δοκιμαστικό χρήστη". That is how Google
Translate translated "test user" into Greek. Because it is not easy
for me to type that name into the Windows login prompt, I simply ran
commands like the following to test it:
runas "/user:δοκιμαστικό χρήστη" "c:\x2gobuilds\x2goclient\dist\x2goclient.exe"

-Mike

---

From: "George Trakatelis" <[email-masked]>
Date: Sun, 7 Sep 2014 13:04:51 +0300

Hi Mike,

1st, Ok.

2nd, You guessed right, so I have attached the very same file.

3rd, Your findings really puzzled me, so I did the following:

a. Installed German keyboard. Hit ';' to produce an accented letter
and -to my surprise- got ö. So I thought ölexandr was the right name to test for username.

b. Created user ölexandr and logged in as that user.
The patch did not work, as you mentioned.

c. As toLocal8Bit() returns the local 8-bit representation of a string,
I changed the system locale for non-unicode programs to German.
Now the patch worked!

I think the patch solves the problem for the Windows users who use English
as a foreign language and prefer having (non-Ascii) usernames in their native language.

-George

PS. A better title for the bug would be

X2Go Client for Windows 4.0.2.1 cannot create C:\Users\<username>\ssh\known_hosts file when the local Windows account username has non-Ascii characters

-----Αρχικό μήνυμα-----
Από: Michael DePaulo [mailto:[email-masked]] 
Αποστολή: Κυριακή, 7 Σεπτεμβρίου 2014 7:44 πμ
Προς: Mike Gabriel; [email-masked]
Κοιν.: [email-masked]; George Trakatelis
Θέμα: Re: [X2Go-Dev] Bug#566: Bug#566: X2Go Client for Windows 4.0.2.1 cannon create C:\Users\<username>\ssh\known_hosts file when the local Windows account username has non-English characters

Hi George,

1st, please use either "diff -c" or "git diff" to prepare the patch.
You can even use "git diff" on a files/folders that are outisde of a git source tree.

2nd, I tried what I think was your patch. I've attached the change I used, it is outputted with "git diff". I used the x2goclient master branch (4.0.3.0.)

Unfortunately, it failed to fix the bug. I did not experience any regressions though.

I tested it on both a Windows XP 32-bit SP3 machine and a Windows 8.1 64-bit machine (with the latest required updates & optional updates from MS.)

3rd, the username I used was "δοκιμαστικό χρήστη". That is how Google Translate translated "test user" into Greek. Because it is not easy for me to type that name into the Windows login prompt, I simply ran commands like the following to test it:
runas "/user:δοκιμαστικό χρήστη" "c:\x2gobuilds\x2goclient\dist\x2goclient.exe"

-Mike

---

From: Michael DePaulo <[email-masked]>
Date: Sun, 7 Sep 2014 09:23:22 -0400

On Sun, Sep 7, 2014 at 6:04 AM, George Trakatelis <[email-masked]> wrote:
[...]
> X2Go Client for Windows 4.0.2.1 cannot create C:\Users\<username>\ssh\known_hosts file when the local Windows account username has non-Ascii characters

Hi Mike#1 and George,

1. What do you think of using toUtf8() instead? I am hoping that will
fix the bug regardless of the locale.

I tried changing it to that (bug566.utf8.test.patch), but it still did
not fix this bug with the greek username on my system with the locale
set to English. It did not introduce a regression for my ASCII user
account at least.

If libssh needs to be recompiled for Unicode, I can do that. I just
recompiled it for bug #590.

FYI: This is the API we are calling:
http://api.libssh.org/stable/group__libssh__session.html#ga7a801b85800baa3f4e16f5b47db0a73d

2. in sshmasterconnection.cpp, we also set "SSH_OPTIONS_USER" . I
think that we should also change its value from to.Ascii() to
toLocal8Bit() or toUtf8(). bug566.test.v2.patch has this change for
toLocal8Bit(), and bug566.utf8.test.patch has it for toUtf8().

-Mike#2

---

From: "George Trakatelis" <[email-masked]>
Date: Mon, 8 Sep 2014 03:13:48 +0300

Hi Mike,

1. If you recompile libssh 0.6.3 with Unicode support and provide me with a download link,
I will happily test it on all Windows platforms (namely Windows 8, 7, and XP).

2. I agree.

-George

-----Original Message-----
From: Michael DePaulo [mailto:[email-masked]] 
Sent: Sunday, September 7, 2014 4:23 PM
To: George Trakatelis; [email-masked]; Mike Gabriel
Subject: Re: [X2Go-Dev] Bug#566: X2Go Client for Windows 4.0.2.1 cannot create C:\Users\<username>\ssh\known_hosts file when the local Windows account username has non-Ascii characters

[...]

Hi Mike#1 and George,

1. What do you think of using toUtf8() instead? I am hoping that will fix the bug regardless of the locale.

I tried changing it to that (bug566.utf8.test.patch), but it still did not fix this bug with the greek username on my system with the locale set to English. It did not introduce a regression for my ASCII user account at least.

If libssh needs to be recompiled for Unicode, I can do that. I just recompiled it for bug #590.

2. in sshmasterconnection.cpp, we also set "SSH_OPTIONS_USER" . I think that we should also change its value from to.Ascii() to
toLocal8Bit() or toUtf8(). bug566.test.v2.patch has this change for toLocal8Bit(), and bug566.utf8.test.patch has it for toUtf8().

-Mike#2

---

From: Michael DePaulo <[email-masked]>
Date: Sun, 7 Sep 2014 22:55:13 -0400

Yeah. I think that libssh isn't being compiled with Unicode support,
or at least not under MingW 4.4. It's not 100% clear how to compile a
C library with Unicode support under MinGW and CMake. For example,
recent versions of MinGW let you specify the linker option -municode.
But I cannot find a single release note that states when exactly that
feature was added. I do not know if 4.4 has it or not.

I'm going to try to compile it with MinGW 4.8. I've been meaning to
upgrade to X2Go to MinGW 4.8 and QT 4.8.6 anyway. (bug: #474)

---

From: Mike Gabriel <[email-masked]>
Date: Mon, 08 Sep 2014 08:52:31 +0000

Hi Mike#2, hi George,

On  So 07 Sep 2014 12:04:51 CEST, George Trakatelis wrote:

> X2Go Client for Windows 4.0.2.1 cannot create  
> C:\Users\<username>\ssh\known_hosts file when the local Windows  
> account username has non-Ascii characters

Now that I think of it more thoroughly... Shouldn't we do a proper  
encode/decode here so that we convert UTF-8 to Windows-CP1251 encoding  
[1]?

Alternatively, one could work with filenames in UTF-16 as shown in  
this [2] example.

I am not sure about the exact position in the code that bugs this up,  
but I fear, the solution is non-trivial.

Either you need to detect the client-side encoding and convert between  
UTF-8 and that encoding, or we may consider addressing file names in  
UTF-16 (if that is possible in Qt).

Unfortunately, it seems that Windows uses different encodings at  
different places (e.g. command.exe vs. Windows Explorer).

Just guessing after a little bit of internet research on this,
Mike#1

[1] http://comments.gmane.org/gmane.comp.lib.qt.general/39868
[2] https://www.mail-archive.com/[email-masked]/msg00099.html

-- 

DAS-NETZWERKTEAM

GnuPG Key ID 0x25771B31

freeBusy:

---

From: Mike Gabriel <[email-masked]>
Date: Mon, 08 Sep 2014 08:58:33 +0000

Hi Mike#2,

> http://api.libssh.org/stable/group__libssh__session.html#ga7a801b85800baa3f4e16f5b47db0a73d

I am getting the suspicion, that this is a libssh issue.

X2Go Client and libssh should communicate via UTF-8 (because it is  
_the_ encoding that applications use internally, nowadays) and libssh  
should handle whatever conversion (probably to UTF-16 wide string?) is  
necessary for creating files, being aware of usernames, etc.

> toLocal8Bit(), and bug566.utf8.test.patch has it for toUtf8().

Same thing. SSH_OPTIONS_USER should be set as UTF-8 and libssh should  
convert to whatever encoding is needed.

Greets,
Mike

-- 

DAS-NETZWERKTEAM

GnuPG Key ID 0x25771B31

freeBusy:

---

From: Michael DePaulo <[email-masked]>
Date: Tue, 9 Sep 2014 09:15:52 -0400

On Mon, Sep 8, 2014 at 4:58 AM, Mike Gabriel
<[email-masked]> wrote:
>>> C:\Users\<username>\ssh\known_hosts file when the local Windows account
>>> username has non-Ascii characters
>>
>>
>> Hi Mike#1 and George,
>>
> X2Go Client and libssh should communicate via UTF-8 (because it is _the_
> encoding that applications use internally, nowadays) and libssh should
> handle whatever conversion (probably to UTF-16 wide string?) is necessary
> for creating files, being aware of usernames, etc.
>
[...]

Hi Mike#1 and George,

1.

I believe the best approach is to try to compile libssh with unicode
support, and pass unicode values to it. There is no guarantee that a
user's username (and home dir path) is in the same language as the
language that is set for non-Unicode programs. After all, the
"Language for non-Unicode programs" is called the "system locale", it
applies to all users on the system.

If we have to for the 4.0.3.0 release of x2goclient, we should fix
this bug for the system locale only, and then fix the bug for all
languages/locales later.

2.

See bug 474 for George's patch for compiling libssh with MinGW 4.8.2.
It worked :)

3.

I made some progress compiling libssh 0.6.3 with Unicode support.

With MinGW 4.4, when I specified the following values in CMAKE, libssh
built successfully, but it did not seem to change anything.
CMAKE_EXE_LINKER_FLAGS:STRING=-municode
CMAKE_MODULE_LINKER_FLAGS:STRING=-municode
CMAKE_SHARED_LINKER_FLAGS:STRING=-municode
CMAKE_STATIC_LINKER_FLAGS:STRING=-municode

With MinGW 4.8.2, when I specify those values, libssh fails to build.
I've attached the output.

This is a positive sign because it implies that MinGW 4.8.2 supports
-municode, whereas MinGW 4.4 did not.

I plan to try to resolve this build failure later tonight.

-Mike#2

---

From: "George Trakatelis" <[email-masked]>
Date: Mon, 15 Sep 2014 08:26:40 +0300

Dear Mike,

I have built libssh 0.6.3 in Visual Studio 2013 (with UNICODE and _UNICODE defined in all projects)
and debugged a tiny test app with function calls analogous to those used in x2goclient.
I'm afraid preprocessor definitions are not enough to make an application/program/library support Unicode.
One could migrate existing programs following a list of guidelines

http://msdn.microsoft.com/en-us/library/cc194801.aspx

For example, a function that fails in libssh (used in ssh_path_expand_tilde(), 
which is always called to replace a ~ with the actual home folder path in a string) is strdup.
To make libssh unicode-friendly strdup has to be replaced by _strdup

http://msdn.microsoft.com/en-us/library/y471khhc.aspx

_strdup would benefit from preprocessor definitions for Unicode support
as it would be converted to the proper Unicode function.

Imho, the only option for now is patch "bug566.test.v2.patch"
as it corrects the problem in the vast majority of Windows installations.
Remember that foreign Windows installations set by default the 
"Language for non-Unicode programs" to that particular foreign language.

-George

-----Original Message-----
From: Michael DePaulo [mailto:[email-masked]] 
Sent: Tuesday, September 9, 2014 4:16 PM
To: [email-masked]
Cc: George Trakatelis; Mike Gabriel
Subject: Re: [X2Go-Dev] Bug#566: X2Go Client for Windows 4.0.2.1 cannot create C:\Users\<username>\ssh\known_hosts file when the local Windows account username has non-Ascii characters

[...]

Hi Mike#1 and George,

1.

I believe the best approach is to try to compile libssh with unicode support, and pass unicode values to it. There is no guarantee that a user's username (and home dir path) is in the same language as the language that is set for non-Unicode programs. After all, the "Language for non-Unicode programs" is called the "system locale", it applies to all users on the system.

If we have to for the 4.0.3.0 release of x2goclient, we should fix this bug for the system locale only, and then fix the bug for all languages/locales later.

2.

3.

With MinGW 4.4, when I specified the following values in CMAKE, libssh built successfully, but it did not seem to change anything.

This is a positive sign because it implies that MinGW 4.8.2 supports -municode, whereas MinGW 4.4 did not.

-Mike#2