Subject: paramiko fails to recognize known_hosts

From: Jan Engelhardt <[email-masked]>
Date: Tue, 17 Dec 2013 14:47:20 +0100 (CET)

Package: x2gobroker
Version: 0.0.2.3

When logging into a x2gobroker-0.0.2.3 with paramiko-1.11.0,
it so happens that, for whatever reason, it rejects the
remote side's key despite that being in
/var/lib/x2gobroker/.ssh/known_hosts.

{u'defsndport': False, u'useiconv': False, u'iconvfrom': 'UTF-8', u'height': 768, u'pack': '16m-jpeg', u'export': '', u'quality': 9, u'fullscreen': False, u'layout': '', 'host=terminalsrv2': '172.18.7.216', 'host=terminalsrv1': '172.18.7.205', u'width': 1024, u'speed': 4, u'soundtunnel': False, u'soundsystem': 'pulse', u'print': True, u'type': 'auto', u'sndport': 4713, u'xinerama': True, u'usekbd': True, u'variant': '', u'applications': ['TERMINAL', 'WWWBROWSER', 'MAILCLIENT', 'OFFICE'], u'host': ['terminalsrv1', 'terminalsrv2', 'terminalsrv3'], u'link': 'lan', u'multidisp': False, 'host=terminalsrv3': '172.18.7.214', u'fstunnel': False, u'sound': False, u'rootless': False, u'name': 'Terminalsitzung starten LAN Einstellungen', u'sshproxyport': 22, u'iconvto': 'UTF-8', u'directrdp': False, u'useexports': True, u'command': 'GNOME', u'dpi': 96, u'sshport': 22, u'setdpi': 0, u'startsoundsystem': 'false', u'user': ''}
['terminalsrv1', 'terminalsrv2', 'terminalsrv3']
['terminalsrv3', 'terminalsrv1', 'terminalsrv2']
{u'hostname': 'terminalsrv2', u'port': 22}
starting thread (client mode): 0xe8ded0L
Connected (version 2.0, client OpenSSH_5.1)
kex algos:['diffie-hellman-group-exchange-sha256', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa', 'ssh-dss'] client encrypt:['aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'arcfour128', 'arcfour256', 'arcfour', 'aes192-cbc', 'aes256-cbc', [email-masked]', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr'] server encrypt:['aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'arcfour128', 'arcfour256', 'arcfour', 'aes192-cbc', 'aes256-cbc', [email-masked]', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr'] client mac:['hmac-md5', 'hmac-sha1', [email-masked]', 'hmac-ripemd160', [email-masked]', 'hmac-sha1-96', 'hmac-md5-96'] server mac:['hmac-md5', 'hmac-sha1', [email-masked]', 'hmac-ripemd160', [email-masked]', 'hmac-sha1-96', 'hmac-md5-96'] client compress:['none', [email-masked]'] server compress:['none', [email-masked]'] client lang:[''] ser
 ver lang:[''] kex follows?False
Ciphers agreed: local=aes128-ctr, remote=aes128-ctr
using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none
Switch to new keys ...
/usr/lib64/python2.6/site-packages/paramiko/client.py:95:
UserWarning: Unknown ssh-rsa host key for terminalsrv2: 4d1d9d46a16da7d8c5a3829c920ff5da
  (key.get_name(), hostname, hexlify(key.get_fingerprint())))
['terminalsrv3', 'terminalsrv1']
{u'hostname': 'terminalsrv1', u'port': 22}
starting thread (client mode): 0xe95cd0L
Connected (version 2.0, client OpenSSH_5.1)
kex algos:['diffie-hellman-group-exchange-sha256', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa', 'ssh-dss'] client encrypt:['aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'arcfour128', 'arcfour256', 'arcfour', 'aes192-cbc', 'aes256-cbc', [email-masked]', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr'] server encrypt:['aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'arcfour128', 'arcfour256', 'arcfour', 'aes192-cbc', 'aes256-cbc', [email-masked]', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr'] client mac:['hmac-md5', 'hmac-sha1', [email-masked]', 'hmac-ripemd160', [email-masked]', 'hmac-sha1-96', 'hmac-md5-96'] server mac:['hmac-md5', 'hmac-sha1', [email-masked]', 'hmac-ripemd160', [email-masked]', 'hmac-sha1-96', 'hmac-md5-96'] client compress:['none', [email-masked]'] server compress:['none', [email-masked]'] client lang:[''] ser
 ver lang:[''] kex follows?False
Ciphers agreed: local=aes128-ctr, remote=aes128-ctr
using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none
Switch to new keys ...
/usr/lib64/python2.6/site-packages/paramiko/client.py:95:
UserWarning: Unknown ssh-rsa host key for terminalsrv1: 4d1d9d46a16da7d8c5a3829c920ff5da
  (key.get_name(), hostname, hexlify(key.get_fingerprint())))
thservice.logEOF in transport thread
['terminalsrv3']
{u'hostname': 'terminalsrv3', u'port': 22}
starting thread (client mode): 0xe95290L
Connected (version 2.0, client OpenSSH_5.1)
kex algos:['diffie-hellman-group-exchange-sha256', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa', 'ssh-dss'] client encrypt:['aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'arcfour128', 'arcfour256', 'arcfour', 'aes192-cbc', 'aes256-cbc', [email-masked]', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr'] server encrypt:['aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'arcfour128', 'arcfour256', 'arcfour', 'aes192-cbc', 'aes256-cbc', [email-masked]', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr'] client mac:['hmac-md5', 'hmac-sha1', [email-masked]', 'hmac-ripemd160', [email-masked]', 'hmac-sha1-96', 'hmac-md5-96'] server mac:['hmac-md5', 'hmac-sha1', [email-masked]', 'hmac-ripemd160', [email-masked]', 'hmac-sha1-96', 'hmac-md5-96'] client compress:['none', [email-masked]'] server compress:['none', [email-masked]'] client lang:[''] ser
 ver lang:[''] kex follows?False
Ciphers agreed: local=aes128-ctr, remote=aes128-ctr
using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none
Switch to new keys ...
/usr/lib64/python2.6/site-packages/paramiko/client.py:95:
UserWarning: Unknown ssh-rsa host key for terminalsrv3: 4d1d9d46a16da7d8c5a3829c920ff5da
  (key.get_name(), hostname, hexlify(key.get_fingerprint())))
EOF in transport thread
EOF in transport thread
starting thread (client mode): 0xe8df50L
Connected (version 2.0, client OpenSSH_5.1)
kex algos:['diffie-hellman-group-exchange-sha256', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa', 'ssh-dss'] client encrypt:['aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'arcfour128', 'arcfour256', 'arcfour', 'aes192-cbc', 'aes256-cbc', [email-masked]', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr'] server encrypt:['aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'arcfour128', 'arcfour256', 'arcfour', 'aes192-cbc', 'aes256-cbc', [email-masked]', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr'] client mac:['hmac-md5', 'hmac-sha1', [email-masked]', 'hmac-ripemd160', [email-masked]', 'hmac-sha1-96', 'hmac-md5-96'] server mac:['hmac-md5', 'hmac-sha1', [email-masked]', 'hmac-ripemd160', [email-masked]', 'hmac-sha1-96', 'hmac-md5-96'] client compress:['none', [email-masked]'] server compress:['none', [email-masked]'] client lang:[''] ser
 ver lang:[''] kex follows?False
Ciphers agreed: local=aes128-ctr, remote=aes128-ctr
using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none
Switch to new keys ...
Traceback (most recent call last):
  File "/usr/sbin/x2gobroker", line 145, in <module>
    tornado.ioloop.IOLoop.instance().start()
  File "/usr/lib64/python2.6/site-packages/tornado/ioloop.py", line 606, in start
    self._run_callback(callback)
  File "/usr/lib64/python2.6/site-packages/tornado/ioloop.py", line 457, in _run_callback
    callback()
  File "/usr/lib64/python2.6/site-packages/tornado/stack_context.py", line 330, in wrapped
    raise_exc_info(exc)
  File "/usr/lib64/python2.6/site-packages/tornado/stack_context.py", line 301, in wrapped
    ret = fn(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/tornado/iostream.py", line 340, in wrapper
    callback(*args)
  File "/usr/lib64/python2.6/site-packages/tornado/stack_context.py", line 330, in wrapped
    raise_exc_info(exc)
  File "/usr/lib64/python2.6/site-packages/tornado/stack_context.py", line 301, in wrapped
    ret = fn(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/tornado/httpserver.py", line 339, in _on_request_body
    self.request_callback(self._request)
  File "/usr/lib64/python2.6/site-packages/tornado/web.py", line 1599, in __call__
    handler._execute(transforms, *args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/tornado/web.py", line 1133, in _execute
    self._when_complete(self.prepare(), self._execute_method)
  File "/usr/lib64/python2.6/site-packages/tornado/web.py", line 1140, in _when_complete
    callback()
  File "/usr/lib64/python2.6/site-packages/tornado/web.py", line 1161, in _execute_method
    self._when_complete(method(*self.path_args, **self.path_kwargs),
  File "/usr/lib64/python2.6/site-packages/x2gobroker/web/plain.py", line 136, in post
    profiles = broker_backend.list_profiles(username)
  File "/usr/lib64/python2.6/site-packages/x2gobroker/brokers/base_broker.py", line 870, in list_profiles
    running_sessions, suspended_sessions = x2gobroker.agent.has_sessions(username, query_mode=agent_query_mode, remote_agent=remote_agent)
  File "/usr/lib64/python2.6/site-packages/x2gobroker/agent.py", line 229, in has_sessions
    _session_list = list_sessions(username, query_mode=query_mode, remote_agent=remote_agent)
  File "/usr/lib64/python2.6/site-packages/x2gobroker/agent.py", line 174, in list_sessions
    return call_remote_broker_agent(username, mode='listsessions', remote_agent=remote_agent)
  File "/usr/lib64/python2.6/site-packages/x2gobroker/agent.py", line 139, in call_remote_broker_agent
    raise x2gobroker.x2gobroker_exceptions.X2GoBrokerAgentException('Query to remote X2Go Broker Agent (user: {user}, hostname: {hostname}, port: {port}) failed'.format(user=remote_username, hostname=remote_hostname, port=remote_port))
x2gobroker.x2gobroker_exceptions.X2GoBrokerAgentException: Query to remote X2Go Broker Agent (user: x2gobroker, hostname: terminalsrv3, port: 22) failed

---

From: Jan Engelhardt <[email-masked]>
Date: Tue, 17 Dec 2013 15:14:59 +0100 (CET)

strace reveals this:

2844  open("//.ssh/known_hosts", O_RDONLY) = -1 ENOENT (No such file or
directory)

Looking at the environ of a newly-started x2gobroker:

# hexdump -C /proc/20044/environ
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000840  00 00 00 00 00 00 00                              |.......|
00000847
#

So the environment was modified inside python and I have no
further clue on what its actual state is, whether the environ
array just moved in memory (plausible), or destroyed in-place
leaving nothing for paramiko.

---

From: Jan Engelhardt <[email-masked]>
Date: Tue, 17 Dec 2013 15:47:16 +0100 (CET)

When started from sysvinit, the HOME environment variable is not set 
(what to, anyway!), and startproc does not set HOME _either_.
Therefore, /etc/init.d/x2gobroker-daemon needs a

  export HOME=/var/lib/x2gobroker

or something nicer of equal footing so that /usr/sbin/x2gobroker has 
something to work with.