X2Go Bug report logs - #272
[X2Go-User] Session resume fails with AFS home directories

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.x2go.org (full text, mbox, reply):

From: Mike Gabriel <mike.gabriel@das-netzwerkteam.de>

To: submit@bugs.x2go.org

Cc: x2go-user@lists.berlios.de

Subject: Re: [X2Go-User] Session resume fails with AFS home directories

Date: Fri, 26 Jul 2013 16:40:06 +0200

[Message part 1 (text/plain, inline)]

Package: x2goserver
Version: 4.0.1.3

Hi Sebastian,

(quoting your complete original mail, so we have it in the bug report  
I create with this/my reply)

On Fr 26 Jul 2013 15:08:50 CEST Sebastian Flothow wrote:

> I've just set up a Debian 7 box with X2Go. It does work in that it  
> is possible to start new sessions, however, resuming a previous  
> session does not work, it always results in this message: "The  
> remote proxy closed the connection while negotiating the session.  
> This may be due to the wrong authentication credentials passed to  
> the server."
>
> I suspect this is due to the fact that home directories are stored  
> in AFS (for regular users, that is; when logging in as root, whose  
> home directory is on a local ext4 FS, resume does work). Accessing  
> AFS requires an AFS token in the user's name, obtaining this in turn  
> requires a Kerberos ticket. PAM is set up to obtain both  
> automatically on login, but I guess something goes wrong there  
> during session resume.
>
> Is it possible to add custom commands to the X2Go login/resume  
> procedure? It would be quite helpful if the client could run klist  
> and tokens through the ssh session, and either log or display the  
> output.

Is there any environment variable that we have to set before we can  
access the home directory of the user?

My guess is that we have to set at least

  export KRB5CCNAME=???

Maybe any other env var for the AFS token?

We should get this issue fixed upstream, so I have switched over to  
x2go-dev and our bug tracker (done by sending my reply). Please reply  
to 272@bugs.x2go.org with your reply. Thanks.

Mike

-- 

DAS-NETZWERKTEAM
mike gabriel, herweg 7, 24357 fleckeby
fon: +49 (1520) 1976 148

GnuPG Key ID 0x25771B31
mail: mike.gabriel@das-netzwerkteam.de, http://das-netzwerkteam.de

freeBusy:
https://mail.das-netzwerkteam.de/freebusy/m.gabriel%40das-netzwerkteam.de.xfb

[Message part 2 (application/pgp-signature, inline)]

Message #10 received at 272@bugs.x2go.org (full text, mbox, reply):

From: Sebastian Flothow <sebastian.flothow@gip.com>

To: 272@bugs.x2go.org, mike.gabriel@das-netzwerkteam.de, x2go-user@lists.berlios.de

Subject: Re: [X2Go-User] Session resume fails with AFS home directories

Date: Mon, 29 Jul 2013 13:06:34 +0200

[Message part 1 (text/plain, inline)]

Am 26.07.2013 16:40, schrieb Mike Gabriel:
> Package: x2goserver
> Version: 4.0.1.3

By now it's 4.0.1.6-0~x2go1+wheezy~main~712~build1, but the problem 
persists.


> Is there any environment variable that we have to set before we can
> access the home directory of the user?
>
> My guess is that we have to set at least
>
>    export KRB5CCNAME=???
>
> Maybe any other env var for the AFS token?

No, that should not be necessary. KRB5CCNAME is set by pam_krb5.so. 
pam_afs_session.so in turn uses this to obtain an AFS token, then 
associates it with a new Process Authentication Group. The PAG ID is 
stored in the group array for the session, i.e. "id" shows an additional 
artificial group id. In fact this all works flawlessly on initial login, 
it's only on resume where it fails.

It occurs to me now that both KRB5CCNAME and PAG are per-session rather 
than per-user, so that might be the cause for this problem (but I'm 
really just guessing here).

Is there a detailed description of the resume process? Does it involve 
any shell scripts or similar I could hook into in order to log 
additional information?


I'm attaching /var/log/user.log as well as the client output from a 
failed resume attempt, maybe this offers some clues.

Thanks,
Sebastian

[client.txt (text/plain, attachment)]

[user.log (text/x-log, attachment)]

Message #15 received at 272@bugs.x2go.org (full text, mbox, reply):

From: Sebastian Flothow <sebastian.flothow@gip.com>

To: 272@bugs.x2go.org

Cc: x2go-user@lists.berlios.de

Subject: Re: [X2Go-User] Session resume fails with AFS home directories

Date: Mon, 16 Sep 2013 16:17:31 +0200

I did some further testing, and the resume failures are indeed due to 
missing AFS tokens. When suspending a session, the SSH connection is 
closed, sshd will call pam_close_session(), which means that pam_krb5 
and pam_afs_session will delete the user's ticket/token (resp.). The 
session therefore loses access to the home directory and appears to 
freeze up, preventing it from being resumed.

Both pam_krb5 and pam_afs_session accept retain_after_close as a 
parameter, which disables the delete-on-close behavior. With this 
parameter set, it becomes possible to resume sessions, unless the AFS 
token has expired.

This solves at least the case where the user reconnects quickly (eg. 
after a short network outage), but it still means sessions will become 
unresumable when left unused for a few days. I guess the only way to 
avoid this is to not store session data in the home directory. Can X2go 
be configured such that it uses eg. /tmp or /var/lib for this purpose?

Message #20 received at 272@bugs.x2go.org (full text, mbox, reply):

From: Mike Gabriel <mike.gabriel@das-netzwerkteam.de>

To: Sebastian Flothow <sebastian.flothow@gip.com>

Cc: 272@bugs.x2go.org, x2go-user@lists.berlios.de

Subject: Re: [X2Go-User] Session resume fails with AFS home directories

Date: Wed, 18 Sep 2013 23:24:38 +0200

[Message part 1 (text/plain, inline)]

Hi Sebastian,

On Mo 16 Sep 2013 16:17:31 CEST Sebastian Flothow wrote:

> I did some further testing, and the resume failures are indeed due  
> to missing AFS tokens. When suspending a session, the SSH connection  
> is closed, sshd will call pam_close_session(), which means that  
> pam_krb5 and pam_afs_session will delete the user's ticket/token  
> (resp.). The session therefore loses access to the home directory  
> and appears to freeze up, preventing it from being resumed.
>
> Both pam_krb5 and pam_afs_session accept retain_after_close as a  
> parameter, which disables the delete-on-close behavior. With this  
> parameter set, it becomes possible to resume sessions, unless the  
> AFS token has expired.

Thanks for digging this out. Good work!!!

> This solves at least the case where the user reconnects quickly (eg.  
> after a short network outage), but it still means sessions will  
> become unresumable when left unused for a few days.

I get that. NFSv4 with Kerberos is very similar to the AFS token behaviour.

> I guess the only way to avoid this is to not store session data in  
> the home directory. Can X2go be configured such that it uses eg.  
> /tmp or /var/lib for this purpose?

In earlier versions of X2Go every session detail was in $HOME. Some of  
the session information has to be accessible by super-user root. Those  
bits, I have already moved out of the home (e.g. the session.log file).

Normally, the AFS token should be immediately restored after SSH login  
(which is the first action taken when resuming a session). However,  
this AFS token does not re-awake the session so it can be resumed. The  
question is why...

Does a session simply not resume (with an x2goagent still being  
present for this session)? Or does the x2goagent crash somewhere on  
the run (i.e. when the session is suspended and the AFS home freezes  
some time later)?

When evoking x2golistsessions, the first field of each output line is  
the x2goagent PID that is associated to that session in the same line.  
With non-resumable sessions, please check if the x2goagent processes  
remain active on the X2Go server or if the x2goagent processes crash  
(disappear). I can only imagine that the x2goagent processes remain  
alive (frozen) until the AFS token gets reinstated by the X2Go  
resuming SSH login. If x2goagent crashes somewhere on the way, we have  
to find out why and how to prevent it.

However, if x2goagent stays functional, we have to investigate, if  
there is anything AFS-critical in /usr/bin/x2goresume-session. If you  
look at the script /usr/bin/x2goresume-session, can you spot anything  
that might fail on AFS?


Greets,
Mike

-- 

DAS-NETZWERKTEAM
mike gabriel, herweg 7, 24357 fleckeby
fon: +49 (1520) 1976 148

GnuPG Key ID 0x25771B31
mail: mike.gabriel@das-netzwerkteam.de, http://das-netzwerkteam.de

freeBusy:
https://mail.das-netzwerkteam.de/freebusy/m.gabriel%40das-netzwerkteam.de.xfb

[Message part 2 (application/pgp-signature, inline)]

Message #25 received at 272@bugs.x2go.org (full text, mbox, reply):

From: Sebastian Flothow <sebastian.flothow@gip.com>

To: Mike Gabriel <mike.gabriel@das-netzwerkteam.de>, 272@bugs.x2go.org

Subject: Re: [X2Go-User] Session resume fails with AFS home directories

Date: Mon, 30 Sep 2013 17:21:17 +0200

Hi,

Am 18.09.2013 23:24, schrieb Mike Gabriel:
> Does a session simply not resume (with an x2goagent still being present
> for this session)? Or does the x2goagent crash somewhere on the run
> (i.e. when the session is suspended and the AFS home freezes some time
> later)?

I did a quick test (note that I removed retain_after_close from the PAM 
config again, so that I can create broken sessions quickly without 
waiting for AFS token expiry). Right after starting a new session, 
things look like this:

giplin101:~# x2golistsessions_root
26521|flothow-50-1380548308_stDXFCE_dp32|50|giplin101|R|2013-09-30T15:38:28|da9be4183a9c7d711f325742963c691e|10.0.0.105|30001|30002|2013-09-30T15:38:31|flothow|112|30003|
giplin101:~# ps 26521
  PID TTY      STAT   TIME COMMAND
26521 ?        S      0:00 /usr/lib/nx/../x2go/bin/x2goagent -extension 
XFIXES -extension GLX -nolisten tcp -D -auth /afs/gip.l


Then, after suspending the session:

giplin101:~# x2golistsessions_root
26521|flothow-50-1380548308_stDXFCE_dp32|50|giplin101|S|2013-09-30T15:38:28|da9be4183a9c7d711f325742963c691e|10.0.0.105|30001|30002|2013-09-30T15:41:32|flothow|186|30003|
giplin101:~# ps 26521
  PID TTY      STAT   TIME COMMAND
26521 ?        S      0:00 /usr/lib/nx/../x2go/bin/x2goagent -extension 
XFIXES -extension GLX -nolisten tcp -D -auth /afs/gip.l


After attempting to resume it:

giplin101:~# x2golistsessions_root
26521|flothow-50-1380548308_stDXFCE_dp32|50|giplin101|R|2013-09-30T15:38:28|da9be4183a9c7d711f325742963c691e|10.0.0.105|30001|30002|2013-09-30T15:43:20|flothow|315|30003|
giplin101:~# ps 26521
  PID TTY      STAT   TIME COMMAND
26521 ?        S      0:00 /usr/lib/nx/../x2go/bin/x2goagent -extension 
XFIXES -extension GLX -nolisten tcp -D -auth /afs/gip.l


It is still in this state now, more than half an hour later.


> If you look at
> the script /usr/bin/x2goresume-session, can you spot anything that might
> fail on AFS?

I already looked at this script a few weeks ago and added a bunch of 
debug statements which log various things to /var/log/x2godebug. When 
the script executes, there is a valid AFS token, $SESSION_DIR and 
${SESSION_DIR}/options are readable, and the script completes successfully.


However, I think that this is not meaningful. What happens is presumably 
this:

When first logging in (before an X2Go session exists), a new SSH session 
is created, which I'll refer to as the first SSH session. This session 
obtains a Kerberos ticket and an AFS token through PAM, and then spawns 
an X2Go sessions which inherits these. The Kerberos ticket is stored in 
a file pointed to by $KRB5CCNAME, while the AFS token is tied to the PAG 
(Process Authentication Group).

When suspending the X2Go session, this first SSH session is terminated. 
Depending on the PAM configuration, the ticket and token are either 
removed immediately, or expire some time later.

Now, when attempting to resume the X2Go session, a new, second SSH 
session is created. This session again obtains a ticket and a token, and 
it seems to be this session in which x2goresume-session is executed; 
however, this ticket/token is in a different file/PAG (resp.) than those 
from the first session, so the X2Go session can't use them.


After figuring this out, I remembered that pam_afs_session recognizes 
the parameter nopag, which inhibits PAG creation. Absent a PAG, AFS 
tokens are tied to user IDs instead, and indeed, when this option is 
set, sessions can be resumed even after their initial token expired - 
without PAGs, the new token from the second session propagates to the 
first session, since the user ID is identical. After resuming, the X2Go 
session still doesn't have a valid Kerberos ticket (because there are 
still two different ticket files), but it does have an AFS token, which 
is all that matters for filesystem access. Obtaining a new Kerberos 
ticket can then be done manually if necessary.

However, I'm a bit wary of using nopag in a production environment, 
because the man page also warns: "Be careful when using this option, 
since it means that the user will inherit a PAG from the process 
managing the login.  If sshd, for instance, is started in a PAG, every 
user who logs in via ssh will be put in the same PAG and will share 
tokens if this option is used."


To fix this so that it works without nopag, we'd need to move an AFS 
token from one PAG to another. I'm not aware of any way to do this 
directly, but it might be possible to copy the Kerberos ticket from the 
new ticket file to the old one, and then call aklog within the old 
session before attempting any file system access.


- Sebastian

Message #30 received at 272@bugs.x2go.org (full text, mbox, reply):

From: Roy Williams <fang64@gmail.com>

To: 272@bugs.x2go.org

Subject: Regarding x2go and afs interaction

Date: Tue, 13 Jan 2015 09:11:26 -0500

[Message part 1 (text/plain, inline)]

Hello Everyone,

I have a suggestion that basically involves using k5start from Russ Albury
which I suspect will no longer be as maintained in the future. Available at
http://www.eyrie.org/~eagle/software/kstart/k5start.html and having that
maintain credentials in the session, then copying the KRB5CCNAME into a new
session and having it renew the Kerberos tickets, so when k5start runs
aklog it'll renew the tokens in the suspended session. It's not ideal but
it does allow you to have session resuming.

I am not sure what the security implications would be doing this since I
suspect this would be frowned on by the Kerberos community. This k5start
tool was intended to keep long running processes from losing their file
system access on a host.

Roy Williams (fang64@gmail.com)

[Message part 2 (text/html, inline)]

Send a report that this bug log contains spam.