Subject: Re: [X2Go-User] Session resume fails with AFS home directories

From: Sebastian Flothow <[email-masked]>
Date: Mon, 16 Sep 2013 16:17:31 +0200

I did some further testing, and the resume failures are indeed due to 
missing AFS tokens. When suspending a session, the SSH connection is 
closed, sshd will call pam_close_session(), which means that pam_krb5 
and pam_afs_session will delete the user's ticket/token (resp.). The 
session therefore loses access to the home directory and appears to 
freeze up, preventing it from being resumed.

Both pam_krb5 and pam_afs_session accept retain_after_close as a 
parameter, which disables the delete-on-close behavior. With this 
parameter set, it becomes possible to resume sessions, unless the AFS 
token has expired.

This solves at least the case where the user reconnects quickly (eg. 
after a short network outage), but it still means sessions will become 
unresumable when left unused for a few days. I guess the only way to 
avoid this is to not store session data in the home directory. Can X2go 
be configured such that it uses eg. /tmp or /var/lib for this purpose?

---

From: Mike Gabriel <[email-masked]>
Date: Wed, 18 Sep 2013 23:24:38 +0200

Hi Sebastian,

On Mo 16 Sep 2013 16:17:31 CEST Sebastian Flothow wrote:

> I did some further testing, and the resume failures are indeed due  
> to missing AFS tokens. When suspending a session, the SSH connection  
> is closed, sshd will call pam_close_session(), which means that  
> pam_krb5 and pam_afs_session will delete the user's ticket/token  
> (resp.). The session therefore loses access to the home directory  
> and appears to freeze up, preventing it from being resumed.
>
> parameter set, it becomes possible to resume sessions, unless the  
> AFS token has expired.

Thanks for digging this out. Good work!!!

> after a short network outage), but it still means sessions will  
> become unresumable when left unused for a few days.

I get that. NFSv4 with Kerberos is very similar to the AFS token behaviour.

> I guess the only way to avoid this is to not store session data in  
> the home directory. Can X2go be configured such that it uses eg.  
> /tmp or /var/lib for this purpose?

In earlier versions of X2Go every session detail was in $HOME. Some of  
the session information has to be accessible by super-user root. Those  
bits, I have already moved out of the home (e.g. the session.log file).

Normally, the AFS token should be immediately restored after SSH login  
(which is the first action taken when resuming a session). However,  
this AFS token does not re-awake the session so it can be resumed. The  
question is why...

Does a session simply not resume (with an x2goagent still being  
present for this session)? Or does the x2goagent crash somewhere on  
the run (i.e. when the session is suspended and the AFS home freezes  
some time later)?

When evoking x2golistsessions, the first field of each output line is  
the x2goagent PID that is associated to that session in the same line.  
With non-resumable sessions, please check if the x2goagent processes  
remain active on the X2Go server or if the x2goagent processes crash  
(disappear). I can only imagine that the x2goagent processes remain  
alive (frozen) until the AFS token gets reinstated by the X2Go  
resuming SSH login. If x2goagent crashes somewhere on the way, we have  
to find out why and how to prevent it.

However, if x2goagent stays functional, we have to investigate, if  
there is anything AFS-critical in /usr/bin/x2goresume-session. If you  
look at the script /usr/bin/x2goresume-session, can you spot anything  
that might fail on AFS?

Greets,
Mike

-- 

DAS-NETZWERKTEAM
mike gabriel, herweg 7, 24357 fleckeby
fon: +49 (1520) 1976 148

GnuPG Key ID 0x25771B31
mail: [email-masked], http://das-netzwerkteam.de

freeBusy:
https://mail.das-netzwerkteam.de/freebusy/m.gabriel%40das-netzwerkteam.de.xfb

---

From: Sebastian Flothow <[email-masked]>
Date: Mon, 30 Sep 2013 17:21:17 +0200

Hi,

Am 18.09.2013 23:24, schrieb Mike Gabriel:
> Does a session simply not resume (with an x2goagent still being present
> for this session)? Or does the x2goagent crash somewhere on the run
> (i.e. when the session is suspended and the AFS home freezes some time
> later)?

I did a quick test (note that I removed retain_after_close from the PAM 
config again, so that I can create broken sessions quickly without 
waiting for AFS token expiry). Right after starting a new session, 
things look like this:

giplin101:~# x2golistsessions_root
26521|flothow-50-1380548308_stDXFCE_dp32|50|giplin101|R|2013-09-30T15:38:28|da9be4183a9c7d711f325742963c691e|10.0.0.105|30001|30002|2013-09-30T15:38:31|flothow|112|30003|
giplin101:~# ps 26521
   PID TTY      STAT   TIME COMMAND
26521 ?        S      0:00 /usr/lib/nx/../x2go/bin/x2goagent -extension 
XFIXES -extension GLX -nolisten tcp -D -auth /afs/gip.l

Then, after suspending the session:

giplin101:~# x2golistsessions_root
26521|flothow-50-1380548308_stDXFCE_dp32|50|giplin101|S|2013-09-30T15:38:28|da9be4183a9c7d711f325742963c691e|10.0.0.105|30001|30002|2013-09-30T15:41:32|flothow|186|30003|
giplin101:~# ps 26521
   PID TTY      STAT   TIME COMMAND
26521 ?        S      0:00 /usr/lib/nx/../x2go/bin/x2goagent -extension 
XFIXES -extension GLX -nolisten tcp -D -auth /afs/gip.l

After attempting to resume it:

giplin101:~# x2golistsessions_root
26521|flothow-50-1380548308_stDXFCE_dp32|50|giplin101|R|2013-09-30T15:38:28|da9be4183a9c7d711f325742963c691e|10.0.0.105|30001|30002|2013-09-30T15:43:20|flothow|315|30003|
giplin101:~# ps 26521
   PID TTY      STAT   TIME COMMAND
26521 ?        S      0:00 /usr/lib/nx/../x2go/bin/x2goagent -extension 
XFIXES -extension GLX -nolisten tcp -D -auth /afs/gip.l

It is still in this state now, more than half an hour later.

> If you look at
> the script /usr/bin/x2goresume-session, can you spot anything that might
> fail on AFS?

I already looked at this script a few weeks ago and added a bunch of 
debug statements which log various things to /var/log/x2godebug. When 
the script executes, there is a valid AFS token, $SESSION_DIR and 
${SESSION_DIR}/options are readable, and the script completes successfully.

However, I think that this is not meaningful. What happens is presumably 
this:

When first logging in (before an X2Go session exists), a new SSH session 
is created, which I'll refer to as the first SSH session. This session 
obtains a Kerberos ticket and an AFS token through PAM, and then spawns 
an X2Go sessions which inherits these. The Kerberos ticket is stored in 
a file pointed to by $KRB5CCNAME, while the AFS token is tied to the PAG 
(Process Authentication Group).

When suspending the X2Go session, this first SSH session is terminated. 
Depending on the PAM configuration, the ticket and token are either 
removed immediately, or expire some time later.

Now, when attempting to resume the X2Go session, a new, second SSH 
session is created. This session again obtains a ticket and a token, and 
it seems to be this session in which x2goresume-session is executed; 
however, this ticket/token is in a different file/PAG (resp.) than those 
from the first session, so the X2Go session can't use them.

After figuring this out, I remembered that pam_afs_session recognizes 
the parameter nopag, which inhibits PAG creation. Absent a PAG, AFS 
tokens are tied to user IDs instead, and indeed, when this option is 
set, sessions can be resumed even after their initial token expired - 
without PAGs, the new token from the second session propagates to the 
first session, since the user ID is identical. After resuming, the X2Go 
session still doesn't have a valid Kerberos ticket (because there are 
still two different ticket files), but it does have an AFS token, which 
is all that matters for filesystem access. Obtaining a new Kerberos 
ticket can then be done manually if necessary.

However, I'm a bit wary of using nopag in a production environment, 
because the man page also warns: "Be careful when using this option, 
since it means that the user will inherit a PAG from the process 
managing the login.  If sshd, for instance, is started in a PAG, every 
user who logs in via ssh will be put in the same PAG and will share 
tokens if this option is used."

To fix this so that it works without nopag, we'd need to move an AFS 
token from one PAG to another. I'm not aware of any way to do this 
directly, but it might be possible to copy the Kerberos ticket from the 
new ticket file to the old one, and then call aklog within the old 
session before attempting any file system access.

- Sebastian