X2Go Bug report logs - #1637
[llm-coauthored] Intermittent SIGSEGV in ssh_select()/ssh_event_add_session() from SshMasterConnection::channelLoop() -- looks like a data race on the channel list

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.x2go.org (full text, mbox, reply):

From: Philipp Klenze <p.klenze@gsi.de>

To: <submit@bugs.x2go.org>

Subject: [llm-coauthored] Intermittent SIGSEGV in ssh_select()/ssh_event_add_session() from SshMasterConnection::channelLoop() -- looks like a data race on the channel list

Date: Thu, 17 Sep 2026 10:47:19 +0200

[Message part 1 (text/plain, inline)]

Package: x2goclient
Version: 4.1.2.3-4

--------------------------------------------------------------------
NOTE ON HOW THIS REPORT WAS PRODUCED
--------------------------------------------------------------------
The debugging (gdb sessions, valgrind run, log analysis) was done
together with an LLM assistant (Claude) at my direction -- I ran
every command myself and verified the findings, but the writeup
itself is LLM-coauthored. Flagging this in the subject line;
disregard or downgrade this note as you see fit.

--------------------------------------------------------------------
SUMMARY
--------------------------------------------------------------------
x2goclient intermittently segfaults with a SIGSEGV inside libssh's
ssh_select() -> ssh_event_add_session(), called from
SshMasterConnection::channelLoop(). Both a gdb post-mortem analysis
of a core file and a live valgrind/memcheck run independently point
to the same thing: the channels[] array that channelLoop() hands to
ssh_select() gets corrupted, almost certainly by a race between the
SSH worker thread iterating it and something else (channel
teardown?) mutating/freeing it concurrently.

--------------------------------------------------------------------
ENVIRONMENT
--------------------------------------------------------------------
- x2goclient: 4.1.2.3-4 (Debian trixie)
- libssh4: 0.11.2-1+deb13u1 (trixie-security), reported SONAME
   libssh.so.4.10.5
- OS: Debian trixie, amd64
- Connection topology: two-hop, via an SSH jump host
   (SshMasterConnection instantiated once for the direct hop to the
   proxy, again for the tunnelled hop to the target host)
- valgrind: 3.24.0

--------------------------------------------------------------------
BACKTRACE (gdb, from a core file)
--------------------------------------------------------------------
Program terminated with signal SIGSEGV, Segmentation fault.
#0  ssh_select (channels=0x7fc590005df0, outchannels=0x7fc590009dc0, 
maxfd=0,
     readfds=0x7fc597349910, timeout=<optimized out>) at 
./src/connect.c:351
#1  0x0000564b01ded639 in SshMasterConnection::channelLoop (
     this=0x7fc5c8007740) at ../src/sshmasterconnection.cpp:1981
#2  0x0000564b01df1304 in SshMasterConnection::run (this=0x7fc5c8007740)
     at ../src/sshmasterconnection.cpp:752
#3  0x00007fc5d32ded21 in operator() (__closure=<optimized out>)
     at thread/qthread_unix.cpp:350
#4  (anonymous 
namespace)::terminate_on_exception<QThreadPrivate::start(void*)::<lambda()> 
 > (t=...) at thread/qthread_unix.cpp:287
#5  QThreadPrivate::start (arg=0x7fc5c8007740) at 
thread/qthread_unix.cpp:310
#6  0x00007fc5d2c9e8bb in start_thread (arg=<optimized out>)
     at ./nptl/pthread_create.c:448
#7  0x00007fc5d2d1c538 in __GI___clone3 ()
     at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

libssh's connect.c at that point (0.11.2 source):

   346         ssh_event event = ssh_event_new();
   347         int firstround = 1;
   348
   349         base_tm = tm = (timeout->tv_sec * 1000) + 
(timeout->tv_usec / 1000);
   350         for (i = 0 ; channels[i] != NULL; ++i) {
   351             ssh_event_add_session(event, channels[i]->session);
   352         }

The crash is on line 351, dereferencing channels[i]->session.

(I reconstructed this snippet from general knowledge of the libssh
0.11.x source rather than fetching the exact trixie-security tarball
-- worth double-checking against the real source for this build if
it matters to you.)

--------------------------------------------------------------------
POINTER INSPECTION
--------------------------------------------------------------------
(gdb) print channels[0]
$9 = (ssh_channel) 0x7fc590005
(gdb) print channels[1]
$10 = (ssh_channel) 0x0
(gdb) print *channels[0]
Cannot access memory at address 0x7fc590005

For reference, the array itself lives at channels = 0x7fc590005df0.
channels[0]'s value (0x7fc590005) is channels right-shifted by 12
bits -- i.e. (address_of_field) >> 12. That's not a coincidence: it
is the exact shape of glibc's "safe-linking" freelist obfuscation
(protected_ptr = (&field >> 12) XOR next; with next == NULL this
collapses to &field >> 12). In other words, the chunk backing this
channels array appears to have already been free()'d by the time
channelLoop()'s thread read it, and what's being read back is
leftover allocator bookkeeping, not a real ssh_channel*.

--------------------------------------------------------------------
CONFIRMED LIVE UNDER VALGRIND (--trace-children=yes)
--------------------------------------------------------------------
Running with:
   valgrind --trace-children=yes --tool=memcheck --track-origins=yes \
       x2goclient --debug --libssh-debug
caught the same class of fault live:

==372207== Invalid read of size 8
==372207==    at 0x4B23B9A: ssh_event_add_session (in libssh.so.4.10.5)
==372207==    by 0x4B0927E: ssh_select (in libssh.so.4.10.5)
==372207==    by 0x1FB638: ??? (in /usr/libexec/x2goclient)
==372207==    by 0x1FF303: ??? (in /usr/libexec/x2goclient)
==372207==    by 0x5C3AD20: ??? (in libQt5Core.so.5.15.15)
==372207==    by ...start_thread / clone
==372207==  Address 0x2a000005c8 is not stack'd, malloc'd or (recently) 
free'd
...
==372207== Process terminating with default action of signal 11 
(SIGSEGV): dumping core

(offsets 0x1FB638/0x1FF303 line up with channelLoop()/run() from the
gdb trace above.) Note this crash happened in a forked child process
running the x2goclient binary image itself (a separate valgrind
==PID== group, not ssh/nxproxy), consistent with x2goclient forking
off a second SshMasterConnection worker for the jump-host hop.

Full (redacted, de-duplicated) valgrind log is attached separately:
x2go-valgrind.sanitized.log

--------------------------------------------------------------------
POSSIBLE CONTRIBUTING FACTOR: SCP-SESSION RETRY LOOP
--------------------------------------------------------------------
The valgrind run that caught the crash also logged 26,670 repeated
attempts of:

   x2go-DEBUG-../src/sshmasterconnection.cpp:1850> Error initializing
   SCP session: Failed to open channel for scp

in a tight loop, right before the crash. This looks like a separate,
unbounded retry loop (worth its own bug report?) -- but it's also
plausibly what's making the race easy to hit: rapid repeated channel
creation/teardown is exactly the kind of hammering that would narrow
the timing window needed to trigger a use-after-free like this.

--------------------------------------------------------------------
PRIOR RELATED REPORTS
--------------------------------------------------------------------
This isn't the first time SshMasterConnection's threading has
produced crashes in this area -- see e.g.:
   #187  "Destroying SshMasterConnection off the main thread leads to
         a diagnostic from Qt debug libs"
   #1460 "Windows client crashes if Jumphost runs NetBSD 6 (probably
         race)"
Both point at the same general area (channel/session lifecycle vs.
thread boundaries) without a definitive fix landing, as far as I can
tell from the bug history.

--------------------------------------------------------------------
ATTACHMENT
--------------------------------------------------------------------
x2go-valgrind.sanitized.log -- de-duplicated, redacted valgrind/
memcheck log (hostnames, usernames, an internal IP, and what looked
like a live session cookie have been replaced with placeholders)

--Philipp with Claude 5 Sonnet

[x2go-valgrind.sanitized.log (text/x-log, attachment)]

Send a report that this bug log contains spam.