Package: x2goclient Version: 4.1.2.3-4 -------------------------------------------------------------------- NOTE ON HOW THIS REPORT WAS PRODUCED -------------------------------------------------------------------- The debugging (gdb sessions, valgrind run, log analysis) was done together with an LLM assistant (Claude) at my direction -- I ran every command myself and verified the findings, but the writeup itself is LLM-coauthored. Flagging this in the subject line; disregard or downgrade this note as you see fit. -------------------------------------------------------------------- SUMMARY -------------------------------------------------------------------- x2goclient intermittently segfaults with a SIGSEGV inside libssh's ssh_select() -> ssh_event_add_session(), called from SshMasterConnection::channelLoop(). Both a gdb post-mortem analysis of a core file and a live valgrind/memcheck run independently point to the same thing: the channels[] array that channelLoop() hands to ssh_select() gets corrupted, almost certainly by a race between the SSH worker thread iterating it and something else (channel teardown?) mutating/freeing it concurrently. -------------------------------------------------------------------- ENVIRONMENT -------------------------------------------------------------------- - x2goclient: 4.1.2.3-4 (Debian trixie) - libssh4: 0.11.2-1+deb13u1 (trixie-security), reported SONAME libssh.so.4.10.5 - OS: Debian trixie, amd64 - Connection topology: two-hop, via an SSH jump host (SshMasterConnection instantiated once for the direct hop to the proxy, again for the tunnelled hop to the target host) - valgrind: 3.24.0 -------------------------------------------------------------------- BACKTRACE (gdb, from a core file) -------------------------------------------------------------------- Program terminated with signal SIGSEGV, Segmentation fault. #0 ssh_select (channels=0x7fc590005df0, outchannels=0x7fc590009dc0, maxfd=0, readfds=0x7fc597349910, timeout=) at ./src/connect.c:351 #1 0x0000564b01ded639 in SshMasterConnection::channelLoop ( this=0x7fc5c8007740) at ../src/sshmasterconnection.cpp:1981 #2 0x0000564b01df1304 in SshMasterConnection::run (this=0x7fc5c8007740) at ../src/sshmasterconnection.cpp:752 #3 0x00007fc5d32ded21 in operator() (__closure=) at thread/qthread_unix.cpp:350 #4 (anonymous namespace)::terminate_on_exception > (t=...) at thread/qthread_unix.cpp:287 #5 QThreadPrivate::start (arg=0x7fc5c8007740) at thread/qthread_unix.cpp:310 #6 0x00007fc5d2c9e8bb in start_thread (arg=) at ./nptl/pthread_create.c:448 #7 0x00007fc5d2d1c538 in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 libssh's connect.c at that point (0.11.2 source): 346 ssh_event event = ssh_event_new(); 347 int firstround = 1; 348 349 base_tm = tm = (timeout->tv_sec * 1000) + (timeout->tv_usec / 1000); 350 for (i = 0 ; channels[i] != NULL; ++i) { 351 ssh_event_add_session(event, channels[i]->session); 352 } The crash is on line 351, dereferencing channels[i]->session. (I reconstructed this snippet from general knowledge of the libssh 0.11.x source rather than fetching the exact trixie-security tarball -- worth double-checking against the real source for this build if it matters to you.) -------------------------------------------------------------------- POINTER INSPECTION -------------------------------------------------------------------- (gdb) print channels[0] $9 = (ssh_channel) 0x7fc590005 (gdb) print channels[1] $10 = (ssh_channel) 0x0 (gdb) print *channels[0] Cannot access memory at address 0x7fc590005 For reference, the array itself lives at channels = 0x7fc590005df0. channels[0]'s value (0x7fc590005) is channels right-shifted by 12 bits -- i.e. (address_of_field) >> 12. That's not a coincidence: it is the exact shape of glibc's "safe-linking" freelist obfuscation (protected_ptr = (&field >> 12) XOR next; with next == NULL this collapses to &field >> 12). In other words, the chunk backing this channels array appears to have already been free()'d by the time channelLoop()'s thread read it, and what's being read back is leftover allocator bookkeeping, not a real ssh_channel*. -------------------------------------------------------------------- CONFIRMED LIVE UNDER VALGRIND (--trace-children=yes) -------------------------------------------------------------------- Running with: valgrind --trace-children=yes --tool=memcheck --track-origins=yes \ x2goclient --debug --libssh-debug caught the same class of fault live: ==372207== Invalid read of size 8 ==372207== at 0x4B23B9A: ssh_event_add_session (in libssh.so.4.10.5) ==372207== by 0x4B0927E: ssh_select (in libssh.so.4.10.5) ==372207== by 0x1FB638: ??? (in /usr/libexec/x2goclient) ==372207== by 0x1FF303: ??? (in /usr/libexec/x2goclient) ==372207== by 0x5C3AD20: ??? (in libQt5Core.so.5.15.15) ==372207== by ...start_thread / clone ==372207== Address 0x2a000005c8 is not stack'd, malloc'd or (recently) free'd ... ==372207== Process terminating with default action of signal 11 (SIGSEGV): dumping core (offsets 0x1FB638/0x1FF303 line up with channelLoop()/run() from the gdb trace above.) Note this crash happened in a forked child process running the x2goclient binary image itself (a separate valgrind ==PID== group, not ssh/nxproxy), consistent with x2goclient forking off a second SshMasterConnection worker for the jump-host hop. Full (redacted, de-duplicated) valgrind log is attached separately: x2go-valgrind.sanitized.log -------------------------------------------------------------------- POSSIBLE CONTRIBUTING FACTOR: SCP-SESSION RETRY LOOP -------------------------------------------------------------------- The valgrind run that caught the crash also logged 26,670 repeated attempts of: x2go-DEBUG-../src/sshmasterconnection.cpp:1850> Error initializing SCP session: Failed to open channel for scp in a tight loop, right before the crash. This looks like a separate, unbounded retry loop (worth its own bug report?) -- but it's also plausibly what's making the race easy to hit: rapid repeated channel creation/teardown is exactly the kind of hammering that would narrow the timing window needed to trigger a use-after-free like this. -------------------------------------------------------------------- PRIOR RELATED REPORTS -------------------------------------------------------------------- This isn't the first time SshMasterConnection's threading has produced crashes in this area -- see e.g.: #187 "Destroying SshMasterConnection off the main thread leads to a diagnostic from Qt debug libs" #1460 "Windows client crashes if Jumphost runs NetBSD 6 (probably race)" Both point at the same general area (channel/session lifecycle vs. thread boundaries) without a definitive fix landing, as far as I can tell from the bug history. -------------------------------------------------------------------- ATTACHMENT -------------------------------------------------------------------- x2go-valgrind.sanitized.log -- de-duplicated, redacted valgrind/ memcheck log (hostnames, usernames, an internal IP, and what looked like a live session cookie have been replaced with placeholders) --Philipp with Claude 5 Sonnet