From unknown Wed Jun 17 09:51:13 2026 MIME-Version: 1.0 X-Mailer: MIME-tools 5.507 (Entity 5.507) X-Loop: owner@bugs.x2go.org From: owner@bugs.x2go.org (X2Go Bug Tracking System) Subject: Bug#1313 closed by Mike Gabriel (Re: [X2Go-Dev] Bug#1313: Bug#1313: there is still a problem in getting a correct value for loadavgXX with loadchecker) Message-ID: References: <20181214150252.Horde.byNG7EXbaadzROu-QLl9djD@mail.das-netzwerkteam.de> X-X2go-PR-Message: they-closed 1313 X-X2go-PR-Package: x2gobroker-agent X-X2go-PR-Source: x2gobroker Date: Fri, 14 Dec 2018 15:05:04 +0000 Content-Type: multipart/mixed; boundary="----------=_1544799904-27214-0" This is a multi-part message in MIME format... ------------=_1544799904-27214-0 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 This is an automatic notification regarding your Bug report which was filed against the x2gobroker-agent package: #1313: there is still a problem in getting a correct value for loadavgXX wi= th loadchecker It has been closed by Mike Gabriel . Their explanation is attached below along with your original report. If this explanation is unsatisfactory and you have not received a better one in a separate message then please contact Mike Gabriel by replying to this email. --=20 1313: http://bugs.x2go.org/cgi-bin/bugreport.cgi?bug=3D1313 X2Go Bug Tracking System Contact owner@bugs.x2go.org with problems ------------=_1544799904-27214-0 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 1313) by bugs.x2go.org; 14 Dec 2018 15:03:08 +0000 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on ymir.das-netzwerkteam.de X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=BAYES_00,RDNS_NONE, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.2 Received: from localhost (localhost [127.0.0.1]) by ymir.das-netzwerkteam.de (Postfix) with ESMTP id E8E685DACB for <1313@bugs.x2go.org>; Fri, 14 Dec 2018 16:03:04 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at ymir.das-netzwerkteam.de Received: from ymir.das-netzwerkteam.de ([127.0.0.1]) by localhost (ymir.das-netzwerkteam.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4UZV4wI3m6zT for <1313@bugs.x2go.org>; Fri, 14 Dec 2018 16:02:58 +0100 (CET) Received: from fregna.das-netzwerkteam.de (unknown [IPv6:2a01:4f8:202:1381::1]) by ymir.das-netzwerkteam.de (Postfix) with ESMTPS id 7BB2C5DAEF for <1313@bugs.x2go.org>; Fri, 14 Dec 2018 16:02:58 +0100 (CET) Received: from grimnir.das-netzwerkteam.de (grimnir.das-netzwerkteam.de [IPv6:2a01:4f8:202:1381::105]) by fregna.das-netzwerkteam.de (Postfix) with ESMTPS id 653B561160; Fri, 14 Dec 2018 15:02:58 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by grimnir.das-netzwerkteam.de (Postfix) with ESMTP id 5ADC4C270C; Fri, 14 Dec 2018 16:02:58 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at grimnir.das-netzwerkteam.de Received: from grimnir.das-netzwerkteam.de ([127.0.0.1]) by localhost (grimnir.das-netzwerkteam.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rvmJYMhsGK3z; Fri, 14 Dec 2018 16:02:52 +0100 (CET) Received: from das-netzwerkteam.de (localhost [127.0.0.1]) by grimnir.das-netzwerkteam.de (Postfix) with ESMTPS id AB845C26F5; Fri, 14 Dec 2018 16:02:52 +0100 (CET) Received: from m-049.informatik.uni-kiel.de (m-049.informatik.uni-kiel.de [134.245.254.49]) by mail.das-netzwerkteam.de (Horde Framework) with HTTPS; Fri, 14 Dec 2018 15:02:52 +0000 Date: Fri, 14 Dec 2018 15:02:52 +0000 Message-ID: <20181214150252.Horde.byNG7EXbaadzROu-QLl9djD@mail.das-netzwerkteam.de> From: Mike Gabriel To: 1313@bugs.x2go.org Cc: Walid MOGHRABI Subject: Re: [X2Go-Dev] Bug#1313: Bug#1313: there is still a problem in getting a correct value for loadavgXX with loadchecker References: <883547417.4000531.1534156346103.JavaMail.root@servicemagic.eu> <1342096826.4000684.1534156440651.JavaMail.root@servicemagic.eu> <20180913131919.Horde.E_bg6JMBLAcV9jdu5upQ4C5@mail.das-netzwerkteam.de> In-Reply-To: <20180913131919.Horde.E_bg6JMBLAcV9jdu5upQ4C5@mail.das-netzwerkteam.de> User-Agent: Horde Application Framework 5 Accept-Language: de,en Organization: DAS-NETZWERKTEAM X-Originating-IP: 134.245.254.49 X-Remote-Browser: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0 Content-Type: multipart/signed; boundary="=_6-78xO-KqXvV_VjDhzMr_NN"; protocol="application/pgp-signature"; micalg=pgp-sha256 MIME-Version: 1.0 This message is in MIME format and has been PGP signed. --=_6-78xO-KqXvV_VjDhzMr_NN Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Control: close -1 On Do 13 Sep 2018 15:19:19 CEST, Mike Gabriel wrote: > Hi Walid, > > On Mo 13 Aug 2018 12:34:00 CEST, Walid MOGHRABI wrote: > >> package: x2gobroker-agent >> version: 0.0.4.0-0~1038~ubuntu16.04.1 >> priority: bug >> >> I don't have a "0" value anymore since latest fixes so the=20=20 >>=20loadchecker process don't crash anymore but still, there is=20=20 >>=20something strange. >> Here is a fragment of my loadchecker logs from this morning. >> Just to give you the context, I have 22 servers which are all=20=20 >>=20automaticaly started at 6 AM (wake on lan) and they are absolutely=20= =20 >>=20the same (blade servers with same CPU, memory amount, bios version,=20= =20 >>=20...). >> I checked our monitoring to see if users were correctly distributed=20= =20 >>=20over the farm and at 7:30AM, I had about 7 or 8 users connected but=20= =20 >>=204 of them were on tce-server-21 where I should have had 1 user on 8=20= =20 >>=20servers. > > Have you seen this issues more often? Does it hop from one server to=20= =20 >=20another or occur on more than one server at a time? > >> Here is the loadchecker log fragment : >> >> root@tce-manager-01 [~] # grep -B 1 'loadavgXX:1;'=20=20 >>=20/var/log/x2gobroker/loadchecker.log >> ... >> 2018-07-24 07:15:01,200 - loadchecker - INFO - Executing agent=20=20 >>=20command on remote host tce-server-21 (10.50.0.221): sh -c=20=20 >>=20'/usr/lib/x2go/x2gobroker-agent foo checkload' >> 2018-07-24 07:15:01,622 - loadchecker - INFO - Broker agent=20=20 >>=20answered: OK; loadavgXX:1; memAvail:23684; myMemAvail:23810;=20=20 >>=20numCPU:16; typeCPU:2400; >> -- >> 2018-07-24 07:17:50,354 - loadchecker - INFO - Executing agent=20=20 >>=20command on remote host tce-server-21 (10.50.0.221): sh -c=20=20 >>=20'/usr/lib/x2go/x2gobroker-agent foo checkload' >> 2018-07-24 07:17:50,779 - loadchecker - INFO - Broker agent=20=20 >>=20answered: OK; loadavgXX:1; memAvail:23686; myMemAvail:23812;=20=20 >>=20numCPU:16; typeCPU:2400; >> -- >> 2018-07-24 07:20:32,550 - loadchecker - INFO - Executing agent=20=20 >>=20command on remote host tce-server-21 (10.50.0.221): sh -c=20=20 >>=20'/usr/lib/x2go/x2gobroker-agent foo checkload' >> 2018-07-24 07:20:32,964 - loadchecker - INFO - Broker agent=20=20 >>=20answered: OK; loadavgXX:1; memAvail:23683; myMemAvail:23809;=20=20 >>=20numCPU:16; typeCPU:2400; >> -- >> 2018-07-24 07:23:21,610 - loadchecker - INFO - Executing agent=20=20 >>=20command on remote host tce-server-21 (10.50.0.221): sh -c=20=20 >>=20'/usr/lib/x2go/x2gobroker-agent foo checkload' >> 2018-07-24 07:23:22,034 - loadchecker - INFO - Broker agent=20=20 >>=20answered: OK; loadavgXX:1; memAvail:23685; myMemAvail:23811;=20=20 >>=20numCPU:16; typeCPU:2400; >> -- >> 2018-07-24 07:26:03,872 - loadchecker - INFO - Executing agent=20=20 >>=20command on remote host tce-server-21 (10.50.0.221): sh -c=20=20 >>=20'/usr/lib/x2go/x2gobroker-agent foo checkload' >> 2018-07-24 07:26:04,286 - loadchecker - INFO - Broker agent=20=20 >>=20answered: OK; loadavgXX:1; memAvail:23684; myMemAvail:23809;=20=20 >>=20numCPU:16; typeCPU:2400; >> -- >> 2018-07-24 07:28:52,917 - loadchecker - INFO - Executing agent=20=20 >>=20command on remote host tce-server-21 (10.50.0.221): sh -c=20=20 >>=20'/usr/lib/x2go/x2gobroker-agent foo checkload' >> 2018-07-24 07:28:53,338 - loadchecker - INFO - Broker agent=20=20 >>=20answered: OK; loadavgXX:1; memAvail:23684; myMemAvail:23809;=20=20 >>=20numCPU:16; typeCPU:2400; >> -- >> 2018-07-24 07:31:35,252 - loadchecker - INFO - Executing agent=20=20 >>=20command on remote host tce-server-21 (10.50.0.221): sh -c=20=20 >>=20'/usr/lib/x2go/x2gobroker-agent foo checkload' >> 2018-07-24 07:31:35,670 - loadchecker - INFO - Broker agent=20=20 >>=20answered: OK; loadavgXX:1; memAvail:23685; myMemAvail:23811;=20=20 >>=20numCPU:16; typeCPU:2400; >> -- >> 2018-07-24 07:34:24,424 - loadchecker - INFO - Executing agent=20=20 >>=20command on remote host tce-server-21 (10.50.0.221): sh -c=20=20 >>=20'/usr/lib/x2go/x2gobroker-agent foo checkload' >> 2018-07-24 07:34:24,842 - loadchecker - INFO - Broker agent=20=20 >>=20answered: OK; loadavgXX:1; memAvail:23683; myMemAvail:23809;=20=20 >>=20numCPU:16; typeCPU:2400; > > The log message "Broker agent answered:" comes directly from X2Go=20=20 >=20Broker Agent. It is basically its raw output. > > This means, that the flaw must be in x2gobroker-agent.pl on the=20=20 >=20remote X2Go Server. Or that the loadchecker stops querying the=20=20 >=20broker agent and re-uses old data. > > Looking at x2gobroker-agent.pl: If we focus on the loadavgXX for=20=20 >=20now, we come to the conclusion, that the load was really "0" or it=20= =20 >=20was negative (both gives us a loadavgXX value of "1". The value=20=20 >=20should normally be greater (system load of 1.0 brings a loadavgXX of=20= =20 >=20100). > > Looking at x2gobroker.agent.py: As the values always change=20=20 >=20slightly, we can't say that Python provides us the same return=20=20 >=20result string all the time. The query to the broker agent must have=20= =20 >=20happened. > > We need to do more debugging if this issue reoccurs: > > * run '/usr/lib/x2go/x2gobroker-agent foo checkload' on the=20=20 >=20affected X2Go Server > and see if the reported values match with what the load checker sees. > > * check if it is reoccuring on the same X2Go Server > > * if /usr/lib/x2go/x2gobroker-agent returns a load of zero, > look at /proc/loadavg > > * and /proc/sys/vm/min_free_kbytes, > /proc/meminfo > /proc/cpuinfo > > ... and report all back here... > >> As you can see, there is only 1 server with a loadavgXX =3D 1 (which=20= =20 >>=20means that in fact, we got a zero value from the broker agent). >> This is not normal, at 7:34, there were 4 users already connected=20=20 >>=20to this server and most of my other servers were empty. > >> Restarting x2gobroker-loadchecker service fixed the issue. > > Considering the above analysis that the issue must come from=20=20 >=20x2gobroker-agent.pl, a restart of the loadchecker can in theory not=20= =20 >=20solve such an issue. > > Can you see the x2gobroker-agent.pl process appear and disappear in=20=20 >=20the process list on the remote X2Go Server? Or does it stay open,=20=20 >=20even zombied? > >> I think there is a problem in retrieving this informations ... even=20= =20 >>=20memAvail seem strange on this server to me ... with 4 connected=20=20 >>=20users, it should have been lower than that. > > Hmmm... Ok... Maybe the wrong server got tested? Two identical IPs=20=20 >=20on the subnet? > >> I also think the number of connected users should be taken into=20=20 >>=20account when calculating the load factor (maybe this is already the=20= =20 >>=20case, not sure about that). > > Yes, we take the number of sessions into account. But that is not=20=20 >=20provided by the broker agent, but is available in the X2Go Server=20=20 >=20database and queried from there. > > Mike Request from Walid on IRC. Not an issue anymore. Thus, closing... Mike --=20 DAS-NETZWERKTEAM mike=20gabriel, herweg 7, 24357 fleckeby mobile: +49 (1520) 1976 148 landline: +49 (4354) 8390 139 GnuPG Fingerprint: 9BFB AEE8 6C0A A5FF BF22 0782 9AF4 6B30 2577 1B31 mail: mike.gabriel@das-netzwerkteam.de, http://das-netzwerkteam.de --=_6-78xO-KqXvV_VjDhzMr_NN Content-Type: application/pgp-signature Content-Description: Digitale PGP-Signatur Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIzBAABCAAdFiEEm/uu6GwKpf+/IgeCmvRrMCV3GzEFAlwTxhwACgkQmvRrMCV3 GzGzIw//e1ghRfsUTamOI1luaPrAn3B1rr3XjZhosT76UOay7w6cRmnu5MmAUWFh rk8JiB/ZSx8ctPSnRJH5jlA0MxPOdpZx9aQDPcL7tg3b0Ty+kNvBy9IpCq+pv2Ek IsuY2Ta7w+t7k9lvOiFszE0l5OR6PXoqS7l4m5NQYxrge49ABOJYO+Mwi0AUfwuX xDAO7Is/7vzY9FdsHkGpRCAuOkRN9NbrhLuBnzukmHDTSf+xJ0Q8OoGgxLISIFOw qZT+IhGtNoJ/BuC2w9m/nEhKVoUvMHy/NHNup40Wf8pDJn/NIKyYXrT2zwd9v0MS gilLQ4jgJcqJgBYlavP/dJ1yT2O3xLQnHSlaLO3J/TJpLmfHXecGyezn/kfDtwOY zaX8eGEP7WpxRgQYHUKdCDjLG7IUQZh6ZPPw98/KXZX+KJoFO8MpUMBShC3eX9/M VuY+Xqi/S5Lx9YN7HdQY8cxDPNKF8PBHyYHkXc93WhtpTvZMJDgXbMsO86rqiIZA xjiAT/ubUnoM4Yu9JkTSDje36Y0gX5I7zXS/nVCLZjwcZeNBcJ0/1trNHe0Qn45E veP12UrolDHbtLbxitpz12ExjUvH0SdT/j31E0BRRfHGV+JXsXZuYQvZXOQXQmKJ UIipivwaTswmN6RnDWmqaku730bMIJHAXm4OIpfLvaelPgT/iOg= =VZG/ -----END PGP SIGNATURE----- --=_6-78xO-KqXvV_VjDhzMr_NN-- ------------=_1544799904-27214-0 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by bugs.x2go.org; 13 Aug 2018 10:40:16 +0000 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on ymir.das-netzwerkteam.de X-Spam-Level: X-Spam-Status: No, score=0.8 required=3.0 tests=BAYES_50 autolearn=ham autolearn_force=no version=3.4.1 Received: from localhost (localhost [127.0.0.1]) by ymir.das-netzwerkteam.de (Postfix) with ESMTP id C6EBE5DAEB for ; Mon, 13 Aug 2018 12:39:57 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at ymir.das-netzwerkteam.de Received: from ymir.das-netzwerkteam.de ([127.0.0.1]) by localhost (ymir.das-netzwerkteam.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QqkBs4jkNaK6 for ; Mon, 13 Aug 2018 12:39:50 +0200 (CEST) X-Greylist: delayed 599 seconds by postgrey-1.35 at ymir.das-netzwerkteam.de; Mon, 13 Aug 2018 12:39:49 CEST Received: from zm-01.servicemagic.eu (zm-01.servicemagic.eu [176.31.236.17]) by ymir.das-netzwerkteam.de (Postfix) with ESMTPS id DEF6A5DAE9 for ; Mon, 13 Aug 2018 12:39:49 +0200 (CEST) Received: from localhost (localhost.localdomain [127.0.0.1]) by zm-01.servicemagic.eu (Postfix) with ESMTP id 5116B807BF1BD for ; Mon, 13 Aug 2018 12:34:01 +0200 (CEST) X-Amavis-Modified: Mail body modified (using disclaimer) - zm-01.servicemagic.eu X-Virus-Scanned: amavisd-new at servicemagic.eu Received: from zm-01.servicemagic.eu ([127.0.0.1]) by localhost (zm-01.servicemagic.eu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 48Z1bUYN2xEi for ; Mon, 13 Aug 2018 12:34:00 +0200 (CEST) Received: from zm-01.servicemagic.eu (localhost.localdomain [127.0.0.1]) by zm-01.servicemagic.eu (Postfix) with ESMTP id A82A5806980E2 for ; Mon, 13 Aug 2018 12:34:00 +0200 (CEST) Date: Mon, 13 Aug 2018 12:34:00 +0200 (CEST) From: Walid MOGHRABI To: submit@bugs.x2go.org Message-ID: <1342096826.4000684.1534156440651.JavaMail.root@servicemagic.eu> In-Reply-To: <883547417.4000531.1534156346103.JavaMail.root@servicemagic.eu> Subject: there is still a problem in getting a correct value for loadavgXX with loadchecker MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [195.200.167.70] X-Mailer: Zimbra 7.2.0_GA_2669 (ZimbraWebClient - GC68 (Linux)/7.2.0_GA_2669) package: x2gobroker-agent version: 0.0.4.0-0~1038~ubuntu16.04.1 priority: bug I don't have a "0" value anymore since latest fixes so the loadchecker process don't crash anymore but still, there is something strange. Here is a fragment of my loadchecker logs from this morning. Just to give you the context, I have 22 servers which are all automaticaly started at 6 AM (wake on lan) and they are absolutely the same (blade servers with same CPU, memory amount, bios version, ...). I checked our monitoring to see if users were correctly distributed over the farm and at 7:30AM, I had about 7 or 8 users connected but 4 of them were on tce-server-21 where I should have had 1 user on 8 servers. Here is the loadchecker log fragment : root@tce-manager-01 [~] # grep -B 1 'loadavgXX:1;' /var/log/x2gobroker/loadchecker.log ... 2018-07-24 07:15:01,200 - loadchecker - INFO - Executing agent command on remote host tce-server-21 (10.50.0.221): sh -c '/usr/lib/x2go/x2gobroker-agent foo checkload' 2018-07-24 07:15:01,622 - loadchecker - INFO - Broker agent answered: OK; loadavgXX:1; memAvail:23684; myMemAvail:23810; numCPU:16; typeCPU:2400; -- 2018-07-24 07:17:50,354 - loadchecker - INFO - Executing agent command on remote host tce-server-21 (10.50.0.221): sh -c '/usr/lib/x2go/x2gobroker-agent foo checkload' 2018-07-24 07:17:50,779 - loadchecker - INFO - Broker agent answered: OK; loadavgXX:1; memAvail:23686; myMemAvail:23812; numCPU:16; typeCPU:2400; -- 2018-07-24 07:20:32,550 - loadchecker - INFO - Executing agent command on remote host tce-server-21 (10.50.0.221): sh -c '/usr/lib/x2go/x2gobroker-agent foo checkload' 2018-07-24 07:20:32,964 - loadchecker - INFO - Broker agent answered: OK; loadavgXX:1; memAvail:23683; myMemAvail:23809; numCPU:16; typeCPU:2400; -- 2018-07-24 07:23:21,610 - loadchecker - INFO - Executing agent command on remote host tce-server-21 (10.50.0.221): sh -c '/usr/lib/x2go/x2gobroker-agent foo checkload' 2018-07-24 07:23:22,034 - loadchecker - INFO - Broker agent answered: OK; loadavgXX:1; memAvail:23685; myMemAvail:23811; numCPU:16; typeCPU:2400; -- 2018-07-24 07:26:03,872 - loadchecker - INFO - Executing agent command on remote host tce-server-21 (10.50.0.221): sh -c '/usr/lib/x2go/x2gobroker-agent foo checkload' 2018-07-24 07:26:04,286 - loadchecker - INFO - Broker agent answered: OK; loadavgXX:1; memAvail:23684; myMemAvail:23809; numCPU:16; typeCPU:2400; -- 2018-07-24 07:28:52,917 - loadchecker - INFO - Executing agent command on remote host tce-server-21 (10.50.0.221): sh -c '/usr/lib/x2go/x2gobroker-agent foo checkload' 2018-07-24 07:28:53,338 - loadchecker - INFO - Broker agent answered: OK; loadavgXX:1; memAvail:23684; myMemAvail:23809; numCPU:16; typeCPU:2400; -- 2018-07-24 07:31:35,252 - loadchecker - INFO - Executing agent command on remote host tce-server-21 (10.50.0.221): sh -c '/usr/lib/x2go/x2gobroker-agent foo checkload' 2018-07-24 07:31:35,670 - loadchecker - INFO - Broker agent answered: OK; loadavgXX:1; memAvail:23685; myMemAvail:23811; numCPU:16; typeCPU:2400; -- 2018-07-24 07:34:24,424 - loadchecker - INFO - Executing agent command on remote host tce-server-21 (10.50.0.221): sh -c '/usr/lib/x2go/x2gobroker-agent foo checkload' 2018-07-24 07:34:24,842 - loadchecker - INFO - Broker agent answered: OK; loadavgXX:1; memAvail:23683; myMemAvail:23809; numCPU:16; typeCPU:2400; As you can see, there is only 1 server with a loadavgXX = 1 (which means that in fact, we got a zero value from the broker agent). This is not normal, at 7:34, there were 4 users already connected to this server and most of my other servers were empty. Restarting x2gobroker-loadchecker service fixed the issue. I think there is a problem in retrieving this informations ... even memAvail seem strange on this server to me ... with 4 connected users, it should have been lower than that. I also think the number of connected users should be taken into account when calculating the load factor (maybe this is already the case, not sure about that). --- DISCLAIMER: This e-mail is private and confidential and may contain proprietary or legally privileged information. It is for the intended recipient only. If you have received this email in error, please notify the author by replying to it and then destroy it. If you are not the intended recipient you must not use, disclose, distribute, copy, print or rely on this e-mail or any attachment. Thank you ------------=_1544799904-27214-0--