Subject: Bug#674: keycode -> keycode translation harmful (makes configuration complex)

From: Robert Siemer <[email-masked]>
Date: Fri, 14 Nov 2014 01:39:31 +0100

Package: x2goserver
Version: 4.0.1.18

This bug report is about keyboard configuration issues: it is very 
challenging to have a custom keyboard configuration over x2go, while it 
is running perfectly on the local machine.

With a Linux X2Go client the keyboard configuration could be entirely 
transparent, like an ssh-session with X11-forwarding is. But it isn’t.

Background: In X11, the keycodes (numbers, e.g. 98) are sent by the 
X11-keyboard driver to the X11-core. These numbers also get reported to 
the applications. The applications in turn ask the X-server what they 
should make out of those keycodes (e.g. 'a', called “symbols”).

The "final" problem I’m describing here is independent of the ‘keyboard 
settings’ of the x2go client. Nevertheless I describe here only the 
effects as seen with a configuration of "leave keyboard settings alone 
and don’t do anything about it”.

With this kind of configuration the X-clients on the X2Go-server get the 
same answer to for keycodes → symbols requests as they get local, but 
they get it from the X2Go-server (nxagent or something). It seems that 
these settings get copied over to the X2Go-server and are left alone. I 
call those settings the “XKB configuration”.

A different X2Go-client keyboard configuration (e.g. auto), by the way, 
would only reconfigure that XKB configuration, which can also be done by 
hand with xkbcomp or setxkbmap.

—But the keycodes coming from the X2Go-client do not get sent over as 
is. The are “mangled” or “translated” in between, especially on any 
Linux client with evdev input device drivers.

There is no X11 way to reconfigure those keycodes, as they normally come 
from the driver and are fixed for each key.

Three questions came to my mind:

1) Why are the keycodes translated?

2) What logic does this translation follow?

3) How can I influence it?

To the first question: it might be an artifact which is needed for other 
platforms.

Second question: It turned out that the keycodes received by the 
applications on the X2Go-server are exactly those that an old pre-evdev 
Linux machine would generate. Those codes are written down and shipped 
with Xorg (X-Server) in the file keycodes/xfree86. The file used these 
days with the evdev driver is keycodes/evdev, where those known values 
are recorded.

After changing my XKB configuration, completely “faking” a key on my 
keyboard to be something else, the X2Go server (or nxagent or whatever) 
still knew how to translate the keycode of it.

After some trial and error I found out that the translation instance 
knowns about my original keyboard with the help of the some X property 
on the root window:

$ xprop -root _XKB_RULES_NAMES
_XKB_RULES_NAMES(STRING) = "evdev", "pc104", "us", "altgr-intl", ""

The first string, here "evdev" dicts the rules file to use for XKB 
configuration changes. Note: this X property is not part of the XKB 
configuration, but setxkbmap read and records those five strings, which 
guide XKB configuration composition so to say. (Those five strings are 
rules, model, layout, variant and options, and they only guide which XKB 
configuration pieces to load. Those pieces are named keycodes, types, 
compat, symbols and geometry.)

I don’t know if X2Go actually reads and parses the rules file and the 
XKB snippets or if the mappings are hardcoded, but changing the first 
string in _XKB_RULES_NAMES (which dictates which rules file to use) 
influences how the translation is done.

the following rules files dictate which keycodes to use:

evdev → evdev
xfree86 → xfree86
base → xfree86

When starting or resuming an X2Go session, the _XKB_RULES_NAMES property 
is initially set to "xfree86", "null", "null", ... in the nxagent. It 
really looks like that this X11 intermediary wants to fake “xfree86 
hardware” (i.e. old PC hardware + kernel). As my current setup has no 
“xfree86 settings” nowhere, I assume that this is fixed.

In effect: changing _XKB_RULES_NAMES to something where the rules is set 
to “xfree86” or “base”, no translation takes place (or the translation 
is 1:1). This has to be configured before X2Go session resume or 
startup. Note that this does not change the XKB configuration (it would 
only influence the next setxkbmap run).

Once no real keycode translation takes place, no XKB keyboard 
(re-)configuration is necessary as the copied local setup is probably 
exactly what you want.

---

From: Mike Gabriel <[email-masked]>
Date: Fri, 14 Nov 2014 04:49:56 +0000

Hi Robert,

On  Fr 14 Nov 2014 01:39:31 CET, Robert Siemer wrote:

> challenging to have a custom keyboard configuration over x2go, while  
> it is running perfectly on the local machine.
>
> With a Linux X2Go client the keyboard configuration could be  
> entirely transparent, like an ssh-session with X11-forwarding is.  
> But it isn’t.
>
> X11-keyboard driver to the X11-core. These numbers also get reported  
> to the applications. The applications in turn ask the X-server what  
> they should make out of those keycodes (e.g. 'a', called “symbols”).
>
> The "final" problem I’m describing here is independent of the  
> ‘keyboard settings’ of the x2go client. Nevertheless I describe here  
> only the effects as seen with a configuration of "leave keyboard  
> settings alone and don’t do anything about it”.
>
> With this kind of configuration the X-clients on the X2Go-server get  
> the same answer to for keycodes → symbols requests as they get  
> local, but they get it from the X2Go-server (nxagent or something).  
> It seems that these settings get copied over to the X2Go-server and  
> are left alone. I call those settings the “XKB configuration”.
>
> A different X2Go-client keyboard configuration (e.g. auto), by the  
> way, would only reconfigure that XKB configuration, which can also  
> be done by hand with xkbcomp or setxkbmap.
>
> —But the keycodes coming from the X2Go-client do not get sent over  
> as is. The are “mangled” or “translated” in between, especially on  
> any Linux client with evdev input device drivers.
>
> There is no X11 way to reconfigure those keycodes, as they normally  
> come from the driver and are fixed for each key.
>
> To the first question: it might be an artifact which is needed for  
> other platforms.
>
> applications on the X2Go-server are exactly those that an old  
> pre-evdev Linux machine would generate. Those codes are written down  
> and shipped with Xorg (X-Server) in the file keycodes/xfree86. The  
> file used these days with the evdev driver is keycodes/evdev, where  
> those known values are recorded.
>
> keyboard to be something else, the X2Go server (or nxagent or  
> whatever) still knew how to translate the keycode of it.
>
> knowns about my original keyboard with the help of the some X  
> property on the root window:
>
> configuration, but setxkbmap read and records those five strings,  
> which guide XKB configuration composition so to say. (Those five  
> strings are rules, model, layout, variant and options, and they only  
> guide which XKB configuration pieces to load. Those pieces are named  
> keycodes, types, compat, symbols and geometry.)
>
> I don’t know if X2Go actually reads and parses the rules file and  
> the XKB snippets or if the mappings are hardcoded, but changing the  
> first string in _XKB_RULES_NAMES (which dictates which rules file to  
> use) influences how the translation is done.
>
> When starting or resuming an X2Go session, the _XKB_RULES_NAMES  
> property is initially set to "xfree86", "null", "null", ... in the  
> nxagent. It really looks like that this X11 intermediary wants to  
> fake “xfree86 hardware” (i.e. old PC hardware + kernel). As my  
> current setup has no “xfree86 settings” nowhere, I assume that this  
> is fixed.
>
> In effect: changing _XKB_RULES_NAMES to something where the rules is  
> set to “xfree86” or “base”, no translation takes place (or the  
> translation is 1:1). This has to be configured before X2Go session  
> resume or startup. Note that this does not change the XKB  
> configuration (it would only influence the next setxkbmap run).
>
> (re-)configuration is necessary as the copied local setup is  
> probably exactly what you want.

I guess this is the best analysis on what happens in NX around  
keyboard stuff, I have ever read. Thanks for that very good tutorial.

Now about fixing the bug.

I really think we should get evdev support into NX. What do you think?

But for now... so that I get a deeper feeling for this... Could you  
provide a recipe for setting the correct / transparent keyboard setup  
in the X2Go Session?

A sequence of commands on the client-side, a sequence of commands on  
the server-side, something like that...

THANKS!
Mike

-- 

DAS-NETZWERKTEAM
mike gabriel, herweg 7, 24357 fleckeby
fon: +49 (1520) 1976 148

GnuPG Key ID 0x25771B31
mail: [email-masked], http://das-netzwerkteam.de

freeBusy:
https://mail.das-netzwerkteam.de/freebusy/m.gabriel%40das-netzwerkteam.de.xfb

---

From: Ulrich Sibiller <[email-masked]>
Date: Fri, 14 Nov 2014 13:36:52 +0100

> keyboard stuff, I have ever read. Thanks for that very good tutorial.

That's what I was thinking when I read it, too ;-) But I'll have to read it again to fully get into it.

> the server-side, something like that...

Man setxkbmap states this:

-------

If you have an Xserver and a client shell running on different computers and XKB
configuration files on those machines are different you can get problems specifying a keyboard map
by model, layout, options
names. This is because setxkbcomp converts these names to names of XKB configuration
files according to files that are on the client side computer, then it sends the file names to the
server where the xkbcomp
has to compose a complete keyboard map using files which the server has. Thus if the sets
of files differ significantly the names that the setxkbmap generates can be unacceptable on the
server side. You can
solve this problem by running the xkbcomp on the client side too. With the -print option
setxkbmap just prints the file names in an appropriate format to its stdout and this output can be
piped directly to the
xkbcomp input. For example, the command

setxkbmap us -print | xkbcomp - $DISPLAY

makes both steps run on the same (client) machine and loads a keyboard map into the server.

-------

Wouldn't this bypass all this setxkbmap names and files altogether?

Uli
_______________________________________________
x2go-dev mailing list
[email-masked]
http://lists.x2go.org/listinfo/x2go-dev

---

From: Robert Siemer <[email-masked]>
Date: Sun, 07 Dec 2014 05:07:14 +0100

> I really think we should get evdev support into NX. What do you think?

No, for keyboard configuration no special evdev support is needed.

 > provide a recipe for setting the correct / transparent keyboard
 > setup in the X2Go Session?

For the commands see below. – At the moment I remove _XKB_RULES_NAMES 
from the root window property before starting/resuming x2go.

 > Wouldn't [running xkbcomp on the client] bypass all this setxkbmap
 > names and files altogether?

I do already run xkbcomp on the client (and local as well) already, but 
it has no influence on keycode translation. – I’m not sure why you 
mentioned it...

I believe I know the reasoning behind this keycode translation: in a 
heterogeneous environment, where you resume an x2go session from 
different workstations, there exists only one way to represent the 
different keyboards as an unchanging virtual keyboard. Yes: with keycode 
translation.

The alternative would be to reconfigure XKB on each session resume, 
which I believe is perfectly reasonable. – This is what I (would) do.

Allow me to draw a little ascii diagram here (look at it with fixed 
width font):

local Xserver    local Xclients    x2go Xserv    x2go Xclients

[1]
|==================xterm
|==================chromium
|==================xlsclients [3]
|==================setxkbmap
|==================xkbcomp [7]
|                                   [2]
|------------------------------------|==============xterm
|------------------------------------|==============gnome-do [6]
|------------------------------------|==============xlsclients [4]
|------------------------------------|==============setxkbmap
|------------------------------------|==============xkbcomp [8]

This is about the X11 protocol level view of an x2go setup. There are 
two Xservers running: the local one [1] and that x2go Xserver (that is 
called nxagent?) [2]. Both are drawn as lines made of |

The x2go Xserver [2] represents its clients to the real local Xserver 
[1] as X11 clients, but draws them in a different way (still X11 
protocol, though).

An xlsclients [4] executed on the x2go server will be run against [2] 
and show only the x2go Xclients. On the local Xserver [1] an execution 
of xlsclients [3] lists all Xclients: the local ones and the remote ones.

Unfortunately the x2go Xserver [2] does not relay all and everything to 
the local Xserver [1]. Examples: the X properties of the root window 
(some kind of “global” storage for X clients) exist twice (in [1] and 
[2]) and can differ. Also global hotkeys (“passive keygrabs”) of the 
x2go Xclients will be stored in [2] and have no influence on [1]. The 
launcher gnome-do [5] registers a hotkey: it will only be delivered to 
[5] if the local Xserver [1] has no local Xclient waiting for it _and_ 
any of the x2go client windows is in focus. Otherwise [2] will not
even see the key event.

Back to XKB. As far as I could observe, the XKB setup consists of an XKB 
configuration (which can be read and written with xkbcomp [7][8]) and 
the root window property _XKB_RULES_NAMES (can be read with xprop and 
setxkbmap and is written as a side effect of setxkbmap). Note that both 
things exist twice, stored in [1] and [2].

Command line examples:

# dump current XKB configuration into one file
$ xkbcomp $DISPLAY full.xkb

# load full single-file XKB configuration
$ xkbcomp full.xkb $DISPLAY

# load from multiple files (xkbcomp will read more than one file):
# main keymap file triggers inclusion of files from system X11 files
$ xkbcomp keymap.xkb $DISPLAY

# example keymap.xkb file
xkb_keymap {
	xkb_keycodes  { include "evdev+aliases(qwerty)"	};
	xkb_types     { include "complete+my"	};
	xkb_compat    { include "complete"	};
	xkb_symbols   { include "pc+inet(evdev)+eurosign(e)"	};
	xkb_geometry  { include "pc(pc104)"	};
};

# load from multiple files and have those searched for in your own tree
# otherwise they will be looked for in XKB root /usr/share/X11/xkb
$ xkbcomp -I/home/blabla/xkb mykeymap.xkb $DISPLAY

# read _XKB_RULES_NAMES
$ xprop -root _XKB_RULES_NAMES
_XKB_RULES_NAMES(STRING) = "evdev", "pc104", "us,us", "altgr-intl,", 
"eurosign:e,caps:backspace,compose:rctrl"

# interpreted version of the above
$ setxkbmap -query
rules:      evdev
model:      pc104
layout:     us,us
variant:    altgr-intl,
options:    eurosign:e,caps:backspace,compose:rctrl

# delete _XKB_RULES_NAMES property (and then read it)
$ xprop -root -remove _XKB_RULES_NAMES
$ xprop -root _XKB_RULES_NAMES
_XKB_RULES_NAMES:  not found.
$ setxkbmap -query
Couldn't interpret _XKB_RULES_NAMES property
Use defaults: rules - 'base' model - 'pc105' layout - 'us'
rules:      base
model:      pc105
layout:     us

# set _XKB_RULES_NAMES only(!)
# I wrote a tool for that (xprop can’t set STRING arrays)
# https://github.com/siemer/xproperty
$ ./xproperty.py _XKB_RULES_NAMES evdev pc104 us altgr-intl
<class 'Xlib.protocol.request.QueryExtension'>
_XKB_RULES_NAMES evdev pc104 us altgr-intl

# let setxkbmap read this property, override a part of it
# and ask which keymap it would load
$ setxkbmap -layout de -print
xkb_keymap {
	xkb_keycodes  { include "evdev+aliases(qwertz)"	};
	xkb_types     { include "complete"	};
	xkb_compat    { include "complete"	};
	xkb_symbols   { include "pc+de+inet(evdev)"	};
	xkb_geometry  { include "pc(pc104)"	};
};

# without “-print” setxkbmap will write the updated property _and_
# instruct the X11 server to load the resulting configuration
# with the help of xkbcomp
# → setxkbmap can’t just update the property alone

On x2go session startup, the x2go Xserver [2] does a superficial 
investigation of what keyboard is used on the local Xserver [1] and sets 
up an incomplete keycode translation to a keyboard known as “xfree86”. 
It further creates an imperfect copy of the XKB configuration from [1] 
to [2]. The _XKB_RULES_NAMES property on [2] is set to rules=xfree86, 
model = pc???, layout = us(?) [I forgot the details].

Why is it a superficial investigation? – It takes the rules name from 
_XKB_RULES_NAMES and deducts the resulting keycodes directly from that 
(I believe). Even though in a common setup rules name “evdev” results in 
one specific set of keycodes and rules name “xfree86” in another 
specific set of keycodes, this is not a must.

Why is the keycode translation incomplete? – As far as I observed it, 
“unusual” or unknown keycodes are not translated and passed as-is.

Why is the XKB configuration copy imperfect? – I did a XKB dump on the 
local Xserver [1] and the x2go Xserver [2] after starting a new session. 
Keys with four levels of symbols turned into two-level-keysyms. – 
_Loading_ (i.e. xkbcomp/setxkbmap) a four-level configuration is no 
problem, though.

Note: the imperfect copy of the XKB configuration is almost always 
useless, because the copied configuration does not match the new 
(translated) keycodes!

Resuming an x2go session triggers the same superficial keyboard 
investigation, the translation kicks in again, but the XKB configuration 
is not touched (i.e. is not copied from [1] to [2]).

Deleting the _XKB_RULES_NAMES property before session startup/resume or 
setting the first part (that is the “rules” part) to some unknown string 
results in untranslated keycodes.

Please pardon me for repeating myself in some instances.

Robert

---

From: Ulrich Sibiller <[email-masked]>
Date: Mon, 8 Dec 2014 00:07:07 +0100

On Sun, Dec 7, 2014 at 5:07 AM, Robert Siemer <[email-masked]> wrote:
> investigation of what keyboard is used on the local Xserver [1] and sets up
> an incomplete keycode translation to a keyboard known as “xfree86”. It
> further creates an imperfect copy of the XKB configuration from [1] to [2].
> The _XKB_RULES_NAMES property on [2] is set to rules=xfree86, model = pc???,
> layout = us(?) [I forgot the details].
>
> _XKB_RULES_NAMES and deducts the resulting keycodes directly from that (I
> believe). Even though in a common setup rules name “evdev” results in one
> specific set of keycodes and rules name “xfree86” in another specific set of
> keycodes, this is not a must.
>
> Why is the XKB configuration copy imperfect? – I did a XKB dump on the local
> Xserver [1] and the x2go Xserver [2] after starting a new session. Keys with
> four levels of symbols turned into two-level-keysyms. – _Loading_ (i.e.
> xkbcomp/setxkbmap) a four-level configuration is no problem, though.
>
> Note: the imperfect copy of the XKB configuration is almost always useless,
> because the copied configuration does not match the new (translated)
> keycodes!
>
> investigation, the translation kicks in again, but the XKB configuration is
> not touched (i.e. is not copied from [1] to [2]).
>

I am not sure if I am getting this right. Will using xkbcomp on the
local server, pushing the result to the nxagent and loading it there
result in a clean keyboard setup or not? Do we need to disable setting
up the translation within the nx code to make that work?

Uli

BTW: You can find the code for the keycode translation in
nx-x11/programs/Xserver/hw/nxagent/Keyboard.c. It seems some kind of
translation from evdev to pc105 is done. You can also see that if
there is no "xfree86" rules file the one called "xorg" is used instead
(if available, of course).

---

From: Robert Siemer <[email-masked]>
Date: Sat, 13 Dec 2014 01:45:02 +0100

On 08/12/14 00:07, Ulrich Sibiller wrote:
> result in a clean keyboard setup or not?

That depends on whether the keycode translation is on or off.

If the translation it is not in effect, then yes, you get a setup like 
you have without x2go.

If the translation is on, you get a skewed setup.

Note that the x2go (with the setting of “do nothing”) does a non-perfect 
copy with the same effect: if the translation is effective, the keyboard 
is skewed and you are force to load at least “something” to fix XKB. If 
the translation is off, you “only” need to load XKB configuration if 
that non-perfect copy annoys you...

Btw, the forme is exactly what I do: dump local XKB configuration, make 
sure I don’t get a translation in x2go and load it on the other side.

> nx-x11/programs/Xserver/hw/nxagent/Keyboard.c.

Thanks for the pointer! I had a look at it. As I suspected: the 
translation is “bad”. Sometimes two different keycodes are translated to 
the same keycode. How are you supposed to tell them apart on the remote 
side? You can’t.

> It seems some kind of translation from evdev to pc105 is done.

The naming is a little funny: the keycodes generated by the kernel 
before evdev were not called like that, were they?

 > Do we need to disable setting

That would be great as a first step.

---

From: Ulrich Sibiller <[email-masked]>
Date: Tue, 16 Dec 2014 10:40:06 +0100

On Sat, Dec 13, 2014 at 1:45 AM, Robert Siemer <[email-masked]> wrote:

> copy with the same effect: if the translation is effective, the keyboard is
> skewed and you are force to load at least “something” to fix XKB. If the
> translation is off, you “only” need to load XKB configuration if that
> non-perfect copy annoys you...

> Thanks for the pointer! I had a look at it. As I suspected: the translation
> is “bad”. Sometimes two different keycodes are translated to the same
> keycode. How are you supposed to tell them apart on the remote side? You
> can’t.
>
> The naming is a little funny: the keycodes generated by the kernel before
> evdev were not called like that, were they?

I don't think so.

> That would be great as a first step.

I was wondering why this translations had been introduced. It emerged
between nxagent 3.2.0-10 and 3.3.0-9. The changelog between these
releases contains (among others) these changes:

nxagent-3.3.0-9

- Changed Keyboard initialization and reset. This change should fix
  TR11F02129, TR11F02131, TR11F02132.

nxagent-3.3.0-4

- Fix the XKB map load in the case of 64 bit server.

nxagent-3.3.0-3

- Fixed TR10F02119. If the remote X display is using evdev keyboard
  then copy maps from remote.

nxagent-3.2.0-12

- Fixed TR08E01814. Added shadow keymap initialization in order to
  enable nxcompshad to translate keycodes across different layouts.

So it MIGHT have been introduced mainly for shadow sessions. The
TR10F02119 refers to Ubuntu 8.10, which - according to some other
posts I found - is the version that introduced evdev. So the fix that
solved evdev problems is causing problems when evdev is enabled.

I also looked at the source and found that clearing the property seems
to be the only way to disable the translation. (apart from disabling
XKB completely).

Uli

---

From: Mike Gabriel <[email-masked]>
Date: Tue, 16 Dec 2014 14:42:32 +0000

Hi Ulrich,

> to be the only way to disable the translation. (apart from disabling

Note that you can view the changes between versions of nxagent via  
nx-libs.git [1].

Use the "commitdiff" links in the right column...

Mike

[1] http://code.x2go.org/gitweb?p=nx-libs.git;a=shortlog;h=refs/heads/nxagent

-- 

DAS-NETZWERKTEAM

GnuPG Key ID 0x25771B31

freeBusy: