Michael says X needs stronger, hostname-independent, authentication.

xhost +SI:localuser:foo:

daniels@ephemera:~% xhost +SI:localuser:nobody
localuser:nobody being added to access control list
daniels@ephemera:~% sudo -u nobody xdpyinfo
name of display: :0.0
version number: 11.0
vendor string: The X.Org Foundation
[...]

This was all part of the ServerInterpreted auth scheme, which I think Sun did the work on. So you're free to concoct arbitrary schemes involving SELinux, and just implement them in the server you're intersted in.