ssh

If you want the X connection forwarded over SSH, you need to enable it on both the server side and the client side. (Depending on the distribution, it may be enabled or disabled by default.) On the server side, make sure that you have X11Forwarding yes in /etc/sshd_config (or /etc/ssh/sshd_config or wherever the configuration file is). On the client side, pass the -X option to the ssh command, or put ForwardX11 in your ~/.ssh/config.

If you run ssh -X localhost, you should see that $DISPLAY is (probably) localhost:10.0. Contrast with :0.0, which is the value when you're not connected over SSH. (The .0 part may be omitted; it's a screen number, but multiple screens are rarely used.) There are two forms of X displays that you're likely to ever encounter:

With ssh -X localhost, you can access the X server through both displays, but the applications will use a different method: :NUMBER accesses the server via local sockets and shared memory, whereas HOSTNAME:NUMBER accesses the server over TCP, which is slower and disables some extensions.

Note that you need a form of authorization to access an X server, called a cookie and normally stored behind the scenes in the file ~/.Xauthority. If you're using ssh to access a different user account, or if your distribution puts the cookies in a different file, you may find that DISPLAY=:0 doesn't work within the SSH session (but ssh -X will, if it's enabled in the server; you never need to mess with XAUTHORITY when doing ssh -X). If that's a problem, you need to set the XAUTHORITY environment variable or obtain the other user's cookies.

To answer your actual question:

Local displays correspond to a socket in /tmp/.X11-unix.

Remote displays correspond to open TCP ports above 6000; accessing display number N on machine M is done by connecting to TCP port 6000+N on machine M. From machine M itself:

(The rest of this bullet point is of academic interest only.)

From another machine, you can use nmap -p 6000-6099 host_name to probe open TCP ports in the usual range. It's rare nowadays to have X servers listening on a TCP socket, especially outside the loopback interface.

Strictly speaking, another application could be using a port in the range usually used by X servers. You can tell whether an X server is listening by checking which program has the port open.

If that shows something ambiguous like sshd, there's no way to know for sure whether it's an X server or a coincidence.