There are several diffrent processes running in Y Window system. They can be divided into 4 categories.
Server. There is just one server process at a time. It's mainly used to maintain communication between other processes.
Display process. is the one that communicates with hardware, it draws unicode characters on the screen, reads keyboard and mouse.
Window Manager. is used to draw window's frames, menus and such. It's special kind of client, so there can be only one WM at a time, it's sent special events, so that it can do its work. For example, when new window is created, WM is sent special events containing title of windows, and various flags, passed by client.
Regular Clients The so called application programs. Examples of such clients include yterm.
Server and Window Manager should be connected with relativly fast link, possibly even running on the same machine, over loopback connection. Fast link is not required as much with display and regualar clients.
Client connects only to server, they can ask server to forward messages between them, but they mainly use their link to draw on their windows.
Server implements windows as rectangular, possibly overlapping, buffers. Server remebers what was written into the window, so it doesn't need to ask client to redisplay it when it's moved. However, server doesn't know anything about widgets, subwindows etc. It's much like X server, run xinit /usr/X11R6/bin/xterm and see what happens. Client shall nevertheless implement redraw method. It's issued when Display process type changes, and therefore some display capabilities could have been added or removed.
Display process is very simple, it's the only system depented thing, under linux it uses /dev/vcs* interface to be faster. It should be possible to easy reimplement it on other unices, possibly using X11 and even on those funky windowing "systems".
TCP/IP using Berkeley socket interface is used as transport layer. Server and clients implements custom RPC interface. Calls are distinguished by ASCII name.