1. Display and mode setting: DRM. DRM defines connector/encoder/crtc to support display(including multi-monitor). And KMS makes mode setting happens in kernel space.
2. 2D acceleration: X server EXA/UXA extension. Vendor should write a X server driver which implements EXA or UXA(should be upstreamed). UXA is an enhancement design of EXA, proposed by intel, so GEM is used in UXA. DRM has no direct 2D acceleration interface definitions(e.g: memory copy/move, blit, color space conversion...), this is defined in EXA/UXA extensions. But vendor's X server driver will use drm functions, mostly is buffer manipulations.
3. 3D acceleration: via Mesa. Mesa has 2 parts:
libGL.so -- the implementation of OpenGL spec, the default OpenGL implementation library in Linux. This libGL.so translates OpenGL callings to Mesa-specified instructions.
DRI driver -- the driver accepts the Mesa-specified instructions generated by libGL.so and call drm functions to get them accelerated in GPU. Should be upstreamed.
DRM: has userspace libdrm & kernel space drm driver.
libdrm: Implemented drm APIs. Vendor can hook it's implementations as well as add more APIs which used by EXA/UXA driver(2D) or DRI driver(3D). That's why the build result of libdrm normally has 2 libraries(e.g: libdrm.so & libdrm_intel.so). Vendor codes of libdrm should be upstreamed.
Kernel drm driver: Works under kernel drm framework, should be upstreamed. Cause vendor can add APIs in libdrm, so kernel drm driver can handle vendor specified ioctls.
GLX: X server extension of OpenGL. X server has DRI/DRI2 extensions as well.
They're working with libGL.so in Mesa. By these extensions, Mesa can do:
1. Direct rendering: libGL.so in Mesa try to figure out whether DRI driver is ready in Mesa. If so, GLX returns some infos(such as window size, position...) or create off-screen buffers(via DRI/DRI2 X extension) for Mesa. The rendering is handled in Mesa's DRI driver -- no relations with X server.
2. Indirect rendering: libGL.so in Mesa can't find out Mesa's DRI driver. So it queries whether DRI is available in X server. If so, it pass the Mesa-specified instructions which generated by libGL.so to X server. And GLX/DRI/DRI2 extensions of X server handles the rest things. Normally it's software rendering which implemented by Mesa as well. AIGLX(Accelerated Indirect GLX) seems has some solutions to accelerate this.
So Mesa handles a lot of works in linux 3D graphics stack. And it has lots of relations with X server as well(GLX and DRI X extensions).
An important reason that why we involve so many components is, decouple the dependencies of all of these guys. According to this design, DRM is not related with X server so it works with other programs as well(e.g: Wayland). Mesa is the same.