I learned that even if you don’t have a display server running, chances are that Linux is still using the graphics driver - which makes sense, framebuffers don’t draw themselves on their own. But that’s still annoying if you want to install a graphics driver without having to reboot.