On my desktop, I don't bother starting X until logged in. If you're using a login manager like gdm, you'll have to do it like how fragfutter has suggested, but if you're skipping that, just add this to your ~/.profile
if [[ -z "$DISPLAY" ]] && [[ $(tty) = /dev/tty1 ]]; then
startx
logout
fi
This says that, if you're on terminal 1, start X; once X finishes, log out. Since you start on terminal 1 by default, this'll make X start unless you switch to another terminal to prevent it. Of course, you'll need a properly set-up ~/.xinitrc. Also, this will only work for your login; logins by root or other users will need to be set up the same way to get the automatic X.