If you prefer a more vim-like workflow (i.e. jumping in and out of your editor from your shell), a good solution is to fire up emacs in daemon mode to run in the background, and then connect to it using emacsclient.

I automatically run

  emacsclient --daemon

when my OS boots up.

In my .zshrc and .bashrc files, I use the aliases

  alias e='emacsclient -t'
  alias vi='emacsclient -t'

to give me quick access to emacs from the shell.

I'm not sure if this addresses your problems entirely, but it does keep you from having to deal with all of the crap that loads on startup.