I've actually been playing around with this stuff all day, pretty much since my last comment above. I've enabled smarter caching on my website, replaced multiple image requests with a single spritesheet, optimized my images, and cleaned up my CSS file to remove unused code. Google's PageSpeed has been an invaluable tool, as well as webpagetest.org which breaks down the data in an intelligent way.

Turns out Google Analytics is actually doubling my page load time, but the data is too valuable to give up.

Anyway, thanks for the tips.