You can install this admin module which will report the load average for most linux server set-ups (Not Localhost WAMP Stacks). This is for
Average Load and Uptime J1.7 or
MiniSystem for J1.5. They should allow you to see if your server has a low loading or not... if it's high then your site will be slow.
You can also install
Yslow in firebug which will give you useful information about page load optimisation.
If your site is normally pretty quick the server or connection speeds are possible causes. If it's constant then it's more likely the page content, image size, image resizing in html, no cache or no gzip etc.
If it's intermittent and or has just suddenly started check the server loading first.