Seems that it is quite outdated but how you measure your server response time? Have you checked how your server responds for different locations? Try these to tools http://www.dotcom-monitor.com/WebTools/website-performance-test.aspx and http://www.dotcom-monitor.com/WebTools/free-server-uptime-test.aspx
