Googlebot Crawler download rate increase with move to Rackspace

Google has been keeping data on sites which aren’t verified on Google Webmaster Tools all this time which isn’t such a bad thing as it sounds. I haven’t used Google Webmaster Tools in a while and decided to re-verify this site today.

download time decreased

80% decrease in download time on 4/7

I began migrating the server and DNS from Hostmonster to Rackspace on 4/5 and got the majority of the website back up and running on 4/6. The 80% decrease in download time occurring on 4/8 was most likely Googlebot taking 48 hours to update its DNS records to point to the Rackspace server.

The pre-4/7/Hostmonster download time looks horrible in retrospect as the Rackspace server is much more responsive and isn’t a shared server. But two months ago, it seemed pretty dandy. I guess you really do get what you pay for. In my previous post on migrating from Hostmonster to Rackspace, my current plan on Rackspace is about 35% greater per year.

Was it worth it? I sure think so. I now have to freedom to configure the server with whatever I need. In the migration article, I also mentioned I am running Nginx on the Rackspace server to serve up pages. Hostmonster uses Apache.

Mashable's 33 More Entertaining 404 Error Pages

Mashable – 33 More Entertaining 404 Error Pages

In 2011, Apparent Etch’s 404 page was featured on Mashable’s 33 More Entertaining 404 Error Pages. It was also hosted on Hostmonster at the time. I remember the site slowing down and grinding to a halt for a few hours from a few hundred visitors (not completely sure due to lack of statistics, I checked Hostmonster’s AWStats for info back then).

Reddit - We need some explaining Blizzard.

Reddit – We need some explaining Blizzard.

Fast forward to this year, on 5/12 I posted a fun find in the SC2 Editor to Reddit, I got about 15,000 views in the following 24 hrs. The lowest cost Rackspace server (1 CPU, 512MB RAM) handled the traffic spike without a problem. …Although the bandwidth charges should be interesting.

Anson Liu