Google has sophisticated algorithms that determine how much to crawl each site. Our goal is to crawl as many pages from your site as we can on each visit without overwhelming your server's bandwidth.
You can change the crawl rate (the time used by Googlebot to crawl the site) for sites that are at the root level—for example, www.example.com
. You can't change the crawl rate for sites that are not at the root level—for example, www.example.com/folder
Note: Crawl rate refers to the speed of Googlebot's requests during the crawl process. It doesn't have any effect on how often we crawl or how deeply we crawl your URL structure.
To change the crawl rate:
1.On the Webmaster Tools Home page, click the site you want.
2.Under Site configuration, click Settings.
3.In the Crawl rate section, select the option you want.
The new crawl rate will be valid for 90 days.