When we come across, everyone wants to rank their site on keywords, optimise their content and want higher rank on Google.
While talking about On-Page SEO, there’s a much more fundamental need that every SEO needs to address.
One of the important factors that we need to consider while optimising our site for SEO is crawl budget. Very few Search Engine Optimisation company and website development company consider SEO crawl budget.
What is a crawl budget?
There is no exact definition that would describe everything that “crawl budget” stands for externally.
But in layman language Crawl budget is the number of pages of a website crawled by Google on any given day.
Why crawl budget optimisation is worth doing?
To understand this first, we will need to know how crawling is done by Google bots.
Google Bot prioritise what to crawl, when, and how many resources the server hosting the site can allocate to crawling is more important for bigger sites or those that auto-generate pages based on URL parameters for example
Crawl rate limit : “Crawl rate limit,” which limits the maximum fetching rate for a given website to make sure it doesn’t degrade the site performance.
The crawler will fetch the pages based on the speed the web site response as well as the time it has to wait between the fetches. The crawl rate can go up and down based
Crawl health: if the site responds really quickly for a while, the limit goes up, meaning more connections can be used to crawl.
Limit set in Search Console: website owners can reduce Googlebot’s crawling of their site. Note that setting higher limits doesn’t automatically increase crawling.
Even if the crawl rate limit isn’t reached, if there’s no demand from indexing, there will be low activity from Googlebot. The two factors that play a significant role in determining crawl demand are:
- Popularity: URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our index.
- Staleness: our systems attempt to prevent URLs from becoming stale in the index.
Additionally, site-wide events like site moves may trigger an increase in crawl demand in order to reindex the content under the new URLs.
Taking crawl rate and crawl demand together we define crawl budget as the number of URLs Googlebot can and wants to crawl.
Few tips for crawler budget optimisation.
#1 Reduce your page size
#2 Increase your server response
Still finding reasons why your website is not ranking?