A question we see popping up time and time again among people on the hunt for hot topics is ‘How do you get Google to crawl your website faster?’.
With that in mind, we thought we would take the time to answer this.
You could be asking this question as the owner of a big site, always on the lookout for bigger and better ways to do things, or you might be optimising your new web pages wondering how to get them crawled and indexed quickly.
In this post, we will cover a few different ways that are tried and tested, that will allow you to improve the crawl speed of your site and get Google to crawl your website faster!
So, let’s look first at why Google may be slow when spidering your content and website.
There are a few obvious reasons out there.
A common reason is that Google can’t detect enough quality links to your content. A site with quality links will often be talked about as having ‘authority’.
Websites lacking inbound and outbound links are deemed to be of little importance to Google, so it’s important to build a strong link portfolio, looking at both your inbound and outbound links.
Other reasons are slightly more technical, however a good SEO agency should be able to deal with these issues quite comfortably.
Your site may have errors (e.g. broken links, formatted incorrectly, duplicate content). You may have a mighty slow website, making it hard for the search engine bots to come and crawl your content or Google may simply find it has too many pages to crawl, with not enough valuable content.
Let’s take a look at website authority first.
Consider the Authority of Your Website
If your site is lacking inbound links, Google won’t crawl it very quickly. A site without inbound links is not deemed important; so brand new sites will generally fall under this category.
One way to ensure more action on your web content and a better crawl speed from Google is to do some link building.
Below we have listed three posts you can read to get you started with link building.
We’ve purposely chosen topics that have previously been covered by industry experts and can be bookmarked for you to refer back to in the future:
If you feel that you are still in need of more, in-depth information on link building, Search Engine Journal has written its very own link building guide, looking at how to acquire and earn links to boost your SEO.
You can download it here.
There are some technical issues that will often result in the slow crawling of a website.
These can be split into three different categories:
- Website speed
- Site Errors
- Too many URLs
One of the most common reasons for Google not crawling a website very fast is that the site is actually incredibly slow to load.
If your site takes a long time to load bulky content or takes forever and an age to respond to requests, Google may lower their crawl depth and frequency intentionally.
Another scenario is that your crawl budget will be limited in a bid to stop things from breaking. If this is the case, you most likely have larger issues with your site speed.
If your website has a lot of errors on it, Google will start crawling more slowly.
The only way you can speed up this process is to fix your errors. Look for 301 errors on your site and redirect them to proper URLs.
To locate 301 errors, we’d recommend logging into Google Search Console and checking your website coverage.
Another trusted method is to access your site via your site logs.
You can find tools online that act as an analyser for your log files, such as Screaming Frog.
To ensure your site continues to be crawled more quickly, you should periodically check for errors and fix them.
For more information on how to fix errors, such as 404 errors, you can follow this link.
If your site is experiencing a lot of downtime, Google will also slow down a lot.
There are various reasons why your site may be experiencing down time, one of them being your hosting.
Switching hosting is a good idea if your site is often down.
Using online tools like Pingdom will help you check your site and monitor its uptime and downtime statistics.
Too Many URLs
Although it is necessary to have links and URLs on your site for it to gain authority, it is also possible to have too many URLs.
Too many URLs on your site can happen for a number or reasons; faceted search navigation and a system on your site that generates too many URLs being two possibilities.
You can figure out whether your site is guilty of having too many URLs by crawling it yourself.
This can be done manually or by using a tool specifically designed for the purpose, such as Ryte.
Tips to Make Google Crawl Your Website Faster
By following a few simple tips, you can successfully increase the crawl speed of your site:
- Make sure you have a fast website.
- Find and fix errors on your site, using the methods we talked about in this article.
- Add an XML sitemap to your web content and submit it to all the different search engines.
- Start Link Building.