9 reasons why pages are not indexed

Imagine you’ve created a great page full of useful and interesting information and great products, and Google says: “No, we won’t index this.” That’s very frustrating, because you’ve just put a lot of effort into it. If Google doesn’t index a page at all, this usually points to a technical issue. We’re not talking about a low ranking here, as that has other causes.

What is the difference between crawling and indexing?

Crawling is the first phase in which the search engine bot, also known as the Googlebot, visits the web pages of a website. During crawling, the bot follows links from one page to another and scans the entire internet for new and updated pages. The aim is to retrieve and understand the content of these pages, such as text, images and links.

Indexing is the process by which a search engine stores pages in a massive database after crawling them.

Once you’ve created a good page, you’ll naturally want it to appear immediately in the search results so that people can find it. You do this by having the page ‘fetched’. When you fetch a page, Googlebot visits the page straight away and indexes it (or not). If a page isn’t indexed, you can check this in Google Search Console. Sometimes the reason why a page isn’t being indexed is listed there straight away, but you’ll often have to investigate it yourself. There are more than 10 technical reasons why a page might be excluded from Google. Some are easier to resolve than others, but all can be fixed with just a few lines of code or a setting in your backend or FTP.

Crawled – currently not indexed

Sometimes Google indicates that a page has been crawled but not indexed. Firstly, it is important to distinguish between ‘crawling’ and ‘indexing’. Crawling and indexing are two different things, and these two terms are often confused. Crawling means that Googlebot examines all the content and code on the page and analyses it. Indexing means that the page is eligible to appear in Google’s search results.

The message ‘crawled – currently not indexed’ is sometimes displayed. When a search engine such as Google visits a website, it goes through a process known as ‘crawling’. During crawling, the search engine scans the website’s content to determine which pages exist and what information they contain. This is the first stage of indexing the website. After crawling, provided everything is correctly configured on the website, the indexed pages are included in the search engine’s index. This means that the pages can appear in search results when users carry out searches relevant to those pages. The message ‘crawled – currently not indexed’ indicates that the search engine has visited the website and crawled the content, but has currently decided not to include the pages in the search engine index. There may be various reasons for this, such as irrelevant content, duplicate content and technical issues.

Below are a number of reasons why a page might not be indexed by Google.

1. Listed in Robots.txt

Have you ever looked at a website’s robots.txt file? The robots.txt is a small file containing rules for crawlers, such as Googlebot. These rules determine which pages the crawlers are not permitted to visit. Pages such as a webshop’s checkout page are often blocked, as is the admin area. Sometimes other pages are also excluded in the robots.txt file.

A page listed in the robots.txt file must not be visited by crawlers. If Googlebot cannot crawl a page, it will not be indexed. So make sure you never include pages that you do want to be indexed in the robots.txt file!

Usually, if you want to edit the robots.txt file, you need to do so via FTP. If this is a foreign concept to you, we strongly recommend that you contact your web developer if you believe your website’s robots.txt file is not set up correctly.

2. No-Index tag

Sometimes, when a new page is published, it is still set to ‘No-Index’. This is a single line of HTML code that allows you to tell Google that you do not want this page to be indexed.

This rule can, for example, be applied to pages that do serve an informative purpose but otherwise gain nothing from appearing in the SERPs. Another example is thank-you pages. These pages fulfil an important function, but there is no added value in having them appear in Google’s rankings.

Adding or removing the ‘No-Index’ setting is a simple selection in all systems. Don’t worry – the ‘No-Index’ setting will always be automatically disabled. Setting a page to ‘No-Index’ is therefore a deliberate action 99 times out of 100.

3. Incorrect Canonical

Many websites automatically add canonical tags to all pages. This is a very useful setting, but also highly prone to errors. For example, this regularly goes wrong with the pagination of a product category or with language settings. An incorrect canonical can cause a page to disappear completely from the rankings or even never be indexed at all.

The canonical tag is also a single HTML line. Just like the No-Index setting, this line is usually easy to adjust. If you’re not sure, get in touch with Traffic Today and we’ll be happy to help!

For pages that you do want to be indexed, it is usually advisable to add a reciprocal canonical tag. This tag therefore points to itself.

4. Excluded via server

Another reason why a page might not be indexed could be due to server settings. Depending on the type of server (Apache, Nginx, IIS, etc.), there are options to block pages or bots. Fortunately, it is not common for Googlebot to be blocked via the server. Nevertheless, it is worth checking these settings and adjusting them if necessary. This is because there are a number of bots on the internet that can slow down your website or scrape its content to publish it elsewhere.

5. Not included in sitemap.xml

Just like the Robots.txt file, there is a sitemap.xml file. The sitemap.xml is a file consisting of a list of URLs. This list serves as a sort of breadcrumb trail for your website for Google. You are, as it were, providing Google with a list of all your website’s URLs in advance, before Google crawls your site.

This ensures that Google can crawl your site more effectively, resulting in better indexing. In the sitemap.xml, you should only include URLs that you actually want to be indexed.

If a completely new page is not listed in the sitemap.xml, this will certainly not help with prompt indexing. So always ensure that all pages you want to have indexed are included in the sitemap.xml.

6. No internal backlinks & No-Follow tag

When Googlebot visits your site, it always starts on the homepage. It then clicks on various links on the page in search of new pages on your website, until it has visited all the linked pages.

And therein lies the problem: brand-new pages often have no inbound links from other pages on the website. If a page has no inbound links and is not listed in the sitemap.xml, it is invisible to Googlebot.

So always ensure you build new backlinks to your new page straight away. This is important not only because Googlebot won’t even see the page without backlinks, but also to direct more value to the new page.

Also, ensure that these inbound links are properly tracked by Google. Just as there is a ‘No-Index’ tag, there is also a ‘No-Follow’ tag. When Googlebot encounters this tag, it registers the link but does not follow it. Googlebot does not, as it were, ‘click’ on the link and does not visit the page it leads to. So, if a page only has inbound links with a No-Follow tag, the page will still not be indexed.

7. Insufficient authority via backlinks

In the previous section, we briefly mentioned the value that is passed on from other pages to the new page you wish to have indexed.

Internal links are very important, but external links also do a great deal of good for an individual page. If you arrange a few external backlinks straight away when publishing a page online, this will significantly improve its ranking. It can provide just the boost needed to send a new page soaring up the rankings.

8. Crawl budget

Before we explain why crawl budget can influence a page’s indexing status, it is first important to explain exactly what crawl budget is.
The crawl budget is the number of pages that Google crawls on your site in a day. This number varies from day to day, but is generally fairly stable. The number of pages crawled by Google each day is determined by a number of factors, such as the number of errors on your site, as well as the number of external links and the authority of your website.
A crawl budget can cause problems particularly if you have a large website with a low crawl budget. Suppose your website has 300,000 pages, and Google crawls 2,000 pages every day. This would mean it would take a very long time for Google to crawl all your pages. As a result, a new page might not be indexed.

9. Duplicate content

Duplicate content refers to the same content appearing on multiple URLs. This can occur because your website is not technically optimised, or because text has been copied to other (possibly external) pages.

Too much duplicate content on a website can confuse search engines and prevent a page from being indexed. As there are several causes of duplicate content, there are also several solutions available to reduce duplicate content on your website.

Conclusion

In this article, we have outlined some of the most common reasons why a page may not be indexed. Are you experiencing problems with your website’s indexing process? Traffic Today can help you identify and resolve these issues.