WebWave logo

WebWave drag and drop website builder lets you create unique websites. You can build your website from scratch by starting with a blank page, or choose various website templates from our free collection and modify them according to your needs. With WebWave website builder, you have a complete website design and hosting system at your fingertips.

This website was created with WebWave.

Find us online



Website builder.

Offer. 

Help. 

Useful articles. 

Company.

Other articles.

Authors: Julia Madraszewska & Weronika Wawrzyniak

What's the point of indexing websites?

The simplest answer is: because you want users to be able to find you in Google search results.

Of course, there are websites that shouldn't be indexed. If you store on a site confidential data or share a file to download, you might not want it to appear in search results and be accessible for everyone, especially if users have to make a specific action ( a purchase, registration, etc.) to enter a particular subpage. It would be abysmal if users could just type "your-address.com PDF" and download whatever they want without fulfilling the required conditions. 

Even though there are exceptions, in most cases indexing is crucial. 

How does indexing work?

Speaking about Google, there is crawling, crawl budget and mobile-first indexing. Each of these methods is responsible for different aspects of indexing. 

 

What is crawling?

In the context of Google, it's a situation when robots enter a website and view it, without indexing it yet. It is used to analyze the website before indexing. 

Not every website will get indexed after crawling, but all websites that are indexed have gone through the crawling. 

 

What is a crawl budget?

Crawl budget stands for the maximum amount of data, which Google can gather from a domain during one visit of the robots. Crawl rate limit and crawl demand specify the frequency limit of indexing and the demand for it.

In case of a situation when crawling overweights your server and causes inconvenience for users on your website, you can set a limit for this service in Google Search Console. It's worth to mention that Google discourages manual manipulation of the limits since it is calculated by the algorithm as optimal. 

 

What is mobile-first indexing?

Information announced in the second half of the year 2018 about indexing order changed positioning forever. If your website has a mobile version, it will be visited by robots first. 

Search results ranking is based on information concluded in the mobile version of the website. That's why it's important to carefully select elements, which will be displayed on different screens. 

If you are not in Google results, you don't exist online.

Is my website indexed?

The easiest way to check if your website displays in Google is to check search results :)

Enter the name of your website and check if you can see a link to it in search results. If yes - congratulations, you are all set.

To check whether the whole website got indexed, type in Google: 

site:http://sitename.com/

Keep in mind that it takes some time for Google robots to see your website. Sometimes it takes a couple of hours, sometimes a couple of days and sometimes months. Don't panic if you won't see your website in Google search results in a few hours after publication. 

 

How to speed up website indexing?

If you want robots to visit your website sooner than later there are a few ways to speed up the process:

  • Well configured robots.txt files are responsible for communication with Google robots. Additionally, not that long ago, webmasters additionally placed indexing information in those files, but like we wrote in the previous article, starting September 1st Google will no longer support noindex in robots.txt files, so keep in mind that this is no longer a good method. 

  • Configuration of Google Search Console has also a great influence on indexing. You can send a request to robots to index URL address and also limit the frequency of indexing if needed (crawl budget).   

  • If you have a website that was published a while ago, adding a link to the new site can speed up the indexing process. Google robots after completed crawling of the website will automatically enter websites from the links and index them.   

 

You are not in Google results? You don't exist online!

Indexing is a key factor to succeed on the Internet. 

I am sure you know a lot about website positioning and getting noticed by Google is the first step of it. You can't improve SEO if you are not in search results. 

Actually, my bad. The very first step is to create a website. You can do it for free in WebWave - white label responsive website builder with no coding, where we make sure that all requirements for your website indexing process to be the fastest are fulfilled. 

Start now ->

Join over 400 000 WebWave users and create websites with a true drag and drop website builder.

Indexing is the process of adding websites to Google search results. It consists of gathering, processing and storing data by Google robots, which visit your website and search for created by you metatag, that can be index or noindex. 

If the metatag is indexable, robots will read the source code of the website and add it to search results. In case of noindex, your site won't appear in search results, but keep in mind that Google stops supporting noindex robots.txt files at the beginning of September 2019.

However, the position of your website in search results is not dependent on robots, but algorithms. They are sequences of formulas and computer processes, whose main task is to find the best answers for users' queries.

Example of portfolio page

Updated: 22nd September 2022 

Reading time: 6 min 

02 September 2019

What's indexing? Indexing, Google robots and algorithms. 

WebWave logo

Webmaster's panel

Sign up

Log in