Robots.txt Tester

Robots.txt files provide search engines with important information about crawling files and web pages. This file is used primarily to manage crawler traffic to your website in order to avoid overloading your site with requests.

You can solve two problems with its help:

• First, reduce the likelihood of certain pages being crawled, including getting indexed and appearing in search results.
• Second, save the crawling budget by closing pages that shouldn’t be indexed.

However, if you want to prevent a page or another digital asset from appearing in Google Search, a more reliable option would be to add the no-index attribute to the robots meta tag.

A quick and easy way to make sure your robots.txt file is working properly is to use special tools.

For example, you can validate your robots.txt by using our tool: enter up to 100 URLs and it will show you whether the file blocks crawlers from accessing specific URLs on your site.

To quickly detect errors in the robots.txt file, you can also use Google Search Console.

• The file is not in the .txt format. In this case, bots will not be able to find and crawl your robots.txt file because of the format mismatch.
• Robots.txt is not located in the root directory. The file must be placed in the top-most directory of the website. If it is placed in a subfolder, your robots.txt file is probably not going to be visible to search bots. To fix this issue, move your robots.txt file to your root directory.

In the Disallow directive, you must specify particular files or pages that should not appear on SERPs. It can be used with the User-agent directive in order to block the website from a particular crawler.

• Disallow without value. An empty Disallow: directive tells bots that they can visit any website pages.
• Disallow without value. An empty Disallow: directive tells bots that they can visit any website pages.
• Blank lines in the robots.txt file. Do not leave blank lines between directives. Otherwise, bots will not be able to crawl the file correctly. An empty line in the robots.txt file should be placed only before indicating a new User-agent.

1. Use the proper case in robots.txt. Bots treat folder and section names as case-sensitive. So, if a folder name starts with a capital letter, naming it with a lowercase letter will disorient the crawler, and vice versa.
2. Each directive must begin on a new line. There can only be one parameter per line.
3. The use of space at the beginning of a line, quotation marks, or semicolons for directives is strictly prohibited.
4. There is no need to list every file you want to block from crawlers. You just need to specify a folder or directory in the Disallow directive, and all of the files from these folders or directories will also be blocked from crawling.
5. You can use regular expressions to create robots.txt with more flexible instructions.
   • The asterisk (*) indicates any value variation.
   • The dollar sign ($) is an asterisk-type restriction that applies to website URL addresses. It's used to specify the end of the URL path.
6. Use server-side authentication to block access to private content. That way, you can ensure that important data is not stolen.
7. Use one robots.txt file per domain. If you need to set crawl guidelines for different sites, create a separate robots.txt for each one.

You can analyze your robots.txt file using the Google Search Console tool.

This robots.txt tester shows you whether your robots.txt file is blocking Google crawlers from accessing specific URLs on your website. The tool is unavailable in the new version of GSC, but you can access it by clicking this link.

Choose your domain and the tool will show you the robots.txt file, its errors, and warnings.

Go to the bottom of the page, where you can type the URL of a page in the text box. As a result, the robots.txt tester will verify that your URL has been blocked properly.