How to Create an ideal Robots.txt File for SEO

Robots.txt is actually a text file created by webmasters. Robots.txt’s basic function is to define rules and instruct search engine robots (web-robots) on how to crawl pages on their website.

The robots.txt file is a basic component of the “robots exclusion protocol” (REP), a set of rules of web standards that control how robots crawl the websites, access and index content, and serve that content material up to users. The REP also includes directions like meta robots, page content, media files, and subdirectory. The REP also includes site-wide instructions for how search engines should treat web links.

Practically robots.txt files allow, whether specific user agents such as “web-crawling software” can or cannot crawl the different parts of a website. These crawl guidelines are specified by allowing or disallowing the patterns of particular user agents.

Basic format:
User-agent: *
Disallow: /

We have a big question here. Why do we need robots.txt?
Robots.txt files actually control (Robots) crawler access to particular fields/URL/directories of your site. It could be hazardous if Googlebot is
accidentally disallowed from crawling. Be careful while creating or editing your robots.txt file; some certain situations in which a robots.txt file can be beneficial.

Some common uses

  • Avoiding duplicate content from appearing in SERPs.
  • Keeping private the entire sections of a website.
  • Keeping internal search results pages from showing up on a public SERP
    Specifying the location of sitemap(s)
  • Blocking search engines from indexing particular files on your website includes any media file like images and PDFs.
  • You can also specify a crawl delay to minimize your servers from becoming overloaded when crawlers load multiple contents at once.

Search Engine Optimization — Top SEO Tips For Beginners

SEO best practices

 

  • You need to make sure that you’re not blocking any specific sections of your website or content that you want to be crawled.
  • Keep in mind that Links on pages blocked by robots.txt will not be followed or crawled.
    That means
  • if any link that you blocked is also linked from any other search engine-accessible pages, the linked asset/content or information will not be crawled and may not be indexed on those search engines like bing or Yandex, etc.
  • No link equity can be crawled from the blocked page to the linked resources. If you have any specific pages to that you want equity to be crawled, use any other blocking system other than robots.txt.
  • Never use robots.txt to protect sensitive data like “private user information” from showing up in SERP results because other web pages may link directly to that page containing private information. Therefore by skipping the robots.txt directives on your root directory (domain or homepage), it may still get indexed. So, If you want to block your specific page or information from indexing, use any method like “NOINDEX meta directive” or password protection.
  • Typically, search engines have multiple user-agents. For example, Google utilizes Googlebot for organic search and Googlebot-Image for image search. The user agents from the same search engine follow the same rules, so there is no need to have specific directives for each of a search engine’s multiple agents.
  • A search engine will cache the robots.txt contents but usually update the cached contents at least once a day. If you change the file and update it more quickly than it is taking place, you can submit your robots.txt URL to Google.

How to Change Default Admin Username In WORDPRESS — 3 Simple Ways

Some basic concepts you must know

  • robots.txt file must be placed in a website’s top-level (root)directory.
  • the file |should always be named the same as “robots.txt”.
  • Some user agents (robots) might decide to ignore the robots.txt file. This is specifically common with more negative crawlers like email address scrapers and malware robots.
  • The www.example.com/robots.txt file is publicly available. just add /robots.txt to the end of any root domain to see that website’s directives, means anyone can see what web-pages you do or don’t want to be crawled,
  • Each domain and subdomain must have separate robots.txt files. That means both www.blog.yoursite.com and www.yoursite.com must have their own separate robots.txt files on their root directories.
  • it is a best practice to add the website’s sitemaps linked with the domain at the bottom of the file.
For example

User-agent: Bingbot
Disallow: /example-subfolder/blocked-page.html
Sitemap: https://www.xyz.com/sitemap.xml

How to speed up your WordPress Website — Tips And Tricks

How to create a robots.txt file

How to create a robots.txt

If you don’t find that file on your website’s root directory then creating one is a simple process. You can have further guidelines from Google’s article, which is specific about the txt file creation process, and this robots.txt testing tool permits you to test if your file is set up correctly.

If you found that your website doesn’t have a robots.txt file, you can easily create it by sung notepad or any other text editor—copy that created robot.txt file into your website’s root directory (example: ww.xyz.com/robot.txt). You can copy this file using a FileZilla FTP client or Cpanel. After copying the file, you need to set 644 file permission.
The simplest and most powerful robots.txt setting/rules, which I recommend to the best of my limited knowledge, are given below.

How to combine images using CSS sprites without a plugin — Tutorial

Example No. 1

User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /readme.html
Disallow: /refer/
Sitemap: https://www.yourdomainname.com/sitemap_index.xml

You can copy that lines into your robots.txt, or you can modify your existing file.
Note: replace www.yourdomainnmae.com/sitemap_index.xml with your own website’s domain name and Sitemap.xml name.

Another robots.txt file example which I want to share with you is:

Example No. 2

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Allow: /wp-content/themes/your theme name/
User-agent: Mediapartners-Google*
Allow: /

User-agent: Googlebot-Image
Allow: /

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Mobile
Allow: /

User-agent: Twitterbot
Allow: /

User-agent: *
Disallow: /search
Disallow: /cgi-bin/
Allow: /

User-agent: *
Disallow: /*.html
Allow: /*.html$
Sitemap: https://www.yourdomainname.com/sitemap_index.xml

But you can see separate rules defined for each User-agent, which is a little complicated for newbies. So, therefore, I recommend the first example for the beginners

;

Related Articles

Website Speed Optimization, Important tools and techniques

Like other factors, website speed optimization is an advanced factor because Google is too conscious about the website's speed and better user experience that...

Top important tips for writing perfectly SEO friendly content

If we talk about search engine optimization, it is a technique to rank higher your web pages in search engines for more readers by...

Importance of images and how to do image optimization for SEO

As images support and explains your content and make your content engaging for your visitors. You should attach SEO friendly images to every article...

Latest Articles

How to Increase the Maximum File Upload Size in WordPress

Are you facing problems while uploading heavy image files, themes, or plugins to the WordPress website? You need to increase the maximum file upload...

On-page SEO Key Ranking Factors – Top 9 Rules

On-page SEO (Search Engine Optimization) is focused on Creating or improving relevance signals for a website’s page according to the niche and search queries....

How to Change Site Permalinks Without Breaking Internal Links

Changing the site permalinks structure on your website is not a big deal. You can Change permalinks structure just through a single click, but...

What is Proofreading? Basic Techniques and Tips

Proofreading is not just editing; it is the process of reviewing the final draft of a piece of writing to ensure consistency and accuracy...

How to Improve HTTP Response Header In WordPress

A text record information that a Web server transmits back to a client's browser in response to receive an HTTP request. The HTTP response...

How To Change WordPress website’s Address [URL]

If you want to change or update your WordPress website's default address, domain name, WordPress Address or URL have any reason. You can change it...