September 28, 2023

How to edit Robots.txt file in WordPress root folder ?

What is Robots.txt?

Before learning How to edit Robots.txt file in WordPress? lets see what is it? Robots.txt is a text file placed on a website that specifies which pages or directories of the website should not be crawled or indexed by search engines or other web robots. It is a simple method used by website owners to communicate with web robots and prevent them from accessing certain parts of the website.

The robots.txt file includes instructions that tell web robots which pages to avoid or which user agents to disallow from accessing the website altogether. It can also contain directives to limit the crawl rate of web robots, which can help prevent the site from being overwhelmed by excessive crawling activity.

Robots.txt is a useful tool for website owners who want to control how their website is indexed and crawled by search engines and other web robots. It is important to note, however, that not all web robots obey the instructions specified in the robots.txt file, so sensitive information should not be placed in directories or pages that should be hidden. Lets see How to edit Robots.txt file in WordPress?

How to edit Robots.txt file in WordPress?
How to edit Robots.txt file in WordPress?
How to edit Robots.txt file in WordPress?

How to edit Robots.txt file in WordPress? There was issue with my default wordpress robots there were multiple sitemaps, I tried Yoast SEO, All in one Seo, Robots.txt editor plugin etc but it was not reflecting on Actual wordpress robots.txt. Finally I found right robots.txt editor for wordpress.

10 minutes 10 minutes

Install Virtual Robots.txt

robots.txt plugin for wordpress How to edit Robots.txt file in Wordpress root folder ?

Go to Plugins > Add New > Install Virtual Robots.txt

Go to Plugins Then Virtual Robots.txt settings

changing robots.txt of root folder of wordpress

Go to Plugins > Already Installed Plugins > Settings of Virtual Robots.txt.

copy your current Robots.txt

How to edit Robots.txt file in WordPress root folder?

Go to yourwebsite.com/robots.txt for me its bloggerguest.com/robots.txt and enter you will get current robots.txt as shown in Image. In my case there were multiple sitemaps present which is not desirable for good SEO. So I needed to remove those 2 cross marked sitemaps and keep only one i.e. sitemap_index.xml.

Just copy current robots.txt.

Paste it in Virtual robots.txt text box and do required changes.

editing robots.txt file in wordpress root folder

I have removed extra sitemaps i.e. News sitemaps, and sitemap.xml and just kept sitemap_index.xml and click on save changes. You can do changes as per your requirements and click on save changes the changes gets reflected in robots.txt file present in your wordpress root folder.

Thanks to virtual robots.txt plugin for making this possible, as i have tried Rankmath and similar plugins but it wasn’t reflecting in actual file. Finally its fixed with this plugin.

Supply:

Tools:

  • Virtual robots.txt file editor

Materials: robots.txt file

You may like – 5 Best Cheap WordPress Hosting 2023

How to open my websites robots.txt?

To open your website’s robots.txt file, follow these steps:

  1. Open your web browser and go to your website’s domain name (e.g., www.example.com).
  2. Add “/robots.txt” to the end of your domain name (e.g., www.example.com/robots.txt).
  3. Press Enter.

This will display the content of your website’s robots.txt file in your web browser. If the file is not found, you will see a 404 error message.

You can also check your website’s robots.txt file using online tools such as the Google Search Console, Bing Webmaster Tools, or other SEO tools. These tools can provide additional information about how search engines are crawling and indexing your website. in How to section we have shown How to edit Robots.txt file in WordPress?

How to create Robots.txt for my website Manually in Notepad?

To create a robots.txt file for your website, follow these steps:

  1. Open a text editor such as Notepad, TextEdit, or Sublime Text.
  2. Type the following code at the beginning of the file:
User-agent: *
Disallow:

This code tells all web robots that they are allowed to access all parts of your website.

3. If you want to disallow web robots from accessing specific directories or pages on your website, add the following code:

User-agent: *
Disallow: /directory/

Replace “/directory/” with the name of the directory you want to disallow. You can also disallow specific pages by adding their file names after the directory name (e.g., “/directory/page.html”).

4. Save the file with the name “robots.txt” in the root directory of your website. The root directory is usually the main folder where your website’s homepage is located.

Once you have created the robots.txt file, you can upload it to your website using an FTP client or through your website’s hosting control panel. It is important to test the file to ensure that it is working properly and that search engines are able to access the parts of your website that you want them to. You can use online tools such as the Google Search Console or Bing Webmaster Tools to check your robots.txt file and see how search engines are crawling and indexing your website. hope this tutorial on How to edit Robots.txt file in WordPress? is helpful for you.

How to generate robots.txt online?

There are several tools available online that can help you generate a robots.txt file for your website. Here are a few options:

How to edit Robots.txt file in WordPress? Present in root folder
How to edit Robots.txt file in WordPress? Present in root folder
  1. Robots.txt Generator by Small SEO Tools: This is a free online tool that generates a robots.txt file based on your website’s URL. Simply enter your website URL, select which search engines you want to allow or disallow, and click on the “Create Robots.txt File” button.
  2. Robots.txt Generator by SEOBook: This is another free online tool that generates a robots.txt file based on your website’s URL. It allows you to customize the file by specifying which directories and files you want to allow or disallow, as well as setting crawl delay.
  3. Robots.txt Generator by SEOOptimer: This is a free online tool that generates a robots.txt file based on your website’s URL. It offers a simple interface and allows you to customize the file by specifying which directories and files you want to allow or disallow.

Once you’ve generated your robots.txt file, you can upload it to your website’s root directory so that search engines can access it. Remember to double-check the file to ensure that it is correctly formatted and contains the appropriate directives for your website. Hope you have enjoyed our post How to edit Robots.txt file in WordPress?

Also Read – WordPress Vs Blogger Which Is The Best?

ketanblogger

I am a welding expert completed diploma in mechanical engineering, Blogging as a hobby, I love to help fellow bloggers to solve their issues and help them monetize their websites. I teach people how to earn money online.

View all posts by ketanblogger →

One thought on “How to edit Robots.txt file in WordPress root folder ?

Comments are most welcome and appreciated.