Saturday, May 25, 2013

How to Enable and Customize Blogger Robots.txt File?

at 12:53 AM
19 Comments

Robots.txt
Hi ! In this tutorial I am going to show you how to edit your Blogger blog robots.txt file. Webmasters use custom robots.txt file to control search engine web robots (also known as Web Wanderers, Crawlers, or Spiders) to crawl some directories and web pages or links of website or blog. when we take settings robots.txt file in blogger then search engine automatically index or remove pages from search engine according to the settings. 

By default each website allows the Search engines robots however if you would like to restricts the robots either to not crawl any bound directory, file or the complete website then you may want the robots.txt file in which you've got to write instructions for search engine bots.

Steps to edit robots.txt on blogger

Your Site Settings › Search preferences › Crawlers and indexing  
Here you will able to see two options Custom robots.txt and Custom robots header tags.These two
 options would offer you the flexibility to customise your robot.txt file.In the last post i told you about
Custom robots header tags.
Now press edit button which is present after  'Custom robots.txt' option. After pressing on edit button
you can see a message " Enable custom robots.txt content? " so press "Yes" and proceed to next step.
Custom robots.txt - Search preferences
Now you can see  text area, type the content which you want to exclude a content from crawling.
Click on Save Changes button.
Enable Custom robots.txt - Search preferences

You are done!
How to block Link from search engines?
No, you have to write the URL. For example, if your want to stop robots from crawling this URL
(www.bloggerknown.blogspot.com/p/about.html) then, in the robot.txt file you will enter this command.
User-agent: *Disallow: p/about.html
Your robots.txt file located under your main blogspot directory as for Blogger Known its located on the following url:
http://www.bloggerknown.com/robots.txt
By default blogger robots.txt contains the following content:
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://www.bloggerknown.com/feeds/posts/default?orderby=UPDATED
I know this post is not a new thing but I hope it might help many newbies to understand the importance of robots.txt.
Get Free Updates:

19 comments:

  1. Bro My Blog Homepage not search on google top search Result..

    Please Search on Google For Tech2Webs (that is my blog)

    What Should I Do??????????????

    ReplyDelete
  2. This is such a very informative post and heartly thanks for sharing your kind knowledge with us and as per knowledge Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site

    ReplyDelete
  3. my blog name is www.fullformof.org and i ma posting butrobots.txt file is blocking my urls
    recently i posted 134 posts and robots.txt blocked 51 url

    please help me to prevent robots.txt to block my urls

    ReplyDelete
    Replies
    1. Dear yor must read about webmaster tools & custom robots header tags for website 1. http://www.bloggerknown.com/2013/03/submit-website-sitemap-to-webmasters.html
      2. http://www.bloggerknown.com/2013/05/how-to-custom-robots-header-tags-for.html

      Delete
  4. If i have a dozen blogs in my blogger account, will this robots.txt file
    "User-agent: Mediapartners-Google
    Disallow:"

    block the search engines to my entire blogs?

    ReplyDelete
  5. nice post please visit my blog :-d http://fun-zone4every1.blogspot.com/

    ReplyDelete
  6. Still confused, After lot of search still i am not found the perfect solution of robot.txt. Please help me How to allow sites to crawl my web.

    ReplyDelete
  7. hey admin please tell me that i have two blog domains and i want to change the first on into second can u help me !

    ReplyDelete
    Replies
    1. Dear go to blogger Settings › Other and Export blog download xml file and go to second blog settings and import xml file. All post of first blog publish on second blog. Thanks.

      Delete
    2. Very thnk u, but is there any drawback or harm in doing this and my first blog data will stay in that blog or not.

      Delete
    3. It will not make any harm to your blog. And your first blog data will be safe.

      Delete
  8. Nice post ! very useful Thanks Muhammad Azeem.
    Am a blogger in http://talkzoom.blogspot.in Thanks!

    ReplyDelete
  9. Good one..
    Thanks
    http://www.neentech.blogspot.com

    ReplyDelete
  10. thanks for this info.. please visit also my blog http://streamhigh.blogspot.com/

    ReplyDelete
  11. hello guys i found on net that this website giving free rs2000/- and earn more than 1,00,000/pm for free without investment see now

    www.makewebsiteearncash.blogspot.com/

    ReplyDelete

Your feedback is always appreciated. we'll try to reply to your queries as shortly as time allows.

Note:
1. To add HTML CODE in comments then please use our HTML Encoder
2. You can always Test the tutorial on our HTML Editor
3. Please don't spam Spam comments will be deleted immediately upon our review.

Regards,
Mohammad Azeem