Generating/Testing Robot.txt using google webmasters tool for your blog/website

In order to generate/test robot.txt for your website, navigate to the link below.

https://www.google.com/webmasters/tools/robots-testing-tool



Select your website if you have already added your website, if not, click on add property and add your website. On selecting/adding, google will generate a robot.txt file as shown below.



Now enter the blog post URL in the field above and select any option from the dropdown box and click on test. This will help you check as to which all URL's of your website will be allowed/disallowed from crawling.

When googlebot was selected from the dropdown and clicked on test, the results appeared as below.




This will help you check as to which all URL's of your website will be allowed/disallowed from crawling