Labels

Suggested Reading

Suggested Reading
Ayurvedic Way of Living - A step towards a Disease free life

Search This Blog

Comments

4-comments

Feed

POPULAR POSTS

Enable/disable proxy using a batch file in Windows

Delete temporary files using command prompt

How to execute MASM programs in Windows 7

Skip to main content
"Short, tweetable sentence from the article." Tweet this

Generating/Testing Robot.txt using google webmasters tool for your blog/website

In order to generate/test robot.txt for your website, navigate to the link below.

https://www.google.com/webmasters/tools/robots-testing-tool



Select your website if you have already added your website, if not, click on add property and add your website. On selecting/adding, google will generate a robot.txt file as shown below.



Now enter the blog post URL in the field above and select any option from the dropdown box and click on test. This will help you check as to which all URL's of your website will be allowed/disallowed from crawling.

When googlebot was selected from the dropdown and clicked on test, the results appeared as below.




This will help you check as to which all URL's of your website will be allowed/disallowed from crawling

Comments