Robots.txt is a simple text file that allows or disallows Google and other search engines from crawling through the website pages. It is a good way of preventing a file or directory from indexing by writing special commands in the file. These simple instructions for robots or search engine spiders control exactly what you want the spiders to crawl.
A robots.txt file can be created in notepad and it must be present in the root directory of your website for search engines to read it.
One Response to “Robots.txt File”
Post A Comment
You must be logged in to post a comment.
September 8th, 2008 at 3:30 am
[…] use a robots.txt file to prevent the search engines from crawling them. Once you have created the robots.txt file, you should upload it to the root directory of your domain. By putting this file in the root […]