Robots.txt: It is a file formed by webmasters is a set of instructions for the search engines robots that index the web pages. This is called The Robots Exclusion Protocol.
It includes User-Agent and Disallow. Here is the meaning of these words as these are very important part of robots.txt file:
User-Agent : It is used for the search engine robots. It gives the permission to a particular or all robots to index the
pages. Here is the syntax :
User-Agent : * ("*" means all robots have permission to crawl all the files/pages)
Disallow : It denies access to robots to directories/files. Here is the syntax:
Disallow: /images/ (it denies access to images directory.)
A simple robots.txt file should be:
User-Agent: *
Disallow:
It means we give permission to all the robots to visit and the directories and files.
It is very easy to create this file. Open notepad editor. Insert these instructions and save it by name robots in text
format. Put it in top-level directory and upload it. Suppose your website name is http://www.abc.com/ then your robots file's path should be here http://www.abc.com/robots.txt
It includes User-Agent and Disallow. Here is the meaning of these words as these are very important part of robots.txt file:
User-Agent : It is used for the search engine robots. It gives the permission to a particular or all robots to index the
pages. Here is the syntax :
User-Agent : * ("*" means all robots have permission to crawl all the files/pages)
Disallow : It denies access to robots to directories/files. Here is the syntax:
Disallow: /images/ (it denies access to images directory.)
A simple robots.txt file should be:
User-Agent: *
Disallow:
It means we give permission to all the robots to visit and the directories and files.
It is very easy to create this file. Open notepad editor. Insert these instructions and save it by name robots in text
format. Put it in top-level directory and upload it. Suppose your website name is http://www.abc.com/ then your robots file's path should be here http://www.abc.com/robots.txt
