Robots -
Also known as Spider / Crawler / Web Crawler / Robots
"Robots is an automated program used to feed the database of their search engines with new information"
Also known as Spider / Crawler / Web Crawler / Robots
"Robots is an automated program used to feed the database of their search engines with new information"
- By Crawling link to link in search of new information.
- By updating the database already index site.
Robots.txt -
This is a file that gives the instructions for all search engine robots. Webmaster use this file for avoid spamming.
To allow all Robots -
User-agent: *
Disallow:
To Stop all Robots -
User-agent: *
Disallow: /
To Stop Robots on Particular Folder or Files -
User-agent: *
Disallow: / Folder Name/ Files Name.html
To allow all Robots Except Google -
User-agent: *
Disallow: /
User-agent: Googlebot
Allow:/
Note - -- *(The Astrick or Wild Card represents a special value that means any robots). The Disalow : line without / (forward slash) tells the robots that they can index the entire site.
After creating robots.txt file use can upload this file under root folder of website server.
Meta Robots -
We write the meta robots under <head> section of webpage.
To allow all robots -
<meta name = "robots" content="all"/>
OR
<meta name = "robots" content="follow, index"/>
To Stop all robots -
<meta name = "robots" content="none"/>
OR
<meta name = "robots" content="nofollow, noindex"/>
If you want to robots follow your all links dont index your page then-
<meta name = "robots" content="follow, noindex"/>
If you want to robots don't follow your links and index your page then -
<meta name = "robots" content="nofollow, index"/>
To allow all robots -
<meta name = "robots" content="all"/>
OR
<meta name = "robots" content="follow, index"/>
To Stop all robots -
<meta name = "robots" content="none"/>
OR
<meta name = "robots" content="nofollow, noindex"/>
If you want to robots follow your all links dont index your page then-
<meta name = "robots" content="follow, noindex"/>
If you want to robots don't follow your links and index your page then -
<meta name = "robots" content="nofollow, index"/>
