Robots Meta Robots Robots.txt

Robots - 

Also known as Spider / Crawler / Web Crawler / Robots
"Robots is an automated program used to feed the database of  their search engines with new information"

• By Crawling link to link in search of new information.
• By updating the database already index site.

Robots.txt -  

This is a file that gives the instructions for all search engine robots. Webmaster use this file for avoid spamming.

To allow all Robots -

User-agent: *

Disallow: 

To Stop all Robots -

User-agent: *

Disallow: /

To Stop Robots on Particular Folder or Files -

User-agent: *

Disallow: / Folder Name/ Files Name.html

To allow all Robots Except Google -

User-agent: *

Disallow: /

User-agent: Googlebot

Allow:/

Note - -- *(The Astrick or Wild Card represents a special value that means any robots). The Disalow : line without / (forward slash) tells the robots that they can index the entire site.

After creating robots.txt file use can upload this file under root folder of website server.

Meta Robots - 

We write the meta robots under <head> section of webpage.

To allow all robots -
<meta name = "robots" content="all"/>
OR
<meta name = "robots" content="follow, index"/>

To Stop all robots -
<meta name = "robots" content="none"/>
OR
<meta name = "robots" content="nofollow, noindex"/>

If you want to robots follow your all links dont index your page then-
<meta name = "robots" content="follow, noindex"/>

If you want to robots don't follow your links and index your page then -
<meta name = "robots" content="nofollow, index"/>