Thursday, January 13, 2011

What is Robots.txt file used for ?

Robots.txt file
1. Helps crawl website.
2. It also lets you decide which page to expose to crawler.

No comments:

Post a Comment