The robots.txt file is used to prevent search engine crawlers from indexing pages you don't want indexed. Drupal comes with a default robots.txt that is fairly complete, except that, if you are using pathauto, you may want to add the line: Disallow: /node/. This is done to prevent crawlers from indexing duplicate content, that is, the www.example.com/node/999 and the www.example.com/pathauto-named version of the page are duplicates, and this may reduce page's score and/or waste site bandwidth with crawlers indexing each page twice.