Discuss about top search engine gogole its optimization techniques including the infamous Google Dance, Google News, Google Page Rank, Google Toolbar. Get a top ranking in Google and see your site traffic soar. Discussion related to the Google search engine only please.
Robots txt file is very usefull to prevent any bot to index your web site, infact each web site should have because i do not think that many crawlers check and index your web site with having robots.txt