The Robots.txt index files contain data about what information can be processed on the site. The stench gives you the opportunity to exchange the number of drinks for a scan, which significantly changes the amount of money for a resource. Also, the stench of the traffic of crawl robots and you can turn off the content, submissions on the side of Google search engine.