A robots.txt file lets you restrict the obtain of internet search engine robots that crawl the net and it may possibly protect against these robots from accessing certain directories and pages. In addition it specifies the place the XML sitemap file is located. Redirecting requests from the non-favored area is essential for the reason that serps