to prevent undesirable information during the search indexes, site owners can instruct spiders to not crawl sure information or directories throughout the regular robots.txt file in the basis directory of the domain. https://www.seoninjas.co.za