Robots.txt

Robots.txt is a file in a website's root directory that tells search engine crawlers which pages or sections they should not crawl. It can keep duplicate or irrelevant areas out of a crawler's path, but it does not protect confidential information and should never be treated as a security control. Reviewing crawl rules is a standard part of a technical SEO audit, especially after a redesign or migration.