Robots.txt basics
Robots.txt tells crawlers which parts of your site they may visit. It does not hide pages from Google — use a noindex meta tag for that. Always include your XML sitemap URL.
Robots.txt examples you can copy
Allow everything and list your sitemap:
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlBlock an admin area only:
User-agent: *
Disallow: /admin/Disallow: / under User-agent: * asks all crawlers to stay away from your entire site. Developers often use it on staging sites, so check that it has gone after launch.Robots.txt does not hide a page
This is the most common misunderstanding. A blocked URL can still appear in Google if other websites link to it. To keep a page out of search results, allow crawling and use a noindex tag. For private content, use a password.
Where does the file go?
It must be at the root of your domain, for example https://www.example.com/robots.txt. It will not work in a sub-folder. Open that address in your browser to confirm, then test it using the robots.txt report in Google Search Console.
Should you allow AI crawlers?
If you want to appear in AI answers, allow search-related crawlers such as OAI-SearchBot and PerplexityBot. Each company documents its own crawler names, so check these before blocking any of them.