robots.txt Generator – Create robots.txt Free

robots.txt is a plain-text file placed at the root of your website that tells web crawlers and bots which pages or directories they are allowed or not allowed to access. It follows the Robots Exclusion Standard and is the first file most search engine bots check before crawling your site.

robots.txt must be placed at the root of your domain — for example, https://yoursite.com/robots.txt. It cannot be placed in a subdirectory. Every domain and subdomain requires its own robots.txt file.

Yes. If you accidentally block important pages with Disallow rules, search engines will not crawl or index them, which removes them from search results. Conversely, using robots.txt to block low-value pages (admin panels, duplicate content) can focus crawl budget on your most valuable content.

Yes. Blocking paths like /admin/, /dashboard/, and /wp-admin/ with Disallow rules prevents search engines from wasting crawl budget on those pages and reduces the chance of sensitive URLs appearing in search results.

If you set Disallow: / for all user agents, search engines will not crawl any page on your site. This will eventually remove your entire site from search results. Only use this setting intentionally, for example during site development before launch.

Yes. Use a specific User-agent name instead of the wildcard *. For example, User-agent: Googlebot followed by rules applies only to Google's crawlers. This lets you allow general crawling while restricting specific bots like scrapers or AI crawlers.

Crawl-delay tells bots how many seconds to wait between requests to your server. For example, Crawl-delay: 10 means the bot waits 10 seconds between page fetches. This reduces server load on smaller sites. Note that Googlebot ignores Crawl-delay — use Google Search Console to control Googlebot's crawl rate instead.

To block AI training crawlers, add separate User-agent rules for GPTBot (OpenAI), ChatGPT-User (OpenAI browsing), Google-Extended (Google AI), CCBot (Common Crawl), and anthropic-ai (Anthropic). Use our Block AI Crawlers preset to add these rules automatically.

No. robots.txt is a public file and is not a security mechanism. It only requests that well-behaved bots comply — malicious crawlers may ignore it. Never rely on robots.txt to protect sensitive data; use authentication and server-side access controls instead.

Use the robots.txt Tester in Google Search Console (Search Console → Settings → robots.txt) to check whether specific URLs are allowed or blocked. You can also use the Fetch as Google tool to see how Googlebot views your pages.