robots.txt Generator
Easily control search engine crawlers and block AI crawlers.
Simply use the form to auto-generate a correctly formatted robots.txt.
⚙️ Default Rule (All Crawlers)
🤖 AI Crawler Protection
Prevents unauthorized scraping by ChatGPT, Claude, Applebot, etc.
🕷️ Individual Bot Settings
✨ Generated robots.txt
About the robots.txt Generator
The robots.txt generator is a utility that allows you to easily create a text file to instruct search engine crawlers (spiders) which pages of your website they are allowed to crawl or forbidden to access.
In addition to traditional search engine control like Googlebot and Bingbot, it features a function to **bulk block unauthorized data collection (scraping) by "AI crawlers"** such as ChatGPT (GPTBot), Claude (ClaudeBot), and Apple's AI, which have been increasing recently. Your settings are reflected in the code in real time and can be copied and used immediately with one click.
How to Use
Select Default Rule
First, choose whether to "Allow" or "Disallow" access to the entire site for all crawlers.
Bot & Directory Settings
Set whether to block AI bots and restrict access to specific directories (e.g., `/admin/`).
Copy Code
Copy the text generated simultaneously on the right (bottom on mobile) using the "Copy Code" button.
Install on Server
Save the copied content as `robots.txt` and place it in the top directory (root) of your site.
robots.txt Glossary
- robots.txt
- A text file placed in the root directory of a website. It contains rules instructing automated programs (crawlers) like search engines on which parts of the site they can or cannot access.
- User-agent
- An identifier representing the identity of the accessing crawler or browser. In robots.txt, writing "User-agent: Googlebot" applies rules to a specific crawler. "*" means all.
- Allow / Disallow
- Commands to "Allow" and "Disallow" crawlers. Writing "Disallow: /admin/" prohibits access to all pages under /admin/.
- AI Crawler (AI Bot)
- Automated programs that collect data on the Internet to train AI models by AI companies like OpenAI (ChatGPT) and Anthropic (Claude). GPTBot is a typical example.
- Sitemap
- An XML file that tells crawlers the page structure of a site. By writing "Sitemap: https://..." in robots.txt, you can efficiently convey the entire picture of your site to crawlers.
Frequently Asked Questions
- Q.What is the difference between robots.txt and HTML meta robots?
- robots.txt is a file that controls which pages on your site crawlers are allowed to crawl. On the other hand, meta robots (like noindex) controls whether a crawled page is indexed (shown) in search results. Use robots.txt to prevent access itself, and meta robots to prevent indexing.
- Q.Will blocking AI crawlers affect normal search rankings?
- No, it will not. Major search crawlers like Googlebot and AI crawlers like ChatGPT (GPTBot) or Claude (ClaudeBot) have different User-agents. Blocking only AI crawlers (Disallow) does not affect your Google SEO evaluation.
- Q.Where should I place the generated robots.txt?
- Upload it to the root directory (the topmost level) of your website with the filename "robots.txt". For example, if your site is https://example.com/, it should be accessible at https://example.com/robots.txt.
- Q.What happens if I configure it incorrectly?
- If you select "Disallow All" in the default rules or accidentally Disallow "/", Googlebot won't be able to crawl your entire site, which could result in your site disappearing from search results. Please check the settings and preview carefully before installing.
Use Cases
Blocking unauthorized AI training
Use the bulk block of AI crawlers when you want to prevent your original content and blog articles from being arbitrarily scraped as AI training data.
Protecting admin and private pages
Bulk deny crawling for directories you don't want to appear in Google search results, such as WordPress /wp-admin/, members-only pages, and dynamic search result pages.
SEO efficiency (Crawl Budget Optimization)
On massive websites, restrict crawling to unimportant pages to guide Googlebot to prioritize crawling important pages (like new articles).
Hiding development/test sites
Disallow the entire site to prevent test environments during site production or before renewal from being accidentally indexed by search engines.
Send Feedback
Please let us know your thoughts to help us improve the tool.
Feedback is temporarily suspended
The server is busy or spam protection is active. Please try again later.