Free robots.txt tester for search and AI crawlers.
Fetch any site robots.txt, inspect its rules and test whether a specific path is allowed for Googlebot, Bingbot or AI crawlers such as GPTBot, ClaudeBot and PerplexityBot.
Test robots.txt rules
Enter a domain and optionally a path to test against a crawler.
What the robots.txt tester shows.
A single wrong line in robots.txt can block your whole site, or let crawlers into places they should not go.
File status
Whether robots.txt exists, returns 200, its size and the sitemap directives it declares.
- HTTP status
- Size check
- Sitemap lines
Rule testing
Allowed or blocked for a chosen user-agent and path, with the exact matching rule shown.
- Most-specific rule wins
- Allow versus Disallow ties
- Wildcards and $
AI crawler control
Test how GPTBot, ClaudeBot, PerplexityBot and Google-Extended are treated, to decide whether AI engines can read your content.
- AI user-agents
- Group matching
- Full file view
Three steps to useful results.
Enter the domain
We fetch /robots.txt from the root of the site.
Pick a crawler and path
Choose a user-agent and enter a URL path to see whether it is allowed.
Adjust and retest
Edit your robots.txt, publish and run the test again.
Robots.txt Tester FAQ
Common questions about this tool and what the results mean.
Ask us about your resultsWhat does robots.txt do?
It tells crawlers which paths they may fetch. It controls crawling, not indexing, so a blocked URL can still appear in results if other pages link to it.
Should I block AI crawlers?
It depends on your goals. Blocking prevents some AI systems from reading your pages but can reduce AI citations. Read our guide to AI SEO before deciding.
Does Allow override Disallow?
The most specific, longest matching rule wins. If an Allow and a Disallow match with equal length, Allow wins.
Where must robots.txt live?
At the root of the host, for example example.com/robots.txt. Each subdomain needs its own file.
Read our guide
Want an expert to fix what the tool finds?
Our team turns diagnostics into a prioritized roadmap and does the work. Start with a free audit request.