ListMySaaSListMySaaS

Technical SEO · /tools/robots-txt-tester

Robots.txt tester

Check whether Googlebot, Bingbot, GPTBot, ClaudeBot or any crawler may fetch a URL, and see the exact rule that decides. Edit the file to test a fix before you ship it.

0 groups · 0 sitemaps

Group applied: none (allow all)
  • Allowed/

    No group matches this crawler, so everything is allowed.

  • Allowed/admin

    No group matches this crawler, so everything is allowed.

  • Allowed/blog/post?ref=x

    No group matches this crawler, so everything is allowed.

Crawler access for /

CrawlerUsed byGroupResult
GooglebotGoogle Search Search—Allowed
Googlebot-ImageGoogle Images Search—Allowed
BingbotBing, Copilot Search—Allowed
ApplebotApple, Siri Search—Allowed
DuckDuckBotDuckDuckGo Search—Allowed
GPTBotOpenAI training AI—Allowed
OAI-SearchBotChatGPT search AI—Allowed
ClaudeBotAnthropic training AI—Allowed
Claude-SearchBotClaude search AI—Allowed
PerplexityBotPerplexity AI—Allowed
Google-ExtendedGemini training AI—Allowed
CCBotCommon Crawl AI—Allowed
BytespiderByteDance AI—Allowed

The robots.txt tester fetches a site's robots.txt and tells you whether a given crawler may fetch a given URL, with the exact rule that decides. It follows the rules Google documents and RFC 9309 standardised: the most specific group for the crawler applies, the longest matching path wins, and Allow wins a tie.

Test search engines like Googlebot and Bingbot, and AI crawlers like GPTBot, ClaudeBot and PerplexityBot. You can edit the robots.txt in the page to try a change before you deploy it.

How it works

  1. Step 1

    Enter a domain

    We fetch /robots.txt from the site, the way a crawler would.

  2. Step 2

    Pick a crawler and URLs

    Choose Googlebot, Bingbot, GPTBot, ClaudeBot or a custom user agent, and list the paths to test.

  3. Step 3

    See allowed or blocked

    Each URL shows Allowed or Blocked and the line that decided it. Edit the file to test a fix.

FAQ

How does Google choose between Allow and Disallow?
The rule with the longest matching path wins. If an Allow and a Disallow match with the same length, Allow wins.
Which group applies to Googlebot?
The group whose User-agent line matches the crawler most specifically. If there is no group for Googlebot, the * group applies. Groups are not combined with *.
What do * and $ mean in robots.txt?
* matches any sequence of characters. $ at the end of a rule means the URL must end there, so Disallow: /*.pdf$ blocks PDFs only.
Does robots.txt stop a page from being indexed?
No. It stops crawling. A blocked URL can still be indexed from links pointing to it. Use a noindex tag on a crawlable page to keep it out of search.
What happens if robots.txt returns 404?
Crawlers treat a missing robots.txt as allow everything. Server errors (5xx) make Google pause crawling the site until the file is reachable again.

More free SEO tools

ListMySaaS Weekly

Spot the next great SaaS

One email every Monday: 5 fresh SaaS launches, 3 tools trending on ListMySaaS, and 1 underrated pick worth trying.

Free, every Monday. Unsubscribe anytime. Privacy