Does blocking in robots.txt remove a page from search?

No — and this trips people up constantly. Blocking stops the page being crawled, but a blocked URL can still be indexed from external links, just without a description. To remove a page, allow crawling and use a noindex tag.

Should I block AI crawlers?

That is a business decision, not a technical one. Blocking them keeps your content out of AI answers — including the answers where a customer is trying to find you.

Free tool

robots.txt tester

Enter a full URL and we'll tell you if crawlers are allowed to fetch it.

Fetches the live robots.txt and tests whether a specific path is crawlable, using longest-match rules the way real crawlers do. Also lists every sitemap the file declares.

How it works

1
We fetch robots.txt from the origin of the URL you give us.
2
The path is matched against the rules using longest-match precedence, which is what search engines actually apply.
3
Any Sitemap: lines are extracted and listed.

Questions about this tool

Does blocking in robots.txt remove a page from search?: No — and this trips people up constantly. Blocking stops the page being crawled, but a blocked URL can still be indexed from external links, just without a description. To remove a page, allow crawling and use a noindex tag.
Should I block AI crawlers?: That is a business decision, not a technical one. Blocking them keeps your content out of AI answers — including the answers where a customer is trying to find you.

How it works

Questions about this tool

Related tools and reading