Should I block AI crawlers?

It depends what you want. Blocking them keeps your content out of AI training and answers — which also means AI answer engines can't recommend you. Most businesses selling to the public want to be findable.

Does robots.txt protect private pages?

No. It's a request, not access control, and it publicly lists the paths you named. Anything genuinely private needs authentication.

Free tool

robots.txt generator

Generate a robots.txt — including whether AI crawlers may read your site.

Website address

Search enginesAllow search engines to crawl the site

AI crawlersblocking these keeps you out of AI answers too

Paths to keep out of crawlsone per line

robots.txt

User-agent: *
Allow: /
Disallow: /admin
Disallow: /cart
Disallow: /checkout

robots.txt is a request, not access control, and it publicly lists the paths you name. Anything genuinely private needs authentication.

Turn your website into an AI representative.

RubyRep learns your whole website and answers your customers on it, 24/7.

Create My RubyRep

Builds a robots.txt with sensible defaults, your sitemap declared, and explicit choices about which AI crawlers may read your site.

How it works

1
Choose whether search engines and AI crawlers are allowed.
2
Add any paths you want kept out of crawls.
3
Copy the generated file to your site root.

Questions about this tool

Should I block AI crawlers?: It depends what you want. Blocking them keeps your content out of AI training and answers — which also means AI answer engines can't recommend you. Most businesses selling to the public want to be findable.
Does robots.txt protect private pages?: No. It's a request, not access control, and it publicly lists the paths you named. Anything genuinely private needs authentication.

How it works

Questions about this tool

Related tools and reading