What is a web crawler?
A crawler is a program that fetches web pages automatically and follows the links it finds, building a collection of a site's content.
Search engines use crawlers to build an index. AI systems use them to build a knowledge base. The mechanics are the same — start somewhere, fetch, extract links, repeat, stay inside the boundaries you were given — and so are the reasons a page ends up missing.
Why it matters
Four things decide whether a page is seen, and they fail in order. Reachability comes first: a page with no link pointing at it and no sitemap entry does not exist as far as a crawler is concerned, however good it is. Then robots rules — a disallowed path is skipped whether or not anything valuable is there. Then rendering: content that only appears after JavaScript runs is invisible to any crawler that does not execute it, which is many of them. Finally budget, which is why large sites find their thinnest pages crawled and their newest ones ignored. Diagnosing a missing page means checking those four in that sequence, because a rendering fix does nothing for a page nothing links to.
Examples
- An orphan page with no internal links, never fetched.
- A returns policy under a disallowed /account/ path, skipped.
- A JavaScript-rendered FAQ that a simple fetcher receives as an empty div.
Not the same as
Scraper
Usually a matter of intent and manners. A crawler follows robots rules and rate limits; a scraper is the word people use when it does not.
Indexing
Crawling fetches a page. Indexing decides whether to store it and show it. A page can be crawled and never indexed.
Related
term
What is a knowledge base?
Traditionally a knowledge base is a help centre humans read. For AI support it is the source material a representative answers from — which means an ordinary website, well written, can serve as one without a separate help centre.
term
What is llms.txt?
The idea borrows its shape from robots.txt and sitemap.xml but answers a different question. A sitemap says what exists. llms.txt says what matters — a curated, annotated list of the pages a machine reader should start from, written in plain markdown with a sentence explaining each.
term
What is structured data?
A page's HTML says how content should look. Structured data says what it means: that this string is a price, that one is an author, that this block is a question and the next is its answer. Search engines and answer engines read it because it removes guesswork.
tool
Website support score
Scores a website on how well it could support customers — and how well an AI representative trained on it would perform. Five dimensions, a total out of 100, and the specific gaps behind the score.
term
What is llms.txt?
The idea borrows its shape from robots.txt and sitemap.xml but answers a different question. A sitemap says what exists. llms.txt says what matters — a curated, annotated list of the pages a machine reader should start from, written in plain markdown with a sentence explaining each.
RubyRep is this, applied to your website.
Give it your URL and it reads your pages, then answers your customers from them.
Train My RubyRep