What is llms.txt?
llms.txt is a proposed convention: a markdown file at a site's root that lists its most important pages in a form intended for language models rather than browsers.
The idea borrows its shape from robots.txt and sitemap.xml but answers a different question. A sitemap says what exists. llms.txt says what matters — a curated, annotated list of the pages a machine reader should start from, written in plain markdown with a sentence explaining each.
Why it matters
Two honest caveats belong with any recommendation to add one. It is a convention, not a standard, and adoption by the major AI companies is partial and changing; nobody can promise a given crawler reads it. And it grants nothing and blocks nothing — a file listing your pages does not stop anyone crawling the rest, so it is not an access control and should not be treated as one. What it does cost is almost nothing: a file, a list, a sentence each. For a business whose support content is worth quoting, that is a reasonable bet on a convention that may become normal, and the exercise of deciding which twenty pages matter is worth doing regardless.
Examples
- A list pointing at the pricing, shipping, returns and support pages with a line describing each.
- An expanded variant inlining the full text of those pages for readers that want it.
- A site that has one and still gets crawled everywhere else, because it is not a fence.
Not the same as
robots.txt
robots.txt tells crawlers where not to go and is widely honoured. llms.txt suggests where to start and is not yet universally read.
sitemap.xml
A sitemap lists everything for completeness. llms.txt lists a few things for usefulness.
Related
term
What is answer engine optimization (AEO)?
Search increasingly returns synthesised answers rather than links. AEO is the unglamorous work of making sure the facts about your business — what it is, who it's for, what it costs — are stated plainly enough to be quoted correctly.
term
What is structured data?
A page's HTML says how content should look. Structured data says what it means: that this string is a price, that one is an author, that this block is a question and the next is its answer. Search engines and answer engines read it because it removes guesswork.
term
What is a web crawler?
Search engines use crawlers to build an index. AI systems use them to build a knowledge base. The mechanics are the same — start somewhere, fetch, extract links, repeat, stay inside the boundaries you were given — and so are the reasons a page ends up missing.
term
What is a web crawler?
Search engines use crawlers to build an index. AI systems use them to build a knowledge base. The mechanics are the same — start somewhere, fetch, extract links, repeat, stay inside the boundaries you were given — and so are the reasons a page ends up missing.
RubyRep is this, applied to your website.
Give it your URL and it reads your pages, then answers your customers from them.
Train My RubyRep