Free tool
Website URL extractor
Get a clean list of every page on a website.
Lists a website's internal URLs by reading its sitemap and following internal links, then separates them into pages worth reading and pages worth skipping.
How it works
- 1
We check the site's robots.txt and sitemap first.
- 2
We follow internal links from the homepage to fill in anything the sitemap missed.
- 3
You get a de-duplicated list, split into content pages and assets or utility URLs.
Questions about this tool
- Is this a full crawl?
- It's a bounded one — enough to give you a representative list quickly rather than an exhaustive index of a large site.
Related tools and reading
tool
Sitemap checker
Finds your sitemap via robots.txt or the usual locations, checks that it parses, expands sitemap indexes, and reports how many URLs it declares.
tool
Sitemap generator
Crawls a bounded sample of your site and produces a valid XML sitemap you can copy or download.
tool
Sitemap checker
Finds your sitemap via robots.txt or the usual locations, checks that it parses, expands sitemap indexes, and reports how many URLs it declares.