Your redirect map is due Friday and the only URL list you have is the one in your head. Miss a page and the migration ships with a 404 where a product page used to be, which you find out about from a customer rather than a report. Search engines see hundreds of URLs you have never opened.
You can crawl a website for all URLs without writing a single line of code. No Python, no terminal, no paid seat. You will point a browser tool at a domain, get back every link it can reach, and export the result as CSV or TXT you can sort, filter, and hand off.
When the crawl lands as a messy CSV or an HTML dump, Word Spinner converts it into a clean file in the browser, no account, nothing to install.
What is the fastest way to get all URLs from a website?
Start by pasting the homepage into a free browser-based extractor such as the Website URL Extractor, which returns every link it can reach and needs no account. You get a direct answer in under a minute, which beats installing a desktop crawler for a one-off SEO check. If you need CSV or TXT output for a spreadsheet, SimpleScraper's URL extractor downloads both.
Switch to a desktop crawler such as Screaming Frog only when the site runs past its 500-URL free ceiling or hides links behind JavaScript.
What you'll need before you crawl a website for all URLs
You can crawl a website for all URLs with nothing but a browser: free URL extractors run in the page and hand back a CSV or TXT file. Six things decide whether that list is usable.
- The root domain, not a deep category page. Starting mid-site hides everything above it.
- A crawl depth you accept in advance. Free extractors often cap a single run.
- JavaScript enabled. Many sites inject navigation links after the first paint; a raw HTML reader misses them.
- A place for the output, usually a CSV or TXT file you can open in Excel or Google Sheets.
- A rough page count to tell a complete crawl from a truncated one.
- Permission, which matters most on client domains you do not own.
The count you get back is not the number of pages you have. Faceted navigation, tag archives, and tracking parameters inflate a 900-page site into 4,000 URLs. On a 40-page brochure site you can work straight from the raw list; past a few thousand rows, remove the parameters first or you will audit the same page a dozen times.
How to crawl a website for all URLs, step by step
You need the site's root address and a decision about scope before you start. Everything else happens in the browser, and none of it requires an install.
- Paste the root address and let the crawler follow internal links: Start at the homepage rather than a deep page, because a crawler discovers URLs by reading the links on each page it loads and queueing the ones it has not seen yet. That is the same approach recommended in scraping threads on finding every URL: request the main page, pull the links, then repeat the process on every new page. Set the crawl depth to 3 or 4 for a typical marketing site, since a depth of 1 returns only what the homepage links to. Run it twice on a large site: once at depth 2 to see the shape of the site, then again at depth 5 once you know which sections matter. Visible result: a table of discovered URLs, each with the page it was found on.
- Set the scope before you run the crawl: Decide whether the crawl stays on one hostname, includes subdomains, or follows links out to other domains. Leave external links off unless you specifically want them, because a crawl that follows every outbound link to social platforms, review sites, and news articles returns thousands of URLs you cannot act on. Query strings need a decision too. A store with filter and sort options can serve the same product page under dozens of parameter combinations, and each combination looks like a separate URL in the export. Most crawlers let you exclude paths by pattern, which is faster than filtering thousands of rows afterward. Visible result: a URL count in the hundreds rather than the tens of thousands.
- Read the status codes before you read the URLs: Sort the export by response code. A 200 means the page was served normally. A 301 or 302 means the URL redirects, and a chain of three or four hops wastes crawl time and dilutes whatever link value the page had. A 404 means something on the site still links to a page that no longer exists. A 403 usually means the server refused the crawler, which happens often on sites sitting behind bot protection. Watch for soft 404s as well: a page that returns 200 but shows an empty category or a "no results" message passes a status check and is still dead weight. Visible result: a shortlist of broken and redirected URLs you can fix in an afternoon.
- Export to CSV and collapse the duplicate variants: The same page often shows up several times in a raw export: with and without a trailing slash, with uppercase and lowercase paths, over http and https, with and without www, and with tracking parameters such as utm_source. Normalize those variants down to one row per page before you count anything, because a raw total that includes duplicates will not match any other number you have. Two people running the same crawl on the same day can get different totals if one filters parameters and the other does not, so write down the settings you used. Visible result: a list where each page appears exactly once. If you would rather ask questions about the file than scroll through it, the AI chat with your website data tool lets you query an exported file in plain language.
- Sort by depth and referring page to find what is buried: Crawl depth tells you how many links deep a page sits from the homepage. Pages at depth 1 are linked from the homepage and get found; pages at depth 5 or 6 are effectively invisible to visitors and get revisited less often by search engines. If a page has a single internal link pointing at it, that link is the only path a crawler has to reach it. Depth is not the same as importance, but it is a decent proxy for how much internal linking a page receives. A crawl also answers a different question than an XML sitemap does: the sitemap lists the pages an owner wants indexed, while the crawl shows which pages are actually reachable by following links. Visible result: a ranked list of pages that exist but that almost nothing points to.
- Turn the export into a working file: Group the URLs by folder, sort each group by depth, and add two columns: who owns the page and what happens to it next. A 200 page with thin content goes into a rewrite queue, a 301 goes into a redirect map, and a 404 goes onto a fix list. Sort the finished file by action rather than by URL, so the fix list and the rewrite queue do not get mixed together. This is where Word Spinner fits: the free tools run in the browser with no account or installation, and the paid platform handles the long-form writing and rewriting that the rewrite queue creates. Visible result: a spreadsheet where every URL has an owner and an action.
The most common mistake is running the crawl with the default scope and treating the raw count as the page total. Filter the duplicate variants first, or you will spend a week chasing URLs that were never separate pages.
Crawl website for all URLs: the implementation checklist
Run these six checks before you trust any exported URL list, because a crawl returns whatever the crawler could reach, not everything your site actually contains.
| Task | Why it matters | Done |
|---|---|---|
| Run the crawler on a page whose navigation loads after JavaScript runs, then confirm the menu and footer links appear in the results. | A plain HTML fetch only sees links already present in the source, so JavaScript-heavy sites come back with a list that is missing entire sections of the navigation, including the footer links most sites rely on. | |
| Set the crawl scope to one exact host, and make a separate decision about whether subdomains belong in the same export or in a second run. | Mixing two hosts into one file makes grouping and deduplication harder, and it hides the fact that a subdomain often runs on a different platform with its own URL rules and its own redirects. | |
| Add the XML sitemap as a second seed alongside the homepage, then compare the sitemap URLs against the list the link crawl produced. | Pages with no internal links pointing at them, which SEOs call orphan pages, appear in the sitemap and nowhere else, so a link-only crawl never surfaces them no matter how deep it goes. | |
| Filter the export down to HTML pages by removing image, stylesheet, script, font, and document file extensions, then sort what is left by path depth. | Asset files usually outnumber real pages several times over, and leaving them in buries the rows you need to review under hundreds of .jpg and .css entries. | |
| Normalize every URL before you count anything: strip trailing slashes, lowercase the host, and remove tracking parameters such as utm_source and fbclid. | One page reached through three different link paths otherwise appears as three separate rows, which inflates your total and hides the gaps you were looking for in the first place. | |
| Open ten random rows from the finished file and confirm each one loads without redirecting to a different address, then note how many of the ten needed a second hop. | A crawler reports what it found, not what works, so dead links, soft 404s, and multi-hop redirect chains all survive into your final list unless you check. |
The two failures that waste the most time are scope and normalization. A crawl that quietly includes a staging subdomain, or that counts a page with and without a trailing slash as two pages, produces a total that looks authoritative and is wrong. Print this table, run it against one site, and both problems show up in under ten minutes, usually before you have opened a spreadsheet.
The fix is normally one setting in the crawler, not a second crawl.
Re-run the checks after every batch of publishes rather than once a quarter. New pages are the ones most likely to have nothing linking to them yet, and old pages are the ones most likely to have picked up a redirect along the way. Keep the previous export so you can diff the two lists and see exactly which URLs appeared. A crawl is a snapshot of a moving site, and it goes stale the moment you ship.
If you want the rest of the workflow to stay browser-based too, the free tools compared in this roundup of AI writing tools run in a tab without an account.
The gap between sitemap and crawl
A crawl result is not the site. It is the set of pages a crawler could reach by following links from the homepage, minus anything it skipped, and the gap between that list and the site's XML sitemap is where the useful information lives.
Pages in the sitemap but missing from the crawl are usually orphaned, excluded by a robots file, or rendered by JavaScript after load. Pages in the crawl but missing from the sitemap are usually pagination, filters, or tracking parameters, which is why an SEO crawl often reports several times more URLs than the site has real pages. Neither list is complete on its own; the union is what you actually want.
The usual advice to just crawl the site changes once your page count passes a few hundred. Below that, a single-page link extractor that reads one URL and returns every link on it is enough. Above it, you want a crawler that queues each new URL it finds and keeps going, with a depth cap so a filter parameter cannot send it into the thousands.
Frequently asked questions about crawling a website for all URLs
Can you crawl a website for all URLs without installing anything?
Yes. Free browser tools such as SiteGPT's URL extractor and SimpleScraper take a domain, follow the internal links they find, and return a downloadable list. You paste the address, run the scan, and export the results as CSV or TXT. You install nothing, and most skip the sign-up step entirely.
Why does a crawl miss pages that exist on the site?
Pages that nothing links to are the usual reason. A crawler moves by following links, so a page reachable only through a search box, a JavaScript dropdown, or a form submission never enters the queue. Orphan pages with no internal links behave the same way, and a crawl and an XML sitemap rarely match.
Do you need an XML sitemap to find every URL?
No, but it makes the job easier. An XML sitemap is a file a site publishes listing the pages it wants found, and crawlers treat it as a starting list rather than a full inventory. Crawling the live site catches pages the sitemap omits; the sitemap catches pages nothing links to.
How many URLs can a free crawler return?
It depends on the tool's page ceiling, not on your site. Diffbot's Crawly stops at 500 URLs per run, and other free extractors cap their scans in similar territory. On a 5,000-page store you will need several passes or a paid plan. Check the stated limit first, because a truncated crawl looks like a complete one.
What should you do with the list once you have it?
Clean it before you trust the count. A raw crawl list contains redirects, paginated duplicates, and tracking parameters that inflate the total, so sort it, drop the repeats, and check each URL's status. The cleaned CSV works as a monthly inventory. Word Spinner converts that CSV into other formats in the browser if you need a different file type.
Final verdict on crawling a website for all URLs
Run one browser-based crawler, export the CSV, and work the list with broken links first. That was the whole job, and it never needed a script.
The crawl is the easy half. Word Spinner is where we would take the list next: its free browser tools need no installation, so you can convert the exported file or rewrite the pages the crawl flagged, then move on. Keep the export, because it is a month of work.
Choose a scheduled crawler instead only if you are tracking hundreds of thousands of pages on a recurring basis; free browser tools handle one-off audits fine. If the list shows you need a site to point those URLs at, this guide to building a free website covers the hosting side.