You know you need to check your site for broken links and redirect chains, but a manual page-by-page click test doesn't scale. That is where a link crawler comes in: it automates the process, visiting every reachable URL and reporting back exactly what it finds.
You will see which links are broken, which pages have no inbound links, and how to fix the problems. To follow the steps in practice, try the free Word Spinner link crawler, no account required.
What is a link crawler?
The best way to understand a link crawler is to use the Link Crawler Chrome extension: it scans any web page and lists every link on it, letting you filter and export them. It is free, requires no account, and works directly in your browser. For deeper technical audits, SiteOne Crawler is a strong open-source alternative that also checks HTML and broken links.
What you actually need
Most link crawler guides assume you need a paid server tool, but for a single-page audit the free Link Crawler Chrome extension handles it. Here is what you need before starting:
- A modern browser (Chrome, Edge, or Brave)
- The Link Crawler extension installed
- The target URL of the page you want to crawl
- Patience: crawling a page with hundreds of links takes 10, 20 seconds
- A spreadsheet or text editor ready for the export list
If you are crawling an entire site with thousands of URLs, switch to a server-side crawler like SiteOne Crawler instead.
How to run a link crawl from start to finish
Before you begin, you need two things: a starting URL to scan and a crawler tool. The free Link Crawler Chrome extension handles single-page audits instantly. For deeper site-wide scans, a dedicated web crawler gives you more control over scope and depth.
- Choose your crawler for the job: For a fast check of one page, install the Link Crawler extension and open it on your target page. It lists every link on that page immediately. For a whole-site audit, use a full crawler such as Website Crawler (real-time analysis with technical issue detection) or SiteOne Crawler (open-source, deep crawl with HTML audit and broken-link checking). Another option is the Website URL Extractor Crawler from Word Spinner, which pulls all URLs from a page or domain without requiring an extension install.
- Enter the starting URL and set scan scope: Paste the full URL into your crawler's input field. For single-page tools, the scope is that page only. For full-site crawlers, you usually set a crawl depth (how many levels of links to follow) and a domain filter (crawl only pages on the same domain). The URL crawler from Redirection lets you enter a URL and get a fast analysis of quality, performance, and content issues. The visible result is a list of discovered URLs, often grouped by status code.
- Run the scan and inspect the raw output: Click "analyze" or "start crawl." The crawler fetches each URL and records its HTTP status code. You see a live or near-live table of URLs with columns for status (200, 301, 404, etc.), link type (internal, external, image, script), and page title. This raw output is the foundation of everything you do next. Scan through it to get a sense of the site's link health before filtering.
- Filter for what actually matters: Most crawls produce hundreds or thousands of URLs. Your goal is to surface only the problematic or high-value ones. Filter by status code to see 404s and 301 redirects. Filter by link type to find external links that might point to dead pages. Many crawlers let you search for specific patterns in the URL string or exclude known-good paths. This step turns noise into an action list you can actually work through.
- Check internal link structure: Use an internal link checker like Word Spinner's Internal Link Checker to validate that your on-site links point to live, relevant pages. A crawl shows you every internal link on the site; the checker tells you whether each destination actually resolves and whether the anchor text is meaningful. This is where you catch orphan pages and broken navigation that a full crawl alone might bury.
- Export the data and fix the findings: Download the filtered results as a CSV or spreadsheet. Each row is a link with its source page, destination URL, and status code. Work through the list systematically: update or replace broken links, remove dead pages, and set proper redirects where needed. For a deeper understanding of why inbound link health matters and how to approach it, see the Inbound Link Checker explained guide.
A common mistake is skipping the filter step. Running a crawl and looking at the full unfiltered list rarely surfaces the actionable issues. Always narrow your view to broken links, redirect chains, or external links before exporting.
Link crawler checklist: what to verify after the crawl
A finished crawl report lists every link the crawler found, but it does not tell you which ones need fixing. That distinction is your job. Work through these six checks in the order shown. Each one targets a failure that the raw data will surface but not resolve for you.
Mark a row done only after you have confirmed the pass condition by inspecting the actual report output. Do not skip a row even if you think the category is clean, because a single missed issue can cascade into larger problems later.
| Task | What to inspect | Done |
|---|---|---|
| Confirm all internal pages were found | Compare the crawled URL list against your sitemap or known site structure. Any missing page means the crawler never reached it, often because no internal link points to it or a redirect chain skipped over it. | - |
| Flag broken internal links | Filter the report for HTTP 4xx and 5xx status codes. Every broken link sends users and crawlers to a dead page, wasting link equity and trust. | - |
| Identify long redirect chains | Look for URLs that redirect two or more times before reaching the final destination. Each hop adds page load time and passes less authority than a direct link. | - |
| Validate external links | Spot-check 5-10 outbound links from the export to confirm they resolve to the correct page. A link to a removed external resource damages your credibility with readers. | - |
| Check for crawl blocks | Review entries marked as blocked by Robots, meta nofollow, or a noindex directive. Pages you intended to audit may be invisible to the crawler and need an exception. | - |
| Verify crawl coverage depth | Confirm the crawler visited roughly the number of pages you expected based on site size. A short count usually means a configuration issue or an early stop. | - |
Each row maps to a specific, fixable issue. Resolve every failure in a row before moving to the next one. A clean pass means zero unexpected errors across all six categories.
One last tip: after you finish all six checks, run the crawl once more. A single pass is a snapshot; a second pass proves your fixes held and nothing new broke.
Expert tip: when your link crawler misses JavaScript links
The free Link Crawler Chrome extension only scans static HTML. If your page loads links inside a JavaScript-rendered accordion or infinite scroll, the crawler will never see them. The usual advice, run a quick link check, fails here because you get a false sense of completeness. For pages that depend on JavaScript, switch to a crawler that supports JavaScript rendering, like SiteOne Crawler (which offers a headless browser option) or Redirection's service.
The tradeoff: JavaScript rendering is slower, so for static pages you're better off with the faster HTML-only scan.
Frequently asked questions about link crawlers
How long does a link crawler take to crawl a full website?
A browser extension like Link Crawler finishes a single page in seconds. A full-site tool like SiteOne Crawler or the free Websitecrawler can take anywhere from 30 seconds for a 10-page site to over 10 minutes for thousands of pages. The bottleneck is page load time: slower hosts and JavaScript-heavy pages increase crawl duration.
Can a link crawler find broken links automatically?
It reports HTTP status codes for every link. Any link returning a 404 Not Found or 410 Gone status is a broken link that needs fixing. The crawler flags these by status code, but you still decide whether to redirect, update, or remove each one.
What is the difference between a link crawler and a sitemap generator?
A link crawler discovers every link it can reach by following hyperlinks from a starting page. A sitemap generator lists the URLs you tell it to include, typically as an XML file for search engines. The crawler finds what actually exists on your site; the sitemap declares what you want found.
Does a link crawler work on JavaScript-heavy sites?
Free browser extensions like Link Crawler see only the static HTML. For single-page applications rendered in JavaScript, you need a headless crawler such as SiteOne Crawler (free and open-source) or a paid tool that executes JavaScript before scanning links.
Do I need a link crawler if I already use Google Search Console?
Search Console only shows URLs that Google has indexed or tried to crawl. A link crawler covers every page regardless of indexing status, including staging environments and pages blocked from search engines. The two tools complement each other rather than replace one another.
Can I use a link crawler on someone else's website?
You can crawl any publicly accessible URL. Ethical practice means respecting the site's `Robots` rules and keeping request volume low. Crawling a competitor's public site does not violate any law, but high-volume scraping may break their terms of service.
Next steps with your link crawler
For most site audits, the free Link Crawler Chrome extension is all you need, it handles static HTML pages quickly and exports clean link lists. But if your site relies heavily on JavaScript-rendered content, that browser extension will miss links that appear only after scripts run. In that case, switch to Word Spinner's AI chat with websites tool, which can parse dynamic pages and flag issues the basic crawler overlooks.
What should you do next? Start by running a crawl on your five most important pages. Check the results against the checklist from this guide, fix any broken or redirected links, then move on to deeper pages. Repeat the process weekly to catch problems early.