Technical SEO crawler
Crawl a public website from one starting URL and inspect response codes, redirects, broken internal links, crawl depth, internal linking and crawl-level indexability signals.
Live site crawl
SERPbeam follows same-host HTML links server-side and records response, redirect, architecture and crawl-level indexability signals.
Crawler guide
The first crawler release focuses on observable crawl signals. It follows same-host HTML links, records the responses it actually receives and keeps source-to-target context for broken links instead of turning every finding into a generic error list.
Start from one public URL and follow same-host anchor links breadth-first.
SERPbeam normalizes discovered HTTP(S) URLs, removes fragments and records crawl depth from the starting page. In this first version it uses source HTML rather than JavaScript rendering, so links that only appear after client-side execution may not be discovered yet.
Distinguish successful pages, redirects, client errors and server errors.
Redirects are handled explicitly so SERPbeam can record the first redirect target and hop count instead of hiding them behind automatic fetch behavior. This makes it possible to see redirected URLs and progressively build full chain and loop diagnostics.
A broken destination matters most when you can see which pages still link to it.
When a discovered internal target is actually checked and returns a 4xx or 5xx response, SERPbeam reports the source page, destination and anchor text. Unchecked URLs outside the crawl limit are not mislabeled as broken.
Report crawl-level blockers without claiming to know Google's final indexing decision.
The crawler checks robots.txt awareness, robots meta directives, X-Robots-Tag headers and HTTP response state. A clean result is labeled “No crawl-level blocker detected” rather than “Indexable” because search-engine indexing depends on signals beyond a single crawl.
Use depth, inlinks and outlinks to understand site architecture.
Each crawled page records its discovered depth plus links to and from other crawled URLs. Larger versions will use this graph for internal-link opportunity analysis, orphan-like URL detection, template issue clustering and prioritization.
This first live crawler is deliberately small while we prove the Cloudflare execution model.
The current crawl is capped at 20 URLs and standard HTML fetching. It does not yet provide queued large-site crawls, JavaScript rendering, external-link checking, sitemap comparison, persistent crawl history or full redirect-chain reporting across thousands of URLs. Those are the next crawler phases.