Changelog

Crawl

• Multi-threaded crawler that follows redirects hop by hop, respects robots.txt (wildcards and crawl-delay included) and stores everything in SQLite, so an interrupted crawl resumes where it stopped.
• Spider and List modes, XML sitemaps, limits for URLs and depth, and — Pro — scheduled crawls that run on their own.
• JavaScript rendering (Pro) with an off-screen WebKit view, screenshots, and a side-by-side of the raw HTML and what Googlebot renders.
• Large crawls stay fast — with 150,000 URLs, counting rows takes 9 ms and jumping to the end 52 ms.

Audit

• Indexability explained with the exact reason a page cannot be indexed.
• Canonicals and directives — self-referencing, canonicalised, missing, conflicting and HTTP-header canonicals; meta robots and X-Robots-Tag with a filter for every directive.
• Hreflang — missing return links, wrong language or region codes (en-UK, jp…), missing self-reference and x-default.
• Titles, descriptions and H1–H6, with pixel widths, skipped heading levels and the full outline of each page; a SERP snippet drawn the way Google shows it.
• Images and links — missing or empty alt, images without dimensions, generic anchors such as "click here", nofollow internal links and deep pages.
• Structured data validated against Google's rich-result requirements, accessibility checked with axe-core, and duplicate and near-duplicate content.
• Issues panel that ranks what to fix first, and Page Tests with Rich Results and Lighthouse for a single URL.

Your data, cross-checked (Pro)

• Search Console, Analytics and PageSpeed next to every URL, with filters for quick wins (positions 4–20 with demand), low CTR, indexable pages nobody sees and non-indexable pages that still get traffic.
• Keyword cannibalisation — URLs of your site competing for the same search query.
• Orphan pages — URLs that Search Console, Analytics or the sitemap know about but no internal link reaches.

Automation and export

• spider-cli to crawl from the terminal or a CI job, and spider-mcp so AI agents can run crawls and read results — set up for 16 clients (Claude Desktop, Cursor, VS Code, Windsurf, Zed…) from Settings with one click.
• Export any tab with its filter to CSV, the crawl graph to JSON and — Pro — to Excel; compare two crawls (Pro).

Free and Pro

• Free: the whole crawler up to 500 URLs per audit, indexability, on-page and structured-data checks, the site audit (robots.txt, sitemaps, llms.txt), the CLI and the MCP server.
• Pro: no limit on URLs, JavaScript rendering and custom extraction, Search Console, Analytics and PageSpeed data, crawl comparison and scheduling, and Excel export — a monthly or yearly subscription through the Mac App Store, restored and managed from your Apple account.