Screaming Frog SEO Spider Website Crawler

The Screaming Frog SEO Spider is a fast and advanced SEO site audit tool. It can be used to crawl both small and large websites, where manually checking every page would be extremely labour intensive, and where you can easily miss a redirect, missing page title, or duplicate page issue. You can view, analyse and filter the crawl data as it’s gathered and updated in real-time in the apps UI.

The SEO Spider allows you to export key onsite SEO elements (URL, page title, meta description, headings etc) to a spread sheet, so it can easily be used as a base for SEO recommendations. Check our out demo video above.

The ‘lite’ version of the tool is free to download and use. However, this version is restricted to crawling up to 500 URLs in a single crawl and it does not give you full access to the configuration, saving of crawls, or advanced features such as JavaScript rendering, custom extraction, Google Analytics integration and much more. You can crawl 500 URLs from the same website, or as many websites as you like, as many times as you like, though!

For just £199 per year you can purchase a licence, which removes the 500 URL crawl limit, allows you to save crawls, and opens up the spider’s configuration options and advanced features.

Alternatively hit the ‘buy a licence’ button in the SEO Spider to buy a licence after downloading and trialing the software.

The SEO Spider crawls sites like Googlebot discovering hyperlinks in the HTML using a breadth-first algorithm. It uses a configurable hybrid storage engine, able to save data in RAM and disk to crawl large websites. By default it will only crawl the raw HTML of a website, but it can also render web pages using headless Chromium to discover content and links.

For more guidance and tips on how to use the Screaming Frog SEO crawler –

Keep updated with future releases by subscribing to RSS feed, our mailing list below and following us on Twitter @screamingfrog.

If you have any technical problems, feedback or feature requests for the SEO Spider, then please just contact us via our support. We regularly update the SEO Spider and currently have lots of new features in development!