How To Crawl Your Site Like a Search Engine

How to Crawl Your Site title image

The SEO Spider tool from Screaming Frog is a free application that lets you crawl your site like a search engine would so you can identify any trouble spots in your site structure and take action to correct them.

The spider tool is free for up to 500 URLs of your site. If you have more than 500 URLs you’ll need to upgrade to the paid version.

In this article, we’ll cover:

What Is a Site Crawler?

But what is a site crawler, and why does it matter? As you probably already know, search engines crawl the web and index the results. If you can test your site’s “crawlability,” and see any potential pitfalls that might block or inhibit search engine bots, you can work to fix them and avoid loss of time that can risk in poor search results and delays.

It can take up to six months for changes made to your website to start showing significant search gains. Imagine if you were unaware that an important part of your site was not being crawled for six months? Here is where a site crawler can save you.

And, up to a certain number of URLs, the Screaming Frog tool is free.

How to Crawl Your Site With SEO Spider

Now it’s time to run a crawl of your site. For this part of the tutorial, just make sure that you have the primary domain of your website handy because we will use it to initiate our crawl.

  1. Open SEO Spider on your computer
  2. Fill in your URL
  3. Click Start

More SEO Resources

Check out these other resources from the support center and blog:

Make sure your business, agency, or reseller clients are always connected and powered on with our optimized Managed VPS Hosting.
Christopher Maiorana Content Writer II

Christopher Maiorana joined the InMotion community team in 2015 and regularly dispenses tips and tricks in the Support Center, Community Q&A, and the InMotion Hosting Blog.

More Articles by Christopher

Was this article helpful? Join the conversation!

Leap into Savings
Leap Ahead with Limited Time Deals