We use a variety of tools to help us do our job as efficiently as possible. One of those tools is Screaming Frog. I keep finding new uses for this rather impressive piece of software and it saves a lot of time.

Post transition checks / redirect audits

When transitioning a site, one of the most common errors we come across is not setting up redirects or setting up the wrong type of 3XX redirect. A 301 redirect passes SEO value to the new page while a 302 does not. A 302 redirect traps the value on a page; if this page is not expected to be used again, this is a waste. This is something that every site changing the URL structure of domain names needs to be aware of but it is such a common mistake to be made.

Crawl errors

The tool can also be used to look for 400 or 500 errors, which can be looked into and addressed. This is really important as users or search engines looking for these pages will not find what they are looking for. A 400 error (most commonly 404 errors) means that a page can no longer be found, and 500 errors suggest issues with the server where the domain is hosted. Both will likely mean that users are arriving at broken/poor pages and will not have the good experience you want them to have.

Collating and checking on page elements

The tool can crawl and pull together many of the most important on page elements such as page title and meta description, giving a nice central location for review and optimisation. Something that was recently added is a SERP snippet emulator which calculates the pixel width of page titles and meta descriptions and can show how the site will show when ranking on Google.

Finding navigable pages

The spider function of Screaming Frog allows us to crawl the site and find all the URLs that are linked to from pages that are crawled. This is a great way to either:
1. Get an idea of how many pages there are on a site (the caveat here is that it will only find pages linked to internally, so any “floating” pages would not be found.)
2. See which pages are linked to and where they are linked from.

This can then be used to see if the XML site map is up-to-date or missing pages. It can also be used to get a better understanding of internal linking, it is great to find pages that are not yet benefiting from internal linking.

Creating XML site maps

Screaming Frog can be used to generate XML site maps, this is currently only supported for HTML sites so does not include images. It is best practice to upload a site map to Webmaster Tools as this is the best way to help Google find all the relevant pages on the site.

Checking Google Analytics tags are in place

Screaming Frog can check for Google Analytics tags.Today I came across a site where the Analytics tag had been added to some pages but not to others. There are a few ways to test if a GA tag is on the page, it is possible to check the page source manually or use a browser plugin to look at the tags added to a page. This is a fairly slow process, there is the possibility for human error (even for a site with only a few hundred pages this is not a quick task). With Screaming Frog, it is possible set up a custom report that looked for the UA number (the unique identifier of the Google Analytics tag) and provided a list of pages missing the correct GA tag.

Screaming Frog can also be configured to look at other elements on the page such as page titles, meta description, H tags. Another great use is looking for missing elements on a page, for example: pages missing page titles; pages with an old phone number that needs to be updated; and so on.

Backlink analysis and checks

When looking at backlink analysis, many of the tools can be slow at picking up changes in the backlink profile, if there has been activity to remove links it can be a long and time consuming task checking each and every site. This is fine if you only have a few links, but if you are monitoring hundreds or even thousands then an automated monitoring system will be very helpful. There are other tools that can help with this kind of thing such as rmoov.

These are just a few of the functions that I regularly use Screaming Frog for. There are many other good reasons to use this tool but, for me, the top reason is that it saves time. This tool allows us to spend more time on the analysis and less time doing the time consuming (and to be frank, boring) tasks. Which is a benefit to both the clients I work for and to me as I can spend more time doing the things that make a difference.

Why I love Screaming Frog:

– It is logical and relatively simple to use (considering what it does)
– It saves time, which gives us more time on the changes that make a difference
– There are not many tools that are so flexible.

So there are just a few reasons I love Screaming Frog. : )

Have you used this tool? It would be great to hear what you think about it. Please leave us a comment.