From the course: Technical WordPress SEO (2019)

The initial crawl - WordPress Tutorial

From the course: Technical WordPress SEO (2019)

Start my 1-month free trial Buy for my team

The initial crawl

“

- [Instructor] I like to start every SEO project by crawling the site, and I use Screaming Frog SEO Crawler to do just that. This tool allow me to quickly identify how the site's structured, all of the meta descriptions, meta keywords, what heading tags are being used, and so on. So I'll start by adding our URL in the top bar here, and then I'll select start. It's this output that we'll use to begin understanding what areas of our site we'll need to improve. Let's take a look at this together. So in the upper left-hand corner, you'll see that I'm on the internal tab, and this is showing me all of the internal pages to our site. The external tab will show us all of the calls that are being made to pages outside of our domain. Now, saltandsimple doesn't link to any other websites, but it does use a variety of CSS and Java Script packages, which is what we're seeing here. If we got back to internal, we have the option to filter. Right now, we're seeing every type of content, from the Java Script, to the CSS, to the images. It's more helpful if we break this down into the individual components, so we'll start by filtering just the HTML. What we see in the left-hand side is the address, and the type of content, in this case it's HTML since we filtered it, and then we have the status code. So, first, we see that there is a 301 status code, and if I select that, I can see down in the bottom bar that the address is saltandsimple.com, the status was 301, which is moved permanently, and it was redirected to HTTPS saltandsimple.com. So right away I can see that our HTTPS redirect is working, and if you do have HTTPS set up, I always start by adding just the base domain, so that I can identify whether that redirect is working. So we can see we have a 301, 200s, and then we have a 302 here, which is the checkout page, and what I can see is that this checkout page is redirecting to cart. Now that 302 might be problematic, however, the checkout page is not a page that I'm interested in indexing, and we can see that because it is identified as non-indexable in the indexability column. So my filter by indexabilty, we can see all of the pages that are non-indexable, and this is helpful because if we had a page that was shown here that should be indexed, we'd know we have a problem. And we can see why it's not indexable, in this case the indexability status says this one's been redirected, and this is set as no index. From here we have a preliminary understanding of what's going on under the hood, and here on the right-hand side we can start to understand each of the SEO elements, and Screaming Frog breaks them down into areas to help us navigate. So we see the internal elements, and we started with our HTML, but we could begin to evaluate each section here. We have the external elements, protocol, our response code, and this is where we start to begin our true technical audit. So we can start by evaluating anything that's been blocked by robots.txt. This is helpful in understanding if our content is not going to be indexed by Google. We can see all of our redirections, we could see if we had any 404 errors that needed to be resolved. We can see if we have issues with our URLs, and then, our page titles. Here we can see where we have duplicate page titles, short page titles, long page titles, and so on. Each of these areas represents something that we'd like to review and evaluate. This tool provides a ton of incredible resources, so I encourage you to evaluate your own site using it, and see what you discover.

Contents