From the course: Technical WordPress SEO (2019)
Join today to access over 22,600 courses taught by industry experts.
Testing robots.txt - WordPress Tutorial
From the course: Technical WordPress SEO (2019)
Testing robots.txt
- [Instructor] An important aspect of managing your robots.txt is, well, testing it. So let's say that you've made some changes and now you want to make sure that the directives you've added actually work the way you've intended them to. That's where a robots.txt test tool comes in handy. Here on the robots.txt file for Salt and Simple and you'll see that we disallow wp-admin, but we allow admin-ajax.php from within that directory. Let's test to see if this is working. I'll start by going to technicalseo.com/tools and then I'm going to choose the robots.txt Tester. What we need to do is enter in the URL that we want to test to see if it's blocked or allowed. So we'll enter in https://saltandsimple.com/wp-admin. We can select a user agent, say Googlebot, and then choose Test. Okay, so we can see that this URL will not be allowed for crawling and it's highlighting the directive that is enabling that. So if you were expecting…
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.