In this screencast, we are going to protect our web site against bad bots. We've seen how to do this with a plug-in in the previous screencast, but there is a better, more efficient, way to protect your site directly, using the htaccess file. Here, in our FTP/file editor, we're looking at our site's web-accessible route htaccess file. To implement this method, grab a copy of the htaccess code that's included in the exercise files of this screencast. Copy everything and then return to your file editor and paste the code beneath any existing rules.
This chunk of code is like a virtual control panel for blocking bad bots and user agents. First, we're blocking blank user agents. Then these lines here collectively block some of the worst known bad bots. Then this last section here is the part that actually does the blocking, based on what you have listed in these previous directives. And best of all, no upfront editing is required for this code to work.
Just save and upload the file to your server. To see it in action, let's return to the browser and visit this ridiculously handy user agent bot-simulation tool. First, let's just see it work by adding the URL of our web site and clicking the Go button. Here is our demo site that we're working with. So we copy the URL from the address bar and return to Bots versus Browsers and paste that URL here.
Then we click the Go button. As expected, our site is accessible when using the legit user agent, specified here. So now let's check that the code is working by spoofing a request from one of our blocked user agents, or blocked bots. Returning to our FTP editor. Let's grab a random user agent, skygrid, copy, and return to our Bots versus Browsers page. Paste it into the user agent field like so and then click Go with your site's URL still in the URL field. A 403: Forbidden means that the request has been blocked.
This is exactly the response we want to send to bad bots. It's is a simple response that's easy on the server. Using a plug-in would have required significantly more resources to deliver the same response, requiring WordPress, plug-in files, and the database just to block a bad bot. Using htaccess lets Apache just make the block directly at the server level, which is the optimal way of doing it. To add new bad bots to the list, return to your file editor, and we can either create a new line or just add the name to an existing line, like so.
Casing shouldn't matter, because of this directive here NoCase. So you can use any combination of upper- and lowercase letters and the result will be the same. Or we can instead just start a new line like so, sort of emulate the previous lines, and then put our new blocked bots on their own line.
And this is a good way to help keep things nice and organized, as you use this method to protect your site. That's all there is to it, so let's save and upload the file and return to our handy Bots versus Browser page to see it work. We type in the name of the user agent that we just added and click the Go button. That's it right there. Our request using this user agent has been blocked.
In this screencast, we've seen how to block bad bots and user agents from accessing our web site. Using htaccess instead of a plug-in, we're able to block bad bots directly, with greater efficiency and better site performance.
Get unlimited access to all courses for just $25/month.Become a member
82 Video lessons · 100996 Viewers
61 Video lessons · 87785 Viewers
71 Video lessons · 71662 Viewers
56 Video lessons · 103551 Viewers
Access exercise files from a button right under the course name.
Search within course videos and transcripts, and jump right to the results.
Remove icons showing you already watched videos if you want to start over.
Make the video wide, narrow, full-screen, or pop the player out of the page into its own window.
Your file was successfully uploaded.