From the course: Advanced SEO: Developing an SEO-Friendly Website

Unlock the full course today

Join today to access over 22,400 courses taught by industry experts or purchase this course individually.

Robots.txt

Robots.txt

From the course: Advanced SEO: Developing an SEO-Friendly Website

Start my 1-month free trial

Robots.txt

- [Narrator] The method for controlling how your crawlers access your webpage is through what's known as the robots.txt file and this is a simple text file that describes how you want the search bots to crawl your site. It will live at the root directory. So in this case, I'm on LinkedIn's robots.txt file and it's simply at LinkedIn.com/robots.txt. Now, if you have several subdomains, you'll need to add a robots.txt file for each subdomain because they're considered different hosts and they'll need to have their own robots file. Now, you'll notice that here in this file we can see that we've described, "Hey, for the user agent google bot, "I want to disallow you "from crawling any of these directories." Now, in some cases, you'll notice that we can say disallow this directory, but then later on, we can say, but you can say, "Hold on a second, I want to allow other subdirectories "that are in this directory." So, while I disallow that entire directory, I've gone ahead and using the…

Contents