Join David Booth for an in-depth discussion in this video Understanding how search engines index content, part of SEO Foundations.
- It's important to understand how search engines discover new content on the web as well as how they interpret the locations of these pages. One way that search engines identify new content is by following links. Much like you and I will click through links to go from one page to the next, search engines do the exact same thing to find an indexed content. Only they click on every link they can find. If you wanna make sure that search engines pick up your new content, an easy thing to do is just make sure you have links pointing to it.
Another way for search engines to discover new content is from an XML sitemap. An XML sitemap is really just a listing of your page's content in a special format that search engines can easily read through. You or your webmaster can learn more about this specific syntax and how to create these XML sitemaps by visiting sitemaps.org. Once you've generated your sitemaps, you can submit them directly to the search engines. And this gives you one more way to let them know when you add or change things on your site. Search engines will always try to crawl your links for as much additional content as they can find to index.
for as much additional content as they can find to index. While this is generally a good thing, there are plenty of times that you might have pages up that you don't want search engines to find. Think of your test pages or members-only areas of your website that you don't want showing up on search engine's results pages. To control how search engines crawl through your website, you can set rules in what's called a robots.txt file. This is a file that you or your webmaster can create in the main root folder of your site and when search engines see it, they'll read it and follow those rules that you've set.
You can set rules that are specific to different browsers and search engine crawlers. And you can specify which areas of your website that they can and can't see. This can get a bit technical. You can learn more about creating robots.txt rules by visiting robotstxt.org. Again, once search engines discover your content, they'll index it by URLs. URLs are basically the locations of web pages on the internet. It's important that each page on your site has a single unique URL, so that search engines can differentiate that page from all the others.
can differentiate that page from all the others. And the structure of this URL can also help them understand the structure of your entire website. There are lots of ways that search engines can find your pages. And while you can't control exactly how the crawlers do their job, by creating links for them to follow, unique and structured URLS, sitemaps for them to read, and robots.txt files to guide them, you'll be doing everything you can to get your pages in the index as fast as possible.
This course is part of a Learning Path approved by the American Marketing Association.
Gain the skills you need to become an AMA Professional Certified Marketer (PCM) in Digital Marketing by using the industry-leading courses and resources in the Learning Path. Take the AMA certification exam to show that you have what it takes to lead the digital transformation.
- What is SEO?
- Understanding how search engines index content
- Researching keywords
- Using SEO tools
- Optimizing pages for keywords
- Optimizing code and site structure
- Building links to your content
- Optimizing nontext components of a webpage
- Analyzing content quality
- Defining your audience, topics, angle, and style
- Promoting your content via social media
- Measuring SEO effectiveness
- Setting up Google+ Local
- Optimizing ecommerce sites for search
- Configuring sites for mobile