From the course: Web Scraping with Python
Join today to access over 22,600 courses taught by industry experts or purchase this course individually.
Recording data - Python Tutorial
From the course: Web Scraping with Python
Recording data
- In the last section, we built this web crawler that prints data to the terminal as it crawls from page to page. This is fun, but typically you don't want to just print data out, you want to collect it, if not in a database, at least some sort of a well-formatted file. Fortunately, this is really easy to do with Scrapy, and in the process of doing it, we're going to be adding a little bit of, you might say, enterprise infrastructure to our crawler that's going to serve as well in building robust, scalable web crawlers in any situation. I've mentioned before, some of these mysterious Scrapy files off to the side that get magically created when we make a new project, and now we're going to be using one of them, items.py. And this is where you define your items. Of course, an item is a type of content that you're scraping. In this case, we're scraping articles. So keep in mind these aren't Wikipedias or Wikipedia…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.