From the course: Data Ingestion with Python
Join today to access over 22,600 courses taught by industry experts or purchase this course individually.
Other considerations - Python Tutorial
From the course: Data Ingestion with Python
Other considerations
- [Instructor] Calling APIs can be more complex than just writing code to access and pass. Here are a few points to consider. Some sites have a limit on how many requests you can make per amount of time, say a day. They usually allow more requests if you pay more. Make sure to read the plans and try to figure out how much is going to cost you for your use case. Calling an API can be slow, the usual approach is either to run query's in parallel, or use a special bulk API these sites have. Another option is to save results locally, known as caching. Some sites will blacklist certain IPs to disallow bots. You might need a special proxy or VPN if you scrap data from a cloud provider. Selenium needs to run a browser, which means, it needs some kind of a screen to attach to, however, when running a script on a server, you won't have a screen, this is known as headless mode, check out solutions, as headless drivers, VNC, XVFB, and others. This is usually something the operations team need to…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.