Learn how to deal with big data or data that’s too big to fit in memory. You can look into the HDF5 file format and see how it can be used from Pandas. Learn also how to use dask for distributed computation.
- [Instructor] So far we work with data that fits in memory.…And in most cases it's the best path to choose.…If you like memory, before you start with a big data…solution, remember that you can rent a machine…with 2TB of memory from Amazon.…But sometimes we do need to deal with data…that is too big to fit in memory.…Switching to a big data system like Spark or Hadoop…is painful and the learning curve is steep.…Sometimes they justify the effort.…But let me show you two easier approaches that might…help you.…These are the HDF5 file format and the Dusk library.…
HDF5 is a format for storing data.…It is designed with scientific applications in mind…and can officially query huge amounts of data.…NASA at one point had 15 pedabyte of satellite data…on HDF5.…HDF5 is supported by many languages including Python.…It is a good way to pass data between say Matlab…and Python.…Pandas as great support for HDF5 and as usual…the documentation is excellent.…Let's see it in action.…We load the text data set which I copied from the XSS files…
- Working with Jupyter notebooks
- Using code cells
- Extensions to the Python language
- Markdown cells
- Editing notebooks
- NumPy basics
- Broadcasting, array operations, and ufuncs
- Folium and Geo
- Machine learning with scikit-learn
- Plotting with matplotlib and bokeh
- Branching into Numba, Cython, deep learning, and NLP
Skill Level Intermediate
1. Scientific Python Overview
2. The Jupyter Notebook
3. NumPy Basics
Manage environments5m 11s
6. Folium and Geo
7. NY Taxi Data
10. Other Packages
11. Development Process
Next steps1m 33s
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.