Learn how to load corrupted CSV data into Pandas DataFrame. You can see methods to debug the cause of error and how to use “read_csv” function options to correctly load data.
- [Instructor] Let's load the data to Pandas.…Df equals pd.read_csv and the file name.…Note that Pandas sees the .pc2 extension…and know how to decompress.…Let's see how many rows we got.…100,000 rows, which seems right.…Remember that Pandas uses the first row…as the column names.…Let's take a look at a single row.…This doesn't look so good.…VendorID looks like a time.…Store and forward flag looks like a coordinate.…Something went wrong, but what?…As we'll read the header line…and one line of data from the file,…with bz2.open…fname and texture mode as fp.…
Header is fp.readline,…and data is fp.readline.…And let's print them out.…Print header…and print data.…If you look closely to data output,…you'll see two extra comas at the end.…Let's use split to see how many fields there are…in every line.…So len of header.split…on a comma,…and this is 21.…Len of data.split…on a comma,…and this is 23.…
We have two more extra fields in the data…as we suspected.…Let's fix this by telling Pandas…to load only the first 22 columns of data.…
- Working with Jupyter notebooks
- Using code cells
- Extensions to the Python language
- Markdown cells
- Editing notebooks
- NumPy basics
- Broadcasting, array operations, and ufuncs
- Folium and Geo
- Machine learning with scikit-learn
- Plotting with matplotlib and bokeh
- Branching into Numba, Cython, deep learning, and NLP
Skill Level Intermediate
1. Scientific Python Overview
2. The Jupyter Notebook
3. NumPy Basics
Manage environments5m 11s
6. Folium and Geo
7. NY Taxi Data
10. Other Packages
11. Development Process
Next steps1m 33s
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.