In this video, Mark Niemann-Ross demonstrates the memory problem associated with copying data objects. Learn how to recognize and avoid this problem with rbind and data.frame.
- [Instructor] R provides a range of data structures,…things like lists, vectors, and matrices.…Data frames are probably the most convenient way…to store data, they behave very much like a spreadsheet,…but R does some strange things, memory wise,…when you start to copy, append, and change data frames.…Let's take a look at how that works,…and then I'll show you a solution for using data frames…without the memory side effects.…The first thing I'm going to do…is create something called aDataFrame,…and it's a subset of the UCS Satellite data set.…
You can see over in the environment, it's,…aDataFrame has 1,886 observations of three variables,…and we can see how big that actually is…using line 11, where it says object.size aDataFrame.…Down in the console, it indicates that that's 39,000 bytes.…Well, I'd like to watch what happens DataFrame…as I make modifications to it,…and we know from a previous video…that tracemem will tell us any time…that an object is copied, so let's use line 13,…where I turn on tracemem for aDataFrame,…
Author
Released
10/26/2018- Accessing memory and processing power
- Visualizing high-volume data
- Profiling and optimizing R code
- Compiling R functions
- Parallel processing with R
- Using R with other big data solutions
Skill Level Intermediate
Duration
Views
Related Courses
-
Data Wrangling in R
with Mike Chapple4h 12m Intermediate -
Code Clinic: R (2015)
with Mark Niemann-Ross3h 24m Intermediate
-
Introduction
-
Sample data set1m 49s
-
1. Problems and Opportunities with High-Volume Data
-
Big data and CPU cycles2m 14s
-
2. Visualizing High-Volume Data
-
Code: rug() and jitter()2m 32s
-
3. Working within the R Programming Language
-
Downsampling1m 42s
-
4. Advanced High-Volume Techniques
-
Compile R functions3m 55s
-
Parallel processing with R3m 39s
-
-
5. Use R with External Big Data Solutions
-
Code: R with databases5m 25s
-
Cloud computing with R1m 9s
-
Sparklyr with R3m 38s
-
Code: R with Sparklyr4m 45s
-
Conclusion
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.
CancelTake notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.
Share this video
Embed this video
Video: Code: Avoid copy-on-modify with data.table