In this video, Mark Niemann-Ross discusses how modifying R objects consumes memory. Learn how to write R code to avoid this common problem.
- [Instructor] High-volume data is memory intensive.…No surprise, it's big data.…Whether it's on a hard drive or in computer memory,…it occupies a lot of space.…R has memory management routines that efficiently…allocate memory for objects such as data frames,…vectors, and lists.…For the most part, memory management happens automatically,…and you, as a programmer, don't need to be aware of it…or be involved in it.…However, there is one aspect of memory management…under your control that will make a difference…in execution speed and available memory.…
When an object, such as a data frame, is created,…R allocates memory for that object,…which is a good thing.…But when that object is modified,…it's actually copied, with the new modifications.…A new block of memory is allocated,…and the new object is created.…For a period of time, both objects exist in memory.…So each time you append an observation to an object,…R allocates a new segment in memory…to contain that new object.…
That allocation takes processing time…and occupies memory.…
Author
Released
10/26/2018- Accessing memory and processing power
- Visualizing high-volume data
- Profiling and optimizing R code
- Compiling R functions
- Parallel processing with R
- Using R with other big data solutions
Skill Level Intermediate
Duration
Views
Related Courses
-
Data Wrangling in R
with Mike Chapple4h 12m Intermediate -
Code Clinic: R (2015)
with Mark Niemann-Ross3h 24m Intermediate
-
Introduction
-
Sample data set1m 49s
-
1. Problems and Opportunities with High-Volume Data
-
Big data and CPU cycles2m 14s
-
2. Visualizing High-Volume Data
-
Code: rug() and jitter()2m 32s
-
3. Working within the R Programming Language
-
Downsampling1m 42s
-
4. Advanced High-Volume Techniques
-
Compile R functions3m 55s
-
Parallel processing with R3m 39s
-
-
5. Use R with External Big Data Solutions
-
Code: R with databases5m 25s
-
Cloud computing with R1m 9s
-
Sparklyr with R3m 38s
-
Code: R with Sparklyr4m 45s
-
Conclusion
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.
CancelTake notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.
Share this video
Embed this video
Video: Avoid the copy-on-modify problem with R