From the course: Spark for Machine Learning & AI
Join today to access over 22,600 courses taught by industry experts or purchase this course individually.
Introduction to preprocessing - Apache Spark Tutorial
From the course: Spark for Machine Learning & AI
Introduction to preprocessing
- [Instructor] There are two types of pre-processing, numeric and text pre-processing. Normalizing maps data values from their original range to the range of zero to one. It's used to avoid problems when some attributes have large ranges and others have small ranges. For example, salaries have a large range, but years of employment has a small range. Standardizing maps data values from their original range to a range of negative one to one. And it also has a mean value of zero. This transformation creates a normal distribution with a standard deviation of one. Now this transforms our data into a bell curve shape formation. It's used when attributes have different scales, and the machine learning algorithm you're using assumes a normal distribution. Partitioning maps data values from continuous values to buckets, like histograms. Deciles and percentiles are examples of buckets. It's useful when you want to work with groups of values instead of a continuous range of values. If you work…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.