# Combining or excluding outliers

## Video: Combining or excluding outliers

When you start looking at your data one of the problems you might have to deal with is outliers. These are extreme scores, like somebody who is 7 feet tall or somebody who has 26 children or unusual categories, like being Nepali or a Latin Poetry Major. Now sometimes these unusual scores or categories are inherently interesting, like with world records or gifted and talented programs in schools. In other situations, however, they can wreak havoc with statistical procedures that might be designed to look at general patterns, or overall trends.

## Combining or excluding outliers

When you start looking at your data one of the problems you might have to deal with is outliers. These are extreme scores, like somebody who is 7 feet tall or somebody who has 26 children or unusual categories, like being Nepali or a Latin Poetry Major. Now sometimes these unusual scores or categories are inherently interesting, like with world records or gifted and talented programs in schools. In other situations, however, they can wreak havoc with statistical procedures that might be designed to look at general patterns, or overall trends.

In the latter case, where you may be interested more in common scores than in uncommon scores, you have a few choices on how to deal responsibly with the outliers. Now the first question is how to define outliers. Now we've already looked at one way of getting a graphical definition of outliers on a scale variable, and it's with a box plot. I am going to come up to Graphs, to Chart Builder, to Boxplot. I will drag in the 1D Boxplot, and let's look at Market Capitalization.

Also, because we have convenient stock symbols over here, I am going to ask for a Point ID so I know who the outliers are. I will just drag that over here and press OK, and what we see is that the variable for Market Capitalization is extraordinarily skewed, and in fact they often call this pathological skewed. We have Apple here with over \$300 billion in market capitalization, Microsoft, Oracle, and Google, and it just goes down. And we have this huge number of companies that are stuck in a tiny level of market capitalization relatively speaking.

In fact, we have no idea what the median or the mean is because those other scores all get squished together so much that there is 2800 companies in the NASDAQ listing, but we have these extreme outliers that are squishing all the others, that is not possible to really see what's going on. So we know that we have outliers here on a scale variable. Now on a categorical variable, like for instance ethnicity, what you then have as a definition for categorical outliers is that any group that has, for instance, less than 10% of the overall sample would be considered a categorical outlier.

In that situation you have the choice of combining them with other categories and creating a sort of Other category except that it has to be very heterogeneous group. That or you simply don't analyze by that variable in the future. But let's talk about what to do with a scale variable. Now if you don't have very many outliers, or that they're not very far away, you can leave them in. You could take them as legitimate values and you could proceed with that understanding, as long as you communicate it adequately with others.

On the other hand, another choice is to exclude them. Now I don't necessarily mean delete them permanently from the data set, but you can create a selector. We've done this before. I should just mention right here, this is \$100 billion, and we still have a huge number of companies right there. I am going to select a much smaller number. I am going to go to \$100 million capitalization. So I am going to go to Data, to Select Cases. Select Cases if your market capitalization is less than 100 million and press Continue.

Now I have the option of just filtering them out. That creates a new variable that temporarily excludes or deleting them permanently, and I don't want to do that. I am just going to filter them out right now. So I am going to press OK, and it tells me that it has done that selection. And in fact, if I go back to the data set I will see that these cases got, for instance, Apple has been selected out. There is a variable here at the end now. There's a filter variable, and if I click on the value labels, I can see there are cases that are selected or not selected. And now I am going to go back, and I am going to do my box plot all over again.

All I have to do is press OK, but this time I don't have any outliers. In fact, this is a pretty normal-looking box plot. I can see that of the 2800 companies in the NASDAQ, the median level of market capitalization is around \$40 million. The first quartile, the first lowest 25% have 20 million or less, whereas the highest quartile have about \$60 million or less. There are of course hundreds of outliers above these, but these give a nice picture of what you'll call the small capitalization market.

Anyhow, the ability to either combine groups or to temporarily exclude outliers is one good way of dealing with them, as long as you can justify your choices. Again, that gets back to a general statistical principle that you can do whatever you feel is most appropriate and that serves your purposes in telling an analytical narrative. You're telling a story about your data, and if temporarily excluding cases or combining them with other groups serves your purposes best, then go ahead and do that, as long as you can justify your decision to others.

Now, in the next video I will look at another way that does not exclude the cases. It leaves them all in, but changes them by doing what's called a transformation, to let you use all of your data and see if you can still find a way of telling a coherent narrative that way.

Show transcript

#### This video is part of

SPSS Statistics Essential Training (2011)

52 video lessons · 19167 viewers

Author

Expand all | Collapse all
1. ### Introduction

2m 58s
1. Welcome
1m 5s
2. Using the exercise files
40s
3. Using a different version of the software
1m 13s
2. ### 1. Getting Started

19m 0s
1. Taking a first look at the interface
11m 49s
7m 11s
3. ### 2. Charts for One Variable

21m 54s
1. Creating bar charts for categorical variables
7m 18s
2. Creating pie charts for categorical variables
2m 54s
3. Creating histograms for quantitative variables
5m 45s
4. Creating box plots for quantitative variables
5m 57s
4. ### 3. Modifying Data

33m 10s
1. Recoding variables
5m 33s
2. Recoding with visual binning
5m 33s
3. Recoding by ranking cases
5m 26s
4. Computing new variables
5m 37s
5. Combining or excluding outliers
5m 21s
6. Transforming outliers
5m 40s
5. ### 4. Working with the Data File

28m 12s
1. Selecting cases
6m 44s
2. Using the Split File command
5m 12s
3. Merging files
5m 33s
4. Using the Multiple Response command
10m 43s
6. ### 5. Descriptive Statistics for One Variable

22m 14s
1. Calculating frequencies
8m 43s
2. Calculating descriptives
5m 31s
3. Using the Explore command
8m 0s
7. ### 6. Inferential Statistics for One Variable

16m 3s
1. Calculating inferential statistics for a single proportion
6m 6s
2. Calculating inferential statistics for a single mean
5m 39s
3. Calculating inferential statistics for a single categorical variable
4m 18s
8. ### 7. Charts for Two Variables

30m 43s
1. Creating clustered bar charts
7m 10s
2. Creating scatterplots
5m 8s
3. Creating time series
3m 24s
4. Creating simple bar charts of group means
4m 17s
5. Creating population pyramids
3m 0s
6. Creating simple boxplots for groups
3m 3s
7. Creating side-by-side boxplots
4m 41s
9. ### 8. Descriptive and Inferential Statistics for Two Variables

45m 28s
1. Calculating correlations
8m 17s
2. Computing a bivariate regression
6m 27s
3. Creating crosstabs for categorical variables
6m 34s
4. Comparing means with the Means procedure
6m 33s
5. Comparing means with the t-test
6m 4s
6. Comparing means with a one-way ANOVA
6m 30s
7. Comparing paired means
5m 3s
10. ### 9. Charts for Three or More Variables

24m 30s
1. Creating clustered bar charts for frequencies
6m 34s
2. Creating clustered bar charts for means
3m 45s
3. Creating scatterplots by group
4m 13s
4. Creating 3-D scatterplots
4m 25s
5. Creating scatterplot matrices
5m 33s
11. ### 10. Descriptive Statistics for Three or More Variables

30m 57s
1. Using Automatic Linear Models
11m 52s
2. Calculating multiple regression
9m 3s
3. Comparing means with a two-factor ANOVA
10m 2s
12. ### 11. Formatting and Exporting Tables and Charts

29m 29s
1. Formatting descriptive statistics
6m 1s
2. Formatting correlations
7m 49s
3. Formatting regression
10m 19s
4. Exporting charts and tables
5m 20s
13. ### Conclusion

51s
1. What's next
51s

### Start learning today

Sometimes @lynda teaches me how to use a program and sometimes Lynda.com changes my life forever. @JosefShutter
@lynda lynda.com is an absolute life saver when it comes to learning todays software. Definitely recommend it! #higherlearning @Michael_Caraway
@lynda The best thing online! Your database of courses is great! To the mark and very helpful. Thanks! @ru22more
Got to create something yesterday I never thought I could do. #thanks @lynda @Ngventurella
I really do love @lynda as a learning platform. Never stop learning and developing, it’s probably our greatest gift as a species! @soundslikedavid
@lynda just subscribed to lynda.com all I can say its brilliant join now trust me @ButchSamurai
@lynda is an awesome resource. The membership is priceless if you take advantage of it. @diabetic_techie
One of the best decision I made this year. Buy a 1yr subscription to @lynda @cybercaptive
guys lynda.com (@lynda) is the best. So far I’ve learned Java, principles of OO programming, and now learning about MS project @lucasmitchell
Signed back up to @lynda dot com. I’ve missed it!! Proper geeking out right now! #timetolearn #geek @JayGodbold
Share a link to this course

### What are exercise files?

Exercise files are the same files the author uses in the course. Save time by downloading the author's files instead of setting up your own files, and learn by following along with the instructor.

### Can I take this course without the exercise files?

Yes! If you decide you would like the exercise files later, you can upgrade to a premium account any time.

How to use exercise files.

Learn by watching, listening, and doing, Exercise files are the same files the author uses in the course, so you can download them and follow along Premium memberships include access to all exercise files in the library.

Exercise files

How to use exercise files.

This course includes free exercise files, so you can practice while you watch the course. To access all the exercise files in our library, become a Premium Member.

Are you sure you want to mark all the videos in this course as unwatched?

This will not affect your course history, your reports, or your certificates of completion for this course.

Congratulations

You have completed SPSS Statistics Essential Training (2011).

Become a member to add this course to a playlist

Join today and get unlimited access to the entire library of video courses—and create as many playlists as you like.

Become a member to like this course.

Join today and get unlimited access to the entire library of video courses.

Exercise files

Learn by watching, listening, and doing! Exercise files are the same files the author uses in the course, so you can download them and follow along. Exercise files are available with all Premium memberships. Learn more

How to use exercise files.

Thanks for contacting us.
You’ll hear from our Customer Service team within 24 hours.

Please enter the text shown below:

The classic layout automatically defaults to the latest Flash Player.

To choose a different player, hold the cursor over your name at the top right of any lynda.com page and choose Site preferencesfrom the dropdown menu.

• Mark video as unwatched
• Mark ALL videos as unwatched
Exercise files

Access exercise files from a button right under the course name.

Mark videos as unwatched

Remove icons showing you already watched videos if you want to start over.

Make the video wide, narrow, full-screen, or pop the player out of the page into its own window.

Interactive transcripts

Click on text in the transcript to jump to that spot in the video. As the video plays, the relevant spot in the transcript will be highlighted.

## Are you sure you want to delete this note?

Thanks for signing up.

We’ll send you a confirmation email shortly.

• new course releases
• general communications
• special notices

Keep up with news, tips, and latest courses with emails from lynda.com.

• new course releases