In the last video, we talked about a few relatively simple ways of dealing with outliers, that is, either leaving them in, if it can be justified; rolling them into other categories, but at the risk of a heterogeneous group; or deleting them or selecting them out temporarily of the analyses. Now while these approaches may make sense if you don't have too many outliers, say for instance no more than 2% or 3% as a rough estimate, they also do some damage to the data and can cause you to lose cases, and you may have worked very hard to get those data.
So another alternative if you have a scale variable is to perform a mathematical transformation on the data. What this does is it modifies all the scores in the variables, generally creating a new variable on the process, using a set formula. Now people are very familiar with transformations, such as multiplying or adding or subtracting a certain amount, and that's taken as common practice. What we're going to be doing in this case, the most common approach for distributions that have a few extremely high scores, like the market capitalization one that we looked at in the last one, is to take the logarithm of the scores.
Now you may remember logarithms from junior high. These have the effect of bringing in extremely high scores. So for instance, the logarithm of 10 is 1, the logarithm of a 100 is 2, the logarithm of a 1,000 is 3, and it brings in the scores in a predictable way. And this is a legitimate way of dealing with outliers, as long as you always specify that you were dealing with the logarithms from this point on. On the other hand, if you have unusual scores at the low end of the distribution, you might want to try squaring the scores, because what that does is it pushes all the scores up but pushes the higher ones even further.
Now in both situations this assumes that you do not have zeros or negative scores, you have all positive scores. There are other ways of dealing with those. You can add a constant to them, but we don't need to deal with that right now. What I'm going to do is I'm going to look at the market capitalization data that we had in our last data set. Now I had filtered out cases of under $100 million market capitalization. I'm going to undo that filter right now. I'm going to Data, to Select Cases, to say please use all of them.
And so now it just tells me that the filter is off, and you can see that none of them are selected out anymore. And I'm going to come back here and let's take another quick look at the box plot for market capitalization that we did before. We have an extremely skewed distribution. Now let's try to find if doing a logarithm could help make this a little less skewed. What we do is we come to Transform, to Compute Variables, and I'm going to create a new variable called LogMarketCap, and that's pretty easy.
It is going to be the logarithm of the market capitalization. Now we've two choices for logarithm. Log10, this is what's called the base 10 logarithm. It takes the number 10 and raises it to a particular exponent to get a number, and that exponent is the logarithm. There's also the natural logarithm, which is on the base e 2.71828, dada, dada, dada, and an irrational number. And while they're very pleasing aesthetic aspects of the natural logarithm, because it's easier to interpret the base 10 logarithm, that's one we usually use.
So what I do is I double-click on that and I bring it up the numerical expression. I just double-click on MarketCap and it fills it so it says Log10MarketCap. Press OK and it tells me that it's created a new variable. If I go to the data set, I can see it right here at the end. You see the numbers are much smaller than most double digits, but that's because we're dealing with very large numbers over here, and that logarithm has to do more with the number of zeros in the number. Now what I'm going to do is I'm going to go back and create another box plot, but instead of doing market capitalization this time, I'll do the log of the market capitalization.
Just drag that in and leave everything else the same. And in this case, what's interesting about it is that we still have outliers, but this time they are symmetrically distributed, that we have outliers on the high end, but we also have outliers on the low end. And in fact, the distribution is remarkably symmetrical. It looks like it's spread out almost exactly the same amount in each direction. And you can see also that Apple, it is an outlier, but look how close it is for instance to Google, whereas here, here's Apple and here's Google down here.
So what we've done is we've taken a extremely asymmetrical skewed distribution and by taking the logarithm, we've pulled it in and made it symmetrical. Now there are still outliers, but they are on both sides and they're not terribly far away like they were before. And so we've taken a variable that really we might not have been able to deal with before or we had to cut awful lot of the scores to make it work, but now we can actually leave all of the scores in, we can use the entire data set, and still come pretty close to meeting the assumptions of most of this statistical procedures.
And so a logarithmic transformation in this case was a huge help in making our data meet the assumptions that we need to make it more manageable for analysis.
Get unlimited access to all courses for just $25/month.Become a member
82 Video lessons · 70309 Viewers
80 Video lessons · 127556 Viewers
52 Video lessons · 62474 Viewers
59 Video lessons · 48183 Viewers
Access exercise files from a button right under the course name.
Search within course videos and transcripts, and jump right to the results.
Remove icons showing you already watched videos if you want to start over.
Make the video wide, narrow, full-screen, or pop the player out of the page into its own window.
Your file was successfully uploaded.