Start learning with our library of video tutorials taught by experts. Get started

Photoshop CS5 Top 5
Illustration by John Hersey

Refining masks


Photoshop CS5 Top 5

with Deke McClelland

Video: Refining masks

One of the most common questions I hear from Photoshop users is how do I extract a person from one background and set them against another, with all of their hair intact? Having written a book and recorded more than 50 hours of video on that very topic, I can assure you, there are lots of ways to do it. And it all starts with creating an accurate mask. Photoshop offers lots of masking options, but they're not what Adobe likes to call discoverable. In other words, there is no way in heck you'd figure out how to mask hair unless a guy like me showed you how to do it.
Expand all | Collapse all
  1. 1h 9m
    1. Common-sense enhancements
      10m 15s
    2. HDR Pro
      11m 33s
    3. Refining masks
      15m 37s
    4. Puppet Warp
      14m 56s
    5. Painting a photograph
      17m 36s

Watch every course in the library with a membership.
Each course includes high-quality videos taught by expert instructors.

Become a member
Please wait...
Photoshop CS5 Top 5
1h 9m Intermediate May 05, 2010

Viewers: in countries Watching now:

In five movies, author Deke McClelland covers five of the most important new features in Photoshop CS5 and shows how these powerful functions can be integrated into workflow immediately and efficiently. Photoshop CS5 Top 5 starts with the small stuff—the Straighten button, the Mini Bridge, and content-aware fill—then builds up to powerhouse features such as High Dynamic Range (HDR) Pro, the new Refine Edge command, and Puppet Warp. The course winds up with a demonstration of how to use the bristle and mixer brushes to convert a portrait photo into a hand-drawn painting. In the end, we hope you'll feel inspired, empowered, and ready to take on Photoshop CS5.

Topics include:
  • Making sense of enhancements
  • Applying HDR Pro adjustments and effects
  • Refining masks
  • Using the Puppet Warp tool
  • Painting a photograph
Design Photography
Deke McClelland

Refining masks

One of the most common questions I hear from Photoshop users is how do I extract a person from one background and set them against another, with all of their hair intact? Having written a book and recorded more than 50 hours of video on that very topic, I can assure you, there are lots of ways to do it. And it all starts with creating an accurate mask. Photoshop offers lots of masking options, but they're not what Adobe likes to call discoverable. In other words, there is no way in heck you'd figure out how to mask hair unless a guy like me showed you how to do it.

Enter Photoshop CS5's revamped Refine Edge command, variously known as Refine Mask, that can grow a selection or layer mask into tendrils of hair, and even get rid of the fringes of color around the edges, so the hair looks right against a different background. I'm not sure the new feature is any more discoverable than a dozen or so other masking options. Who among us equates hair with Refine Edge? But whatever you think of the name, the new and improved Refine Edge command lets you perfect your masks with a significantly higher degree of control than you've had in the past.

In this video, I'm going to demonstrate the dramatically enhanced Refine Edge function inside of Photoshop CS5. The idea is that it allows you to refine the edges of your selection outline or your mask, in order to achieve the absolute best results. For example, let's say what I want to do here is want to take this portrait shot from Jason Stitt of the Fotolia Image Library, about which you can learn more at I want to remove the blue backdrop as well as the blue filtering into the model's hair.

I want to composite her against this sky and sea background, and I want to achieve absolutely credible effects. Well, the approach to take here inside of CS5 is to generate some sort of base selection and then apply the Refine Edge command, in order to make that Selection Outline, or layer mask best, suit the needs and contours of the image. So I'm going to start things off by going up to the Select menu and choosing the Color Range command. Now, this command has been with us for years and years now. However, if you're not familiar with it, it's basically the Magic Wand tool on steroids.

It's not much more difficult to use, although, as you'll see, it's a little different, but it delivers much better results. So I'll go ahead and choose this command. Up comes the Color Range dialog box. I'll move my Eyedropper into the image window, and I'll click in the blue area in order to select it. I can tell that it's selected because of this Selection Preview here inside of the dialog box. Wherever we see white, that represents the selected portion of the image; wherever you see black, that's a deselected portion. So notice that I have a few straight gray areas up in the upper-right region of the image.

I'll go ahead and press the Shift key, which allows you to add to a selection inside of Photoshop. I'll drag over that area in order to select all of those colors. Now notice that I have exactly the opposite effect of what I want. The background is selected. The foreground is deselected. That means if I were to convert this selection into a layer mask that the model would become transparent and her background would remain visible. In order swap that, I'll turn on the Invert check box. Now the background is black, and now the foreground is white.

That's exactly what I want. I'll go ahead and click on the OK button in order to create that selection outline. To see what it looks like, I'll drop down to the bottom of Layers panel, and I'll click on the Add layer mask icon. We've now added a layer mask. So that means any selected portions of the image are now opaque; any deselected portions are now transparent. Now, it looks like we have a pretty good composition at this point, but if we zoom in, we can begin to see some of the problems. Notice that we have some ratty edges associated with her shoulder, also along the knuckles right here.

We've got a lot of blue filtering into the hair. This becomes even more obvious if we preview the image against a white background. That's why I've set up this layer of white. So what we need to do now is refine those edges. Here's how we're going to do it. I'm going to turn off that white layer, so that we can see the image against its new background. I'll scroll her down just a little bit. Then I'll go up the Select menu, and I'll choose this command right there. If I were working on a Selection Outline, it would say Refine Edge. Because I'm working on a layer mask, it says Refine Mask.

However, it's the exact same command, and it brings up the exact same dialog box of options. The two commands also happen to have the same shortcut, which is Ctrl+Alt+R on the PC or Command+Option+R on the Mac. As soon as I choose that command, I bring up the Refine Mask dialog box. Again, it would be called Refined Edge, if I was working with the Selection Outline. Now then, if you've worked with this command in previous versions of the program inside of CS3 or CS4, you'll recognize some of the options, like those down here in the Adjust Edge area, and others will seem completely foreign.

This is a very different command these days. First thing I want to do is determine how I'm going to preview the effects of my modifications. So as things stand now, I'm seeing the selected image against a white background. I could preview it against a black background, if I preferred. Either way I'm going to be able to easily make out those bad edges. You can also choose to see the effect of all layers blended together. I'm going to go ahead and stick with On White, which is the default setting. I'm also going to modify a few of my Edge settings.

For example, I'm going to take the Smooth value up to 20. And that's going to go ahead and round off any of the jagged corners inside of my layer mask. I could go ahead and raise the Feather value, if I wanted to make my mask blurry. However, I regard the Feather Option as something of a crutch. It's a great way to take a bad selection outline and make it look less bad, but less bad is not the same as good. So generally speaking, I urge you to leave that value set to 0. Next, we have the Contrast option. Notice we have some pretty soft edges going, thanks to Smooth.

If we want to firm them up, we would increase that Contrast value, like so. At 100%, we have some very firm edges indeed. I'm going to leave that value set to 0% for now. You finally have the option of shifting the edges outward in order to incorporate more of the old background, or shift those edges inward in order to choke the edges closer to the hair detail. Again, I'm going to leave that value set to 0 for now. I would now like to direct your attention to the real magic of this command, which resides up here in this Edge Detection area.

The first thing you want to do is you want to raise your Radius value. And I'm going to take the Radius up to 70 pixels. And this is analogous to a Blur value, but it works much differently than Feather. Rather than blurring the edges, you instead establish an area in which the Refine Edge command can evaluate the mask. So for example, when I set the Radius value to 70 pixels, I'm saying, anywhere 70 pixels outside the previous edge of the mask, or inside that previous edge is up for grabs where this command is concerned. In other words, the command can go ahead and redress those pixels.

Anywhere outside of that area should be left alone. In order to see what that looks like, go ahead and switch your View from On White to Reveal layer. Then turn on the Show Radius check box. Anywhere that appears black is outside the range of the Refine Edge command. Anywhere where you see imagery is inside the range. So in other words, it's inside of this Radius. Notice the difference. If I was to crank the Radius value down to 0 pixels, we would not see anything. So no automatic refinement is happening. As soon as you increase that Radius value, that's the area of automation right there.

But notice that the area is equally thick around the hair detail, as well as around the smooth details, such as the knuckles, and even the sweater detail along the shoulder. What you'd really, ideally want is a thick radius around the hair, so that you can incorporate all those little filigree details, and then a very fine Radius value around the smooth stuff. You can ask the Refine Edge command to do that automatically for you by turning on the Smart Radius check box. Notice that that's going to go ahead and reduce the Radius value around the Smooth details and keep it thick around the hair, so that the Refine Mask command has room to do its work. All right! Let's go ahead and turn off Show Radius this time around.

I'm going to go ahead and switch back to On White, so that we can see our mask in progress. Now, what do we do about the areas that aren't working out so well? For example, we have some blue inside of the fingers, both in this region and this region there. In order to evaluate what approach to take, you want to turn on the Show Original check box right there. That's the opposite of a Preview check box. So most dialog boxes inside of Photoshop have a Preview check box, and when you turn it on, you're previewing. When you turn it off, you're not previewing. In this case, when you turn Show Original on, you're not previewing.

You're seeing the original version of the masked image. When you turn it off, you are previewing the effects of the options inside the dialog box. Anyway, you can turn the option on and off by pressing the P key, if you like. I'm going to go ahead and turn it on for a moment, so that we can see the original details below the finger were actually in better shape than they are now, same in this region right there. So what I'd like to do is paint some of that radius away. I'm going to do that using the Refine Radius tool. So for starters, we'll turn off Show Original. There are two ways to use this tool.

If you want to add to the radius, then you would paint over an area, like so. Then if you want to subtract from the radius, which is what we want to do, remember that the original version was in better shape, so we need to get the radius out of there, turn that off again, you press and hold the Alt key or the Option key on the Mac, and you paint inside of an area, like so. That paints away the radius of that location. So in other words, you're painting in and out custom radius areas inside of which the Refine Edge command can do its thing.

So don't think of it as being a matter of adding to the selection when you paint, and deleting from the selection when you Alt+Drag or Option+Drag. That's not how it works. You're just affecting the radius. All right! In my case, having removed the radius from this location, it's great for the fingers, but it's not so great for the hair. So I'll paint back in the radius over the hair. You're not going to see anything happen when you first paint. You're going to have to wait for the changes to update. Then down here, I'm going to Alt+ Drag as well, Option+Drag on the Mac to paint that area away. I also have a little bit of a glow, leaking into the fingers around this location here.

So I need to paint in that area with a smaller cursor. I can change the size of my cursor on the fly using the Square Bracket keys, which are located to the right of the P as in Paul key. So I'll press the Left Bracket key a few times in order to reduce the size of my cursor. And then I'll Alt+Drag or Option+Drag over this area, like so, in order to remove it from the equation. I'll also Spacebar+Drag in order to move the image over to the left-hand side, so that I can check out what's going on here. I've got a little bit of haloing along the edge of the neck, so I will Alt+Drag or Option+Drag along that neck as well.

I'm not dragging into it, just along it. That goes ahead and firms up those details quite nicely. Now, I wouldn't go so far as to say everything looks exactly right. In fact, I still have some blue leaking into the hair. That's the reason for this Decontaminate Colors check box. If you turn it on, then you are going to remove the aberrant colors around the edge of the mask or selection outline. However, what that means is that Photoshop is going to go in and recolor the pixels inside of the image, which is why, notice this Output To option.

Right now, we're just going to output to the layer mask, meaning we are going to modify the layer mask and nothing more. But if you turn on Decontaminate Colors, then it changes to New layer with layer Mask, because it's trying to protect the contents of your old layer. So that's just something to bear in mind, that pixels will be recolored. You can see it happening here inside the image window. So when I turn off Decontaminate Colors, the hair looks pretty blue. When I turn the check box back on, the hair becomes a little more brown. However, it's not exactly what I want, and I don't want to generate a new layer. I don't want to change the colors inside of my image layer.

So I'm going to turn that check box off. I'm going to switch Output To back to layer mask, like so. Then I'm going to make one more slight modification here. I'm going to shift the edges inward, toward the hair. So I'm going to change that Shift Edge value to -40%, like so. And then, I'll click on the OK button in order to accept my changes. Things are looking pretty darn good, but in order to get a real sense of what's going on, I'll turn on the white layer, once again. Notice that we still have some blue details inside the outer edges of the hair, down toward the bottom, on the right- hand side of the image as well, as down here in the left-hand side of the image.

We can get rid of those, if you so desire, with another application of Refine Edge. So you can apply that command multiple times to a layer mask if you like. I'll show you how you can achieve some very nice results that way. With the layer mask still selected, I'll go back up to the Select menu, choose the Refine Mask command, Ctrl+Alt +R, Command+Option+R on the Mac. This time, I'm not going to change any of the numerical settings. Instead, I'm just going to paint with the Refine Radius tool. I'll press the Right Bracket key a few times in order to make my cursor larger onscreen, and then I'll paint inside of this blue region.

Notice that the blue begins to get leached away. Then I'll paint again to add more to that Radius value, and sure enough, I get rid of more and more of that blue. I'll go ahead and zoom out, scroll up, paint over this area up here. Notice I'm doing a pretty sloppy job. At this point, when you're making a second pass at a mask like this, you don't need to be nearly as careful, at least that's my experience. Then I'll go ahead and scroll over to the left-hand side and paint down in this region as well. Everything is looking pretty darn good. All right! So now, I'll click OK in order to accept those modifications.

Just a few more things that I need to do. By the way, at this point, you can bring your old school masking techniques to bear, if you so desire. For example, we've got some rounded corners underneath this finger. I'm going to fix those corners using the Smudge tool. So with the layer mask still selected, I'll go ahead and zoom in on that detail, and I'll paint upward to paint some of the mask, just to smudge that mask into those corners, like so. It may take a couple of passes. I'm trying to be very careful as I work.

I'll go ahead and paint down into this corner as well, and paint up into this one. Notice that I have this strange, little point on that figure there. So I think, I'll go ahead and paint up into it. Then we need to firm up a few details. Let's go ahead and turn off the white layer, so that I bring back my background. I've got this strange translucency, this ghosting, over the shoulder. So I'll switch to my Brush tool. Make sure that the mode up here in the Options bar is set to Overlay, as it is. With my foreground color set to white, I'll go ahead and paint over the shoulder to firm it up, like so.

It may take a couple of passes in order to make it look the way it needs to. I might also paint over these knuckles in order to firm up those details. If I need to shift this edge slightly over, as I do, then I'll grab my Smudge tool, once again, and paint into this edge. There are a lot of different ways to work with masks inside of Photoshop. But I just want you to get a sense that you don't need to rely entirely on the Refine Edge command, in order to refine the edges of those masks. You can work any way that you like. This is the final version of my masked image, set against a new background, thanks largely to the power of the enhanced Refine Edge function here inside of Photoshop CS5.

There are currently no FAQs about Photoshop CS5 Top 5.

Share a link to this course
Please wait... Please wait...
Upgrade to get access to exercise files.

Exercise files video

How to use exercise files.

Learn by watching, listening, and doing, Exercise files are the same files the author uses in the course, so you can download them and follow along Premium memberships include access to all exercise files in the library.
Upgrade now

Exercise files

Exercise files video

How to use exercise files.

For additional information on downloading and using exercise files, watch our instructional video or read the instructions in the FAQ.

This course includes free exercise files, so you can practice while you watch the course. To access all the exercise files in our library, become a Premium Member.

Upgrade now

Are you sure you want to mark all the videos in this course as unwatched?

This will not affect your course history, your reports, or your certificates of completion for this course.

Mark all as unwatched Cancel


You have completed Photoshop CS5 Top 5.

Return to your organization's learning portal to continue training, or close this page.

Become a member to add this course to a playlist

Join today and get unlimited access to the entire library of video courses—and create as many playlists as you like.

Get started

Already a member?

Become a member to like this course.

Join today and get unlimited access to the entire library of video courses.

Get started

Already a member?

Exercise files

Learn by watching, listening, and doing! Exercise files are the same files the author uses in the course, so you can download them and follow along. Exercise files are available with all Premium memberships. Learn more

Get started

Already a Premium member?

Exercise files video

How to use exercise files.

Ask a question

Thanks for contacting us.
You’ll hear from our Customer Service team within 24 hours.

Please enter the text shown below:

The classic layout automatically defaults to the latest Flash Player.

To choose a different player, hold the cursor over your name at the top right of any page and choose Site preferencesfrom the dropdown menu.

Continue to classic layout Stay on new layout
Welcome to the redesigned course page.

We’ve moved some things around, and now you can

Exercise files

Access exercise files from a button right under the course name.

Mark videos as unwatched

Remove icons showing you already watched videos if you want to start over.

Control your viewing experience

Make the video wide, narrow, full-screen, or pop the player out of the page into its own window.

Interactive transcripts

Click on text in the transcript to jump to that spot in the video. As the video plays, the relevant spot in the transcript will be highlighted.

Thanks for signing up.

We’ll send you a confirmation email shortly.

Sign up and receive emails about and our online training library:

Here’s our privacy policy with more details about how we handle your information.

Keep up with news, tips, and latest courses with emails from

Sign up and receive emails about and our online training library:

Here’s our privacy policy with more details about how we handle your information.

submit Lightbox submit clicked

Terms and conditions of use

We've updated our terms and conditions (now called terms of service).Go
Review and accept our updated terms of service.