Easy-to-follow video tutorials help you learn software, creative, and business skills.Become a member
In this movie, I'll introduce you to what I generally call the Refine Edge command. And the reason I say that is because it technically goes by two names. If you have a layer mask selected, it's called Refine Mask. If you have a selection outline active, then it's called Refine Edge. Either way, it does the exact same thing. I am going to start things off by giving this guy a new background. So I will switch over to this flat JPEG image. I will convert the background to an independently layer by double-clicking on it here inside the Layers panel, and I will just call it sky, and click OK.
Now, armed with my Rectangular Marquee tool, I will right-click inside the image window, and choose Duplicate Layer, and then I will switch the document to Rough base mask.psd, which is my composition in progress, and then I will click OK. All right. Now if I switch back to that composition, I can see the sky is in front, and the portrait shot is in back. So I will drag the sky to the bottom of the stack, and then I will click on the side view layer to make it active. Now I am going to go ahead and zoom in here; notice that we have not only some very rough edges, of course, but some very clumsy edges over here on the left-hand side of the guy's face.
The easiest way to take care of those problems is to use the Magic Wand tool; it's just going to serve us best. So you can go ahead and select the Magic Wand from the Quick Selection tool flyout menu, and then press the Enter key in order to highlight the Tolerance value, and I am going to take it up to 16 luminance levels. And you can go ahead and turn on the Anti-alias checkbox as well, for all the good it does us. Then go ahead and click right here above the guy's shoulder in order to select a great deal of the background, as you can see. I will go ahead and zoom out to take in the marching ants, which are tracing the entirety of the front of the guy's face, as well as up along the top of the hair, and out a ways into that gradually lit background. All right.
I am going to go ahead and zoom back in, so I can see those edges up close and personal, and I will click on the layer mask thumbnail here inside the Layers panel to make sure the layer mask is active. Black is my foreground color, so I will press Control+Backspace, or Command+Delete on the Mac, to fill that selection with black. All right. Now that we have something that the Refine Edge command can work with, I will go ahead and double-click on the layer mask thumbnail in order to bring up the Properties panel. Now, normally the Mask Edge button would not be dimmed; it would be available to us, and I could go ahead and click on it in order to bring up the Refine Mask dialog box.
In my case, however, I can't, because there's a selection active. Let me go ahead and hide the Properties panel. I will show you another way to get to the command, which is to go up to the Select menu, and choose Refine Edge in this case, or you can press a keyboard shortcut, Control+Alt+R, or Command+Option+R on the Mac. Now, the fact that the command is called Refine Edge instead of Refine Mask tells us that Photoshop is going to apply the effects of the command to the selection outline, as opposed to the layer mask. That's not what we want. So I will press the Escape key in order to hide that menu, and I will press Control+D, or Command+D on the Mac, in order to deselect the image.
Now if I go ahead and double-click on the layer mask thumbnail, the Mask Edge button is available, and I can click on it to bring up the Refine Mask dialog box. Problem is, then if I try to scoot the dialog box out of the way, the Properties panel is blocking part of the image in the background. That's not what I want, so I will cancel out, and hide Properties, and instead of using that button, I'll go up to the Select menu, and choose what is now Refine Mask, thereby telling me that I will affect the layer mask, because there is no selection. So if there is a selection, it gets priority.
If there's no selection, you can choose the Refine Mask command. And that's what I will go ahead and do in order to bring up the Refine Mask dialog box. Now, I want you to see here that you've got this View option, and it represents a tiny preview of our composition. In our case, we're seeing the mask guy against a white background. If that's not what you want, you can click on that icon there, and you can switch to, for example, On Black to see him against the black background, or Black & White in order to see the layer mask itself. If you choose On Layers, you'll see the guy against the cloud background, and if you click on Reveal Layer, you'll see the active layer by itself.
In my case, I think On Black is going to represent things the best. All right. Now I will click away from the menu to hide it. For now, we're going to focus our attention on the Adjust Edge options here; Smooth, Feather, Contrast, and Shift Edge. They are in large part modeled after the commands that are available under the Select menu. You go down to Modify, and then normally, if I were working on a selection outline, I could gain access to commands like Smooth, and Feather, and so forth. These options right here serve the same purpose, except for one big difference: you can preview your modifications as you apply them, which is not true when using the select commands.
So in other words, these options right here are better. For example, let's start with something familiar, like Feather. I will go ahead and click inside of it, and I will increase that Feather value to, say, 20, and that goes ahead and blurs the mask, and we can see the degree to which it blurs the mask on the fly. And this would be true if we were working with a selection outline as well; we would get this exact same preview if we had the view set to On Black, and we could see the effects of feathering the selection as we do here. The Contrast value allows you to increase the sharpness of that edge, so now that I've gotten done blurring it, if I take the Contrast value up to 80%, for example, I end up with this much sharper version of the edge.
Notice how that rounds the heck out of the corners, however, including things that aren't really corners; just divots along the edge. So a combination of a high Feather and Contrast value ends up approximating that selection edge. If you want to shift the edge inward or outward, you use the Shift Edge option. And if you apply a positive value by dragging the slider triangle over to the right, then you move the edge outward. If you drag the triangle over to the left in order to apply a negative value, then you move the edge inward.
I don't want to do any of that. I am going to zero out all those values there by Shift+Tabbing my way back. I'm interested in adjusting the Smooth setting, because after all, we have very ratty, jagged edges. I will go ahead and zoom in a little farther even, so we can see him here. And that's not his razor stubble, even though he has got some. That is just some bad edge detail. So notice, if I start taking this Smooth value up by pressing Shift+up arrow, then I will smooth that edge dramatically. However, if I take it too far, then I end up, once again, rounding off the corners.
That's not what I want. So I am going to take the Smooth value back down to 0, and then I'll press the up arrow key in order to nudge that value upward until I get a very smooth edge, as you can see here, which happens for me as low as 5. Then I'll tab my way down to Shift Edge, and I will press Shift+down arrow in order to shift that edge inward, like so. And I'm ultimately going to take it, let's say, to something like -50 might work out pretty well. Now, if you want to compare the before and after, you turn on Show Original to turn the preview off, and then you turn show original back on to turn the preview setting on.
And you know, I think this is the wrong combination of values, so I am going to try a higher Smooth value here. So I will set that guy to 10, and then I will take Shift Edge down to -20. I think that ends up being a better combination, because what you're doing with Shift Edge is you're shifting the edge inside of the gray space. So in other words, wherever you have gray pixels in the mask, that's the area in which you can shift that edge back and forth. So you can't shift white into black; you can just make the gray pixels brighter, or in our case, darker, because we're shifting the edge inward.
And when I say gray pixels, I mean the gray pixels in the mask, not the gray pixels in the image. All right. That looks pretty good to me, so I'm going to go ahead and click OK in order to accept that modification, and just to give you a sense of what were able to accomplish here, I will press Control+Z, or Command+Z on the Mac. There's our very jagged selection outline at 400%. And if I press Control+Z or Command+Z again, we can see that it is much smoother now, thanks to some very basic modifications applied using the Refine Mask command here inside Photoshop.
Get unlimited access to all courses for just $25/month.Become a member
Access exercise files from a button right under the course name.
Search within course videos and transcripts, and jump right to the results.
Remove icons showing you already watched videos if you want to start over.
Make the video wide, narrow, full-screen, or pop the player out of the page into its own window.
Click on text in the transcript to jump to that spot in the video. As the video plays, the relevant spot in the transcript will be highlighted.