Join Tim Grey for an in-depth discussion in this video Using the Refine Mask command, part of Photoshop: Creating Composites (2012).
The basic concepts of layer mask are relatively straightforward, but in actual practice, some layer mask are easier than others. And when you're dealing with fuzzy objects, hair, feather, fur, a layer mask can be a particular challenge. Fortunately, there are some tools for refining our masks that help make it alot easier to deal with these tricky subjects. I'll go ahead and start off by creating a selection of this horse. I'll choose the Quick Selection tool, and then adjust the brush size as needed. And then paint over the horse itself, identifying the areas of the image that I actually want to select. You'll notice that as soon as I go over the mane here, then I get the sky included in the selection as well as the grass in the background. But I can simply hold the Alt key on Windows or the Option key on Macintosh while painting across that area in order to remove it from the selection. Now, in general concept, I have an accurate selection. The horse is selected and the background is not selected, but we can see around the mane things are a little bit more tricky.
And of course, it would be very difficult to fine tune the selection here, because the selection edge only shows us the difference between greater than or less than 50% selected. And so I want to work in a way where I can actually evaluate the results, while I'm improving the layer mask. So I'll go ahead and create a layer mask based on this selection. I'm working with a single image with a Background image layer. So first, I need to double-click on the thumbnail for the Background image layer, and then click OK in the New Layer dialog, in order to convert that background image layer to a normal layer, so that I can add a layer mask to it.
I'll then go ahead and click on the Add Layer Mask button, the circle inside of a square icon at the bottom of the Layers panel. And because I have a selection active, that will cause the layer mask to based on that selection. You can see that overall, the result is not too bad. But again, in the mane, things aren't looking all that good. And so I'm going to use the Refine Mask tool in order to improve my results. On the Properties panel, I'll go ahead and click on the Mask Edge button, and that will bring up the Refine Mask dialog. I'm currently viewing the image with an On White option, meaning that I'm seeing the image set against a white background. There are a variety of different options.
In many cases, I'll use the On Layers option for example. But because I'm working on a single image, I've not yet added multiple image layers to create the composite. This option will reveal a checker board pattern where pixels are not there, in other words, where there is transparency in the image. So, in this case I think the On White or On Black option will produce the better result. And I'll go ahead and stick with On White in this case. I can then apply some feathering and that will help to blur the edge of the layer mask, and hopefully cause the horse to blend in a little bit more with whatever background I'm going to set it against. In this case just a white background, but of course that doesn't really help the mane up above.
I can also shift the edge inward or outward as needed. So if I drag to the left the layer mask pushes outward into the horse itself. And if I drag the slider over to the right, then the selection essentially expands away from the horse. So in this case then I'm revealing more of the grass in the background. And so when I feather, I will almost always need to shift that edge in or out just a little bit. And so you'll want to fine tune the setting to try to find the best result. The Contrast option by the way will reduce the amount of feathering, so if a selection had been feathered and you need to tighten things up just a little bit, you can use that Contrast option. But generally speaking, you'll have achieved that feathering by use of the Feather slider, and so you can just reduce the value as needed. We can also smooth out the edge of that mask, you'll see here, for example, things are a little bit rough, and so I can increase the value for Smooth, in order to smooth out that line.
But you do need to be careful, because this will cause that mask edge to no longer align perfectly with certain texture details in the image. And so, sometimes it's a little bit more problematic, and you might instead want to simply paint on the layer mask, in order to clean it up. Of course all of these settings are very helpful but they are not doing much for that mane. So, let's take a look at how we can work with the mane. I'll go ahead and zoom in a little bit on the mane itself, so that we can get a better sense of the results we're achieving while we're working. We can increasing the radius for edge detection, and that will cause Photoshop to evaluate a little bit larger area along the edges of our original selection.
In this case though, that's not really going to help too much because we would need to have a very large radius indeed in this specific area of the photo, but not in the rest of the image. I can turn on the Smart Radius checkbox, and that will cause the radius to expand and contract in various areas, based on how fuzzy the object appears to be. But still, that's not quite producing the best result. But I can actually manually paint to define the radius for edge detection, and that's with the Refine Radius tool, which is available by default in the Refine Mask dialog.
All I have to do is move out into the image, and then I can paint with that tool, in order to define additional areas that the tool should evaluate while determining the best position for that layer mask. So I'll paint some of the fur into position here, adjusting this layer mask while I'm working. And then once I release the mouse, that area will be analyzed and the result will be improved, and you can see Improve significantly. I can continue painting in additional areas as needed with that Refine Radius tool.
But you can see the results are very, very impressive indeed. I'll go ahead and zoom out, and I'll click OK in order to apply those changes. And then I can add another layer so we can see what the image will look like, set against a different color for example. I'll hold the Ctrl key on Windows, or the Cmd key on Macintosh, while clicking o the Create New Layer button at the bottom of the Layers panel. And then I'll choose Edit > Fill and then I'll set the use popup to black, and click OK. And that will fill that layer with black. And we can see for most areas here, that is a very very good layer mask.
I can always go back to my layer mask and click that Mask Edge button once again, and perhaps paint in some additional areas in order to improve the overall mask effect. But clearly, we're getting a very very good blending of the image into that background. So, I'll go ahead and click OK there, and we can see quite a bit of an improvement. And I can continue working to refine that mask to perfection. So, with just a little bit of attention to detail, and thanks to that Refine Mask dialog, we're able to really cleanup even very challenging mask, in order to create a very realistic composite image.
- Composite concepts
- Creating automatic composites
- Image compositing
- Refining layer masks
- Matching images
- Adding effects to composites
- Using layer groups