Easy-to-follow video tutorials help you learn software, creative, and business skills.Become a member

Automated edge detection

From: Photoshop Masking and Compositing: Fundamentals

Video: Automated edge detection

In this exercise, I'm going to introduce you to the automated edge detection features inside the Refine Mask dialog box. And you'll see how they allow you to take a very rough selection outline and make it infinitely better. I'm working inside a file called Sunny and sky.psd found inside the 07_refine folder. Now if you've worked through one of my previous Photoshop courses, you may recognize this file. Be aware however that we'll be employing this file in a very different way. Now the Refine Edge command works with a selection created using any of the selection tools, so the Marquees, the Lasso tools, the automated tools, such as Quick Selection and Magic Wand, as well as the Color Range command.

Automated edge detection

In this exercise, I'm going to introduce you to the automated edge detection features inside the Refine Mask dialog box. And you'll see how they allow you to take a very rough selection outline and make it infinitely better. I'm working inside a file called Sunny and sky.psd found inside the 07_refine folder. Now if you've worked through one of my previous Photoshop courses, you may recognize this file. Be aware however that we'll be employing this file in a very different way. Now the Refine Edge command works with a selection created using any of the selection tools, so the Marquees, the Lasso tools, the automated tools, such as Quick Selection and Magic Wand, as well as the Color Range command.

However, it was originally designed with the Quick Selection tool in mind, so that's the tool we'll start with. Go ahead and select that tool or press the W key. Also, make sure that the Auto-Enhance check box is turned on inside the Options bar. Now I'm going to increase the size of my cursor by pressing the right bracket key a few times. And then I'll paint inside the sky, because after all, it's a heck of a lot easier to select that plain background than the multicolored foreground. However, the Quick Selection tool immediately misbehaves, by selecting the white portion at the top of the sweater, go ahead and press the Alt key or the Option key on the Mac and scrub over that area.

You'll also have to paint along the edge between the hand and that white area of the sweater in order to altogether get rid of those extra selection edges. All right. Now I'm going to zoom in on this area inside of her fingers and I'm going to reduce the size of my cursor by pressing the left bracket key a few times, paint inside that area like so, and then paint inside this other area. You may just be able to get away with just clicking. I'm going to reduce the size of my cursor even farther and then click inside that area right there to make sure I snag it.

Now you don't need to try to paint inside the individual bits of hair. That'll never work with the Quick Selection tool, and that's the whole point of these edge detection functions. They'll automatically detect the hair, as you'll see in just a moment. All right. I'm going to go ahead and zoom out once again. And this time, I'll increase the size of my cursor and I'll paint inside of this area over here on the left- hand side of her neck. That's it! Now we're ready to employ the Refine Edge command. Now you can do so on a couple of different ways. You can go up to the Select menu and choose Refine Edge, or you can click on the Refine Edge button up here in the Options bar.

And it's always available in the Options bar when you're using any of the selection tools. Now what that'll do is it'll automatically return a layer mask for this layer. However, you'll also end up getting rid of the original selection. So what I suggest you do is create a layer mask in advance by dropping down to the Add layer Mask icon at the bottom of the Layers panel and clicking on it. Now I've gone ahead and masked in the sky and masked away the model. That's exactly the opposite of what I wanted. So I'll go up to the Image menu, choose the Adjustments command, and choose Invert, or you can press Ctrl +I or Command+I on the Mac.

And that goes ahead and selects the portion of the image we want. It's not a good selection of course, because we have all this blue gunk inside of her hair. I'm going to take care of that using Refine Mask. So with the layer mask selected, notice, if you go up to the Options bar, Refine Edge is now dimmed because we don't have a selection outline. So what you want to do is go to the Select menu and choose Refine Mask, or once again you can press that keyboard shortcut, Ctrl+Alt+R or Command+Option+R on the Mac. Now there's no sense in applying any of these adjust edge functions. You can fool around with them and if you want, you can increase the Feather value, for example, to blur those edges, but that's not going to get you a credible mask.

Let's go ahead and reduce that setting to 0. Instead what you want to do is play around with these edge detection functions, starting with the Radius value. Now you may recall that when you're applying Gaussian blur, for example, or the Feather command, you have a Radius value and that blurs the selection. That is not what it does here. If you crank the selection value up and I'm going to take it up to 70, what you're doing instead is you're telling the Refine Mask command that it has 70 pixels to work with around the original selected edge.

And that's the area in which it can redefine the selection. To get a sense of what I'm talking about, go ahead and turn on the Show Radius check box up here at the top of the dialog box and you'll notice that you see the radius against the white background. Everything that's not white is the area that's inside of your edge detection radius. And that's the portion of the mask that the Refine Mask command will re-evaluate. Now a 70-pixel radius is great for encompassing most of the hair. However, it's too much radius for the sharp transitions, which include the edges around the fingers and the sweater and so forth.

So if you want Photoshop to automatically analyze that radius and make adjustments, then you turn on the Smart Radius check box. And notice that redraws the radius like so, and we get a big area of radius around the hair and a tight area of radius around the knuckles, and the sweater, and so on. I'm going to turn off the Show Radius check box so that we can see the original image once again. And I just want you to see how much of a difference this command has made so far. If I turn on Show Original, that's the original quick selection mask.

Obviously, it looks like trash. By comparison to what we've done now with just a few clicks, I'll turn Show Original off and we end up getting this accurate hair selection. Now, it incorporates too much of the blue, so we'll need to change that and we can do so using this Refine Radius tool, which is selected by default. And notice if you move your cursor out into the image window, you have a brush cursor with an inset plus sign. And what that tells you is that brushing with this tool will add to the radius. I'm going to increase the size of this brush by pressing the right bracket key, so that same keyboard trick works.

And incidentally if it's not working for you on the PC, it's because one of the values is active. Notice if I click inside the Radius value and try to press the right bracket key, nothing happens. What you need to do is turn on and off one of the check boxes. So I'll just go ahead and turn Show Original on, then turn it back off. And now I can use those bracket keys to once again change the size of my cursor. All right. I'm going to increase the size and just brush over some of this hair and then release and the Refine Mask command goes ahead and re-evaluates that area further.

Now I'll reduce the size of my cursor and paint over this region, further reevaluation. In this area right here, it's done a bad thing. Refine Mask has gone ahead and added back in some blue where previously we had white. So we need to unpaint some of that radius. And you can do that by pressing and holding the Alt key or the Option key on the Mac and notice that gives you a little minus sign inside the brush cursor. Then go ahead and paint in that region and it will restore the original selection. This area however needs to be reevaluated, so I'll just go ahead and paint over it and release and we get rid of some more of the blue.

This area is no good so I need to press the Alt key or the Option key on the Mac and paint over it. We end up getting the white back, which is what we want because we want to mask it away. And then I'll reduce the size of my cursor and I'm going to go ahead and Alt+Drag or Option+Drag over that little bit of hair right there in order to more or less reinstate it. That ends putting some of the blue back into play, so I'll increase the size of my cursor, maybe just click right there in order to try to get rid of some of that blue. I believe actually I need to click inside of the blue itself in order to remove it.

Scroll over to the left a little bit, increase the size of my cursor once again, drag over that bit of hair right there, Photoshop goes ahead and re-evaluates the mask, does a terrific job as well. Notice that we have some white sort of bleeding into the neck right there. That's translucency. I'm not sure that's necessarily such a bad thing, but you can get rid of that, you can reestablish the Opacity of the neck by Alt+Dragging or Option+Dragging over this region, like so. And that looks pretty darn good to me.

I'm going to go ahead and click OK, in order to apply the command and notice what a difference it makes. If I press Ctrl+Z or Command+Z on the Mac, this is the original Quick Selection Mask, just looks awful. And then if I press Ctrl+Z or Command +Z again, that is the automatically modified version of the mask. Thanks to the edge detection functions inside the Refine Mask dialog box.

Show transcript

This video is part of

Image for Photoshop Masking and Compositing: Fundamentals
Photoshop Masking and Compositing: Fundamentals

128 video lessons · 31083 viewers

Deke McClelland
Author

 
Expand all | Collapse all
  1. 15m 25s
    1. Welcome
      1m 12s
    2. Loading my custom dekeKeys shortcuts
      3m 45s
    3. Adjusting the color settings
      4m 29s
    4. Setting up a power workspace
      5m 59s
  2. 1h 0m
    1. The channel is the origin of masking
      1m 54s
    2. The Masks and Channels panels
      4m 48s
    3. How color channels work
      7m 7s
    4. Viewing channels in color
      3m 24s
    5. How RGB works
      4m 12s
    6. Single-channel grayscale
      5m 12s
    7. Mixing a custom "fourth" channel
      5m 15s
    8. The other three-channel mode: Lab
      5m 45s
    9. A practical application of Lab
      4m 55s
    10. The final color mode: CMYK
      7m 5s
    11. Introducing the Multichannel mode
      5m 56s
    12. Creating a unique multichannel effect
      5m 18s
  3. 44m 27s
    1. The alpha channel is home to the mask
      1m 40s
    2. The origins of the alpha channel
      3m 40s
    3. How a mask works
      7m 10s
    4. Making an alpha channel
      4m 2s
    5. Using the new channel icons
      6m 27s
    6. Saving an image with alpha channels
      4m 23s
    7. Loading a selection from a channel
      4m 7s
    8. Putting a mask into play
      3m 55s
    9. Loading a selection from a layer
      4m 27s
    10. Loading a selection from another image
      4m 36s
  4. 1h 0m
    1. The mask meets the composition
      1m 8s
    2. Viewing a mask as a rubylith overlay
      6m 13s
    3. Changing a mask's overlay color
      5m 34s
    4. Painting inside a mask
      6m 3s
    5. Cleaning up and confirming
      5m 18s
    6. Combining masks
      5m 10s
    7. Painting behind and inside a layer
      5m 27s
    8. Blending image elements
      6m 1s
    9. What to do when layers go wrong
      6m 3s
    10. Hiding layer effects with a mask
      4m 22s
    11. Introducing clipping masks
      5m 29s
    12. Unclipping and masking a shadow
      3m 50s
  5. 1h 35m
    1. The seven selection soldiers
      52s
    2. The marquee tools
      6m 31s
    3. The single-pixel tools (plus tool tricks)
      6m 48s
    4. Turning a destructive edit into a layer
      5m 34s
    5. Making shapes of specific sizes
      7m 7s
    6. The lasso tools
      5m 49s
    7. Working with the Magnetic Lasso tool
      7m 19s
    8. The Quick Selection tool
      8m 13s
    9. Combining Quick Selection and Smudge
      4m 52s
    10. The Magic Wand and the Tolerance value
      6m 55s
    11. Contiguous and Anti-aliased selections
      6m 58s
    12. Making a good selection with the Magic Wand
      6m 34s
    13. Selecting and replacing a background
      6m 55s
    14. Resolving edges with layer effects
      7m 52s
    15. Adding lines of brilliant gold type
      7m 28s
  6. 1h 11m
    1. Selections reign supreme
      55s
    2. Introducing "selection calculations"
      4m 19s
    3. Combining two different tools
      7m 29s
    4. Selections and transparency masks
      5m 17s
    5. Selecting an eye
      7m 1s
    6. Masking and blending a texture into skin
      5m 1s
    7. Painting a texture into an eye
      4m 19s
    8. Combining layers, masks, channels, and paths
      4m 54s
    9. Moving selection outlines vs. selected pixels
      5m 36s
    10. Transforming and warping a selection outline
      7m 45s
    11. Pasting an image inside a selection
      7m 26s
    12. Adding volumetric shadows and highlights
      6m 54s
    13. Converting an image into a mask
      4m 42s
  7. 1h 5m
    1. The best selection tools are commands
      1m 5s
    2. Introducing the Color Range command
      5m 59s
    3. Working in the Color Range dialog box
      7m 7s
    4. Primary colors and luminance ranges
      4m 12s
    5. A terrific use for Color Range
      4m 57s
    6. Introducing the Quick Mask mode
      7m 43s
    7. Moving a selection into a new background
      5m 43s
    8. Smoothing the mask, recreating the corners
      8m 43s
    9. Integrating foreground and background
      4m 44s
    10. Creating a cast shadow from a layer
      2m 51s
    11. Releasing and masking layer effects
      3m 11s
    12. Creating a synthetic rainbow effect
      4m 30s
    13. Masking and compositing your rainbow
      4m 46s
  8. 1h 17m
    1. The ultimate in masking automation
      1m 6s
    2. Introducing the Refine Mask command
      6m 58s
    3. Automated edge detection
      8m 23s
    4. Turning garbage into gold
      6m 19s
    5. Starting with an accurate selection
      7m 11s
    6. Selection outline in, layer mask out
      7m 48s
    7. Matching a scene with Smart Filters
      4m 29s
    8. Cooling a face, reflecting inside eyes
      4m 45s
    9. Creating a layer of ghoulish skin
      4m 28s
    10. Adding dark circles around the eyes
      5m 20s
    11. Creating a fake blood effect
      5m 38s
    12. Establishing trails of blood
      7m 40s
    13. Integrating the blood into the scene
      7m 3s
  9. 1h 48m
    1. Using the image to select itself
      1m 37s
    2. Choosing the ideal base channel
      5m 7s
    3. Converting a channel into a mask
      6m 34s
    4. Painting with the Overlay mode
      7m 27s
    5. Painting with the Soft Light mode
      5m 55s
    6. Mask, composite, refine, and blend
      4m 40s
    7. Creating a more aggressive mask
      7m 2s
    8. Blending differently masked layers
      7m 0s
    9. Creating a hair-only mask
      6m 0s
    10. Using history to regain a lost mask
      3m 42s
    11. Separating flesh tones from hair
      8m 28s
    12. Adjusting a model's color temperature
      4m 30s
    13. Introducing the Calculations command
      7m 22s
    14. Extracting a mask from a Smart Object
      6m 34s
    15. Integrating a bird into a new sky
      5m 40s
    16. Creating synthetic rays of light
      6m 4s
    17. Masking and compositing light
      7m 39s
    18. Introducing a brilliant light source
      7m 5s
  10. 1h 34m
    1. The synthesis of masking and compositing
      1m 36s
    2. White reveals, black conceals
      6m 45s
    3. Layer masking tips and tricks
      5m 8s
    4. Generating a layer mask with Color Range
      5m 38s
    5. The Masks panel's bad options
      5m 18s
    6. The Masks panel's good options
      3m 50s
    7. Creating and feathering a vector mask
      3m 42s
    8. Combining pixel and vector masks
      3m 50s
    9. Working with path outlines
      7m 10s
    10. Combining paths into a single vector mask
      7m 52s
    11. Sharpening detail, reducing color noise
      4m 27s
    12. Recreating missing details
      8m 49s
    13. Masking glass
      5m 50s
    14. Refining a jagged Magic Wand mask
      5m 53s
    15. Masking multiple layers at one time
      5m 15s
    16. Establishing a knockout layer
      6m 6s
    17. Clipping and compositing tricks
      7m 37s
  11. 1m 17s
    1. Next steps
      1m 17s

Start learning today

Get unlimited access to all courses for just $25/month.

Become a member
Sometimes @lynda teaches me how to use a program and sometimes Lynda.com changes my life forever. @JosefShutter
@lynda lynda.com is an absolute life saver when it comes to learning todays software. Definitely recommend it! #higherlearning @Michael_Caraway
@lynda The best thing online! Your database of courses is great! To the mark and very helpful. Thanks! @ru22more
Got to create something yesterday I never thought I could do. #thanks @lynda @Ngventurella
I really do love @lynda as a learning platform. Never stop learning and developing, it’s probably our greatest gift as a species! @soundslikedavid
@lynda just subscribed to lynda.com all I can say its brilliant join now trust me @ButchSamurai
@lynda is an awesome resource. The membership is priceless if you take advantage of it. @diabetic_techie
One of the best decision I made this year. Buy a 1yr subscription to @lynda @cybercaptive
guys lynda.com (@lynda) is the best. So far I’ve learned Java, principles of OO programming, and now learning about MS project @lucasmitchell
Signed back up to @lynda dot com. I’ve missed it!! Proper geeking out right now! #timetolearn #geek @JayGodbold
Share a link to this course

What are exercise files?

Exercise files are the same files the author uses in the course. Save time by downloading the author's files instead of setting up your own files, and learn by following along with the instructor.

Can I take this course without the exercise files?

Yes! If you decide you would like the exercise files later, you can upgrade to a premium account any time.

Become a member Download sample files See plans and pricing

Please wait... please wait ...
Upgrade to get access to exercise files.

Exercise files video

How to use exercise files.

Learn by watching, listening, and doing, Exercise files are the same files the author uses in the course, so you can download them and follow along Premium memberships include access to all exercise files in the library.


Exercise files

Exercise files video

How to use exercise files.

For additional information on downloading and using exercise files, watch our instructional video or read the instructions in the FAQ .

This course includes free exercise files, so you can practice while you watch the course. To access all the exercise files in our library, become a Premium Member.

Are you sure you want to mark all the videos in this course as unwatched?

This will not affect your course history, your reports, or your certificates of completion for this course.


Mark all as unwatched Cancel

Congratulations

You have completed Photoshop Masking and Compositing: Fundamentals.

Return to your organization's learning portal to continue training, or close this page.


OK
Become a member to add this course to a playlist

Join today and get unlimited access to the entire library of video courses—and create as many playlists as you like.

Get started

Already a member ?

Become a member to like this course.

Join today and get unlimited access to the entire library of video courses.

Get started

Already a member?

Exercise files

Learn by watching, listening, and doing! Exercise files are the same files the author uses in the course, so you can download them and follow along. Exercise files are available with all Premium memberships. Learn more

Get started

Already a Premium member?

Exercise files video

How to use exercise files.

Ask a question

Thanks for contacting us.
You’ll hear from our Customer Service team within 24 hours.

Please enter the text shown below:

The classic layout automatically defaults to the latest Flash Player.

To choose a different player, hold the cursor over your name at the top right of any lynda.com page and choose Site preferences from the dropdown menu.

Continue to classic layout Stay on new layout
Exercise files

Access exercise files from a button right under the course name.

Mark videos as unwatched

Remove icons showing you already watched videos if you want to start over.

Control your viewing experience

Make the video wide, narrow, full-screen, or pop the player out of the page into its own window.

Interactive transcripts

Click on text in the transcript to jump to that spot in the video. As the video plays, the relevant spot in the transcript will be highlighted.

Learn more, save more. Upgrade today!

Get our Annual Premium Membership at our best savings yet.

Upgrade to our Annual Premium Membership today and get even more value from your lynda.com subscription:

“In a way, I feel like you are rooting for me. Like you are really invested in my experience, and want me to get as much out of these courses as possible this is the best place to start on your journey to learning new material.”— Nadine H.

Thanks for signing up.

We’ll send you a confirmation email shortly.


Sign up and receive emails about lynda.com and our online training library:

Here’s our privacy policy with more details about how we handle your information.

Keep up with news, tips, and latest courses with emails from lynda.com.

Sign up and receive emails about lynda.com and our online training library:

Here’s our privacy policy with more details about how we handle your information.

   
submit Lightbox submit clicked
Terms and conditions of use

We've updated our terms and conditions (now called terms of service).Go
Review and accept our updated terms of service.