Easy-to-follow video tutorials help you learn software, creative, and business skills.Become a member

Masking with the Pen tool

From: After Effects Apprentice 05: Creating Transparency

Video: Masking with the Pen tool

In the previous chapter we used the mask shape tool to drag out some simple mask shapes to create transparency for a layer. However, if we need a more complex shape, for example to cut out a flower such as this, we need to use a different tool, the Pen tool. Its shortcut is G. I select the Pen tool and make sure the first option is currently selected. You can go ahead and switch to these other options in context while you're editing. So make sure that first one is the currently active one. And for now we're going to turn off the RotoBezier option.

Masking with the Pen tool

In the previous chapter we used the mask shape tool to drag out some simple mask shapes to create transparency for a layer. However, if we need a more complex shape, for example to cut out a flower such as this, we need to use a different tool, the Pen tool. Its shortcut is G. I select the Pen tool and make sure the first option is currently selected. You can go ahead and switch to these other options in context while you're editing. So make sure that first one is the currently active one. And for now we're going to turn off the RotoBezier option.

We'll demonstrate RotoBezier a little bit later on. If you have the exercise files that came with this project, open up the comp 02a-Bezier*starter. Bezier is the name of the type of curve you'll be creating with the Pen tool. And just as before make sure your layer is selected before you start drawing out a mask. Otherwise you'll just creating a brand-new shape layer rather than cutting out your desired layer. We're going to initially just click around to get familiar with using the Pen tool then we'll use it more carefully to try to cut out this flower.

If you just click the mouse, move it, and click again, you'll draw straight line segments for your mask shape. However, if you move to another location to press the mouse button and before you release start to dragging, you'll get curved mask shapes. Now you go ahead and drag on this handle to create its length, which is its influence and its angle. I'll click and drag on another point, then drag at another point, at another point. Now it's quite common while you're dragging out a mask shape that you realize may be you made a mistake with one of these previous points.

You can wait till you're done creating a mask. This little O by the Pen tool says you're about to close the mask. And then edit it later. That's what I am going to show you in the next movie. Or if you just can't stand looking at your mistakes you can go ahead and edit it before you've finished a mask. If you want to delete a previous point, one that's not currently selected, just hover your cursor over it. You'll see a little minus symbol appear next to your cursor. Click that and it will delete that point. If you want to add a point along an existing line segment place the cursor over that segment and now it'll turn to a plus symbol.

And now you have a brand-new point. If you want to move the location of a previous mask point or vertex or re-drag one of the handles, you need to hold down Command on Mac or Ctrl on Windows and watch the cursor change to this solid black arrow. Once you have that, you've got the Selection tool which means you can select and relocate your mask points as well as drag out the handles again. When you release the Command or Ctrl key, you get this upside down V cursor. That is the Change Direction tool.

If you were to drag on a handle in this condition you would break the handle and create a discontinuous curve through that segment. Now you don't always get that Change Direction tool just by hovering your cursor over it. If you do want to change direction of a vertex hold down the Option key on Mac or Alt key on Windows and then you'll get that inverted V. Clicking on a straight line segment point will convert it to a rounded segment. Clicking on a rounded segment point we will convert it back to a straight corner again. If you have discontinuous points like I have here where I've dragged out the handle, you'll need to convert it back to either a straight line segment or a curved segment.

Again you'll hold down Option or Alt, get the Change Direction cursor. Click once to get a straight segment; click again to go ahead and get now your Bezier handles. And if you want to edit those Bezier handles, hold down Command on Mac or Ctrl on Windows. In general hover your cursor over existing points or handles. Observe the cursor shape. For example, this inverted V is the Change Direction tool. And then on Mac, use Command and Option or on Windows use Ctrl and Alt to change the purpose of that tool. Solid arrow is the Selection tool.

Inverted V is the Change Direction tool. To continue drawing your mask, you need to reselect your last point that you drew along your mask shape, which means to holding down Command or Ctrl, clicking on it, and now you've selected it again. And now you can go ahead and continue clicking at your mask shape. To close your mask path hover the cursor over your first mask point. It's the one with 2 boxes around it instead of one. You'll see a little O appear next to the Pen tool and that indicates you're about to close your mask path.

Click and now you've got a closed mask shape that reveals just the portion of the layer that's inside that mask. By the way, you don't need to create closed mask shapes. We'll show you later on. A good reason is to create open that mask shapes. To do that, when you're almost ready to close, just switch back to the Selection tool and now you have an open mask path. So now that you have a general idea, I am going to give you a quick demonstration of how to do masking for real. And I'll go ahead and select the mask and delete the whole shape.

I am going to go ahead and zoom up to 200% so I can see my flower in greater detail. Hold down the spacebar to temporarily switch to the Hand tool and center it up. Select my Pen tool and start dragging. Okay. Let's say I want to start here. I've got a sharp corner so I only need to click once to create a sharp corner there. And I'll start dragging out some little curves right along this edge as I go.

If I have a sharper corner I'll have very short handles. If I have a more sweeping corner, I'll go ahead and drag out longer handles. If I have a discontinuous edge like a corner, I'll just do single click to add a point without any handles. Let's just keep dragging here out. That's kind of smooth. It has kind of a square edge there. Once I get these longer segments, I'll go ahead and drag out much longer handles for longer influence, like that.

Hold down spacebar to temporarily get the Hand tool and keep dragging along the edge of my tulip. Sharp corner there, sharp corner there, and continue to draw around the shape. I am not going to make you sit there and watch me draw all the rest of this mask shape. However, it would be educational to see how I correct this mask shape and that's what I am going to do in the next movie.

Show transcript

This video is part of

Image for After Effects Apprentice 05: Creating Transparency
After Effects Apprentice 05: Creating Transparency

38 video lessons · 13195 viewers

Chris Meyer and Trish Meyer
Author

 
Expand all | Collapse all
  1. 3m 17s
    1. Overview
      1m 17s
    2. Using the exercise files
      2m 0s
  2. 28m 53s
    1. Creating basic mask shapes
      5m 7s
    2. Using advanced parametric shapes
      3m 35s
    3. Basic mask shape editing
      4m 35s
    4. Masking in the Layer panel
      1m 55s
    5. Working with mask parameters
      4m 0s
    6. Animating masks
      5m 52s
    7. Creating vignettes
      3m 49s
  3. 25m 56s
    1. Masking with the Pen tool
      6m 44s
    2. Editing a mask path
      6m 19s
    3. Using RotoBezier masks
      4m 32s
    4. Targeting masks for animation
      3m 13s
    5. Setting the first vertex point
      5m 8s
  4. 19m 7s
    1. The basics of variable-mask feathering
      7m 33s
    2. Exploring variable-mask feathering options
      5m 56s
    3. Exploring visual effects applications
      5m 38s
  5. 7m 55s
    1. Exploring mask modes
      3m 58s
    2. Fading mask opacity
      3m 57s
  6. 11m 48s
    1. Using targeted treatments
      2m 35s
    2. Filling mask shapes
      3m 8s
    3. Following mask paths
      6m 5s
  7. 13m 31s
    1. Using alpha mattes
      3m 48s
    2. Grouping track matte pairs
      3m 40s
    3. Working with luma mattes
      3m 52s
    4. Animating mattes
      2m 11s
  8. 10m 0s
    1. Using Stencil Luma
      3m 5s
    2. Using Stencil Alpha
      2m 5s
    3. Using effects with stencils
      2m 20s
    4. Stacking stencils
      2m 30s
  9. 8m 23s
    1. Quizzler challenges
      1m 18s
    2. Quizzler solution one: One word at a time
      3m 53s
    3. Quizzler solution two: Stroke drawing direction
      3m 12s
  10. 11m 18s
    1. Idea corner one: More masks and effects
      2m 50s
    2. Idea corner two: Transition effects
      3m 44s
    3. Idea corner three: Sequenced layers as mattes
      4m 44s
  11. 16m 29s
    1. Understanding track matte rendering order
      5m 48s
    2. Exploring mask interpolation
      10m 41s

Start learning today

Get unlimited access to all courses for just $25/month.

Become a member
Sometimes @lynda teaches me how to use a program and sometimes Lynda.com changes my life forever. @JosefShutter
@lynda lynda.com is an absolute life saver when it comes to learning todays software. Definitely recommend it! #higherlearning @Michael_Caraway
@lynda The best thing online! Your database of courses is great! To the mark and very helpful. Thanks! @ru22more
Got to create something yesterday I never thought I could do. #thanks @lynda @Ngventurella
I really do love @lynda as a learning platform. Never stop learning and developing, it’s probably our greatest gift as a species! @soundslikedavid
@lynda just subscribed to lynda.com all I can say its brilliant join now trust me @ButchSamurai
@lynda is an awesome resource. The membership is priceless if you take advantage of it. @diabetic_techie
One of the best decision I made this year. Buy a 1yr subscription to @lynda @cybercaptive
guys lynda.com (@lynda) is the best. So far I’ve learned Java, principles of OO programming, and now learning about MS project @lucasmitchell
Signed back up to @lynda dot com. I’ve missed it!! Proper geeking out right now! #timetolearn #geek @JayGodbold
Share a link to this course

What are exercise files?

Exercise files are the same files the author uses in the course. Save time by downloading the author's files instead of setting up your own files, and learn by following along with the instructor.

Can I take this course without the exercise files?

Yes! If you decide you would like the exercise files later, you can upgrade to a premium account any time.

Become a member Download sample files See plans and pricing

Please wait... please wait ...
Upgrade to get access to exercise files.

Exercise files video

How to use exercise files.

Learn by watching, listening, and doing, Exercise files are the same files the author uses in the course, so you can download them and follow along Premium memberships include access to all exercise files in the library.


Exercise files

Exercise files video

How to use exercise files.

For additional information on downloading and using exercise files, watch our instructional video or read the instructions in the FAQ .

This course includes free exercise files, so you can practice while you watch the course. To access all the exercise files in our library, become a Premium Member.

Join now Already a member? Log in

Are you sure you want to mark all the videos in this course as unwatched?

This will not affect your course history, your reports, or your certificates of completion for this course.


Mark all as unwatched Cancel

Congratulations

You have completed After Effects Apprentice 05: Creating Transparency.

Return to your organization's learning portal to continue training, or close this page.


OK
Become a member to add this course to a playlist

Join today and get unlimited access to the entire library of video courses—and create as many playlists as you like.

Get started

Already a member ?

Become a member to like this course.

Join today and get unlimited access to the entire library of video courses.

Get started

Already a member?

Exercise files

Learn by watching, listening, and doing! Exercise files are the same files the author uses in the course, so you can download them and follow along. Exercise files are available with all Premium memberships. Learn more

Get started

Already a Premium member?

Exercise files video

How to use exercise files.

Ask a question

Thanks for contacting us.
You’ll hear from our Customer Service team within 24 hours.

Please enter the text shown below:

The classic layout automatically defaults to the latest Flash Player.

To choose a different player, hold the cursor over your name at the top right of any lynda.com page and choose Site preferences from the dropdown menu.

Continue to classic layout Stay on new layout
Exercise files

Access exercise files from a button right under the course name.

Mark videos as unwatched

Remove icons showing you already watched videos if you want to start over.

Control your viewing experience

Make the video wide, narrow, full-screen, or pop the player out of the page into its own window.

Interactive transcripts

Click on text in the transcript to jump to that spot in the video. As the video plays, the relevant spot in the transcript will be highlighted.

Learn more, save more. Upgrade today!

Get our Annual Premium Membership at our best savings yet.

Upgrade to our Annual Premium Membership today and get even more value from your lynda.com subscription:

“In a way, I feel like you are rooting for me. Like you are really invested in my experience, and want me to get as much out of these courses as possible this is the best place to start on your journey to learning new material.”— Nadine H.

Thanks for signing up.

We’ll send you a confirmation email shortly.


Sign up and receive emails about lynda.com and our online training library:

Here’s our privacy policy with more details about how we handle your information.

Keep up with news, tips, and latest courses with emails from lynda.com.

Sign up and receive emails about lynda.com and our online training library:

Here’s our privacy policy with more details about how we handle your information.

   
submit Lightbox submit clicked
Terms and conditions of use

We've updated our terms and conditions (now called terms of service).Go
Review and accept our updated terms of service.