Easy-to-follow video tutorials help you learn software, creative, and business skills.Become a member

Refining a layer's transparency with rotoscoping

From: Maya Rendering for After Effects Composites

Video: Refining a layer's transparency with rotoscoping

We have applied the motion tracking data to the render layer, and now the motion tracking is working where the spyglass follows the hand of the actress. One thing we had to do though is spend a little bit more time working with the anchor points. Because there is a little bit of slide based in the way the spyglass is rendered in terms of it's sliding left, right, up, or down compared to her hand, we had to keyframe the anchor point, and it took me a total of six additional keyframes to make the motion look better. Let's play it back there and see what it looks like. So the spyglass is following her hand fairly well.

Refining a layer's transparency with rotoscoping

We have applied the motion tracking data to the render layer, and now the motion tracking is working where the spyglass follows the hand of the actress. One thing we had to do though is spend a little bit more time working with the anchor points. Because there is a little bit of slide based in the way the spyglass is rendered in terms of it's sliding left, right, up, or down compared to her hand, we had to keyframe the anchor point, and it took me a total of six additional keyframes to make the motion look better. Let's play it back there and see what it looks like. So the spyglass is following her hand fairly well.

It looks like it's right there where her fingertips are. Now, one problem of course is the fact that the spyglass handle covers up her hand. This should be really in between her fingers. This is a common problem with visual effects compositing. You have a CG Render, you have to put it behind something that was shot in the real world. So, how do you do that? Well, the best way to fix that is through rotoscoping, which is a very common task. With rotoscoping, you can create a mask, and that mask will affect the Alpha channel of the layer you apply it to. Therefore, we can draw a mask to cut out the bottom of this handle.

So, let's give that a try. I'll go to frame 1, or frame 0, zoom in. What I can do is select the render layer, and then draw a mask with the Pen tool up at the top. With the Pen tool selected, I can click in the Viewer, and each time I click, I get a mask point. So, I'm going to outline the top of the fingers here, because this is where it's going to cut, and then drop down, and just make it extra big down here. Now, I do need to make sure it's a closed path all the way around. So, if I click on the very first point, I get that little circle icon, and I can close that path.

As soon as it's closed, that mask is going to start working. Now, in this case, it saved the lower handle, and cut out the rest. What happens is whatever is inside that closed path becomes solid Alpha, in other words, it's kept. Anything beyond the mask becomes transparent Alpha and is tossed away. So we want the opposite result. We want to keep the top to get rid of bottom. You can do that by going to the Mask options. Once you draw a mask, and go to the Masks section on that layer, and there is Mask 1, expand that, and you have a few options here.

In this case, you want to check on the Inverted Checkbox to get the opposite result. So now it's starting to work. Now I'm going to have to keyframe this mask's changing shape over time because her hand is in different position, therefore this rotoscoping has to change over time too. Before we get to that though, I want to adjust the mask a little bit further. Now, one thing you can do to see what the quality is, is hide this from the Viewer. So I can go to the Toggle Mask button, click that, and that yellow line goes away. However, the result is still there, I can still see what's happening.

So it looks pretty good. However, there's a really hard edge there in terms of the transition. So another thing you can do is go to the Feather on the mask, and increase that, and go from 0 to 3, and that gives me a softer transition. That actually looks better, and is a better match. I will zoom out again and then turn on the Mask. Now, right now the mask has some very hard corners. There are different ways to make softer transitions between the points. One I like to use is called RotoBezier. So, what I'll do is pick the entire mask, and then go to the Selection tool, double-click at the mask lines until the entire mask is selected.

I can tell it's selected because all the points are solid, which means they're selected, and also there is a big Transform Box around it, but I just want to pick the entire thing. I then go up to layer, go to Mask and Shape Path and go to RotoBezier. This converts the mask into a special version, where it's a little bit easier to get smoother, or harder, transitions interactively. Once I have converted it, I can click off this to pick nothing, and then for instance pick one point by clicking on it so it becomes solid and then change that to become smoother, or harder, and there's a special tool for that.

I'll go up to the Pen tool, click and hold down. There's something called Convert Vertex tool. When I pick that, I get the special arrow up here, and I can then place it over at that selected point. I'll see that arrow there in the Viewer also. Then I can click-drag left or right. It interactively makes it harder to the left, or smoother to the right. Once it's converted to RotoBezier, it becomes really easy to convert these points, and interactively change whether you have a hard corner, or a soft corner. So I can click on one at a time with this tool, click-drag left or right, and affect the way that point is transitioning.

So, this is actually a better shape, it's smoother, and it matches the fingers better here. So I'm going to go back to my Selection tool. Now again, I am going to have to keyframe this, but I think we're ready to do that now. So, while I'm on the first frame, frame 0, I'm going to click the Time icon beside Mask Path. That gives me a keyframe right there. Now, the keyframe stores the entire shape of that mask. So now I can go to additional frame, or a later frame, and change the shape, and get additional keyframes. So, I'm going to go to frame 10, and then change the shape. In order to change the shape, what I can do is click off of the mask to pick nothing, and with my Selection tool selected, click on one point at a time and interactively move it to reshape that path.

As soon as I reshape the path by moving points, I get a new keyframe. So, I want to do this for the entire timeline. Now, I don't need to do it for every single frame. In fact, there are several ways to approach it, one is to simply jump ahead every 10 frames, maybe another would be to bisect where you go to the end frame. Next, you shape it, and then go to a center frame. You bisect what you already have. Go to some center position, change the shape, and then go to another position that's in the center of your old keyframe, so you bisect old ones.

Again, reshape your mask, and then continue to bisect. In this case, because the motion is not that big, it's relatively subtle, this works fairly well. I'm doing it fairly quick though, you defiantly want to spend some time to make sure that the mask is following the fingers accurately for the entire duration. So, it might take a few more keyframes. Another thing you can do when you adjust the mask is move the entire thing as one unit, and we have already seen that when we select the mask by double-clicking, get the Transform Box. If you see the Transform Box, you can click-drag in the center, and move the entire thing as a single piece.

This is often very useful. Again, if I were to do that, I get a new keyframe, or it overwrites old keyframe. So, let's turn off the mask so we can't see it here, and let's play it back, fingers, and the rotoscoping is left in, so it's okay. Now, you will probably want to spend more time on yours, but that's a general idea in terms of how to apply it, and how to make it cut out the handle. rotoscoping has allowed us to place the spyglass behind her hand, or at least make it look like it's behind her fingers. We're now ready to move on to other steps where we further integrate the spyglass, so it better matches the footage.

Show transcript

This video is part of

Image for Maya Rendering for After Effects Composites
Maya Rendering for After Effects Composites

34 video lessons · 5331 viewers

Lee Lanier
Author

 
Expand all | Collapse all
  1. 2m 0s
    1. Welcome
      47s
    2. Using the exercise files
      1m 13s
  2. 29m 49s
    1. Working with image sequences
      7m 24s
    2. Importing reference video as an image plane
      5m 13s
    3. Matching the 3D camera to the video footage
      4m 23s
    4. Lighting the model
      5m 35s
    5. Creating mattes and shadows in preparation for rendering
      7m 14s
  3. 15m 38s
    1. Using the Render Layer Editor
      4m 21s
    2. Splitting a scene into multiple render passes
      6m 6s
    3. Adding flexibility by assigning material and render overrides
      5m 11s
  4. 15m 2s
    1. Creating render passes using mental ray
      3m 50s
    2. Batch rendering render passes: Project one
      5m 24s
    3. Batch rendering render passes: Project two
      5m 48s
  5. 19m 4s
    1. Importing render passes into After Effects
      6m 25s
    2. Recombining render passes in a composition
      6m 31s
    3. Transforming multiple render passes as a single unit
      6m 8s
  6. 48m 7s
    1. Setting up a motion tracker
      5m 17s
    2. Using a tracker to analyze motion in footage
      3m 56s
    3. Adjusting tracker options for better results
      7m 2s
    4. Matching layer motion by applying tracker data
      6m 26s
    5. Refining a layer's transparency with rotoscoping
      6m 45s
    6. Improving layer movement with the Smoother tool
      5m 7s
    7. Improving the CG by adding blur and effects
      8m 7s
    8. Adding shadow to make the composite believable
      5m 27s
  7. 32m 36s
    1. Recombining render passes for project two
      5m 17s
    2. Removing unwanted elements with a garbage mask
      4m 57s
    3. Applying motion tracking data to a null layer
      6m 38s
    4. Adjusting shadows and matte edges
      8m 12s
    5. Using color correction to improve layer integration
      7m 32s
  8. 25m 46s
    1. Stabilizing shaky video with the Tracker
      8m 2s
    2. Tracking rectangular elements with the Perspective corner pin option
      5m 31s
    3. Adjusting corner pin points and paths
      6m 56s
    4. Applying corner pin data to multiple layers
      5m 17s
  9. 1m 16s
    1. Next steps
      1m 16s

Start learning today

Get unlimited access to all courses for just $25/month.

Become a member
Sometimes @lynda teaches me how to use a program and sometimes Lynda.com changes my life forever. @JosefShutter
@lynda lynda.com is an absolute life saver when it comes to learning todays software. Definitely recommend it! #higherlearning @Michael_Caraway
@lynda The best thing online! Your database of courses is great! To the mark and very helpful. Thanks! @ru22more
Got to create something yesterday I never thought I could do. #thanks @lynda @Ngventurella
I really do love @lynda as a learning platform. Never stop learning and developing, it’s probably our greatest gift as a species! @soundslikedavid
@lynda just subscribed to lynda.com all I can say its brilliant join now trust me @ButchSamurai
@lynda is an awesome resource. The membership is priceless if you take advantage of it. @diabetic_techie
One of the best decision I made this year. Buy a 1yr subscription to @lynda @cybercaptive
guys lynda.com (@lynda) is the best. So far I’ve learned Java, principles of OO programming, and now learning about MS project @lucasmitchell
Signed back up to @lynda dot com. I’ve missed it!! Proper geeking out right now! #timetolearn #geek @JayGodbold
Share a link to this course

What are exercise files?

Exercise files are the same files the author uses in the course. Save time by downloading the author's files instead of setting up your own files, and learn by following along with the instructor.

Can I take this course without the exercise files?

Yes! If you decide you would like the exercise files later, you can upgrade to a premium account any time.

Become a member Download sample files See plans and pricing

Please wait... please wait ...
Upgrade to get access to exercise files.

Exercise files video

How to use exercise files.

Learn by watching, listening, and doing, Exercise files are the same files the author uses in the course, so you can download them and follow along Premium memberships include access to all exercise files in the library.


Exercise files

Exercise files video

How to use exercise files.

For additional information on downloading and using exercise files, watch our instructional video or read the instructions in the FAQ .

This course includes free exercise files, so you can practice while you watch the course. To access all the exercise files in our library, become a Premium Member.

Join now Already a member? Log in

Are you sure you want to mark all the videos in this course as unwatched?

This will not affect your course history, your reports, or your certificates of completion for this course.


Mark all as unwatched Cancel

Congratulations

You have completed Maya Rendering for After Effects Composites.

Return to your organization's learning portal to continue training, or close this page.


OK
Become a member to add this course to a playlist

Join today and get unlimited access to the entire library of video courses—and create as many playlists as you like.

Get started

Already a member ?

Become a member to like this course.

Join today and get unlimited access to the entire library of video courses.

Get started

Already a member?

Exercise files

Learn by watching, listening, and doing! Exercise files are the same files the author uses in the course, so you can download them and follow along. Exercise files are available with all Premium memberships. Learn more

Get started

Already a Premium member?

Exercise files video

How to use exercise files.

Ask a question

Thanks for contacting us.
You’ll hear from our Customer Service team within 24 hours.

Please enter the text shown below:

The classic layout automatically defaults to the latest Flash Player.

To choose a different player, hold the cursor over your name at the top right of any lynda.com page and choose Site preferences from the dropdown menu.

Continue to classic layout Stay on new layout
Exercise files

Access exercise files from a button right under the course name.

Mark videos as unwatched

Remove icons showing you already watched videos if you want to start over.

Control your viewing experience

Make the video wide, narrow, full-screen, or pop the player out of the page into its own window.

Interactive transcripts

Click on text in the transcript to jump to that spot in the video. As the video plays, the relevant spot in the transcript will be highlighted.

Are you sure you want to delete this note?

No

Your file was successfully uploaded.

Thanks for signing up.

We’ll send you a confirmation email shortly.


Sign up and receive emails about lynda.com and our online training library:

Here’s our privacy policy with more details about how we handle your information.

Keep up with news, tips, and latest courses with emails from lynda.com.

Sign up and receive emails about lynda.com and our online training library:

Here’s our privacy policy with more details about how we handle your information.

   
submit Lightbox submit clicked
Terms and conditions of use

We've updated our terms and conditions (now called terms of service).Go
Review and accept our updated terms of service.