Learn to Master the Roto Brush Tool in After Effects

show more Using the Roto Brush tool provides you with in-depth training on Video. Taught by Chad Perkins as part of the After Effects CS5 Essential Training show less
please wait ...

Using the Roto Brush tool

There is a powerful new type of brush in AfterEffects CS5 called the Roto Brush. And what this does this allows you to go in and select certain objects and remove them from the background. So, I have here this motion graphics back on, for example, with this snowboarder over it, but this is the same photo you were looking at before, which looks like this. So, what we're going to do in this movie is we're going to use the Roto Brush to take regular footage, and isolate the snowboarder so that he appears to jump off of our motion graphics background and then go back to the regular snowy environment to which he feels accustomed.

So, kind of a cool feature to be able to isolate objects like that. So, let's go over to the Start composition and see how this works. I've created some comp markers here for you so we know where to begin and where to end the Roto Brush. Go ahead and double-click this layer to open it up in the Layer panel. Just like every other brush, we need to be in the layer panel to use it, select the Roto Brush tool, which is merely to the right of the Brush tool, which we looked that in the last movie. Oh, yeah also, before you get started, make sure that the resolution is in Full resolution. The Roto Brush really needs to be in full resolution to work properly.

I am going to zoom in here and I am just going to do a rough drag around this object at one second and 12 frames in and actually I might want to be a little bit before then. We are kind of creating our master reference frame we can go back and change this if we want to-- I'll show you how do that momentarily, but what I'm doing here is I am clicking the areas that I want to be selected by the Roto Brush. Now you'll notice that I am not going exactly to the area, say for example, right here where we have this purple outline that tells us where the Roto Brush is seeing our selection.

You will notice that it's very much like the Quick Selection tool in Photoshop and right here at the edge of the snowboard is not being selected. So, I'm just going go ahead and click outside. You'll notice that as I zoom in here that my cursor is not going to the edge. Just a little part of my cursor is going that way. But if I click here, then it automatically just kind of guesses where I want that to go. So, you don't have to go right to the edge and it's probably better that you don't, so you better let AfterEffects kind of guess where we are going to go. So, I could click over here, for example, and then it kind of guesses where it wants it. So again best not to go all the way to the edge.

Now right here, I want to select the glove, but my cursor is a little bit too big. So, just as we saw in the last movie, hold the Command key on the Mac or the Ctrl key on a PC and drag downwards to reduce the size of the brush. Likewise you could drag upwards to increase the size of the brush and just click anywhere to be able to resize that. This is looking pretty good here. Now there are going to be some rough edges here. Do the best that you can to smooth those out, but there are some auto smoothing features built into this feature that we'll be looking at in just a moment.

So, I am just going to go around the edge, get a good selection, as good as I'm able to at the moment. It looks pretty good. Now one of the things that we want to do is we want to use the Page Down and Page Up keys one at a time so we can go to the next frame, and we could see if there's any problems. So, there are a few problems here. So, I am just going to click here to reselect those areas that kind of got lost for whatever reason. Usually these occur around the edges and so they're some of the shirt and back that didn't quite get covered.

That's looking pretty good. So, Page Down. At first, this will seem like a tedious process, although if you have ever done any rotoscoping, which is typically what they call when you have to mask out something frame by frame. That's extremely tedious, and this is a great timesaver for that. You don't have to do every single pixel by hand. It's a great start. And because this is such an intelligent tool, you really don't have to go and make that many changes frame by frame. It seems like after a while the Roto Brush starts to anticipate your very move and so it seems like you've to adjust less with every frame that you go to.

It's not going to be perfect. I'm going to show you some things in a just second and it will make things go even faster for you, but there this frame is almost perfect as is. There is a little bit of a problem around the edges, almost don't even need to worry about that, but with each frame we have less and less to correct. It seems like that frame looks pretty good and a little bit here on the snowboard needs to be fixed. So, what I am going to do is hit the Page Down key and keep advancing frame by frame until each frame looks pretty decent. Now, right here starting so that there is part of the helmet that's not covered so you could zoom in here and make sure that that's not getting covered.

Page Down. It's kind of getting the gist of it. So, I can just keep doing Page Down and I'll let you master this, but I want to pay attention to this. There is a kind of little mini timeline that's been created for us here in the Layer panel when you are working with the Roto Brush, and there's this first frame when we started Roto Brush, and that's where this little like orange dot is here. And we actually want to move the Current Time Indicator to where the Begin Roto Brush Marker was. And what am I actually going to do is trim this from the left edge, so that the Roto Brush starts at this frame.

Then I am going to move out to where our end Roto Brush is right about there and actually there are some pieces that I don't want here. And when you want to get rid of a piece of something that the Roto Brush is selected you hold the Alt key down the PC or the Option key on the Mac and just click, and it will remove stuff that the Roto Brush tool has selected. So, this looks pretty good, so I am going to trim the end of the Roto Brush selection, and now we have this little tiny Roto Brush area.

You can compare them with these icons down on the bottom left-hand corner. You could see our man little bit better so there is a white/black showing of our transparency where white is the opaque parts of our mask, and the black is stuff that will be removed. You could also see the original alpha boundary. We could also see it with like a quick mask type selection with this red overlay if we preferred that. Now I am going to go to the black-and -white selection, the alpha channel. Now what we once thought was a fairly decent mask is really a not very good mask at all.

So, what we could do is go over to the Effect Controls panel and notice that that there is a Roto Brush effect. And so we could do is we could check Refine Matte and just by checking that even without adjusting any other settings, we see a great improvement to what's going on. There's even been some motion blur automatically added for us, which is really getting clean things up a bit. We could also increase the smooth value if we want to smooth out the mask even more. We take it up to four or something. You want to be careful with this though because sharp corners will no longer be sharp if Smooth is high, so be careful about that.

If you want a softer edge, you could increase Feather to something like 40%. Increasing the Choke value will actually restrict the edge pixels bringing them in. I don't want to do that. I also want to show you this Reduce Chatter property, which is really cool. A lot of times when you are working with these automatic keying tools where you trying to remove a background, the noises will kind of jitter and shake from frame to frame. So, if you increase this value-- let's take all the way to 100 percent-- it's going to reduce the chatter. You might not see that much of a difference on the given frame that you are on, but from frame to frame, it's going to be a little bit more of a smooth edge.

Once you got everything the way you want it for the Roto Brush tool and you have the timeline correct here, what I am going to do is click the Freeze button that kind of lock this in place. So, now that we've frozen this selection, if we wanted to unfreeze it we could click this button to unfreeze it and basically when that's done it's cached our little segment of the Roto Brush time span. And so if we to adjust this what we want to do is unfreeze this. But if we go back to our composition here, we'll see that for the Roto Brush span the background has been removed. So here it is with the background and once we've started with the Roto Brush, then our snowboarder is now over nothing basically.

If I take off these layers, we could see that he's over a black background and these are the layers that I created as the background here. But this opens up a lot of possibilities. One, it's cool for motion graphics. We can put maybe some titles up here, something like that, maybe like outline this guy or create some of the motion graphic elements around him, which is a very cool effect. But it also is good for color correction and that type of thing where if we wanted to maybe just bring out this guy and maybe not necessarily isolate him and remove the background. Let me just go ahead and select this and hit Command+D to duplicate the layer, Control+D on the PC. On this back clip, what I'm going to do is I'm going to select it and then delete the Roto Brush effect.

So, basically, we have the snowboarder and then we have the regular background, but because he is isolated on the top layer we could do something like maybe invert. I'll apply the Invert effect to the top layer here and so we've inverted him for that time where he's flipping. So, we could add a color correction. Maybe we could just lighten him or darken him or something a little bit more subtle, but I just want to show you that if you wanted to perform some kind of extra color correction on just one isolated object, this is a great feature to use for that.

Now be careful that you don't over- estimate the power of the Roto Brush feature. For things like this like a little wispy hair or semi transparent objects, you're much better off using a keying tool to do that. So, don't think that the Roto Brush is like this be-all, end-all, cure-all problem for all things that you might run across with AfterEffects. But it can get you a majority of the way there and for times when the edge does not have to be perfect, like a lot of times when you are doing color correction, you really can't beat the new Roto Brush feature in AfterEffects CS5.

Using the Roto Brush tool
Video duration: 9m 25s 8h 39m Beginner


Using the Roto Brush tool provides you with in-depth training on Video. Taught by Chad Perkins as part of the After Effects CS5 Essential Training

After Effects
please wait ...