Applying the 3D Camera Tracker

We have the video of the street as an image sequence loaded into After Effects and we have our first composition. Let's take a look at that footage first. (video playing) So you can see that the camera is shot hand- held, the cameraman is walking down the street. Now our goal for this project is to replace this building right here in the center. And also we'll do a few extra things like add a billboard over here on the side to this building and then also replace the sky.

So this is going to require some type of motion tracking. Now the regular tracking that you might have done which is through the regular Tracker Tool is Transform Tracking and that is where features are tracked as they move up-down in Y or left-right in X. So there is no depth to that, it's just simply up-down or left or right. That's not going to work for this footage. Since the camera is walking down the street and we have perspective and parallax change we really need a 3D camera track. Because, for instance, if we do want to stick that billboard on the side right here and replace this entire building, but have the sides of the building pick up the correct perspective.

So how do we do that? Well there is a separate tool inside After Effects for 3D camera tracking, If I go ahead and select my layer, I can go up to Animation and apply Track Camera. Now the regular Transform Tracking is Track Motion, so this time we want Track Camera instead. I'll apply that and right away it's going to start working, you'll see a blue bar that says Analyzing in background. I'm going to go ahead and cancel that however, because I want to change some options. So in the Effects panel I'm going to click Cancel. Now there's a few options that are very important to check before we analyze and use the result.

One is a Shot Type. This is a type of camera shot we're working with. It defaults to Fixed Angle of View; however I want to check these options. There is also Variable Zoom if there is a zoom lens that changes and there is a place we can also specify a very particular Angle of View. An Angle of View simply means the lens in terms of millimeters. Now I know there is no zoom here and unfortunately I don't know the lens that was used to shoot this so I can't use these two options in this case. So Fixed Angle of View does work, but just remember, if you're working with other footage, to double-check these menus.

The other thing that's important to check is under Advanced. I'll expand that and there is a Solve Method. This is what After Effects uses to look at the track points and to develop an actual After Effects camera. Now it's set to Auto Detect which works sometimes and sometimes it doesn't. Now the options here are Typical, which is your average scene. Mostly Flat, where there's very little depth or perspective change, and the third option is Tripod Pan where the camera is fixed and simply panning left or right.

Now in our case Typical will work the best, since we have a moving camera and there is a significant amount of perspective change, this will probably work. It's not going to work with Tripod Pan and it really won't work with Mostly Flat Scene either, so I'm going to switch this to Typical. Once I've checked these two menus, then I can go ahead and Analyze. I'll click the Analyze button. Now this will take a few minutes, we'll see the progress right here at the left. The second step of this process will show an orange bar and that's where it's solving the camera.

It's looking at all the tracked feature points and trying to produce a camera that matches a real world. Once it's finished you'll see those bars go away in the screen and there will be a bunch of little dots. Now dots are pretty tiny, but what I can do though is go to Track Point Size and scale those up just so I can see them better. Now if the tracking worked correctly, you'll see that the various dots are actually Xs and those Xs range in size. The ones that are close to the camera are large and then they get smaller and smaller in the distance.

Now I know this is working because the Xs have their own perspective, in other words they diminish in size as it get further away. Let's play this back. I'm going to use the Spacebar to play back to Timeline. Now some of these Xs will pop on or pop off as the features that were being tracked disappear or become hard to find because of the colors in the scene, that's okay. What's important though is that there are still a lot of these Xs that are left that remain on the screen and then stick to whatever this should be sticking to, in other words the ones on the building stay on the building, the ones on the trees, at least some of them stay on the trees and so on.

What this means is, it's being able to track these point and determine their relative position in 3D space, in other words in the XYZ space so they have distance and they're far apart. Now what do you do with these points? While at this step once you've produced these points, you can create a camera. There is a Create Camera button and that will produce a 3D camera layer. I'll click that and there is our 3D Tracker Camera. This is a regular After Effects 3D Camera which you can use in After Effects' 3D environment. So now that we have a camera, what we do now? Well I can use these points in several different ways.

You can select some points and either attach a null, which is just an empty which is a point in space which you can use to parent other things too or you can create and a solid and this solid will be positioned in 3D space and it will be scene by the 3D camera, so it picks up perspective and looks like it takes on the move of the original camera. Well how do you pick points? Let's zoom in, I'll use my middle-mouse button to zoom in, or I can use the menu down here. Now what happens is as you drag your mouse over the screen, you'll see that a target appears.

Any time this set of three points is crossed, you'll see the target will be in the center. Now sometimes a perspective with a target lines up with what you would expect, like the building. This time it's not quite right. But the goal is to find points and make the target line up with the feature like the building, for instance this one right here and then attach that null or solid to that. So the first method is simply to drag your mouse over the scene. Once you find a target that looks correct, say right here, you can click and those points are highlighted yellow.

Now once those are highlighted, you can put your mouse over the center and you get the Move tool. Then you can move that target where it slides in that plane up or down, until you offset that point where the target is. And that point is where the null or the solid is going to go eventually. You can always click off and pick other points. Now the second method of selecting points is to click them individually and then Shift+Click additional ones. As soon as you get 3 you get another target. But you can click more if you need to.

Then you have to experiment to figure out which ones make a target that seems to line up with the feature. Again you want to line up say with this building, so it looks like that target is on the side of the building. So those are the two methods; drag your mouse or click and Shift+Click points manually. We've applied the 3D camera tracker, we've analyzed the scene, we've solved the camera, and we've created a 3D tracker camera layer. We've seeing a couple of different ways to select points. Now we're ready to attach things to those points and start adding new things to this scene.