What kinds of NUKE tips do you need? Join the Steve Wright's NUKE NUGGETS group on LinkedIn, and post your requests there.
Skill Level Intermediate
- [Steve] Hi, this is Steve Wright, welcoming you to this weeks Nuke Nugget how to track a 3D object to a CGI render. The idea is to use the position pass, which contains the XYZ location of each pixel to position a 3D object. Nuke Nuggets are often short little techniques but occasionally I like to mix in some next level techniques like this. Okay, to do this, you have to have a CGI render with a position pass plus the camera. Here's the camera that did the original CG render.
This is just a 2D EXR file. There's no geometry here, and I can show you my position pass right there. Okay, so let's get to it. What we want to do is sample from this position pass the XYZ location frame by frame and feed that to a piece of geometry so it'll track right on top of this surface. All right, here we go. In order to see what the original geometry looked like, we can use the position to points node.
So select the read node, go to 3D, Geometry, Position to Points. And then I'll tell it to go look for the position pass in the position pass. Okay, so this is the point cloud from just the rendered surface of our CGI object. Of course there's nothing in the back because we're only seeing what the camera saw. Okay so the first step is we are going to track on our 2D render.
Jump to frame one. Let me go get a tracker. Hook that up here. I'm going to place a track right there, and push in real close and position it right on top of my yellow cross. We can see right there. So I'm really sampling that yellow pixel dead center. Okay, so let's just track on my 2D render here, and it's only going to take it about two seconds.
Looking good. So let's cruise in and just check. Make sure it seems to be staying on target. Yup, lookie there. So I'm right on target. Now my tracking data's going to be a little bit jumpy because I made a very low res image here. This is only 600 by 338, in order to save disk space. So you didn't download a 12 megabyte file for this one little Nuke Nugget. Okay, what I want to do next is I want to crop that one pixel right there that's under my tracker.
To do that, with the tracker selected, we'll come up to the transform and get it crop node. Now the X and Y are the lower left hand corner of the crop, and the R is the right side, and the T is the top. So I want to set the X and the Y to be the tracker. This guy right here. I got that data right there. So to get that put into my crop node, we'll go to the curve editor, open up the tracks.
So here's the track X data for the length of the shot. That'll be this point right here in X. So let us copy those curves, come up to the crop box X, and say paste absolute. Notice that the left hand side of my crop window now jumped up to where my tracker is. We'll go grab our track Y, Edit, Copy, Copy Curves. Go to box Y and paste absolute.
Okay, now the lower left hand corner of my crop window exactly follows the tracker like this. Let me resize that window. So as I scrub through the play head you can see that the crop window is following right there. Okay, what I want to do now is set the crop window to be exactly one pixel. That's easy to do. I want the right here to be the same as box X plus one. So I'll just say, give me an expression that is box X plus one.
You'll notice box X is 224. Box X plus one is 225. So when I say okay, the right edge should jump over to my tracking point. There we go. Now to do the top. Same drill. Give me an expression that is box Y plus one. Boom, okay. So I am now cropping just one pixel right there under the tracker. Now this is critical.
This is one pixel cropped out of the original window. You see that? For what we want next, this one pixel has to fill the entire frame. It cannot be one pixel inside a 600 by 338 window. So to fill the entire frame with this I'm going to click reformat. I now have a one pixel image, but the entire image is my pixel. Now I've noticed that several of the Nude nodes do not like working with a one pixel image.
So what we'll do here is we're going to help Nuke out by adding a reformat node. Going to make this just a little bit larger. So let's set it to box, force the shape, and let's make a 10 tab 10. 10 by 10. So I now have a 10 by 10 image that contains that one pixel under the tracker. Now I'm going to use the curve tool to read the position pass data, but the curve tool only reads the RGBA layer, not the position layer.
So I'm going to clear my property bin. With reformat selected, I have to shuffle the position pass to RGB. So Shuffle, Position to RGB. Okay now my RGB layer here contains the position pass. You can see I'm scrubbing through it there. What I need to do now is to collect the RGB values of this window over the length of the entire shot. That's exactly what the curve tool does.
So with shuffle selected, go to Image, Curve Tool. Okay so we have our curve tool. By default it's going to collect the average intensities, and what that means is the RGB value for every pixel in this frame over the length of the shot. Well the only pixel here is my position data. So all I have to do is click go and say yes, do the entire shot, okay. This is just going to take a second. One of the advantages of using low res images.
Okay, data collection complete. When we go look at the intensity data and these are the X, Y, and Z locations that it copied from here. So as you can see, look at the RGB values down here. From the position passing the screen, exactly match what my curve tool sampled here. So I now have the XYZ position for that surface over the length of the entire shot in my little curve tool.
All right, let me clear things away. Now we're ready to put it into the sphere. So let's switch to our 3D view. Here's my little red ball. And of course as I scrub the play head, it's not doing anything, not going anywhere. Now all I have to do is link my curve tool data to my sphere translate data, and that's real easy to do. With a Command or Control, click drag and drop my intensity data to my translate X.
Then the Y to Y, and Z to Z. Okay, and now my little red ball is flying. If we look at it relative to the position to points, we can see it's actually attached quite nicely to the surface of the point cloud. Now remember, it's going to put the center of my little red ball on the surface. So the inside of the sphere is going to be in here.
Okay, in order to render this red ball, I'm done with my point cloud. In order to render the sphere, I have to have the original rendering camera. Don't forget, camera very important. Okay so we'll go to our scan line render now. Push in, and as I scrub through the length of the shot, there is my little red ball render. Now to get that over the background I'll add a merge node.
Hook up the render to the original clip, merge to comp, let's set it for a little bounce, and play, and there you have it. A 3D object tracked to the position pass of a 2D render. Be sure to check in next week so you don't miss out on my next Nuke Nugget.