Join Scott Pagano for an in-depth discussion in this video MIDI device mapper, part of TouchDesigner & Unreal: Interactive Controllers.
- [Voiceover] We're going to start off with the scene that we built in the TouchDesigner intro course, and you can see we have our little three-D floating island here. I'm going to press play on our timeline, and I'm going to hit F1 to go into perform mode. You can see we have all that working here. We've got some buttons in the lower left corner and some sliders to do some control of our scene. And what we're going to do in this chapter is that we're going to build a post-processing effects block, where we have a few different effects that process this image that we're seeing here and that we use a MIDI controller to control that post-processing.
Alright, so I'm going to get out and back into designer mode here, make this fullscreen, and pause it. We're just going to do a quick overview of MIDI. So, let me go over to Photoshop here and MIDI is a protocol that was first debuted in '83, and it stands for Musical Instrument Digital Interface. And it was a way originally of connecting different synthesizers and different pieces of musical gear to each other, and now it's used for all sorts of different kind of communication between different kinds of controllers and computers and media software. This is a MIDI cable.
It's a five pin din cable, and this was the standard for connecting everything up to a certain point, and now we have a lot of MIDI controllers that actually just have direct USB connections to computers. This is a MIDI interface that has the five pin din port on it, which we could then connect traditional MIDI devices to a computer with. What we're going to be using for this course is a little korg nano controller, which is just a little device with some buttons and some sliders and knobs on it. And like a lot of modern MIDI controllers, this is a direct USB connection to the computer and we don't need that other MIDI interface when we aren't using those older, bulkier MIDI cables.
Now if I go over here, I have just a little editor up. This is an editor for the korg nanokontrol. I just wanted to show you an example of what's going on here. We have little values here where we can change different controller assignments and note numbers and such to various buttons and controllers. So whatever controller you're using has its own editor, and this is just to show that we can customize those controllers and determine what note numbers or what controller values or controller numbers are coming out of the different elements on the hardware controller.
So now I'm going to go back over to TouchDesigner and I'm going to show you how we can link up our hardware controller to Touch. So I'm going to press play here, just let this thing run. I'm going to go up to our Dialogues menu here, and I'm going to go to MIDI Device Mapper. You can see first off we have this ID one row and it says IN Device none, Out Device none. And what I'm going to do is under this In Device area I'm going to select this drop-down arrow, and you can see this is going to show all the MIDI inputs on this computer. The audio interface that's connected to this computer has a few MIDI ports on it, but what we want is we want this nanokontrol 1, so select that, and now if I move a slider on the controller, you can see down here we're getting various values.
And you can see there's different indexes cuz I move different sliders and then we're getting values. And the values go from zero to 127 for MIDI, and then we're going to rearrange those into whatever we want. So that just sets up TouchDesigner to listen to this MIDI device. And now what we're going to do is we're going to go into our project, and this is everything we had set up before in our scene file from the intro video. And we have our little block here for bringing in OSC messages and what we're going to do is just make another container for bringing in MIDI information. So I'm going to stop the scene.
I'm going to go up here and I'm going to hit tab, go to My Components, grab a container, lay this container down. I'm going to call this MIDI. Again, doing this nice and neatly in different containers so we're also building these modular building blocks we can bring between different projects. I'm going to go inside the MIDI container. I'm going to hit tab, go to our Chop area, type M-I-D and we get MIDI In. I'm going to lay that down and all we can leave everything here at default right now. I'm going to hit play.
Now I'm moving this slider and this is coming in on MIDI Channel One and it's master controller two, second slider's controller three, third slider is controller four, and so on. And so as I hit different buttons on here you can see that we're getting note on and offs that come in as just ones or zeros, and then as I move various things we can just see different values. As you can see, it's really straight forward to get MIDI data into TouchDesigner. Once we've brought in our device via the MIDI Device Mapper and laid down our MIDI IN Chop this data just flows right in. So what we're going to do next is we're just going to flesh out our MIDI component that we're building here, so we're prepping our data exactly the way we want it, so then have it flow down the line to control different things in our project.
This course focuses on using interactive hardware controllers to drive real-time 3D scenes in TouchDesigner and Unreal Engine. In TouchDesigner, look at a MIDI controller with sliders that you can use to drive the creation of effects in a real-time rendering system. Next, learn about using TouchOSC to build a custom UI that you can transfer to an iPad that sends OSC messages to TouchDesigner. Then, learn how to use those messages to move particles and objects around on screen. To wrap up, learn how to get data off of a Kinect sensor into TouchDesigner so you can control that same particle and object movement. The course also covers MIDI, OSC, and Kinect as they apply to Unreal Engine.
- Using TouchOSC to build a custom UI
- Transferring a custom UI to an iPad
- Controlling objects with OSC
- Setting up Kinect
- Getting data off of a Kinect sensor and into TouchDesigner
- Using OSC messages to move particles and objects around
- Installing VS and downloading the plug-in
- Creating a C++ Unreal project and compiling a plugin
- Installing the Kinect 4 Unreal plugin