Learn how to add speech and keyword recognition capabilities to your MR application.
- So, there's a number of different things that you can do and different ways of doing it. Should we dig into some code and see how you're doing it? Let's cut over to your machine and... You've got a Unity Scene set up. - [Man In White Shirt] Yep. So I've got my Unity Scene here. Essentially, it's very blank at the moment. I've got my Speech Manager Script Component on my Main Camera just as the default location along with my Gaze Manager. So, that's probably already been covered by a previous... - [Man In Black Shirt] We already talked about Gaze, which is awesome. - [Man In White Shirt] So, we're gonna kind of hook into that, we're not gonna look in that script, but we're gonna hook into that, which is very helpful.
So I'm gonna open my Speech Manager Script. And here's Visual Studio 2017. And you'll notice some sections up here in my class. So I've got the keywordRecognizer, which essentially allows those different keywords to be recognized when they're spoken. And then I've a dictionary here, which I feed these keywords essentially through to match them with actions. Yeah. - [Man In Black Shirt] So, you're really building up a dictionary of keywords to action. When you hear a keyword, it triggers that behavior.
- [Man In White Shirt] Yep. Perfect. Yeah. So, I only use the one event, of course, which if we go down to my Start Method here, we've got the single event, which is the keyword recognizer, on Phrase Recognize. So, if I go down towards Definition, we can see this keywordAction being invoked, once it's been tested against through my dictionary. So, that allows as many, essentially, keywords as you want to go through your dictionary and be recognized with just a small amount of code.
In my start section, I initialize all my different keywords. So, I've got this first one, is Reload Scene, and actually you will notice that a lot of them use this lambda expression within the Dictionary Method. And so this first one does a broadcast message. So, I know I use this a fair bit within the HoloLens, wherein I could broadcast a message, and it would actually invoke on every child of an object, which is very handy, especially for, so in this case, I've got Reload here.
So perhaps you've moved a whole bunch of objects, and you want them to move back into their original location, particularly if you're playing with the physics. That was really helpful because all the scripts that were connected to those children, they all applied that same command and very useful. - [Man In Black Shirt] So essentially what you've done all the children have got set up listening for that broadcast message. As soon as they receive that broadcast message, they go ahead and trigger whatever behavior they want. They could do something different. They could all behave in different ways.
Kind of, if you want, you could think about it almost as a polymorphic behavior within, using speech, which is kinda nice. - Very comparable. Yeah. - [Man In White Shirt] I suppose a key thing here is this section here, which is this string, which is the Reload Scene string. So, if we go just slightly back to my dictionary here, you'll notice that string section there is the first part of that dictionary, and that is actually the words that will be recognized by the keyword recognizer. So, once they're spoken in Scene, that will trigger this, you can't see that, this action here, which is linked via lambda expression.
I've got another one underneath here, which is the Start App... I suppose, call. So, you say that, and it'll run, but you'll notice that I've also got here, and this is where our Gaze Manager is very handy, because it's already picking up focused objects, so I can actually test if I've got a focus object, and then I can try to invoke my method on there through Send Message, and that's its method there that it possibly could run. You could do another test in here because if you've got a whole bunch of different focused objects, you don't want them all to be running a method in case they run that same method in a different way, so maybe you can also do other, like a switch or something, that would be quite handy there, but otherwise, very powerful and allows you to do all sorts of different things, particularly makes it more interactive in your scene.
- [Man In Black Shirt] Nice. - [Man In White Shirt] The last one, which is the one I'm gonna demonstrate in a second, is this one here: White Christmas. And essentially this is very much a hard code, and one wherein I've just got some code that's gonna directly run into my Scene. I'm not going off to any other objects and no other scripts. It's just all internal. So, I've got my local game object here, which if we go slightly up again, is this particleSystemLocal, and I set this in the Inspector, so if I quickly move back to Unity, we'll see down here in my Speech Manager I've set this particleSystemLocal to Snow Object, which you can see in my hierarchy.
And, currently, it is set as inactive, so you cannot see it. So the idea here, if I go back to Visual Studio, is when this keyword is recognized, the action it will invoke is that it will set that particleSystem to active. So here, I'm using the particleSystem's current state, and I'm essentially doing the opposite with this fantastic exclamation mark. - [Man In Black Shirt] Excellent. That's super useful.
And so, while it's not something we often get in Australia, a white Christmas is often something that people would like to be able to command to just happen. So, should we see this running? - [Man In White Shirt] Yeah, absolutely. Okay, so let's go back over to Unity. Gonna press play. Actually I might monitor just this Main Camera bit. I don't have to select any of these, as long as it's its Scene, it'll be fine. Okay. The Scene's running, as we can see. And then if you say the words, so, "White Christmas." - [Man In Black Shirt] Ah-ha.
(both men laugh) Excellent. So, yeah, so it's picking up your voice, and it's using that to trigger that behavior. Of course now we need a "Bring on the Sun" command. - (laughing) There's plenty of sun here. - Yes.
- Mixed reality
- Setting up the camera and scenes
- Gaze in Unity
- Building movement and teleporting
- Setting up audio in Unity
- Conditional compilation in Unity
- Creating simple models using Paint 3D
- Topology and polygon count
- Normal maps, bump maps, and CrazyBump
- Fixing issues using Maya and Visual Studio
- Exploring the frame debugger
- Building an application in Visual Studio
- Submitting an app to the Windows Store