The Microsoft HoloLens is packed with sensors. In this lesson, learn to identify all parts and see how it all works.
- Let me introduce you to the HoloLens. As you can see, it resembles a somewhat heavy pair of ski goggles. You may have used other virtual reality devices before in which case you'll notice one of the biggest differences. You can actually look through the display and see the outside world. It's translucent with a bit of a shade on it. Let me talk you through the physical parts. First of all, you'll notice these cameras. There are two on this side and two on this. These cameras are there to tell the device of any movement the user has made.
It does so by taking snapshots of the environment and comparing those with previous-made snapshots. By doing all sorts of calculations, it can figure out if and how you move and adjust the image the device is showing you accordingly. The net result is that when you move to one side, the image appears to stay in the same place. Now, this is where the display is projected. However, it's very hard to see from the outside. Other people won't be able to see what you see, we have to look at the inside of the device.
There's a light engine here at the top of the visor that beams the image to these two displays. Now, since there are two displays, one for each eye, we can project a stereoscopic or three dimensional image. The sensors and the displays together make it so we can achieve true 3D. What do I mean by that? Well, imagine you watch a 3D movie in a theater. At some point, something is sticking out of the screen towards you.
If you move your head, the object sticking out moves with you. Now, this makes sense in a movie theater. Every audience member needs to have the same kind of experience. It would be weird if people sitting in the middle of the theater see the object straight ahead while the people sitting near the aisles see it from the side. But this is exactly how it would look like if there was actually something sticking out of the screen. People sitting at the edges would see the object from the side. Now, your mind recognizes this.
It sees the effect, but it also notices that it's moving with you as you move your head, so your brain will reject it. It will probably like the experience, but it will not really accept it as a real 3D experience. The HoloLens is different. You can watch something going towards you in 3D, but when you step to the side, it will go passed you instead of at you. This time, your brain will tell you, "Hey, this is the real thing!" The result is that your body will react to objects as if they are real.
These two sensors form the depth sensor. This is an infrared light emitting device and a sensor that reads reflections off this infrared light. And by measuring the time the light takes to return to the device, HoloLens can calculate how far away objects are. We can use this to determine where objects are in the real world. Now, it takes about 0.2 seconds to do a scan, so it does a refresh five times per second. This isn't enough to track moving objects, for instance, people walking by, but it's very useful for detecting floors, walls, ceilings, tables, and so on.
Now, the HoloLens also has got a regular webcam right here. It's more or less hidden behind the visor so you can't see it, but we'll use this later on to stream videos and take pictures. These are the spatial sound emitters. This is what enables the device to let us hear things, and the nice thing is that it is in stereo sound, it's spatial sound, and we can use this to play sounds in a 3D world. You can actually hear where the sounds are coming from, and you can find objects in your virtual world.
Now the speakers are placed just above your ears. This is done so it doesn't block out the real world sounds. You're still able to hear what's going on, but you get additional sounds. Just like the visor where you can see the real world, but you also have additional images added. Now, in the back, you see a USB port. This is used for charging the device and also for uploading your apps as you write them. And that's it, you can't connect a 3D printer for instance.
On the other side, you see a 3.5mm audio jack. You can plug in headphones here if you want to, but I haven't met anyone who actually does that yet. With headphones, you block out the real world sounds which goes against the principles of the device. Now, the batteries are located here and here, three on each side, so it makes a nicely balanced device. The device is Bluetooth and Wi-Fi on board which we can use to connect to all the devices and networks. Be warned though, the Bluetooth only works with human input devices such as your keyboard or an Xbox controller.
Again, don't try to hook up a 3D printer to the HoloLens, it won't work. Last but not least, behind this piece here is the computer. Yes, it does contain a fully self-contained Windows 10 device that has a CPU, a GPU, memory, and SSD. It also has something called a HPU which stands for holographic processing unit. This custom-made chip is what takes all the data coming from the sensors and translates that into events for the CPU.
This is the big secret. This HPU makes the HoloLens work without being connected to an external device. And there you have it! The HoloLens hardware in detail. Now, it's time to investigate what we can actually do with it.
Join instructors Dennis Vroegop and James Ashley for this introduction for HoloLens app development. They show you how to set up your development environment, including Unity and the HoloToolkit; how to deploy to HoloLens and the HoloLens emulator; and how to build apps that accept user input via gazes, tapping, and speech. Then discover how to use spatial mapping to detect your surroundings and analyze your environment with spatial understanding. Learn how to implement spatial sound that adds to the user's existing environment, and synchronize data to create shared experiences. Along the way, learn how it all comes together in two simple 3D apps: a visualization for enterprise business and a platform game.
- Installing Unity, Visual Studio, and the HoloToolkit
- Building an app in Unity
- Deploying to HoloLens or the HoloLens emulator
- Gaze and tapping
- Spatial mapping and spatial understanding
- Spatial sound
- Shared holograms
- Creating 3D enterprise apps
- Creating 3D games