Storage performance overview

- [Rick] In this video, I'll explain some concepts of storage virtualization and we'll take a look at some different storage architectures and how we can effectively troubleshoot them. So when we think about a virtual machine we've got really two parts of a virtual machine. We've got the running state of a virtual machine, which exists on our hypervisor, and we've got the files that make up our financial machine. And those exist on some sort of storage system. So a virtual machine is similar to a physical machine in many ways, right? We have an operating system. And we have to give this virtual machine resources. So our operating system has no idea that this VM is running as a VM. Right, Windows doesn't have an awareness that it's running on top of ESXi. So we basically have to trick Windows into thinking that, "Hey, you've got memory, you've got CPU." Even though we're not actually dedicating any physical hardware to this VM, we want it to appear as if it has its own memory, its own CPU. We want it to think it has its own SCSI controller. And this is really the critical part for storage. So what we're going to do is present drivers to the virtual machine that it can use to access storage just like a physical machine would have. A physical machine would have some sort of SCSI controller. And so we're going to trick our virtual machine. We're going to give it a driver. We're going to allow it to use something like a para-virtual SCSI controller or a LSI logic SCSI controller. We're going to trick that machine into thinking it has access to physical storage when really it's simply going to take those storage commands, dump them into the hypervisor, and the hypervisor will take it from there. And same thing with our virtual NIC. We're tricking the virtual machine into thinking it has a Network Interface Card when in reality it just has a driver for a VMXNET3 network interface or an E1000 network interface. It's not real hardware. It's just taking that network traffic, dumping it into the hypervisor, and the hypervisor takes it from there. And so the beautiful thing about this is that the hypervisor then gives us a layer of abstraction. It doesn't matter to the virtual machine what the actual underlying hardware is. So in the case of storage, the virtual machine never sees whether it's dealing with an NSF storage device, whether it's dealing with VSAM, whether it's dealing with iSCSI, Fibre Channel, local physical storage. It has no awareness of any of that. Windows sees a virtual SCSI controller, it sends those storage commands, those SCSI commands to that virtual SCSI controller, and that's all it really knows. So let's take a look at this process. We have a few different options when it comes to our virtual disks. Just like CPU and memory, the VM doesn't really actually have any physical storage hardware. It's accessing a shared resource. And in this case, our resource is called a datastore. So Windows needs to see the storage hardware. Windows needs to think, "Hey, I've got a SCSI controller. "I've got something that I can send these SCSI commands to." So we're going to trick it. We're going to give it a virtual SCSI controller. We're going to give it a driver. And then, when Windows need to read or write data those SCSI commands are sent to the virtual SCSI controller. And from there, those storage commands hit the hypervisor and they're redirected to the appropriate VMDK file for this virtual machine. This is what gives me a lot of the possible storage features that we have with vSphere. I can do something like a storage VMotion. I could move this VMDK to some other datastore and the virtual machine will be completely unaware that that has happened because all the virtual machine really sees is that SCSI command going out on the virtual SCSI controller. What the hypervisor does with that SCSI command after the fact is completely hidden from the VM. Now, there's also some options for the type of virtual disk that we create for our VMs. And the most common choice is a thin provision to disk. So let's assume that we have a VM created with an 80-gig virtual disk but the VM only has 40 gigs of actual data. This means that only 40 gigs of actual storage capacity is consumed on the datastore itself. The VM thinks it has an 80-gig disk but on the actual physical storage system, it's only consuming 40 gigs of space. And we have to recognize that that comes with a little bit of risk. If I create too many thin-provision disks on a datastore, it's possible that as those virtual machines continue to add more data and their virtual disks expand and take up more space, I could accidentally fill up that datastore. So you want to make sure that you have your datastore usage alarms, free space alarms in vCenter, configured properly and that they're actually set up to either send you an email or an SNMP trap or something like that. If you use thin-provision disks, you have to take those sort of preventative measures to make sure that you don't fill up a datastore because when a datastore becomes full, all the virtual machines on it are impacted. They'll be unable to write data and those virtual machines will halt. So it is service-impacting when a datastore becomes full. And that's one of the risks you run with thin-provisioned disks. And that's one of the reasons why some customers choose to leverage a thick-provisioned disk. So when you create a thick-provision disk all of the space for that virtual disk is immediately set aside. So here my VM has an 80-gig virtual disk. 100% of that storage capacity is immediately consumed on the datastore. And so it's not as efficient with its space usage but you're not really running the risk of accidentally filling up a datastore. And if you choose to do thick provisioned there are two ways you can do it. You could do it either lazy zeroed or eager zeroed. So when I create a VMDK, I can choose whether I want to do thin provisioned, thick-provisioned lazy zeroed, or thick-provisioned eager zeroed. Most of the time, the best option is thick-provision eager zeroed or thin provisioned. One of those two. I really don't recommend using thick-provisioned lazy zeroed. So here we see a thick-provisioned eager zeroed VMDK. And you can see zeroes have been written to the entire VMDK. So we know that a thin-provision disk saves space and is very efficient and a thick-provision disk allocates 100% of the space upfront. But in either case, before space can be used, the blocks must have zeroes written to them. And this can impact applications that write a lot of data, like for example a database virtual machine. Thin-provisioned disks zero out blocks on demand. So do lazy zeroed discs. They will zero blocks as the application is trying to write to them. And so for things like write-intensive databases this can really slow down the way that they perform. But a thick-provision disk is zeroed out upfront so when you create that virtual disk, it's going to take a little longer 'cause zeroes are going to be written to every single block of that virtual disk. But now when your application tries to write to new blocks on that disk, it's not going to have to wait for them to be zeroed. So this is a good option for things like exchange database availability groups or SQL databases. Now, let's take a moment to look at the overall storage architecture and where problems can occur and where latency can occur. And so here you see I have a nice little storage diagram. And on the left, I've got my virtual machine. And I've got a virtual machine with a virtual SCSI controller. So if I'm troubleshooting some sort of problem with storage, number one, I want to know the scope of the problem that I'm having. Is it on every VM or is it specific to one particular VM? And if it's just one VM, maybe I can kind of zero in on that VM and see, hey, is there something wrong inside the operating system of the VM? Is it something going on with this virtual SCSI controller or something else within Windows that could potentially be causing that problem? Now, if I find out the problem to local to my ESXi host, if all the VMs on a host are impacted, what could it be? Well, I could have kernel latency. Kernel latency means the hypervisor itself is introducing latency to my storage commands. And this is expressed in ESXTOP as KAVG, kernel average latency. So if the kernel average latency is over one millisecond you've got a problem on that ESXi host. If the ESXi host is taking more than a millisecond to process these storage commands, that's a problem. And then we can really focus our troubleshooting efforts on the host itself. Another thing that could potentially be harming the performance of a host in an NFS environment is some sort of problem with the physical network adapter. So with NFS, all my storage commands are being funneled through a virtual switch. And so as such, if I'm having a problem at that virtual switch level or maybe one of these physical adapters is having some sort of problem or maybe these connections to the ethernet switch are having some sort of problem. In those cases, that'll cause a host-level slowdown in storage as well. So that's my next step for troubleshooting. And the way I've diagrammed this is really not ideal. Hopefully, if I'm setting up NFS maybe this second adapter is connected to a second physical switch. That would really be ideal. So in this diagram, I've got both adapters going to the same physical switch. In real life, I probably wouldn't do that and I'd probably have the adapters going to two different physical switches for failover purposes. But the other problem that can occur here is at the ethernet switch level, right? So I could have some kind of problem here in the ethernet switch. I could have incorrect MTU settings. I could have a switch that's just plain overwhelmed. Maybe the CPU and memory are overwhelmed or something along those lines. That's another potential spot where my problem might exist. I could have a congested link between the switch and my actual storage device. The processors on my NAS device could be overwhelmed. And then I could have physical storage issues in the NAS device itself. Now, these sorts of problems are typically going to manifest on multiple ESXi hosts. So maybe I don't have a high enough spindle count. I sort of think of spindle count like bottles of ketchup. If I'm squirting ketchup out of one bottle I'm limited to the throughput of one bottle. But if I'm squirting ketchup out of four bottles at once I can squirt four times as much ketchup simultaneously. I know it's kind of a weird analogy but that's sort of like disks. If I'm pulling data from one disk versus pulling it from four, I can pull it four times as fast. That's why spindle count's important. So maybe I don't have a high enough spindle count. Maybe my disks aren't fast enough. Maybe the cache is inadequate. Whatever the case may be, there are a bunch of potential problems that you could have on the actual storage device itself as well. So having a diagram like this is really useful. Let's take a look at one more diagram. This one's for iSCSI. So again, the diagram doesn't really change that much. But there's some different pieces to the puzzle. Again, if the problem is isolated to a VM look at the operating system of that VM. When the virtual machine generates a SCSI command the SCSI command is relayed to the storage adapter of the ESXi host. So in this case, we're using iSCSI. And so I've got some kind of storage adapter here. It could be a dedicated hardware iSCSI initiator. It could be a software iSCSI initiator. It could be a dependent-hardware iSCSI initiator. In this case, I've chosen to make it a software iSCSI initiator. So the job of the storage adapter is to receive SCSI commands in their raw format from the virtual machine and then take those SCSI commands and prepare them as iSCSI packets so that they can traverse the physical network. So what do I want to potentially troubleshoot here? That storage adapter is only going to perform well if there are adequate CPU resources on the ESXi host. It's a software component of ESXi. The virtual switch, VM kernel port, those things are only going to perform well if the host itself has adequate processing resources. And then on the virtual switch or potentially multiple virtual switches, so with iSCSI what I could do is I could create a VM kernel port on one virtual switch and a VM kernel port on a second virtual switch. And I can round robin, have the traffic use both of those VM kernel ports. That's a great way to ensure that number one, you're spreading your traffic out across multiple physical adapters but number two, if one of these switches fails, if one of these network connections fails, our traffic still continues to flow. So if I'm having performance problems and they are local to this host, I'm looking at the CPU of this host. I'm looking at the physical network connections for this host. Are they overwhelmed? And if those are not the scenarios that we're experiencing, if those are not the problems, then I can move my way into the network itself, right? So are the physical switches being overwhelmed with CPU or memory or is there just too much traffic hitting 'em? Have one of these connections to a storage processor failed? Are all of my connections actually up and running? Is there some kind of network problem that's creating latency? And then I can look at my storage array. Now again, if I'm having storage array or network issues I'm probably seeing problems on multiple ESXi hosts. So if the CPUs of my storage processor are overwhelmed that's going to cause all sorts of latency. I might even see storage commands being aborted. You'll see those in ESXTOP as well as aborts per second. Aborts are bad news. If you have storage commands that are being aborted they're sitting out there for so long that they're basically just being dropped. So aborts are bad. And they can happen if the storage array is overwhelmed or again, maybe I have inadequate spindle count. Maybe I don't have fast enough disks. You know, maybe I'm using 7,200-RPM SATA when I should really be using 15,000-RPM SAS or I should be using flash or SSD. So key to troubleshooting and problem determination with storage is always a solid understanding of my storage network topology so that I can kind of systematically work my way through it and determine what the root cause of my storage problem is.