Memory state in ESXTOP/RESXTOP

Often times, you will question if you have enough room for another VM on your host. Now before I begin, let me clarify in a larger environment, you should certainly use capacity analysis tools. But what if you are a small shop and can’t afford one of those tools and you are only an owner of a small cluster and dont mind running ESXTOP/RESXTOP to figure this out. You can look at TPS and other areas but the memory state of the host will indicate the kind of the stress this host is under. This will be your best friend.

MemoryState

As you can tell my host is in the ‘High” state. What does this really mean? Your host can be in one of the following states: “high”, “soft”, “hard” or “low”. Your host will be in either one of these states based on the following:

high state = if the free memory is greater than or equal to 6%

soft state = if the free memory is at 4%

hard state = if the free memory is at 2%

low state = if the free memory is at 1%

As you can tell, high state is what will keep your host happy. One thing to note is in the high and soft states, ballooning is favored over swapping, in hard and low states, swapping is favored over ballooning. Of course TPS and other techniques will enable you to efficiently use the memory on your host and allow you to overcommit.  Another thing to point out is that your host maybe in ‘high’ state but you may notice your VM is still swapping. It’s not  the host, its really the limit on your VM or your RP settings that is causing this VM to swap.

The good news is that DRS will move your VM over to another host (based on your setting) if its gets under stress and moving a VM will guarantee to better its performance. But I have always found ESXTOP/RESXTOP to be an excellent tool to get an insight on whats really happening on your host. Remember a holistic view is great, and when we talk about a cloud a single host may not mean much. However, each host is a building block that forms your cloud. Understanding how memory is handled on a host level will give you better insight on the holistic stats of memory in your cloud.

Optimize performance of Win VMs using RSS in vSphere

Recently VMware published a new white paper about network performance for vSphere 4.1. Duncan posted a link to it on his blog and I decided to take a look at what it had to offer. Without a doubt, it had some very useful information and most importantly its an easy read. So I recommend you read the white paper as well. Along with the possibility of a Linux VM receiving packets at 27Gbps, I thought the take on Windows VM was very interesting.

As mentioned in the white paper, the Linux VMs performed better than the Windows VMs as they leveraged the Large Receive Offload (LRO) which is not available for Windows. This started to make me think about some of the issues that could be addressed just by having a simple understanding of what this means. A VM that does not support LRO, its receiving packets are processed by the vCPU that it has been assigned. One important thing to note is that by default, only vCPU 0 will be used by your Windows VM. This means that even if you have a VM that has been assigned 8 vCPUs, when it comes to processing received packets, that task will only be handled by vCPU 0 as the other 7 sit back and relax. Basically your VM will still wait for it to schedule all the vCPUs before it does anything, however, when all the vCPUs have been engaged, only one will do the job.

As mentioned in the white paper as well, what you can do is enable Receive-Side Scaling (RSS) and this enables the windows VM to utilize all its assigned vCPUs when processing received packets. Your VM will wait to schedule all the vCPUs assigned, why not make use of all of them while you have ’em. This can enhance your VMs performance. Not to mention multiple vCPU should only be assigned to a VM if the application supports it and assigning multiple vCPUs will enhance the VMs performance. In a highly taxed host, a VM with multiple vCPUs for no reason will only suffer.

In a non RSS enabled windows VM where you see a spike in processor due to network utilization, you will notice adding another vCPU doesn’t solve your issue. What might happen is that if your single vCPU VM was at a 100% CPU utilization, now it will be at 50%. If you increase the vCPUs to 4, now the utilization will only be about 25%. But the performance is still the same. Whats going on? Only 1 vCPU is doing all the processing for received packets. ESXTOP will solve the mystry for you as well. By enabling RSS on this VM, you can benefit from using all the vCPUs assigned. Again, be sure that assigning more vCPUs is not causing scheduling issues in your environment. That will depend on how busy your host is.

You can find out if RSS is enabled by running netsh int tcp show global in the command line and it will show you the status.

rss enabled

Its enabled by default on Windows 2008 R2 and can be enabled on windows 2003 sp2 and up. You will also have to enable RSS in the driver setting for the VMXNET3 inside your VM and you are all set. You will need to use VMXNET3 to enable RSS, VMXNET2 will not cut it. Simple things like this can certainly assist in optimizing your environment and put you at ease with what lives in your cloud.

vmkTree and ESXi

If you are in the virtulization arena, you have to be following Duncan. And you should know what he says is the word of God when it comes to virtualization. Going through his blog this morning, I came across ‘vmktree’ (tool to monitor performance counters on your hosts and VMs). As he has mentioned, it’s the tool from back in the days written by, Lars. He spoke highly of it and I had to give it a go. I was amazed at what this tool has to offer, you can run it off your vMA appliance and it has a nice web interface as opposed to the numbers you see in ESXTOP. As Duncan mentioned, those numbers don’t really help you see a trend. I won’t go into much further details. Lars has very simple installation instructions that works. Below is the link:

http://vmktree.org/?p=14

The God himself has endorsed this tool below:
http://www.yellow-bricks.com/2010/12/23/cool-tool-vmktree/

 

e1000 packet loss

While going through the network stats in esxtop, I noticed huge percentage of recieved packets being dropped. For a second I thought I am not looking at the right screen but I was. The numbers were all over the place and didnt make any sense at all. They jumped from 20%, to 90% to 3%, 48% all with 5-6 sec. Sadly this was having to almost all the VMs. I didnt know if this was really happening or the host didnt really know what was going on as we are using the 1000v.

 

I asked our network team to use their tools to see if they are noticing the same thing on their end. They are still researching the issue. In order to restore some sanity, I decided to increase the receive buffer of a VM to see if that would make any difference. It didn’t! Finally I decided to start looking at a few VMs that weren’t experiencing this issue. To my surprise they were all XP machines. I started to think it is guest related in some way. Upon further investigation, I noticed the XP VMs were running the flexible NICs vs the e1000 that was running on the other 2003/2008 VMs. My next step was to replace the NIC on one of the VMs with the vmxnet NIC to see what happenes. WOLLLLAA! the drop packets went down the 0 and stayed there.

I am waiting on the network team to confirm that the e1000 was in fact loosing actual packets and the esxtop wasn’t simply nutty. Once they do confirm this, I plan on replacing the NICs on the other VMs as well. The vmxnet will give you better throughput, they are less CPU intensive and lastly according to ESXTOP they aren’t loosing packets like the e1000 were. The e1000 were great until the vmxnet came around. I think its about time we start looking into implementing this. One thing I don’t like is that the vmxnet NICs appear as removable objects in your system try like a USB drive would . I am sure there must be a way to fix it, I just haven’t figured it out yet.

EDIT: 11/24

VMware released a patch in Sept to address this issue. It turns out that the packet loss being reported on ESXTOP for e1000 nics was not correct.

 

vCPU worlds

While going through series of esxtop or should I say resxtop stats, I noticed that certain VMs had a higher count of worlds they were involved with. Considering I was researching an issue with performance of some of the VMs in the envirnoment, it only made sense to look further into this. Typically a vSphere VM with a single CPU would see 3 worlds (VMX, MKS and the vCPU), however a couple of my single vCPU VMs saw four, the additional world was “Worker” in one instance and “psharescan” in the other. Google didnt help much neither did the endless pdf I went through to figure out what was their purpose. However, upon vMotioning these trouble makers to a different host, the number of worlds got cut down to 3. The number never went back up but not knowing what those unknown worlds were is driving me crazy. One day I will figure it out.

Moral of the story; vCenter may have improved over the years and with 4.1 now we have more stats than ever with storage and networking I/O and what not. With all that in mind, esxtop/resxtop still remains a very nifty tool in every VMware Admin’s toolkit. If you don’t use it, get used to using it because at some point you will find yourself playing with it.