If you’re an application owner or a member of the OS team, what is it that you should know about VMware that will make your life easier? A fellow virtualization evangelist, Tommy Patterson, wrote about exactly that in a recent two part series (Part 1 and Part 2). However, there are a few additional bits that I’d like to add to the story.
Limits, Ballooning, and OS Memory Utilization
If a VM is given a 4GB allocation and a 1GB limit, the OS will still report access to 4GB even though VMware will not allow the VM to get more than 1GB. For any resource the OS tries to use beyond 1GB, VMware will invoke the balloon driver (part of VMware Tools). The balloon driver acts as a process that demands resources from the OS much like any other application.
Here’s a screenshot of an Ubuntu virtual machine with 4GB allocated:
Notice it’s baseline usage sits around 600MB. Now let’s see what happens when I set a 1GB memory limit on it:
The memory utilization went up?!?! Wouldn’t VMware deny the VM any memory greater than the 1GB limit? Yes and yes. Let’s show you a screenshot of what is going on from VMware’s perspective and then we’ll discuss.
The VM started at 4GB of “Consumed” memory. In other words, the host server had mapped 4GB of physical memory to that particular VM. When the Limit was created, the memory consumed rapidly dropped below 1GB, showing the VM was not allowed more than 1GB, as expected. However, the balloon driver (shown as “Balloon”) jumped from 0 to 2.6 GB. Because this 2.6 GB is the size of the balloon driver process inside the OS, this explains why the OS memory usage jumped from 600MB to 3.2GB.
Hypervisor (Host) Swapping
Another important detail is the “Swapped” memory metric. This is not OS swapping, it’s hypervisor (or host) swapping – much worse. When the OS swaps, it gets to choose which memory pages get moved to disk. When the host swaps, it blindly pulls memory pages from a VM and swaps them to disk. This has potentially catastrophic results to performance and is not visible to the OS. Because of the performance degradation caused by host swapping, it is a last resort memory management technique by VMware.
So why is host swapping taking place if it is only supposed to be used as a last resort? In most scenarios, ballooning – which often doesn’t cause a hit to performance – will take care of reclaiming memory from the VM. However, unbeknownst to many, there’s a threshold for ballooning set to 65% of the memory allocated to the VM. In this situation, 65% of 4GB is 2.6GB; therefore, once the VM is already ballooning 2.6GB of memory, the host needed to find other ways – e.g. swapping, memory compression, etc. - to make up for the remaining 400MB difference, forcing the VM below that 1GB limit.
Sharing is Caring
Whether you’re a member of the OS team, an application owner, or a VMware administrator, hopefully you have found this information useful. If you have questions or comments, please share them below.