-Inside the OS, the task manager showed all the RAM was used
-Under the processes tab, the task manager showed store.exe using 8 GB of RAM and everything else approximately 0.5 GB of RAM in total
-VMware showed that 8 GB of RAM was ballooned
Then I discovered that the Exchange VM was limited to 8 GB under its resource allocation. Once the limit was removed, everything looked much better. (It is still not clear why would someone assign a VM 12 GB of RAM and then limit it to 8 GB. The possible scenario could be that the VM was assigned (and limited to) 8 GB of RAM initially, then RAM was increased to 12 GB, but the limit was not changed.)
Besides the fact that 8 GB of RAM was ballooned, quite a lot of it was shared and swapped as well.
Couple of days later, swapping and sharing dropped close to 0, although still not 0. At this time, the host had 12 GB of RAM free which was 25% of the total installed RAM.
It seems that the OS shows the entire RAM allocated to the VM, but the usage per process presents the real picture. Now, RAM usage changed in the task manager as well:
Intrigued by the case, I looked further into the matter and found some interesting stuff.
MS Exchange 2010 database cache
Database cache is used to improve performances using RAM instead of disks for data processing and thus minimising disk I/O. Microsoft recommendation for RAM allocation for a mailbox server is 4 GB + 3 to 30 MB per mailbox. The precise number of MB per user depends on the number of processed messages per user.
If you have SCOM you may get email flow statistics by running a report called “Server Daily Mailflow Statistics Summary” that comes with the management pack for MS Exchange 2010. I am not aware of a way to get these statistics from Exchange itself, so if you don’t have SCOM a third-party log parser could help.
Note: In my case, 15 000 emails a day were processed serving 350 local users, that is, less than 50 messages per user. So per the Microsoft table, the required amount of RAM per user for database cache is 3 MB, which is approximately 1 GB for the whole system. Then according to Microsoft recommendation, this server needs 5 GB of RAM in total for optimal performances, while in reality it was quite struggling with 8 GB.
Microsoft Support Policies and Recommendations for Exchange Servers in Hardware Virtualization Environments (MS Exchange 2007)
Basically, Microsoft recommends turning Dynamic Memory Allocation off because Exchange will not be able to use database caching if the RAM is taken away as unused resource.
"Dynamic Memory Allocation Considerations
Many hypervisors include features to dynamically adjust the amount of RAM that is available to one or more virtual machines. This functionality lets the hypervisor allocate RAM to virtual machines based on the current perceived RAM requirements of the particular virtual machines.
Generally, this functionality is appropriate for virtual machine workloads that use a lot of memory for brief periods and then resume typical operations. In this scenario, the hypervisor can allocate memory to meet the needs of the particular workload, and then retrieve the memory for other virtual machines. However, this functionality may not be suitable for workloads that are designed to use a particular memory pool on an ongoing basis.
Many of the performance improvements in recent versions of Exchange are based on the efficient use of an appropriately-sized RAM allocation. This is particularly true of improvements that are related to reductions in I/O operations. The performance optimizations rely on Exchange caching data in RAM. When RAM is dynamically reduced, the expected performance of the system cannot be achieved. In this scenario, Exchange may exhibit reduced performance, or end-users may experience reduced performance when connecting to Exchange. Therefore, for virtual machines that are running Exchange in a production environment, it is best to turn off memory oversubscription or dynamic memory allocation. Instead, configure a static memory size that is based on the appropriate values for Exchange 2007."
Exchange 2010 System Requirements
"Exchange Server Memory Requirements and Recommendations
Some hypervisors have the ability to oversubscribe or dynamically adjust the amount of memory available to a specific guest machine based on the perceived utilization of memory in the guest machine as compared to the needs of other guest machines managed by the same hypervisor. This technology makes sense for workloads in which memory is needed for brief periods of time and then can be surrendered for other uses. However, it doesn't make sense for workloads that are designed to use memory on an ongoing basis. Exchange, like many server applications with optimizations for performance that involve caching of data in memory, is susceptible to poor system performance and an unacceptable client experience if it doesn't have full control over the memory allocated to the physical or virtual machine on which it is running.
Many of the performance gains in recent versions of Exchange, especially those related to reduction in I/O, are based on highly efficient usage of large amounts of memory. When that memory is no longer available, the expected performance of the system can't be achieved. For this reason, memory oversubscription or dynamic adjustment of virtual machine memory should be disabled for production Exchange servers."
Understanding Memory Resource Management in VMware® ESX™ Server
"ESX maintains four host free memory states: high, soft, hard, and low, which are reflected by four thresholds: 6 percent, 4 percent, 2 percent, and 1 percent of host memory respectively.
In the high state, the aggregate virtual machine guest memory usage is smaller than the host memory size. Whether or not host memory is overcommitted, the hypervisor will not reclaim memory through ballooning or swapping. (This is true only when the virtual machine memory limit is not set.)
If host free memory drops towards the soft threshold, the hypervisor starts to reclaim memory using ballooning. Ballooning happens before free memory actually reaches the soft threshold because it takes time for the balloon driver to allocate and pin guest physical memory. Usually, the balloon driver is able to reclaim memory in a timely fashion so that the host free memory stays above the soft threshold.
If ballooning is not sufficient to reclaim memory or the host free memory drops towards the hard threshold, the hypervisor starts to use swapping in addition to using ballooning. Through swapping, the hypervisor should be able to quickly reclaim memory and bring the host memory state back to the soft state.
In certain scenarios, host memory reclamation happens regardless of the current host free memory state. For example, even if host free memory is in the high state, memory reclamation is still mandatory when a virtual machine’s memory usage exceeds its specified memory limit. If this happens, the hypervisor will employ ballooning and, if necessary, swapping to reclaim memory from the virtual machine until the virtual machines host memory usage falls back to its specified limit.
Reservation is a guaranteed lower bound on the amount of host physical memory that host reserves for a virtual machine even when host memory is overcommitted."
How to turn off Dynamic Memory Allocation in VMware ESX 4:
VMware says that Exchange should be okay if we reserve all the RAM allocated to the VM. This would also reduce the swap file associated with the VM to no space at all. In case this is not acceptable, here are the steps for disabling dynamic memory management.
3 methods can be disabled either per VM or host, but memory compression can be disabled only per host.
Disabling balloon driver
Disabling page-sharing
Remove virtual machine swap space on disk
Enable or Disable the Memory Compression Cache (per ESX host only)
More info:
Advanced Virtual Machine Attributes
Memory Behavior when VM Limits are Set - Revisited
Understanding Memory Compression in vSphere 4.1
vCenter Performance Counters
No comments:
Post a Comment