I recently attended a Webinar on the topic of Virtualizing Business Critical Oracle systems presented by a very large company that had started their journey back in 2004 and so far successfully virtualized 86% of their systems, including some of the most critical Oracle systems. After the Webinar I decided to go back and re-read the whitepaper they had published regarding their journey a few months ago to refresh myself on it’s contents. The whitepaper is one of the best I have read, and the presentation on the Webinar was one of the best I had attended. What they laid out was a successful blueprint and framework for how any company might successfully virtualize business critical Oracle systems, and it is definitely a must read. I’ll discuss some of the main points I think are important and what I got out of it, then you can read it yourself and make up your own mind.
The Webinar was presented by Ramesh Razdan, Senior Director and EMC Distinguished Engineer on EMC IT’s virtualization journey – EMC IT’s Virtual Oracle Deployment Framework. Ramesh is a very smart guy and brilliant presenter. Both the Webinar and the whitepaper present a solid, pragmatic, disciplined and methodical approach to virtualizing large scale Oracle applications with differing levels of criticality, which includes the most critical requiring five 9’s availability. A version of the Webinar presented by Darryl Smith (who I have been talking to re the results) and Ramesh is available at http://www.emc.com/events/2012/q1/01-19-12-virtual-oracle-deployment.htm. One of the most valuable aspects of the paper and Webinar was EMC’s candidacy criteria and how they modeled the requirements of each system and application layer before applying these to the best fit virtualization solution. Here are some of the raw numbers covered in the paper:
50K Users, 400K Customers and Partners, ~850 DB’s, 52M Transactions per day, 5 Datacentres, 10PB Storage, 500 Applications, 6.5K OS Images, 80 Countries, 20 Languages
The process EMC undertook to classify their application components and systems by both availability and scalability and then applying those requirements to the appropriate solution. This is very close to the process I take customers through when virtualizing business critical apps and in Unix to Linux on vSphere Migrations. These two dimensions are very important. When planning the journey however I add additional dimensions that I believe are needed in the classification process, such as complexity to migrate, and financial classification in terms of ROI. By using a multidimensional classification approach you can build a clear roadmap and project plan with measurable and achievable results.
Databases and applications previously supported by RAC in some cases don’t need RAC anymore. Applications and systems that required 99.9% availability could be supported by single instance Oracle DB’s deployed in VMware HA/DRS Clusters as the VMware HA functionality provided sufficient availability. RAC was still required for systems that needed very high availability, four or five 9’s, or where resource requirements exceeded a single virtual machine capability, 8 vCPU 255GB RAM on vSphere 4 and 32 vCPU 1TB RAM on vSphere 5. Reduced complexity from having to manage fewer RAC instances is not the only benefit, it may also be possible to lower license costs as you don’t necessarily need to license the same Oracle edition or same features (Oracle Enterprise to Standard Edition) . However there is a cost to managing many different licenses and license types and ensuring compliance. This is why many organizations choose to sign Enterprise License Agreements. For some workloads single instance Oracle DB’s provide better performance. In 2011 I completed a Oracle DB migration project (Sun E25K to RHEL Linux on vSphere 4) for an organization with as many users as EMC. One of the reasons they were migrating and virtualizing their Oracle DB’s was so they didn’t have to deploy RAC for availability. For them the additional cost and complexity was significant.
EMC has been on the journey for 8 years and has learned a lot during the process that can be leveraged by others. Their experience as documented in the whitepaper and presented in the Webinar can be used as a blueprint for other organizations and tweaked where needed. EMC and VMware Professional Services Organizations and partners experienced and accredited with Virtualizing Business Critical Applications can help customers reproduce similar excellent results by using the collective best practices as a starting point to work from.
The process of virtualizing your business’s most critical applications is a journey and does take a disciplined methodical approach, but can have spectacular financial and operational rewards. Plan and design the virtualization project to meet business requirements from the start. Start with low risk, low impact systems, test, build confidence, then move on to other more critical systems. Virtualize Dev/Test before Prod. This is the standard virtualization journey. One aspect that is not discussed in the paper is how large the cost efficiencies can be from being able to run valid tests on systems that are much more a replica of their production counterparts than may ordinarily have been the case. In large organizations with large numbers of applications projects the costs associated with testing are very significant. Accelerating and improving the testing lifecycle can make a very significant improvement to the bottom line, not to mention much faster time to market.
You can use DRS Host Groups to separate workloads for licensing, but be prepared to be audited and prove your use. VMware DRS Host Groups and VM to Host Rules can be an effective way of separating groups of VM’s within a cluster for licensing or availability reasons. If you’re using it for licensing reasons care should be taken to ensure that it will be recognized by the software vendors involved. I would normally recommend a conservative approach is taken and if there are any possible grey areas a separate cluster be used. However this isn’t as efficient from an infrastructure perspective as additional clusters do require additional management and additional failover resources. There is a very good thread of discussion on this topic here posted by Jeff Browning, aka the Oracle Storage Guy. He also posted a blog article on the topic titled Comments by Dave Welch of House of Brick on Oracle on VMware Licensing and here. I would encourage you to read this if you plan to use DRS Host Groups for Licensing separation.
Pay careful attention to the lessons learned, but beware! Every environment is different and best practices and lessons learned from other environments may be a great place to start from, but they may not all apply to your unique environment. I agreed with and use as best practices all of EMC IT’s recommendations accept one, which was Transparent Page Sharing (TPS). Disabling TPS can have disastrous consequences, including causing additional host swapping, which can result in extremely poor performance, much worse than disabling it could ever possibly gain. In every system I’ve virtualized, and in every environment, including those similar scale to EMC’s, I have never had to disable it to achieve acceptable performance. The small amount of CPU required to calculate memory page hashes by itself should not result in any measurable performance difference, and a properly designed and managed environment should not be breaking large pages into small pages for Oracle Databases or Applications in the first place.
I was so concerned by this recommendation that I have followed up with EMC to try and understand the situation and data that lead them to this in the first place. Based on my inquiries I understand the original testing that lead to this recommendation was conducted on ESX 3.5 (not sure which build or update version), in an overcommitted cluster. With that specific test set up of the application layer and the hardware at the time a 5% improvement in performance was measured. EMC’s biggest reason for disabling TPS appears to be to avoid breaking down of large pages and in their scenario the environment is carefully controlled and isolated. They are managing usage of system memory very carefully and use memory very aggressively for Oracle and the hosts are being used for a single purpose and in some cases a single VM. Disabling the TPS option in this scenario may not be as risky because of the expertise they commit to configuring and monitoring their environment. I have not yet been able to determine if the TPS setting was the only setting changed that achieved the 5% measured improvement.
I was advised that subsequent testing on vSphere 5 did not show any improvement by disabling TPS. Unfortunately the whitepaper did not put this recommendation in context with the version of hypervisor tested, and even though the paper was published in November 2011, this recommendation was not updated to reflect the latest testing. There have been a few fundamental changes since the original ESX 3.5 that would in my opinion eliminate any possible improvement that may have resulted form disabling TPS when the other best practices are applied (e.g. VM reservations), such as Intel Extended Page Table (EPT) Support (not supported until ESX 3.5 U4), vSphere backing guest pages with large host pages by default on hardware supporting EPT (Nehalem and above), and major improvements in the hypervisor efficiency. VMware published tests of EPT alone showed 48% improvement in performance for MMU intensive workloads. I have not had time yet to independently verify the findings. My advice would be to not implement this recommendation without your own testing given the serious consequences that could result from disabling TPS, the additional management overheads and low probability of performance gains on recent hardware with a recent vSphere version. Scott Drummonds has posted an article regarding the TPS discussion titled Transparent Page Sharing and Performance, I would recommend you read it.
[Updated:24/10/2012] Subsequent testing on vSphere 5 has shown no difference between having TPS Enabled or Disabled. My very strong recommendation is to not modify the default memory management behaviour when it comes to TPS, so keep it enabled (default setting). Make sure you follow the best practices around memory reservations and using Huge Pages for your databases.
This is the only point in the EMC IT whitepaper that I thought required further investigation and explanation. I am grateful that the authors of the EMC IT paper have been willing to provide more of an understanding of how they got to their results with regard to TPS. Whenever there is something that doesn’t seem quite right, or there is a technical disagreement, it opens up an opportunity to learn something new. Both EMC and VMware take this approach and they are very much learning and collaborative organizations between themselves and also with their respective ecosystem partners. We all benefit from this approach and the engineering culture that results from it.
Here is the link again to the EMC IT Virtual Oracle Deployment Framework. It’s one of the best written whitepapers in my opinion and presents a very strong case for and method to virtualize Oracle applications and databases in very large enterprise environments. Thanks to Ramesh, Darryl and the entire EMC IT Team that were involved in the project of virtualizing their Oracle systems, and especially for publishing the whitepaper so others can all have a great chance of achieving the same success.
EMC IT’s Virtual Oracle Deployment Framework
My company and I have worked closely with EMC and VMware on similar projects and achieved extremely good results for our clients. If you’re a customer interested in how you might start on your journey to virtualizing your Oracle critical business applications, or a partner wanting to lead their customers on the journey contact me from the Author page, or contact your local EMC or VMware Professional Services representative or Account Manager.
For additional information on virtualizing Oracle visit my Oracle Page.
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2012 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
Have you seen TPS's effectiveness on Linux ? In my experience it's basically worthless. I don't disable it because it doesn't really matter much(and I'm not CPU bound on any workload) but I see typically sub 5% savings with TPS on vSphere 4 at least and Linux. Consistently across clusters, hardware, linux versions and vmware versions (and companies as well). Much different case than with windows.
Even with two VMs running the same OS, running the same applications taking the same traffic from a load balancer the amount of memory that TPS reclaims is startlingly tiny with Linux.
Looking at one of my servers which has 192GB of memory, has 17 VMs on it, 99GB of host memory in use. 15 of the 17 VMs have the same OS/version – all are Linux(Well except one, I see my vCenter is on there too with win2k8). TPS savings on this box are 4,806 MB.
– 4 VMS run the same app "app 1"
– 3 VMs run the same app "app 2"
– 2 VMs run the same app "app 3"
– Rest of VMs on the box run other various apps.
– The largest consumer of memory is a MySQL DB which has 32GB of memory
One would think that at least the common OS, and the VMs running the same apps the amount saved would be higher but for whatever reason it's not. This trend goes back at least to 2009 if not earlier(memory foggy on this topic prior to that)
I don't swap on any of my systems. Last thing I want is for VMs to be actively swapping taking up valuable I/O on my storage. I give my VMs standard 512MB of swap in case of emergency, but if they have to swap more than that I'd rather have them fail and crash then thrash my storage.
I used ESX 3.5 and Oracle a bunch at one company back in 2007, it worked very well, and for a while allowed us to slash our licensing by going "unsupported". 2007 was I believe the first release of the Intel quad core processors, so we had deployed a few DL380G5s with single socket quad cores. Combine that with Oracle SE we split our ESX licenses(which came in packs of 2 sockets at the time) between systems. vmware didn't "support" single socket systems(I don't think I ever had to reach out to vmware support), and Oracle didn't support running on vmware (again no tickets filed to oracle either). It obviously dramatically increased our flexibility and slashed our Oracle licensing vs our previous model of each database instance running on separate physical hardware (typically on a dual proc single core system).
I haven't been at a company since that uses Oracle – though I do miss it, MySQL just doesn't come close in so many areas, operationally MySQL is much more complicated to support and troubleshoot.
For production we stuck with bare metal mainly since there was no real compelling case to not use bare metal to override the lack of support we might get from Oracle, and incurring the ESX licensing fees on those boxes.
Hi Nate, Firstly thanks for the comment, it's great to have this discussion. I am aware of the effectiveness of TPS in general and especially on Linux and Windows in different environment, with different hardware, and different versions of ESX/ESXi. The effectiveness of TPS really isn't the point of the article. The point is there is a big danger if you disable it, which can be disastrous. The dangers of disabling TPS should not be downplayed as it can have a big impact not just on the VM or host where it has been implemented but also the rest of the environment. This is because it eliminates TPS as a possibility and this may cause unnecessary host swapping in the case of memory contention, which will in turn cause a big impact on storage, as you've pointed out. We don't want any swapping at any level (again as you've pointed out), but disabling TPS eliminates one of the options that may prevent some host swapping in situations of host memory contention.
vSphere 4 when combined with EPT enabled hardware will back VM's with large pages by default and will not break those pages until there is resource contention. This will have the appearance of higher host memory utilization and lower TPS savings. That is until there is resource contention. NUMA also has implications for TPS, but the hypervisor is smart enough to deal with it (vSphere 4+). vSphere 5 has a number of enhancements in the area of memory management and TPS in particular that you will notice once you start deploying it.
For production applications and Oracle DB's in particular there should be very little to no benefit from TPS during normal operations because I try to right size all the VM's for optimal performance and expect the VM's to use all of that memory and resources assigned. In addition I'd normally be using huge pages in the OS, and tuning the kernel for optimal performance. I will most likely be providing guaranteed service levels and resource assignments to these VM's by way of a minimum level memory reservation and where required providing 100% memory reservations (for Tier 1 apps). But there are still situations, such as host failures, and other unintended situations that may cause the need to enact various memory management techniques, and you want to have TPS available as an option. Not the resource guarantees apply generally to production and tier 1 environments and would not generally be used in non-production environments where higher overcommitment levels are desired and acceptable.
I designed and deployed a large financial system on ESX 3.5 that required 800 Linux VM's. It wasn't entirely smooth sailing for a number of reasons and we had to do a lot of testing and tuning to get the best possible performance, but it was incredibly successful and none of the tuning required disabling TPS. My goal wasn't to use TPS, my goal was to get the best availability and performance possible that met the business requirements, which I achieved. This was a very demanding environment dealing with billions of dollars, and very leading edge at the time. I used TPS as a bonus only for the regions of memory that we knew we could sacrifice and we sized everything accordingly.
Oracle DB and Applications, Including Oracle RAC are fully supported on vSphere (see my other articles below for details) and there is no licensing penalty if you design your environment optimally. In many cases there are license savings. The benefits of virtualizing production systems on vSphere are very compelling, and aren't just limited to hardware savings or efficiencies. Significant opex savings (including Admin time), SDLC / Testing lifecycle saving, DR savings and simplified DR, reduced complexity and greatly increased efficiency. Hardware independence is a big benefit even if you don't leverage SRM, you can restore your VM's anywhere that supports the hypervisor. Testing savings alone can be extremely significant to big projects.
You may like to read these other Oracle on vSphere related articles that I've posted on this blog:
http://longwhiteclouds.com/2011/11/22/deploying-e… http://longwhiteclouds.com/2012/01/22/oracle-rac-… http://longwhiteclouds.com/2011/08/28/oracle-rac-…
yeah that is true that disabling TPS takes away one thing that vmware can use to prevent host/guest swapping, I guess I was just saying at least in my case it really wouldn't have much of a noticeable effect.
Something I'd love to see, something I've written about on more than one occasion at least with regards to linux is to have the memory balloon driver specifically target memory that is used for buffer cache since by default linux uses all memory it can for the cache, frequently this cache isn't even needed but the guest OS isn't aware if it freed the memory then other guests could benefit.
This doesn't apply much to Oracle systems since oracle tends to be configured with direct i/o and a large enough memory footprint that there isn't much left behind for anything else.
So oracle is fully supported in vmware now ? I haven't looked in a while. Last time I saw they said they would try to help but may insist the customer replicate the issue on bare metal. This was a few years ago though.
Looking here it seems as if vmware provides support for Oracle internally so rather than going to Oracle direct customers can go to vmware –
good post though!
Thanks Nate. You are quite correct VMware provides a single point of contact for all Oracle support incidents and will coordinate with Oracle to resolve customers issues on behalf of customers through it's special support relationship.
There is a way to achieve the balloon driver just targeting the filesystem buffer cache, and that is by using huge pages within the guest as the OS can't then swap that memory as it is locked by the process that is using it. This assumes that your software supports huge pages. But this is an effective mechanism I have used in the past where I wanted to have the filesystem cache to improve performance but know that it would be the first and hopefully only target of ballooning. I then protected the memory using huge pages by setting a reservation on the VM to cover the core Java/Oracle processes and a little room for standard OS processes also. This worked well.
[…] Michael Webster's TPS post. […]
[…] For guidance and ides for design and architecture for your Oracle Databases on vSphere here are two great articles. Deploying Enterprise Oracle Databases on vSphere, Blueprint for Successful Large Scale Oracle Virtualization on vSphere. […]
[…] of TPS on Oracle DB virtual machines has been discussed in many place (reference: here, here and here). Yet I still hear from customers and partners, most recently at VMworld in San Francisco, that […]
[…] time ago I wrote an article about EMC’s Blueprint for Successful Large Scale Oracle Virtualization on vSphere. Now Cisco IT has published a similar whitepaper and study after having virtualized a large number […]