| Unique Visitors |
Powered By 
| Unique Visitors |
vSphere 6 includes a new certificate authority that issues certificates for all of the different components within your environment to ensure communications are secure. If you want to use VMCA as a Subordinate CA from an Enterprise CA in your environment you need to change the VMCA Root Certificate on the Platform Services Controller (PSC) prior to installing vCenter or adding any new components to the environment. This article covers the steps in brief assuming you have a Windows based PSC and a Windows Enterprise CA (based on Windows 2012 R2).
The following assumes you are executing these commands from a Windows Platform Services Controller with vSphere 6.0. Note: Anywhere you see a single dash on this page it should be two dashes without a space – -. You may get errors if you just copy the text from this page.
Back up and edit “C:\Program Files\VMware\vCenter Server\vmcad\certool.cfg”, the default looks like this:
Country = US Name= Acme Organization = AcmeOrg OrgUnit = AcmeOrg Engineering State = California Locality = Palo Alto IPAddress = 127.0.0.1 Email = email@acme.com Hostname = server.acme.com
“C:\Program Files\VMware\vCenter Server\vmcad\certool” –initcsr –privkey=privatekey.pem –pubkey=publickey.pem –csrfile=myrequest.csr
Submit the request to your CA, if you’re using a Windows CA (for example 2012 R2), you may be able to use the web interface or the command line tools. Ensure the certificate request is submitted as a Subordinate Certificate Authority request. For the web certificate submission use http://<YourCA>/CertSrv. In most enterprise environments you would likely send the request to your security team or the team that manages the PKI environment.
When your request is approved download the certificate and certificate chain in base 64 encoded format. Download the Root CA Cert also in base 64 encoded format.
Now we have to create the combined VMCA and Root CA certificate chain file in pem format. To do this we copy the VMCA certificate and then the Root CA certificate to a new file.
For example:
type vmcacert.crt >> vmcaroot.pem
type rootcacert.crt >>vmcaroot.pem
This creates the combined pem file with both the VMCA Certificate and the Root CA certificate.
Your should now stop all services and start only those services to do with certificate management:
“C:\Program Files\VMware\vCenter Server\bin\service-control” –stop –all
“C:\Program Files\VMware\vCenter Server\bin\service-control” –start VMWareAfdService
“C:\Program Files\VMware\vCenter Server\bin\service-control” –start VMWareDirectoryService
“C:\Program Files\VMware\vCenter Server\bin\service-control” –start VMWareCertificateService
Now you add the new root VMCA certificate to the Platform Services Controller.
“C:\Program Files\VMware\vCenter Server\vmcad\certool” –rootca –cert=vmcaroot.pem –privkey=privatekey.pem
If you’ve done everything right you should get a message saying Status: Success
Now you can start all of the services again.
“C:\Program Files\VMware\vCenter Server\bin\service-control” –start –all
Note: For those wondering, yes the W in the service names is capital. Whoever wrote these service names didn’t talk to the marketing department about the capatalisation standards.
To verify the new root cert has been applied correctly run the following command:
“C:\Program Files\VMware\vCenter Server\vmcad\certool” –getrootca
If you want to remove the original root certificate then you will have to refresh the Security Token Service (STS) Root Certificate, and replace the VMware Directory Service Certificate.
Final Word
Now your VMCA as part of your PSC will issue certificates with the correct certificate chain and be trusted by any members of your domain. The communications between the components in your VMware vSphere 6 environment will be secure, and you shouldn’t get those annoying warning messages popping up.
—
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2015 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
If you’re upgrading from vSphere 5.1 to vSphere 5.5 and you ARE NOT using Custom CA SSL Certificates then you might run into an error. The error will be encountered during the upgrade of SSO, and specifically the Lookup Service, and only occurs in specific conditions, such as when using the default VMware Self-Signed Certificates. If you run into this problem your upgrade process will roll back, but leave behind some upgrade files that need to be cleaned up. This article will briefly touch on the recommended solution to this problem.
Many of you will recall the many articles that I wrote regarding updating the default self-signed SSL Certificates in vSphere 5.0 and 5.1 to Custom CA Certificates. If you haven’t seen these articles and you’re interested in SSL Security you can check out Updating CA SSL Certificates in vSphere 5.1 and Updating CA SSL Certificates in vSphere 5. To make the process of updating certificates easier VMware created the VMware Certificate Automation Tool and VMware Partner VSS Labs created vCert Manager. If you’re using CA Signed Certificates for your SSL communications between the various vCenter components then you won’t strike the problem I described above. So now might be a good time to review my previous articles and/or use one of the automation solutions that are available.
This issue during the upgrade from vCenter 5.1 to vCenter 5.5 is described in the VMware KB Article – Upgrade from vSphere 5.1 to vSphere 5.5 rolls back after importing Lookup Service data (2060511). You will likely see an error such as “Warning 25000. Please verify that the SSL certificate for your vCenter Single Sign-On 5.1 SSL is not expired. If it did expire, please replace it with a valid certificate before upgrading to vCenter Single Sign-On 5.5.” or the vCenter Upgrade will simply fail and roll back. In the vim-sso-msi.log you will see an error message like the following:
“Action 10:06:03: PostInstallScripts. Importing Lookupservice data…
CustomAction DoUpdateAndMigrateTasks returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)”
As described in the VMware KB this issue does not affect you if:
For the recommended fix please refer to the VMware KB – Upgrade from vSphere 5.1 to vSphere 5.5 rolls back after importing Lookup Service data (2060511). The fix will require modifying the Windows Registry. Before any upgrade of vCenter is attempted it is recommended that you take a backup and potentially have a snapshot in place for the vCenter Database and the vCenter Server system itself so you have a point you can roll back to. This should be standard practice in most VMware environments, as should testing the upgrade process, as should upgrading test environments and management environments before upgrading production. Given that this impacts environments that have self-signed certificates it has the potential to impact a large number of customers, however as it only impacts customers upgrading from vCenter 5.1 prior to version Update 1a to vCenter 5.5, the number of impacted customers is reduced.
Final Word
The easiest way to get around this problem if you’re using vCenter 5.1 would be to either run through the registry fix described in the KB article. Upgrading to vCenter 5.1 U1b will not correct the issue as it doesn’t correct the certificate. You may also choose to completely rebuild your vCenter with a fresh install of vCenter 5.5 against your existing database. Alternatively you may choose to update your vCenter Server to use CA Signed Certificates, which will also improve the security of your critical management infrastructure. Regardless of the option you choose make sure you have a backup of the vCenter Database and vCenter so you can roll back if needed.
—
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2013 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
Want to be able to change the SSL Certificates on your vCenter Servers and vSphere Hosts to properly signed CA certificates at the click of a button? Want to be able to automatically apply certificates to Auto Deployed Hosts? Need a solution that covers vSphere 4.0 through to 5.1? Can detect and alert you to expiring or revoked certificates and change them for you? Something that provides complete SSL Certificate lifecycle management for your vSphere environment with all the reporting, alerting and audit logs you’d expect? Then vCert Manager might be the tool for you and I’ve got a demo of the beta version to wet your appetite.
I wrote an article regarding a few months ago titled vCert Manager – Changing VMware SSL Certs Made Easy, which included a demo of a very early prototype that I presented at VMworld USA in August 2012. The prototype showed how in concept vCert Manager could automatically replace SSL Certificates in vSphere environments with integration back to a windows CA. A few months on now and VSS Labs, who is the company developing the solution, has released a demo of the beta version. The beta is now being tested by select customers who have signed up to the early adopter program. Check out this demo to see how the product has progressed and get an idea of it’s capabilities.
I think you’ll agree that the vCert Manager Beta has come a long way since the very early prototype I presented at VMworld and wrote about previously. The work is not over though. There is still a little way to go to get everything feature complete and enterprise ready and before it is ready to be generally available. The work is progressing well. The demo below is only 16 minutes long, so won’t take up much of your time.
Final Word
I know how much pain changing SSL certificates is and the goal of vCert Manager is to take this pain away and provide a solution that makes it affordable for all vSphere environments to become more secure. I’d greatly appreciate your feedback on the vCert Manager beta demo. Your suggestions and feedback will help VSS Labs make vCert Manager a great product.
—
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2013 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
Recently I wrote about Updating CA SSL Certificates in vSphere 5.1 which applied to the Windows installable version of the vCenter 5.1 and it’s supporting components including SSO. VMware has now also released the instructions to update the CA SSL certificates in the vSphere 5.1 vCenter Virtual Appliance.
While there are a total of 136 steps in the process to update the CA SSL Certificates in the Windows vCenter 5.1 there are only 81 steps to update it in the vSphere 5.1 vCenter Virtual Appliance. But note that these steps do not include update manager. I will include the link below to the KB regarding update manager also.
Here are the KB articles required to update the vSphere 5.1 vCenter Virtual Appliance and Update Manager.
Configuring certificates signed by a Certificate Authority (CA) for vCenter Server Appliance 5.1 – http://kb.vmware.com/kb/2036744
Configuring CA signed SSL certificates for vSphere Update Manager in vCenter 5.1 – http://kb.vmware.com/kb/2037581
It was another great team effort across the globe within VMware to put these instructions together and test them. Hopefully you find this information useful.
—
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2012 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
Over the past few weeks I have been working behind the scenes with a team of people at VMware spread around the globe on the process to successfully change out the self-signed certificates in vSphere 5.1. With the introduction of Single Sign-On in vSphere 5.1 the process is somewhat more complicated than vSphere 5 (ok quite a lot more complicated). But now I’m able to bring you some of the solutions you’ve all been waiting for.
This work covers vCenter, and all the related core components such as SSO, Inventory Service, Update Manager etc. The great news is that this work has resulted in KB’s that I and a number of others have tested and verified to work with vSphere 5.1 GA for the Windows installable version of vCenter. There are also updates to some previously released KB’s for vSphere 5.0. These processes will also work with the recently released patches to vCenter. The KB articles for the vCenter Virtual Appliance edition will also be published shortly and I will update this article when they are available.
Below are the links to all of the articles and a note with regard to Update Manager. I want to say a massive thank you to all of the people at VMware that made this happen. It was a big team effort. I’m glad I could make a contribution to the effort. I will be making sure the process is automated for you as part of the vCert Manager project that I’m working on. My goal would be to automate both the Windows Installable and Virtual Appliance editions for vSphere 5.1.
Note you should start with KB 2034833 – Implementing CA signed SSL certificates with vSphere 5.1.
| Configuring CA signed certificates for VMware vCenter Server 5.0.x – http://kb.vmware.com/kb/2015421 |
| Configuring CA signed SSL certificates for vSphere Update Manager in vCenter 5.1 – http://kb.vmware.com/kb/2037581 |
| Creating certificate requests and certificates for the vCenter 5.1 components – http://kb.vmware.com/kb/2037432 |
| Configuring CA signed SSL certificates for vCenter SSO in vCenter 5.1 – http://kb.vmware.com/kb/2035011 |
| Configuring CA signed SSL certificates for the Web Client and Log Browser in vCenter 5.1 – http://kb.vmware.com/kb/2035010 |
| Configuring CA signed SSL certificates for the Inventory service in vCenter 5.1 – http://kb.vmware.com/kb/2035009 |
| Configuring OpenSSL for installation and configuration of CA signed certificates in the vSphere environment – http://kb.vmware.com/kb/2015387 |
| Configuring CA signed certificates for ESXi 5.x hosts – http://kb.vmware.com/kb/2015499 |
| Configuring CA signed certificates for vCenter 5.1 – http://kb.vmware.com/kb/2035005 |
| Implementing CA signed SSL certificates with vSphere 5.0 – http://kb.vmware.com/kb/2015383 |
| Implementing CA signed SSL certificates with vSphere 5.1 – http://kb.vmware.com/kb/2034833 |
VMware has also put out a blog article on these KB’s titled Implementing CA Signed SSL Certificates with vSphere 5.1.
Note: I have found a problem with Update Manager when vCenter system is an all in one configuration with everything on the same VM and using a local MS SQL Server database. Update Manager will not be able to log into or register with vCenter when the SSL certificates have been changed. This prevents you from updating the SSL certs for Update Manager and Update Manager may no longer work. This does not appear to occur when the MS SQL Server database is remote. I have not tested this with a local Oracle or other supported local database. I am continuing to work with VMware on this issue and will update this article when it is resolved. In the meantime I would recommend placing the databases for vCenter and it’s other core components on a separate VM, even in small environments.
Final Word
Although changing out the self-signed SSL Certificates is not simple, and is very time consuming to do manually, the above articles make it possible and give you a tested and verified process. I will be automating the processes to take this pain away as part of the vCert Manager project. In the meantime I would recommend you start with KB 2034833 – Implementing CA signed SSL certificates with vSphere 5.1 and work your way through the rest. I hope you get a lot of value out of these articles and the effort that the team has put in. As always your feedback is appreciated.
Derek Seaman has put together a great series of articles on VMware vCenter 5.1 Installation that includes coverage of SSL certificates. I would highly recommend you check it out. Derek has made a great contribution to the process for SSL Certificate Replacement in vSphere 5.1.
—
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2012 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
The twitter wires and blogosphere were ablaze with news out of VMworld US 2012 (August 27 – 30th). This was my first ever VMworld (with hopefully many more to come), and I greatly enjoyed it and I also enjoyed meeting many of you. My direct flight home to Auckland from San Francisco on Air New Zealand was the best flight I’ve ever had, and I got a full 8 hours sleep so I didn’t have any jetlag (Thanks Air New Zealand). But this article is all about my take on the event, what I learned, and vSphere 5.1. I’ve decided to do something slightly different to others, to take it all in, and then write this roundup post VMworld. I’m also going to target this towards the relevance to production and business critical applications environments. I’ll also give you some insight into the sessions I presented, the results and my lessons learned.
vRAM and Cloud Suites: Ding Dong! The Witch is dead. Which old Witch? The vRAM Witch! (The movie this is from is well before my time – Can you guess which movie?) Yes the vRAM Witch is now definitively dead. VMware Announced in the first Keynote on Monday 27th August that vRAM licensing is no more and they will instead be introducing vCloud Suites containing a bundle of products based on per CPU Socket, unlimited cores, unlimited RAM, and unlimited VM entitlement per licensed socket. Although the suite is a collection of products right now over time will become ever more integrated, and the licenses can’t be broken apart into their individual components. This is all very good news for VMware customers. The vRAM announcement was made in the context of vSphere 5.1, but it also applies to vSphere 5. So even if you’re running a vSphere 5 environment vRAM is no longer relevant. Personally I didn’t have a problem with the concept of vRAM as everything is moving towards a consumption based model, but it did cause a lot of extra things to consider during design, especially when every customer I ever engaged with had no impact as a result of vRAM. The free VMware vSphere Hypervisor will still be limited to running on hosts with 32GB physical RAM, but there are no longer any vRAM limitations (think configured memory / overcommitment is unlimited). Everyone with vSphere Enterprise Plus will get entitlement to vCloud Suite Standard, and VMware will be running promotions to get customers to upgrade to the other editions, so watch out for those. Information and comparisons between the vCloud Suites. This is great news for all environments, especially those with Monster VM’s. So now you don’t have to worry about VMware vRAM licensing for your business critical applications you can go back to only worrying about your ISV licensing and the best solution to meet all your other requirements.
vSphere 5.1 and the Mega Monster VM: 64 vCPU, 1TB RAM, 1M IOPs per VM, less network jitter, lower latency, Zero-downtime upgrade for VMware Tools (from 5.1 onwards), Dump Collector works with vDS. These are just some of the highlights of vSphere 5.1. VMware has taken the Monster VM and turned it into a more Mega Monster VM. Best of all the efficiency is still what you come to expect form VMware. So if you can configure 64 vCPU’s you know you can get within a few percentage points of native. IMHO it’s not good saying your architecture supports a huge number of vCPU’s if you can’t utilize them efficiently. VMware does a lot of work to ensure they optimize their architecture to get best efficiency as well as scalability. 1 Million IOPS per VM is great if you have a single VM that you can run off an entire fully FLASH array. But realistically this is just to eliminate any possible thoughts that the hypervisor is the bottleneck when it comes to storage. The test was conducted with 4k IO size and produced very low latency. The IO Size and Latency being important factors as I outlined in Storage Sizing Considerations when Virtualizing Business Critical Applications. See What’s New in VMware vSphere 5.1 – Platform and What’s New in VMware vSphere 5.1 – Performance.
VMware vSphere Distributed Switch: Config Backup/Restore, Rollback and Recovery, Network Healthcheck, BPDU Filter. There is now no reason to run a mixed vSS / vDS environment. With the config backup / restore and automatic rollback and recovery you can be confident that the vSphere Distributed Switch (note name change) will be reliable and available, and easy to recover when things go wrong. The Rollback and Recovery will revert any change that has the consequence of disconnecting the hosts from vCenter or vice versa. Network Healthcheck will periodically check the network for configuration errors such as incorrect VLAN trunking, incorrect MTU, uplink erros etc and alert you to these issues before they become a major problem. This should greatly reduce the effort required in quality assurance when provisioning new hosts and operating hosts as the environment changes. The new vDS also supports Netflow v10 (IPFIX), LACP (IP Hash Only), and also RSPAN/ERSPAN. BPDU Filter is important as it stops the accidental or malicious configuration of a bridged VM from causing a physical host port down event and cascading failure across the cluster. BPDU filter will filter out any BPDU packets. As mentioned above the Network Dump Collector, which collects Purple Screen of Death (PSOD) Core dumps from ESXi hosts now works with vDS. In vSphere 5 this only worked with the standard vSwitch. See What’s New in VMware vSphere 5.1 – Networking.
VMware vSphere Storage Enhancements: All Paths Down (APD), Permanent Device Loss (PDL), Storage IO Control Enhancements, Parallel Storage vMotion, Combined vMotion / Storage vMotion without Shared Disks. The APD and PDL behaviour has been again enhanced in vSphere 5.1, which will see far more predictable behaviour under what should be very rare storage failures. Storage IO Control has been enhanced to be more self tuning. Storage vMotion now supports up to 4 parallel disk copies per VM. See What’s New in VMware vSphere 5.1 – Storage. With the combined vMotion / Storage vMotion and no need to have shared storage we can say goodbye to the concept of a swing datastore or jump datastore. Duncan Epping does a great job of covering this in his article “Say Goodbye to the Transfer LUN aka Swing LUN aka Stepping Stone“.
vCloud Networking and Security: HA, SSL VPN, Load Balancer, 10 NIC’s per Edge, VXLAN Gateway, Endpoint included with vSphere 5.1. All of the new features of vCloud Networking and Security are a major leap forward from the previous version of vShield, which this supersedes. The HA functionality for vShield Edge combined with support for 10 NIC’s, which are user configurable between internal / external means that you can realistically replace a large number of enterprise firewalls very cost effectively. This also means you can very cheaply set up realistic testing and validation environments to test multi-tier applications and their firewall rules before you apply the firewall rules to production physical firewalls. With HA if one host with the primary Edge device fails the firewall state will failover to stand by Edge, this is a real active / passive firewall cluster. Load balancing has been greatly improved to include health checks and can now support HTTPS pass through and any custom TCP ports. SSL VPN is a very convenient way of allowing end user access to the vApps and infrastructure protected by the Edge or for management of the infrastructure. The admin user interface has been greatly enhanced and so has it’s capability, including the logging functionality. Many will be pleased that rules now have a rule ID and this flows through into syslogs. The interface is much more intuitive when it comes to App Firewall also and is simplified removing the rule precedence that existed in the previous version. Flow monitoring is improved and you can now get statistics per rule to determine which rules are being used in addition to the top rules that are used. With Endpoint now included with the Hypervisor I predict that most organizations will start moving to VMware’s Endpoint protection and partner integrated solutions. Service Insertion now allows parters to integrate virtual editions and physical editions of their components with vCloud Networking and Security and also vCloud Director. This will allow many organizations to further differenciate their services and offerings. The automation capabilities that are possible through vCloud Director, vCloud Connector and the REST API’s mean that vCloud Networking and Security is a major step forward with capabilities that really deliver on the software defined datacenter and software defined networking and security. See VMware vCloud Networking and Security Overview.
vCloud Director: SDRS Integration, Storage Profiles / Storage Tiering, Elastic VDC, Linked clones on VMFS across 32 hosts, vApp Snapshots, HA Edge Devices. See What’s New in VMware vCloud Director 5.1. There are so many improvements in vCloud Director 5.1 that I’m only going to cover a few very briefly. Storage DRS and Storage Profile integration is a big one. You will no longer require a separate Provider VDC just to support a different tier of storage. For smaller environments this made the design very tricky as you might in a single 2 or 3 node cluster have to support 2 tiers of storage. This forced you to break with some best practices and use resource pools instead of clusters as the demarcation for the Provider VDC compute resources. This will help greatly improve resource utilisation efficiency in vCloud Director environments. It will be interesting to see the new designs incorporating this and how they are now differentiating their service offerings. With vCloud Director 1.5 you could configure an Elastic VDC across multiple clusters only with the PAYG resource model, but all the vShield Edge devices stayed in the original cluster. With 5.1 you can now do this also with the Allocation Pool resource model and vShield Edge and system resource pools will be split across clusters. With the addition of VXLAN it is now also easier to stretch VDC’s across clusters and this adds improved performance to the isolation networks.
New Certifications: VMware launched a number of new certifications for the Desktop and Cloud tracks. We now see certification paths right up to VCDX-Cloud and VCDX-Desktop. The existing VCP and VCDX have been renamed slightly to VCP-DV and VCDX-DV to designate Datacenter Virtualization. The actual path to VCDX-Cloud and VCDX-Desktop is not quite clear yet and neither is the migration path for existing VCDX qualified individuals. But it is great to see these two new certification paths that will allow everyone to demonstrate their mastery of these technology areas in addition to Datacenter Virtualization. See VMware Certification Roadmap. If you think this looks similar to how Cisco’s certification works you’re right. This is intentional and it just happens the man who designed Cisco’s certification tracks is now in charge of doing that at VMware.
Oracle Virtualization Architecture and Performance Deep Dive: I presented two sessions at VMworld US regarding Oracle Virtualization. The first one APP-BCA1432 – Virtualizing Oracle Across the World — Success Stories from University of Auckland and Indiana University covered the process of how to go about virtualizing Oracle when migrating from traditional Unix platforms and how to engage the DBA’s and keep them happy. My content was based on a large project that I had delivered on behalf of VMware Professional Services. I had Don Sullivan (Oracle Certified Master) from VMware and Dan Young from Indiana University as co-presenters. In my second session APP-BCA1624 Virtualizing Oracle: An Architectural and Performance Deep Dive we really drilled into how to architect Oracle databases for maximum performance and how the hypervisor helped. In this session I had Mark Achtemichuk from VMware (Performance Technical Marketing) and Don Sullivan again. I took the same project as my previous session but this time really drilled down in the technical details of how we delivered 5x performance improvement from the source systems and as such a high ROI. Both would give you a very good understanding of how you really can virtualize Oracle Databases in large organizations successfully and ensure you meet the business requirements and performance requirements.
I received some pretty good ratings (4.39 and 4.3 respectively) for these sessions so a big thank you to all of the people that attended these sessions. You all thought we hit the mark with the content. This is very encouraging and I’ll try and do even better next year if I get a session selected. Based on the feedback a lot of people thought the sessions weren’t long enough. We could have talked for a lot longer and gone a lot deeper. This is the challenge when the sessions are only 60 minutes.
Automating Security and Compliance with DR: I presented this session INF-SEC1282 Automating Security and Compliance with Disaster Recovery Using VCM, vCOps, vShield, VIN and SRM along side Gargi Keeling who is the Product Manager for Security at VMware. This presentation was loosely based on a customer project I had been involved with where we had designed automated security and compliance processes along with DR. In addition to the learning from the actual customer project we enhanced the presentation with a partner solution (Catbird) that allows for automated syncing of vShield polices across multiple datacenters. The presentation covers all of the process and technology steps you need to take and gave an example of a technical architecture that would allow you to implement this, all using out of the box functionality from vShield, vCenter Configuration Manager, vCenter Operations, Virtual Infrastructure Navigator, and vCenter Site Recovery Manager, and supplemented if required with the Catbird solution. This presentation was also the worldwide premier of the SSL Management Solution mentioned below vCert Manager, which was very well received.
I received pretty good rating for this session of 4.15. Not quite as good as my Oracle sessions, so I will try and do better next time. This one was pitched as just a technical session not advanced technical. I also received a lot of feedback that the session wasn’t long enough and it would have been good to have the time to go deeper. What I’ve learned from the presentations I gave is that I probably need to narrow the scope and go a lot deeper. This will allow a lot more to get into a 60 minute presentation. Feel free to comment on this article and let me know your thoughts on this.
SSL Management – vCert Manager: My demo of the vCert Manager prototype was very well received and everyone in the audience of the Automating Security and Compliance with DR session agreed it would greatly simplify the process of managing SSL Certificates in VMware environments. I have published the Demo online and written about it in article vCert Manager – Changing VMware SSL Certs Made Easy.
The below sessions I highly recommend you review. I attended these sessions and thought they were a real highlight. Note I only had very limited time so I wasn’t able to attend many great sessions. I would have liked to have gone to the vCenter Technical Deep Dive and also Jason Nash’s vSphere Distributed Switch Deep Dive also. Jason got the top session of VMworld this year. I think it might be the first year a non-VMware employee has had the top spot.
Virtualizing SQL 2012: APP-BCA1516 Virtualizing SQL 2012 : Doing It Right. Jeff Szastak of VMware and Michael Corey of Ntirety managed to get through 160 slides of a very entertaining and deep technical presentation in just 60 minutes. I think they finished on time to the minute even with questions. I was very flattered that Jeff and Michael borrowed a quote from my Oracle Virtualization Architecture and Performance Deep Dive – “Your database is just an extension of your storage”. It is definitely relevant to SQL just as it is to Oracle or any other database. Optimizing storage performance is critically important and Jeff and Michael covered it well in the context of SQL Server 2012 and the relevant best practices.
SMP FT a.k.a. Multi-vCPU Fault Tolerance: INF-BCO2655 VMware vSphere Fault Tolerance for Multiprocessor Virtual Machines—Technical Preview and Best Practices. Presented by Jim Chow, Shrinand Javadekar, Srinivas Kotamraju, all from VMware. There was no timeframe or commitment given on when or if this might actually make it into the product given given how good it was I really hope it’s sooner rather than later. One of the attendees said this technology would literally save peoples lives as he worked in the 911 system as a systems admin and they could not leverage VMware FT currently due to it’s limitations. I can see many and varied applications for this. I can’t wait to get it into my lab environment when if it gets released.
Stretched Metro Clusters: INF-BCO1159 Architecting and Operating a VMware vSphere Metro Storage Cluster. Duncan Epping and Lee Dilworth did a great job of covering all the key points of architecting and operating a vSphere Metro Cluster environment. This is becoming a very popular solution for many environments these days, but it is not without its challenges.
Storage DRS Datastore Clusters: INF-STO1545 Architecting Storage DRS Datastore Clusters. Frank Denneman and Valentin Hamburger highlighted a number of key considerations when architecting Storage DRS datastore clusters, including some important limitations and considerations around storage IO control and array auto tiering. I wouldn’t operate a Storage DRS Datastore Cluster environment without reviewing this session first.
Oracle RAC Cluster Build Automation: APP-BCA1333 Virtualizing Oracle RAC. Rick Lindberg, Don Sullivan and Bryan Wood of VMware took the audience through the ins and outs of successfully virtualizing Oracle RAC on vSphere. Including the fully automated deployment of a new Oracle RAC Cluster in under 30 minutes (cut down demo recording was 7 minutes). The automation, which is available via a VMware Professional Services engagement allows not only new Oracle RAC Cluster creation but also node addition and node removal from existing clusters that have been created through this process. This will be especially valuable in Test and Development environments. The session also covered what VMware IT is doing in the process of virtualizing all their Oracle RAC systems and the necessary best practices to ensure the process is successful.
Final Word
It was great to see Oracle actually had an official presence at VMworld this year. They had a booth in the Solutions Exchange, which I stopped by for a chat and they gave me a nice T-shirt, and also had taxis and branded cars taking customers from their hotels to VMworld. This is another great show of support for VMware, which is a great place to run Oracle databases and applications. Oracle also confirmed at VMworld that running their applications and databases in a large cluster and using DRS Must Affinity Rules is a perfectly acceptable solution, provided the rules are not violated and the Oracle software is not installed and/or run on an unlicensed host. They also completely clarified the support situation with VMware vSphere. I think all of this is absolutely great news for Oracle and VMware customers. Now if you don’t believe that this actually happend why not just review the video, which is in an article on the License Consulting blog – VMworld TV – Richard Garsthagen Oracle Licensing and Support in VMware Virtualized Environments.
This was my first ever VMworld and it will definitely not be my last. I had a great time presenting to over 650 people and got great feedback. I met so many great people and was able to hang out with some of the VMware virtualization royalty. The only problem I had with VMworld was that it went way too fast. Mind you it was really hard work getting up at 6am every day and not getting to bed until after midnight most days. I would like to once again thank everyone that attended my sessions and gave feedback through the surveys, it was greatly appreciated. I’m looking forward to seeing some of the great people again in a couple of weeks at VMworld Barcelona, which I will be presenting a session titled APP-BCA1751 – Oracle Virtualization: Caging the Licensing Dragon with a great lineup of co-presenters. I hope to see some of you there. I also hope to meet a lot more new people.
—
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2012 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
During my VMworld session presentation INF-SEC1282 Automating Security and Compliance with DR (VMworld account required to access recording) I gave a world premier glimpse of a prototype solution that will allow completely automated management of SSL Certificates in a vSphere environment. The solution is still under development. But if you’d like to peak into the future of an easy and completely automated SSL management world for vSphere then this article is for you.
[Updated 14/09/2013] vCert Manager is now Generally Available! This was announced at VMworld USA 2013 in San Francisco. If you’d like to see how the prototype changed into the full product please check out my article VMworld USA 2013 By The Numbers. You can obtain an evaluation version of vCert Manager by visiting VSS Labs.
The session was an outstanding success, we received a massive response from the audience and subsequent to the session. As a result of this positive feedback we’ve decided to make the demo video available to the public on YouTube here and displayed below. I’m the lead architect of the solution and I’m working with VSS Labs based in Singapore and Philippines. If after reviewing the demo you’d like to become part of the early adopter / beta program please visit the VSS Labs web site and register your expression of interest by filling in the Early Adopter Form.
Some things you should know about the demo before you watch it:
Once you have watched the demo please complete the brief survey below.
Please let us know what your thoughts are on the most critical components we should support when we release vCert Manager 1.0.
Final Word
Managing SSL Certs in a VMware environment is a very complicated, time consuming, error prone, and costly task. My hope is that vCert Manager will revolutionize SSL Management in VMware environments, make it simple, easy, and cost effective to change and maintain SSL certificates throughout their lifecycle, for all customers. Providing a more secure platform to many customers that wouldn’t or couldn’t currently change their SSL certificates. If after reading this article and seeing the demo you still want to do your certificates manually then please feel free to check out my article on Updating SSL Certificates in vSphere 5. I look forward to receiving some good feedback and comments.
—
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2012 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
I’ve written a few articles now on how to change the self-signed SSL certs in a few of the VMware components, such as vCenter Server 5, vSphere Web Client, and ESXi 5 Hosts. All without any discussion about why you would want to do it at all. So why do you bother going to all the trouble of changing out the self-signed SSL certs for Org CA or Public CA signed SSL certs?
If you want a way to fully manage the certificate lifecycle and replace certs automatically then you’ll want to check out vCert Manager – Changing VMware SSL Certs Made Easy. This will completely automate the SSL certificate process in vSphere environments.
One of the main reasons it is generally recommended is to prevent or reduce the risk of Man in the Middle (MiTM) attacks. This is where someone impersonates a valid system or communication using an untrusted or self-signed SSL certificate and in the process intercepts encrypted traffic. This is one of the things the Public Key Infrastructure (PKI) has been built to prevent. However this is one of many reasons why you would want to change the certs for valid CA signed certs.
Changing out certificates for all of the software components in a vSphere infrastructure requires a lot of effort. It can also be quite painful, although I hope my articles go some way to helping with the difficulty level. But in a lot of environments it is definitely worthwhile or required.
Instead of giving you all the reasons you might want to go to this trouble and change the certs for your organization I have instead developed a quick poll. I’d like to know from you why you do it? What motivates your organization to implement Org CA or Public CA signed SSL certs for your vSphere environment and the various components?
Let me know your top 5 reasons below (Select up to 5 options from the list):
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2012 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
I’ve been updating my vCenter and ESXi certificates recently and I ran into one particular system so far that had absolutely no documentation or KB articles to help with changing default SSL certificates for CA signed ones. The system was my vCenter Server Virtual Appliance. You might remember that I wrote about this as a means of using it as the vSphere Web Client without needing an additional Microsoft Windows License and then I used it with a load balancer to Increase vSphere Web Client Availability and Scalability. But a lack of documentation wasn’t going to stop me. Being a SLES based virtual appliance though meant things were quite different when changing the certs. If you want to save yourself a lot of time changing the SSL Certificates for the vCenter Server Virtual Appliance then read on.
If you want a way to fully manage the certificate lifecycle and replace certs automatically then you’ll want to check out vCert Manager – Changing VMware SSL Certs Made Easy. This will completely automate the SSL certificate process in vSphere environments.
I tried to find information on changing the SSL certificates for the vCenter Server Virtual Appliance (VCVA) via Google and also on the VMware web site. The only information I was able to find that was in the correct context was the following:
vCenter Server Appliance: Where Do I Upload SSL Certificate on the VMware Communities Site
vSphere 5 vCenter Server Virtual Appliance Quick-Start Guide
Automating vCenter Server Virtual Appliance Configurations by William Lam
vCenter Management Appliance Certificates by Doug Baer
Now before we begin, because the vCenter Server Virtual Appliance is Suse Linux Enterprise Server based you will have to be used to a Linux command line, using scp, and generally navigating around in order to successfully change your certificates. All operations will be done as root. The default password is vmware. Like in my previous articles regarding changing SSL Certificates I have included an example OpenSSL configuration file that you can use to generate your certificates.
The following directories on the VCVA contain SSL certificates in one form or another:
/opt/vmware/etc/lighttpd/
/etc/vmware-vpx/ssl
/usr/lib/vmware-vpx/inventoryservice/ssl
/usr/lib/vmware-vsphere-client/server/config
I will go through what needs to go where after I’ve given you what you need to create the certificates.
Step-by-Step Process for Changing SSL Certificates on VCVA
You could execute a similar process to the one I’m about to describe using an OpenSSL or Public CA and using the Unix/Linux version of OpenSSL, however this is how I did it successfully in my lab and with my customer. As mentioned in the vSphere 5 Security Guide VMware uses X.509 v3 SSL certificates (base-64 encoded) for encrypting traffic between various components. If you CA has been set to support only SHA512 hash that is fine, it will work. The three key files for an VCVA are rui.crt, rui.key and rui.pfx.
In order to generate the certificates you’ll need to get a copy of OpenSSL x86 v0.98r or higher, and have access to a Microsoft CA (2003 or higher). The certificates will use a clone of a standard web server request template with Subject Alternative Name added, for my lab I modified the default Web Server Certificate Template to accept up to 15 years for certificates. On the system where you will generate the certificate signing request rui.csr) you will need to ensure you have Microsoft Visual C++ 2008 Redistributable Package (x86) before installing OpenSSL. For the purposes of this process you will use the Microsoft CA Web Pages to submit the certificate request and download the resulting base-64 encoded certificate. You can use the certreq command if you wish also (not covered here). Before applying the certificates to your environment you should ensure that your clients and vCenter server trust your CA, if it’s an AD integrated CA this should be automated (if using Internet Explorer), else you may have to pre-trust the Root or Intermediary CA by loading the CA public cert into your clients and vCenter server (not covered in this process).
Note: If you are using a keypair provided by a commercial CA or already have a key and certificate generated outside of the VCVA you will need to import the keypair into the keystore to cover steps 21 – 23. Example instructions of how to do this you can follow Importing private keys into a Java keystore using keytool by Graham Leggett.
Prerequsites:
Microsoft CA (2003 or above, with Web Server Template with Subject Alternative Name included and configured to your liking).
Microsoft Visual C++ 2008 Redistributable Package (x86) on the system where you will generate the certificate signing request (CSR).
OpenSSL 0.98r or above on the system you will use to generate the CSR vCenter 5.0.
A VCVA Deployed and configured the way you like it.
A backup or snapshot of your VCVA prior to beginning this process would be recommended.
Process Step by Step:
Based on William Lam’s comments below it is possible to simplify the above process as follows:
Step 15 and 16 can be replaced with:
vpxd_servicecfg service stop
vpxd_servicecfg certificate change rui.crt rui.key
I would recommend that you check out his post on Automating vCenter Server Virtual Appliance Configurations.
Please let me know if you have any trouble with the above process, and also if it works for you, your comments and feedback are appreciated.
Example OpenSSL Configuration file (openssl.cfg) without most of the normal comments and white space that is included:
# Example vSphere Web Client OpenSSL Configuration File HOME = . RANDFILE = $ENV::HOME/.rnd oid_section = new_oids
[ new_oids ]
[ ca ] default_ca = CA_default # The default ca section
[ CA_default ]
dir = ./demoCA # Where everything is kept certs = $dir/certs # Where the issued certs are kept crl_dir = $dir/crl # Where the issued crl are kept database = $dir/index.txt # database index file. new_certs_dir = $dir/newcerts # default place for new certs. certificate = $dir/cacert.pem # The CA certificate serial = $dir/serial # The current serial number crlnumber = $dir/crlnumber # the current crl number # must be commented out to leave a V1 CRL crl = $dir/crl.pem # The current CRL private_key = $dir/private/cakey.pem# The private key RANDFILE = $dir/private/.rand # private random number file x509_extensions = usr_cert # The extentions to add to the cert name_opt = ca_default # Subject Name options cert_opt = ca_default # Certificate field options default_days = 5475 # how long to certify for default_crl_days= 30 # how long before next CRL default_md = sha512 # which md to use. preserve = no # keep passed DN ordering policy = policy_match
[ policy_match ] countryName = match stateOrProvinceName = match organizationName = match organizationalUnitName = optional commonName = supplied emailAddress = optional
[ policy_anything ] countryName = optional stateOrProvinceName = optional localityName = optional organizationName = optional organizationalUnitName = optional commonName = supplied emailAddress = optional
[ req ] default_bits = 2048 default_keyfile = privkey.pem distinguished_name = req_distinguished_name attributes = req_attributes x509_extensions = v3_ca # The extentions to add to the self signed cert
input_password = testpassword output_password = testpassword encrypt_key = no prompt = no string_mask = nombstr
req_extensions = v3_req # The extensions to add to a certificate request
[ req_distinguished_name ] countryName = NZ
stateOrProvinceName = Auckland
localityName = Auckland
0.organizationName = IT Solutions 2000 Ltd
organizationalUnitName = IT
commonName = vsphere-web.homedns.org
emailAddress = admin@homedns.org
[ req_attributes ]
[ usr_cert ]
basicConstraints=CA:FALSE nsComment = “OpenSSL Generated Certificate” subjectKeyIdentifier=hash authorityKeyIdentifier=keyid,issuer
[ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment extendedKeyUsage = serverAuth, clientAuth subjectAltName = DNS:vsphere-web.homedns.org, DNS:vcva.homedns.org, DNS:vcweb.homedns.org, DNS:vsphere-web, DNS:vcva, DNS:vcweb
[ v3_ca ] subjectKeyIdentifier=hash authorityKeyIdentifier=keyid:always,issuer:always basicConstraints = CA:true
[ crl_ext ] authorityKeyIdentifier=keyid:always,issuer:always
[ proxy_cert_ext ] basicConstraints=CA:FALSE nsComment = “OpenSSL Generated Certificate” subjectKeyIdentifier=hash authorityKeyIdentifier=keyid,issuer:always proxyCertInfo=critical,language:id-ppl-anyLanguage,pathlen:3,policy:foo
# End of vSphere Web Client OpenSSL Configuration File
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2012 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
For those of you that follow me on Twitter you’ll know that I’ve been having some fun this week with changing out the default VMware generated SSL certificates on a greenfields deployment of vSphere 5 that will be supporting a large public cloud. Changing certificates is nothing new, and in environments that are concerned with security it is common practice. However it has been my experience that changing certificates with ESX(i) and vCenter has always been a bit of a challenge (I have done it on vSphere 4.x before this). It can be very time consuming and error prone, especially if you haven’t done it before. One of the things that makes it hard for people to get this right is that there is no one document or source of truth that explains in sufficient detail what the requirements and supported configurations are or how to implement CA signed ssl certificates in ESX(i) and vCenter Server. This has tripped up many organizations both large and small. I’m hoping that the information in this article will help and encourage more people to change out the default certs (to improve security), and make the process far more reliable and easier to achieve with vSphere 5. This article will focus on successfully changing the default VMware SSL certificates on ESXi 5 hosts with CA signed certificates using a Microsoft CA (it will also work with public and OpenSSL CAs, but I have not tested it yet).
If you want a way to fully manage the certificate lifecycle and replace certs automatically then you’ll want to check out vCert Manager – Changing VMware SSL Certs Made Easy. This will completely automate the SSL certificate process in vSphere environments.
General Information on X.509 Certificates
For anyone that doesn’t know what an X.509 certificate is here are a couple of links that will explain it. The first one is a good human readable explanation and the second is the actual specification published by the Internet Engineering Taskforce (IETF).
Each component in your vSphere Infrastructure uses these X.509 SSL certificates for secure encrypted communications. Each SSL certificate is uniquely generated for each component and ties to the FQDN of the component. So this means every ESXi 5 server has a certificate generated for it based on it’s unique FQDN, as does vCenter, vCenter Update Manager, SRM, vShield Manager and any other components you may be using. This ensures non-repudiation. That is, every system knows that is communicating with the other system that it expects, and it’s not an imposter. This means you can’t just take one cert generated for vCenter for example and apply it to all of your hosts. I have not tested using wildcard certificates (*.domain) with vSphere 5, but in earlier versions some components didn’t support them. From a security standpoint it is much better to have a single SSL cert tied to a single host by FQDN.
VMware CA SSL Certificate Resources
While I was working through generating and applying the certificates in the environment I was working in over the last week I ran through all of these resources below. All have some good information. But none is a complete end to end guide on how to generate and apply the certificates that will work reliably with vSphere 5 (which is the reason I had to review them all). The reason why I went through some of the older material on this is because it is largely still relevant. But there are some subtle changes to the way that vSphere 5 works that you need to know about to be successful. You will also notice as you read through all of these documents and kb articles that there is a lack of consistency.
How to use CA certificate to replace VMware certificate on ESX(i) 4 and vCenter
Replacing vCenter Server 4.1 Certificates
Generating Domain Root CA signed certificates for vCenter Server
The Trouble with CA SSL Certificates and vCenter 5
Import an OpenSSL CSR into a Windows CA
Now for the Trouble
You’ll notice in the above list of resources are a couple of VMware KB articles referring to issued with the new VMware HA after you change the SSL certificates for CA signed certificates, or after an upgrade where it was previously done. This is the problem that I ran into both in my lab and also in my customers environment (both are new builds with CA SSL certificates). However following the steps in the KB was not successful.
The reason these errors occur in the first place is that FDM, which is the New VMware HA, enforces SSL certificate verification for communication whenever a host is configured for VMware HA. So you need to make sure that you also have vCenter set to verify host certificates (vSphere Client, Menu Bar – Administration > vCenter Server Settings > SSL Settings > tick vCenter requires verified host SSL certificates, which is the default setting), otherwise you won’t be able to use HA. This is fantastic for security, but unfortunately there is a bug (is expected to be fixed in vSphere 5.0 U1) that means the new thumbprints on hosts that have had their SSL certificates changed don’t end up in the vCenter database (not good). As a result when you try and configure VMware HA after an upgrade where the certs have been changed, or straight after you’ve changed them in a new environment, VMware HA configuration will fail.
You might see an error message such as this:
And also see something like this, an HA election that never ends:
You might also see this in your fdm.log in /var/log on your ESXi Host (related to the above picture):
Feb 3 23:38:37 vmserver12 Fdm: [7D620B90 info ‘Cluster’ opID=SWI-6cc6b9b8] Change state to Startup:0
Feb 3 23:38:38 vmserver12 Fdm: [7D620B90 info ‘Cluster’ opID=SWI-6cc6b9b8] Change state to SlaveConnecting:146874673025
Feb 3 23:38:38 vmserver12 Fdm: [7D620B90 info ‘Election’ opID=SWI-6cc6b9b8] Slave to host @ 192.168.3.222
Feb 3 23:38:42 vmserver12 Fdm: [7D59EB90 info ‘Cluster’ opID=SWI-f5f44234] [ClusterManagerImpl::MainLoop] curState 4 lastState 3
Feb 3 23:38:44 vmserver12 Fdm: [7D620B90 info ‘Cluster’ opID=SWI-6cc6b9b8] Change state to Slave:146874673025
Feb 3 23:38:56 vmserver12 Fdm: [7D7E7B90 info ‘Election’] MasterShutdown
Feb 3 23:38:59 vmserver12 Fdm: [7D6E3B90 info ‘Message’] Destroying connection
Feb 3 23:39:00 vmserver12 Fdm: [7D620B90 info ‘Cluster’ opID=SWI-6cc6b9b8] Change state to SlaveConnecting:146874673025
Feb 3 23:39:04 vmserver12 Fdm: [7D59EB90 info ‘Cluster’ opID=SWI-f5f44234] [ClusterManagerImpl::MainLoop] curState 3 lastState 1
Feb 3 23:39:04 vmserver12 Fdm: [7D59EB90 info ‘Cluster’ opID=SWI-f5f44234] [ClusterManagerImpl::MainLoop] curState 4 lastState 3
Feb 3 23:39:24 vmserver12 Fdm: [7D5DFB90 warning ‘Libs’ opID=SWI-f67a1d5c] SSL_VerifyX509: Certificate verification is disabled, so connection will proceed despite the error
Feb 3 23:39:24 vmserver12 Fdm: [7D5DFB90 warning ‘Libs’ opID=SWI-f67a1d5c] SSL_VerifyX509: Certificate verification is disabled, so connection will proceed despite the error
Feb 3 23:39:44 vmserver12 Fdm: [7D620B90 info ‘Election’ opID=SWI-6cc6b9b8] Slave to host @ 192.168.3.222
Feb 3 23:39:46 vmserver12 Fdm: [7D6E3B90 warning ‘Libs’ opID=SWI-a257b9a0] SSL_VerifyX509: Certificate verification is disabled, so connection will proceed despite the error
Feb 3 23:39:58 vmserver12 Fdm: [7D620B90 info ‘Election’ opID=SWI-6cc6b9b8] Slave to host @ 192.168.3.222
Feb 3 23:40:13 vmserver12 Fdm: [7D620B90 info ‘Cluster’ opID=SWI-6cc6b9b8] Change state to Startup:0
Feb 3 23:40:21 vmserver12 Fdm: [7D620B90 info ‘Cluster’ opID=SWI-6cc6b9b8] Change state to Startup:0
Feb 3 23:40:38 vmserver12 Fdm: [7D59EB90 verbose ‘Cluster’ opID=SWI-f5f44234] [ClusterManagerImpl::CheckElectionState] Transitioned from Startup to SlaveConnecting
Feb 3 23:40:40 vmserver12 Fdm: [7D661B90 warning ‘Libs’ opID=SWI-15378378] SSL_VerifyX509: Certificate verification is disabled, so connection will proceed despite the error
Feb 3 23:40:40 vmserver12 Fdm: [7D661B90 warning ‘Libs’ opID=SWI-15378378] SSL_VerifyX509: Certificate verification is disabled, so connection will proceed despite the error
Feb 3 23:40:44 vmserver12 Fdm: [7D620B90 info ‘Election’ opID=SWI-6cc6b9b8] [ClusterElection::ChangeState] SlaveConnecting => Slave : SlaveConnectingStateFunc
Feb 3 23:40:50 vmserver12 Fdm: [7D765B90 warning ‘Libs’ opID=SWI-16099a2a] SSL_VerifyX509: Certificate verification is disabled, so connection will proceed despite the error
Feb 3 23:40:54 vmserver12 Fdm: [7D6A2B90 info ‘Election’] MasterShutdown
Now for the Fix
To resolve this situation you need to add one additional step to the process that is outlined in After upgrading to vSphere 5, you see the HA error: vSphere HA Cannot be configured on this host because its SSL thumbprint has not been verified. The step is this:
After changing the certificates, restarting the management agents on the host, and existing maintenance mode, wait for HA to configure and fail. Once the exit maintenance mode task is completed disconnect and reconnect the host to vCenter.
Now you can use either of the methods mentioned in KB 2006210 to fix the SSL certificate thumbprint problem. My preference is to use the pearl script via the vSphere API as this doesn’t require vCenter to be shut down. Once the fix has been applied as per the KB you will once again need to reconfigure VMware HA on the host. You will now notice that it is functioning correctly. Now for the step by step process I used.
Step by Step ESXi Host SSL Certificate Replacement using Windows and a Microsoft CA
You could execute a similar process to the one I’m about to describe using an OpenSSL or Public CA and using the Unix/Linux version of OpenSSL, however this is how I did it successfully in my lab and with my customer. As mentioned in the vSphere 5 Security Guide VMware uses X.509 v3 SSL certificates (base-64 encoded) for encrypting traffic between various components. If you CA has been set to support only SHA512 hash that is fine, it will work, although the VMware documentation doesn’t mention it. The two key files for an ESXi host are rui.crt and rui.key.
In order to generate the certificates you’ll need to get a copy of OpenSSL x86 v0.98r or higher, and have access to a Microsoft CA (2000 or higher). The certificates will use a standard web server request template. On the system where you will generate the certificate signing request (rui.csr) you will need to ensure you have Microsoft Visual C++ 2008 Redistributable Package (x86) before installing OpenSSL. For the purposes of this process you will use the Microsoft CA Web Pages to submit the certificate request and download the resulting base-64 encoded certificate. You can use the certreq command if you wish also (not covered here). Ensure you have a vSphere Management Appliance v5 (vMA) deployed in your environment, you will use this to execute the HostReconnet.pl script to save you having to shut down vCenter during the process (hopefully won’t be needed when vSphere 5.0 U1 is available). Before applying the certificates to your environment you should ensure that your clients and vCenter server trust your CA, if it’s an AD integrated CA this should be automated, else you may have to pre-trust the Root or Intermediary CA by loading the CA public cert into your clients and vCenter server (not covered in this process).
Prerequsites:
Microsoft CA (2000 or above, with Web Server Template configured to your liking)
Microsoft Visual C++ 2008 Redistributable Package (x86) on the system where you will generate the certificate signing request (CSR)
OpenSSL 0.98r or above on the system you will use to generate the CSR
vSphere Management Assistant v5 (vMA)
FinalHostReconnect.rar, which contains HostReconnect.pl and can be obtained from VMware KB 2006210
Putty or other SSH client
WinSCP or other SFTP / SCP client
vCenter 5.0
ESXi 5.0
Assumes that the ESXi 5.0 hosts are in a cluster with VMware HA enabled.
Process Step by Step:
Please let me know if you have any trouble with the above process, and also if it works for you, your comments and feedback are appreciated. Note: Steps 16, 18 – 23 are not be needed when vSphere 5.0 U1 or later is used.
Example OpenSSL Configuration file (openssl.cfg) without most of the normal comments and white space that is included:
#OpenSSL Configuration Start
HOME = .
RANDFILE = $ENV::HOME/.rnd
oid_section = new_oids
[ new_oids ]
####################################################################
[ ca ]
default_ca = CA_default # The default ca section
####################################################################
[ CA_default ]
dir = ./demoCA # Where everything is kept
certs = $dir/certs # Where the issued certs are kept
crl_dir = $dir/crl # Where the issued crl are kept
database = $dir/index.txt # database index file.
#unique_subject = no # Set to ‘no’ to allow creation of
# several ctificates with same subject.
new_certs_dir = $dir/newcerts # default place for new certs.
certificate = $dir/cacert.pem # The CA certificate
serial = $dir/serial # The current serial number
crlnumber = $dir/crlnumber # the current crl number
# must be commented out to leave a V1 CRL
crl = $dir/crl.pem # The current CRL
private_key = $dir/private/cakey.pem# The private key
RANDFILE = $dir/private/.rand # private random number file
x509_extensions = usr_cert # The extentions to add to the cert
name_opt = ca_default # Subject Name options
cert_opt = ca_default # Certificate field options
default_days = 5475 # how long to certify for (e.g. 15 years)
default_crl_days= 30 # how long before next CRL
default_md = sha512 # which md to use.
preserve = no # keep passed DN ordering
policy = policy_match
# For the CA policy
[ policy_match ]
countryName = match
stateOrProvinceName = match
organizationName = match
organizationalUnitName = optional
commonName = supplied
emailAddress = optional
[ policy_anything ]
countryName = optional
stateOrProvinceName = optional
localityName = optional
organizationName = optional
organizationalUnitName = optional
commonName = supplied
emailAddress = optional
[ req ]
default_bits = 2048
default_keyfile = privkey.pem
distinguished_name = req_distinguished_name
attributes = req_attributes
x509_extensions = v3_ca # The extentions to add to the self signed cert
input_password = testpassword
output_password = testpassword
string_mask = nombstr
[ req_distinguished_name ]
countryName = Country Name (2 letter code)
countryName_default = NZ
countryName_min = 2
countryName_max = 2
stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = Auckland
localityName = Locality Name (eg, city)
localityName_default = Auckland
0.organizationName = Organization Name (eg, company)
0.organizationName_default = IT Solutions 2000 Ltd
organizationalUnitName = Organizational Unit Name (eg, section)
organizationalUnitName_default = IT
commonName = Common Name (e.g. server FQDN or YOUR name)
commonName_max = 64
emailAddress = Email Address
emailAddress_max = 64
emailAddress_default = admin@yourdomain.com
[ req_attributes ]
challengePassword = A challenge password
challengePassword_min = 4
challengePassword_max = 20
unstructuredName = An optional company name
[ usr_cert ]
basicConstraints=CA:FALSE
nsComment = “OpenSSL Generated Certificate”
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
[ v3_ca ]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer:always
basicConstraints = CA:true
[ crl_ext ]
authorityKeyIdentifier=keyid:always,issuer:always
[ proxy_cert_ext ]
basicConstraints=CA:FALSE
nsComment = “OpenSSL Generated Certificate”
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer:always
proxyCertInfo=critical,language:id-ppl-anyLanguage,pathlen:3,policy:foo
#OpenSSL Configuration End
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2012 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.