VMware KB Articles that have been or will be updated

In addition to the KB’s below a new general KB article with regard to changing SSL certificates in vSphere 5 will be published. This KB will bring together the relevant steps and will hopefully cover the full VMware Cloud Infrastructure Management (CIM) suite. As I become aware of new or updated articles I will include them here. So check back regularly to monitor progress.

VMware KB 2009857 – Certificate warning is reported even after replacing vCenter Server 5.0 default SSL certificates with custom SSL certificates – Updated based on my work

VMware KB 1023011 – Replacing SSL certificates for VMware vCenter Update Manager by using the Update Manager Utility

VMware KB 2007824 – After upgrading to vCenter Server 5.0, the vCenter Service Stats and Hardware Status tab cannot be accessed

Long White Virtual Clouds Articles on CA SSL Certificates

This list below contains links to all of the relevant articles I have posted regarding changing SSL certificates in vSphere 5 and related products. Each link will open in a new window.

Virtual Infrastructure Navigator breaks when vCenter SSL Cert Changed

vCenter Server Virtual Appliance – Changing SSL Certs Made Easy

vSphere Web Client SSL Cert not updated after vCenter SSL Cert Changed

The Trouble with CA SSL Certificates and vCenter 5

The Trouble with CA SSL Certificates and ESXi 5

If you have trouble following any of the above articles or you have a request with regard to changing SSL certificates in another VMware product please get in touch via the feedback form on the Author Page. As always your feedback and comments are greatly appreciated.

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +.

If your company is considering virtualizing your most business critical applications (Oracle, PeopleSoft, Siebel, SAP, Exchange, MS SQL, Sybase, Websphere, Weblogic etc) and you need proven expertise to help you through the journey please get in touch via the feedback form on the Author Page.

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +.

Previous testing I had conducted in customer environments with 10G switches and NIC’s had shown anywhere from 10 – 30% improvement in raw throughput as well as lower latency and improved CPU efficiency. A lot of the performance characteristics are OS dependent, and in the case of Linux will depend on how you’ve tuned your kernel. Both switching equipment and NIC’s have improved a lot over the last couple of years, so it’s possible the differences I found in performance between MTU 9000 and MTU 1500 reflect that as well.

For the testing in my lab I wanted to do as close to valid testing as possible without changing more than necessary, as I have quite a lot of stuff that is deployed and relying on Jumbo Frames (like my storage). My entire underlying network infrastructure in my lab, including my routing switches are all configured for Jumbo Frames. The vSwitches used for the VM’s and VMK ports are also configured for Jumbo Frames, and were not modified during the tests. So my testing was limited to changing the MTU settings for the VMK Ports for vMotion and the NIC MTU settings for the Guest OS’s I tested.

Lab Test Hardware:

Host Type: 2 x Dell T710, 72GB RAM, Dual Intel Xeon X5650, Intel X520-T2 Dual Port 10G NIC
vSphere Version: vCenter 5.0 GA, ESXi 5 – Build 515841
Network Switch: Dell PowerConnect 8024 – 24 Port, 10G-BaseT
Test VM’s:

1. Windows 2008 R2 Enterprise 64bit – 32GB RAM, 3 vCPU, VMXNET3
2. SLES Linux 11 SP1 64bit – 16GB RAM, 3 vCPU, VMXNET3
3. Windows 2003 Standard 32bit – 4GB, 2 vCPU, VMXNET3

Additional details regarding My Lab Environment.

Lab Test Script:

For the vMotion Tests I used the Windows 2008 R2 systems while they were running a Prime95 x64 Torture Test. This ensures that as many memory pages as possible are changing as fast as possible. This places a lot of stress on vMotion, which will extend the migration times and should fully utilize the 10G NIC’s. The Hosts start with a single VMKernel NIC port configured for vMotion, and configured for Jumbo Frames. A second VMKernel port is configured on a separate port group ready when needed for the testing. I conducted multiple tests and used the average from the best test as the results.

vMotion Tests

1. Start RESXTOP from vMA against both hosts in batch mode and record the output from both Test vSphere Hosts.
2. Power on two Test VM’s on Server 1, start torture test on both VM’s
3. Migrate by vMotion both VM’s to destination host Test Host 2 at the same time
4. Migrate by vMotion both VM’s back to source host Test Host 1
5. Repease step 3 and 4 again
6. Enable the second VMKernel vMotion port on each of the test hosts
7. Repeat steps 3 – 5
8. Modify VMKernel Port MTU to 1500 on both VMKernel ports on both test hosts
9. Repeat steps 3 – 5
10. Disable the second VMKernel vMotion port on each of the test hosts
11. Repeat steps 3 – 5
12. Reset hosts to original configuration

For the Guest OS Network Performance Tests I used iPerf, which is an open source network performance test tool. Due to Windows 2003 not supporting receive side scaling I used 10 parallel streams to get the performance results, with both SLES and Windows 2008 R2 I used a single stream.

Guest OS Network Performance Tests

1. Power on First Test VM on Test Host 1
2. Power on Second Test VM on Test Host 2
3. Configure each VM to MTU 1500
4. Start iPerf in Server Mode on Test VM on Test Host 2
5. Start iPerf on Test VM on Host 1 to commence the test
6. Record the results
7. Configure each VM to MTU 9000
8. Repeat steps 4 – 6

For each of the Guest OS’s being tested execute the steps above. Below are the iPerf commands I executed during my tests.

Receiver Node: iperf -s -i 60 -w 1m -f m
Sender Node: iperf -i 5 -w 1m -f m -c <receiver_node_ip>

The Results:

Before starting this testing process I thought I was going to get a 15 – 20% difference between Jumbo and Non-Jumbo. I based this on previous experience and also that the offload capability of 10G NIC’s, Server CPU’s and 10G switches have all improved over the last couple of years. The difference was a bit less than I expected. But still a decent amount compared to what might be expected on a 1Gb/s network. I was not able to test the Jumbo Frames performance on Windows 2008 R2 due to a bug in ESXi 5 VMware Tools and VMXNET3 that prevents Jumbo Frames from functioning, see my previous post Windows VMXNET3 Performance Issues and Instability with vSphere 5.0.

The SLES 11 SP1 VM has had quite a lot of tuning from the out of the box configuration. The tuning probably resulted in the performance of that VM that is roughly the same as the vMotion throughput. If you have not tuned your Linux kernel I wouldn’t expect you’d get the same performance. The Windows 2003 and 2008 R2 were both out of the box configurations with only the VMXNET3 driver MTU modified on the 2003 system.

As you can see from the test results the Linux VM and VMKernel port used for vMotion can saturate a 10G link when using Jumbo Frames. The difference between Jumbo and Non Jumbo on Linux is probably higher than with vMotion due to vMotion VMKernel port being highly tuned for one purpose. The Non Jumbo performance of Windows 2008 R2 was quite close to the Linux Non-Jumbo performance, which shows the improvements that Microsoft has made to their IP stack since Windows 2003.

The Bottom Line:

Using Jumbo Frames requires that all devices from end to end in the network path between source and destination are configured correctly to support MTU 9000, i.e. switches, routers, vSwitches and Servers/VM’s. If Jumbo Frames is not enabled throughout the network path from source to destination you will get packet fragmentation, which will reduce performance back to that of Non-Jumbo. In an existing network if Jumbo Frames was not enabled when it was constructed it could involve considerable effort to change it. However you don’t necessarily have to change it everywhere or on everything, depending upon how your network is segmented. You might consider just enabling it on the segments of the network and servers/switches that could benefit the most from using Jumbo Frames.When implementing new 10G infrastructure it may be worthwhile configuring all new network infrastructure for Jumbo Frames, which is very simple during initial configuration. Even though modern NIC’s and switching equipment have reduced the difference between Jumbo and Non-Jumbo it can still be worthwhile in a number of cases.

My results suggest you could get anywhere from 10% to 13% for normal Guest OS traffic flows and between 8% and 19% for vMotion traffic flows. You will need to decide if the additional throughput, lower CPU usage on servers and network switches/routers and less latency is worth the effort. Two traffic flows that can benefit a lot from the implementation of Jumbo Frames are vMotion and also the Oracle RAC Private Interconnect Network. These types of traffic are normally isolated onto separate switches or non-routed VLANs and could be a prime candidate to implement Jumbo Frames in isolation from the rest of the network. In my vMotion 2 NIC tests the 19% improvement in throughput reduced the migration times by 10 seconds (from 50 seconds down to 40 seconds) for my 2 x 32GB RAM VM’s.

For Oracle RAC in particular Jumbo Frames is recommended even on 1Gb/s networks as a single DB block can then fit into a single IP packet, which reduces DB latencies across the private interconnect. With the latest version of Oracle RAC 11G R2 up to 4 private interconnect networks can be used to provide load balancing and high availability. For databases that make heavy use of the interconnect this can provide a big performance boost without having to completely re-architect the database.

A 10% performance degradation might not sound like much, but when you’re talking about a 10Gb/s network that’s like losing the performance of an entire 1Gb/s link. When you use multiple links it quickly adds up to be a substantial loss of performance. The benefit of Jumbo Frames is only going to grow with the new 40G and 100G Ethernet standards.  Let’s just hope that the OS IP stacks are improved enough to cope with the new standards when they start to become mainstream.

I would encourage you to test it out and implement it where appropriate. Not every application or use case needs Jumbo Frames, but there are a couple of good ones that do.

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +.

After raising this with VMware I have been informed they are working on a supported solution and they will develop a KB article that explains how to change the vCenter Heartbeat SSL Certs for CA signed SSL certs. In the meantime if you require changing the vCenter Heartbeat self-signed SSL certs in your environment you will need to log a support request with VMware and they will help you through the process. As soon as the VMware KB article goes live I will update this post with a link through to it.

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +.

Simply put, if you install VIN prior to changing your SSL certs, it will cease to function and be completely broken. It will require that you delete the VIN instance and redeploy from scratch. Fortunately the rediscovery afterwards will fairly quickly get the inventory service mapping and dependencies back. But this is not great from an end user experience perspective. There is also no documented way to change the default self signed SSL cert on VIN itself. Given that VIN is a great tool for a secure environment to identify what services are where and connected to what I’m hoping it will work better in the future when SSL certificates are updated and that there is an easy process provided to update the VIN SSL cert.

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +.

I tried to find information on changing the SSL certificates for the vCenter Server Virtual Appliance (VCVA) via Google and also on the VMware web site. The only information I was able to find that was in the correct context was the following:

vCenter Server Appliance: Where Do I Upload SSL Certificate on the VMware Communities Site

vSphere 5 vCenter Server Virtual Appliance Quick-Start Guide

Automating vCenter Server Virtual Appliance Configurations by William Lam

Now before we begin, because the vCenter Server Virtual Appliance is Suse Linux Enterprise Server based you will have to be used to a Linux command line, using scp, and generally navigating around in order to successfully change your certificates. All operations will be done as root. The default password is vmware. Like in my previous articles regarding changing SSL Certificates I have included an example OpenSSL configuration file that you can use to generate your certificates.

The following directories on the VCVA contain SSL certificates in one form or another:

/opt/vmware/etc/lighttpd/

/etc/vmware-vpx/ssl

/usr/lib/vmware-vpx/inventoryservice/ssl

/usr/lib/vmware-vsphere-client/server/config

I will go through what needs to go where after I’ve given you what you need to create the certificates.

Step-by-Step Process for Changing SSL Certificates on VCVA

You could execute a similar process to the one I’m about to describe using an OpenSSL or Public CA and using the Unix/Linux version of OpenSSL, however this is how I did it successfully in my lab and with my customer. As mentioned in the vSphere 5 Security Guide VMware uses X.509 v3 SSL certificates (base-64 encoded) for encrypting traffic between various components. If you CA has been set to support only SHA512 hash that is fine, it will work. The three key files for an VCVA are rui.crt, rui.key and rui.pfx.

In order to generate the certificates you’ll need to get a copy of OpenSSL x86 v0.98r or higher, and have access to a Microsoft CA (2003 or higher). The certificates will use a clone of a standard web server request template with Subject Alternative Name added, for my lab I modified the default Web Server Certificate Template to accept up to 15 years for certificates. On the system where you will generate the certificate signing request rui.csr) you will need to ensure you have Microsoft Visual C++ 2008 Redistributable Package (x86) before installing OpenSSL. For the purposes of this process you will use the Microsoft CA Web Pages to submit the certificate request and download the resulting base-64 encoded certificate. You can use the certreq command if you wish also (not covered here). Before applying the certificates to your environment you should ensure that your clients and vCenter server trust your CA, if it’s an AD integrated CA this should be automated (if using Internet Explorer), else you may have to pre-trust the Root or Intermediary CA  by loading the CA public cert into your clients and vCenter server (not covered in this process).

Prerequsites:

Microsoft CA (2003 or above, with Web Server Template with Subject Alternative Name included and configured to your liking).
Microsoft Visual C++ 2008 Redistributable Package (x86) on the system where you will generate the certificate signing request (CSR).
OpenSSL 0.98r or above on the system you will use to generate the CSR vCenter 5.0.
A VCVA Deployed and configured the way you like it.
A backup or snapshot of your VCVA prior to beginning this process would be recommended.

Process Step by Step:

1. Before you start this process you should log into vCenter Server and check that all the services linked with Web Services are working, such as Hardware Status Tab, vCenter Service Status, and also Profile Driven Storage. These are the areas that are very likely to get broken if the process is not followed correctly.
2. After having installed Microsoft Visual C++ 2008 Redistributable Package (x86) and Open SSL 0.98r or later on a management system (vCenter or other system, not the CA) open a command prompt (As Administrator if on Windows 2008) and change to the OpenSSL\bin folder. Use the same command prompt opened As Administrator for all the OpenSSL actions in this list.
3. Edit the openssl.cfg file and ensure it looks similar to the one included at the bottom of this article but with your organization specific information, save the configuration.
4. Execute the following command – openssl req -new -nodes -out rui.csr -keyout rui.key -config openssl.cfg. Note: there will be no prompts as all the information is contained within the configuration file. This is a bit different than when generating the CSR for the ESXi hosts, but that is because there are more of them, and you may not want to have to generate entire config files for each host. For ESXi hosts it is much easier to just hit enter a few times and then specify a common name (fqdn) and then more on. However for vCenter and Update Manager it is better to have everything in the config file, especially as you will likely be specifying multiple Subject Alternative Names (SAN’s – not to be confused with storage area networks).
5. Copy or submit rui.csr to your CA, submit an advanced certificate reqeust, using the Web Server template that you modified, and download the base-64 encoded certificate to the system with OpenSSL that was used to generate the CSR (Screenshots of this available here: How to use CA certificate to replace VMware certificate on ESX(i) 4 and vCenter or here: vSphere 5 Certificates – Replacing the Default vCenter 5 Server Certificate.)
6. Download your root CA certificate and save it as root.cer, copy it to the same location where you generated the CSR.
7. Execute the following command – openssl pkcs12 -export -in rui.crt -inkey rui.key -name “rui” -passout pass:testpassword -out rui.pfx. This will create a rui.pfx file which we will now verify.
8. Execute the following command – openssl pkcs12 -in rui.pfx -info. When prompted enter the password testpassword. You should see an base64 encoded string or characters displayed on the screen and information about the file. Note you will be asked to enter the password twice when it is displaying the private key.
9. Log into the VCVA through the console or via SSH as root and create a directory called /root/certs where you will copy the new certificate files and root CA files.
10. Using WinSCP or another SCP/SFTP client tool copy rui.crt, rui.key, rui.pfx and your root CA cert (root.cer) to /root/certs on the VCVA.
11. On the VCVA console (or SSH login) as root execute cd /root/certs.
12. Execute mv /opt/vmware/etc/lighttpd/server.pem /opt/vmware/etc/lighttpd/server.pem.bak, this will backup the lighthttpd cert.
13. Execute mkdir /etc/vmware-vpx/ssl/backup.
14. Execute mv /etc/vmware-vpx/ssl/rui.* /etc/vmware-vpx/ssl/backup, this will backup the vCenter Server SSL certs.
15. Execute cat rui.key rui.crt >/opt/vmware/etc/lighttpd/server.pem, this will create the new lighthttpd cert.
16. Execute cp rui.* /etc/vmware-vpx/ssl, this will update the vCenter Server SSL certs.
17. Execute mkdir /usr/lib/vmware-vpx/inventoryservice/ssl/backup.
18. Execute mv /usr/lib/vmware-vsphere-client/server/config/keystore /usr/lib/vmware-vsphere-client/server/config/keystore.bak, this will backup the keystore used by the vSphere Web Client.
19. Execute mv /usr/lib/vmware-vpx/inventoryservice/ssl/rui.* /usr/lib/vmware-vpx/inventoryservice/ssl/backup, this will backukp the Inventory Service SSL certs.
20. Execute cp rui.* /usr/lib/vmware-vpx/inventoryservice/ssl, this will update the Inventory Service SSL certs.
21. Execute /usr/lib/vmware-vpx/jre/bin/keytool -keystore /usr/lib/vmware-vsphere-client/server/config/keystore -storetype JCEKS -storepass testpassword -genkey -keyalg rsa -alias s2dmk.
22. When prompted enter the information corresponding to your environment, where the first and last name is the fqdn of the VCVA. An example is as follows:
    What is your first and last name? [Unknown]:vcva.example.com
    What is the name of your organizational unit? [Unknown]:Engineering
    What is the name of your organization? [Unknown]:Example Corporation
    What is the name of your City or Locality? [Unknown]:Palo Alto
    What is the name of your State or Province? [Unknown]:California
    What is the two-letter country code for this unit? [Unknown]:US
    Is CN=vcva.example.com, OU=Engineering, O=”Example Corporation”, L=”Palo Alto”, ST=California, C=US correct?[no]:yes
    Enter key password for <http> (RETURN if same as keystore password):
23. Execute /usr/lib/vmware-vpx/jre/bin/keytool -keystore /usr/lib/vmware-vsphere-client/server/config/keystore -storetype JCEKS -storepass testpassword-certreq -alias s2dmk – file s2dmk.csr.
24. Execute /usr/lib/vmware-vpx/jre/bin/keytool -storetype JCEKS -storepass testpassword-keystore /usr/lib/vmware-vsphere-client/server/config/keystore -import -alias root -file root.cer.
25. Using WinSCP or SCP or another SFTP client copy /root/certs/s2dmk.csr to your CA and submit an appropriate certificate request and download a base64 encoded certificate, save it as s2dmk.crt.
26.  Using WinSCP copy s2dmk.crt to /root/certs on your VCVA system.
27. Execute /usr/lib/vmware-vpx/jre/bin/keytool -keystore /usr/lib/vmware-vsphere-client/server/config/keystore -storetype JCEKS -storepass testpassword -import -alias s2dmk -file s2dmk.crt.
28. Execute /usr/lib/vmware-vpx/jre/bin/keytool -keystore /usr/lib/vmware-vsphere-client/server/config/keystore -storetype JCEKS -storepass testpassword -list to verify that the s2dmk and root certificate have been successfully loaded.
29. Reboot the VCVA by executing the command reboot, or using the vSphere Client to restart the VCVA guest OS.
30. Log back into the VCVA as root using the console or thruogh SSH, as we now need to re-register the vCenter Server with the vSphere Web Client to update the SSL certificate thumbprint.
31. Execute the following command to unregister the local vCenter system from the vSphere Web Client: /usr/lib/vmware-vsphere-client/scripts/admin-cmd unregister https://localhost:9443/vsphere-client localhost root <rootpw>.
32. Now we will execute the command to re-register the local vCenter server again with this vSphere Web Client: /usr/lib/vmware-vsphere-client/scripts/admin-cmd register https://localhost:9443/vsphere-client localhost root <root pw>.
33. Type CRTL-D to log out of the VCVA.
34. When the VCVA has finished restarting you can log into it using the vSphere Web Client, or vSphere Client and check that the certificates are correct. Provided your client trusts your root CA you should not be given a warning message.

Based on William Lam’s comments below it is possible to simplify the above process as follows:

Step 15 and 16 can be replaced with:
vpxd_servicecfg service stop
vpxd_servicecfg certificate change rui.crt rui.key

I would recommend that you check out his post on Automating vCenter Server Virtual Appliance Configurations.

Please let me know if you have any trouble with the above process, and also if it works for you, your comments and feedback are appreciated.

Example OpenSSL Configuration file (openssl.cfg) without most of the normal comments and white space that is included:

# Example vSphere Web Client OpenSSL Configuration File HOME   = . RANDFILE  = $ENV::HOME/.rnd oid_section  = new_oids

[ new_oids ]

[ ca ] default_ca = CA_default  # The default ca section

[ CA_default ]

dir  = ./demoCA  # Where everything is kept certs  = $dir/certs  # Where the issued certs are kept crl_dir  = $dir/crl  # Where the issued crl are kept database = $dir/index.txt # database index file. new_certs_dir = $dir/newcerts  # default place for new certs. certificate = $dir/cacert.pem  # The CA certificate serial  = $dir/serial   # The current serial number crlnumber = $dir/crlnumber # the current crl number      # must be commented out to leave a V1 CRL crl  = $dir/crl.pem   # The current CRL private_key = $dir/private/cakey.pem# The private key RANDFILE = $dir/private/.rand # private random number file x509_extensions = usr_cert  # The extentions to add to the cert name_opt  = ca_default  # Subject Name options cert_opt  = ca_default  # Certificate field options default_days = 5475   # how long to certify for default_crl_days= 30   # how long before next CRL default_md = sha512  # which md to use. preserve = no   # keep passed DN ordering policy  = policy_match

[ policy_match ] countryName  = match stateOrProvinceName = match organizationName = match organizationalUnitName = optional commonName  = supplied emailAddress  = optional

[ policy_anything ] countryName  = optional stateOrProvinceName = optional localityName  = optional organizationName = optional organizationalUnitName = optional commonName  = supplied emailAddress  = optional

[ req ] default_bits  = 2048 default_keyfile  = privkey.pem distinguished_name = req_distinguished_name attributes  = req_attributes x509_extensions = v3_ca # The extentions to add to the self signed cert

input_password = testpassword output_password = testpassword encrypt_key = no prompt = no string_mask = nombstr

req_extensions = v3_req  # The extensions to add to a certificate request

[ req_distinguished_name ] countryName   = NZ

stateOrProvinceName  = Auckland

localityName   = Auckland

0.organizationName  = IT Solutions 2000 Ltd

organizationalUnitName  = IT

commonName   = vsphere-web.homedns.org

emailAddress   = admin@homedns.org

[ req_attributes ]

[ usr_cert ]

basicConstraints=CA:FALSE nsComment   = “OpenSSL Generated Certificate” subjectKeyIdentifier=hash authorityKeyIdentifier=keyid,issuer

[ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment extendedKeyUsage = serverAuth, clientAuth subjectAltName = DNS:vsphere-web.homedns.org, DNS:vcva.homedns.org, DNS:vcweb.homedns.org, DNS:vsphere-web, DNS:vcva, DNS:vcweb

[ v3_ca ] subjectKeyIdentifier=hash authorityKeyIdentifier=keyid:always,issuer:always basicConstraints = CA:true

[ crl_ext ] authorityKeyIdentifier=keyid:always,issuer:always

[ proxy_cert_ext ] basicConstraints=CA:FALSE nsComment   = “OpenSSL Generated Certificate” subjectKeyIdentifier=hash authorityKeyIdentifier=keyid,issuer:always proxyCertInfo=critical,language:id-ppl-anyLanguage,pathlen:3,policy:foo

# End of vSphere Web Client OpenSSL Configuration File

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +.

The default location for the vSphere Web Client certificates is C:\Program Files\VMware\Infrastructure\vSphere Web Client\DMServer\config\ssl. You will need to restart the vSphere Web Client, or reboot the vCenter Server to load the new certificates into memory. You will then be able to log into the vSphere Web Client to test that it is still functioning.

You may notice that when you change the vCenter SSL Certificate that vSphere Web Client will pop up a warning box the when you attempt to log in. The warning box will say that secure communication can’t be verified. This is due to the thumbprint of the vCenter Server SSL Certificate being different to what vSphere Web Client recognized when it was registered with vCenter. If you click install the certificate and ignore to continue it will not prompt you again on this system. You will need to unregister the vCenter system on the vSphere Web client using the admin-app url, and then re-register it again. To do this you will need to log into the vSphere Web Client system using RDP (Assumes Windows Version), then opening https://localhost:9443/admin-app in a web browser. Once the vCenter System is registered with the new thumbprint the warning dialog box should not be displayed again.

WARNING: Under normal circumstances you should not blindly ignore these types of warning messages and should not automatically just install certs and ignore to continue. You need to institutionalize Standard Operating Procedures that question every time a warning dialog such as this is present and you must verify the authenticity of the certificate. Here is an example of the warning box with the vCenter Server and SHA1 thumbprint obscured.

So you don’t have to jump back to my previous article just to find the default location for the Inventory Service SSL Certs it is C:\Program Files\VMware\Infrastructure\Inventory Service\ssl.

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +.

VMware vCenter CA SSL Certificate Resources

As with my previous article I had to run through a bunch of different resources, some of them the same, some of them different. There are some differences in the way you need to generate certificates between vCenter (and VUM) and the ESXi hosts. The below resources are in no particular order or importance. The all contained important information, which I have distilled into a successful process for this article.

vi_vcenter_certificates.pdf

vSphere 5 Security Guide

Replacing vCenter Server 4.1 Certificates

Generating Domain Root CA signed certificates for vCenter Server

VMware KB 1023011 – Replacing SSL certificates for VMware vCenter Update Manager by using the Update Manager Utility

VMware KB 2007824 – After upgrading to vCenter Server 5.0, the vCenter Service Stats and Hardware Status tab cannot be accessed

VMware KB 2009857 – Certificate warning is reported even after replacing vCenter Server 5.0 default SSL certificates with custom SSL certificates

Creating a Certificate with Multiple Hostnames

vSphere 5 Certificates – Replacing the Default vCenter 5 Server Certificate

vSphere 5 Certificates – Replacing the Default Update Manager Server Certificate

Import an OpenSSL CSR into a Windows CA

Replace SSL Certificates: Replace vCenter SSL Certificates

Replacing vCenter 4.1 SSL Certificate with Active Directory Issued One

Replacing vCenter SSL Certificate with Certificate Issued by Microsoft Certificate Authority

Special thanks to the author of WoodITWork.com, Julian Wood for the excellent articles that he posted that were a great help in putting this together.

Now for the Trouble

With a bit of trial and error you could easily enough replace the vCenter Server certificate with a CA signed certificate with a similar process that I showed you for ESXi. However this won’t do you any good as all of your web services will fail. There is a high likelihood that even if you followed the information in VMware KB 2007824 for vCenter 5 that you would be in no better position. The reason for this is that you would likely not have the correct options in either your CA certificate template, or in your OpenSSL configuration file. There are some slight changes in both of these places that will most likely trip you up. It’s not until you read vSphere 5 Certificates – Replacing the Default vCenter 5 Server Certificate and vSphere 5 Certificates – Replacing the Default Update Manager Server Certificate and then VMware KB 1023011 that you might pick up on the necessary fields that you need to concern yourself with. If you don’t take good care you will find yourself without functioning web services, which is were a lot of the vCenter goodies are. Updating Update Manager is actually on the whole a lot easier than vCenter, provided you have the correct options in your CA Template and also your OpenSSL configuration file. As in my previous article I have invested considerable effort to bring you a solution, that I hope will save you a lot of time and frustration. Also note that just updating the SSL Certificate in the vCenter SSL folder is not good enough. You will also need to replace the SSL Certificate files in the Inventory Service SSL directory and also the SSL directory of any other installed components, such as vSphere Web Client.

Now for the Fix

To ensure that the certificates in vCenter Server actually work and function correctly with the Web Services you actually need to add some additional fields into your CA template and your Certificate Signing Request via the OpenSSL configuration file. The information you need is actually contained in VMware KB 1023011. But honestly this isn’t the first place you’re going to look when trying to change out vCenter Server Certificates as it relates to Update Manager. Why this critical information is not present in the security guide I’m not sure. To save you having to read it all the key bits of configuration for OpenSSL are as follows:

Section [ req ]

encrypt_key = no

Section [ x509 ] in the KB and [ v3_req ] in my example configuration

keyUsage = digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth,clientAuth

I found that the default VMware generated certificate had an additional keyUsage parameter, so in my example I have added nonRepudiation and  dataEncipherment, so my config line is as follows:

keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment

You must also include a Subject Alternative Name, which should be the FQDN of the vCenter Server. You can optionally include multiple DNS names in the Subject Alternative Name section, as I have done in my example, to include also the short name of the server. This will ensure that no certificate warning box will be presented if a truly valid server name is used to access it. If you have ever configured certificates for SRM you will know that it uses Subject Alternative Name, but in the SRM case it is normally set to “SRM” and must be the same on both sides. This is quite a different use compared to vCenter.

Because we are specifying a Subject Alternative Name for the vCenter Server you must ensure that your CA certificate template includes this option. This options may not be available in all Microsoft CA versions (I have tested 2003 and 2008 successfully).

Although the KB is a good source of information it is still quite easy to get confused by the way it is written. It also doesn’t have all the details, but it’s good that it has more details than many of the other sources of information. For example if you tried issuing the command that includes the -x509 statement you would not have good results if you are sending the request to a CA, as it needs to be base64 encoded.

There is one piece of very important information in the KB that is quite easy to gloss over and it is this:

openssl pkcs12 -in ./mycert.p12 –info #To see the information in pfx file

In my example I actually just ran the command directly against rui.pfx instead of having an additional step. But this step is critical to test the pfx file and ensure it’s integrity and that it can be deciphered using the correct password. If this part of the process fails you know your pfx file is no good and you should not proceed to replace the certificates on the vCenter or Update Manager Servers.

The key pieces of information you need from KB 2007824 are steps 10 through 12, which I have included in my step by step process below.

Step by Step vCenter Server SSL Certificate Replacement using Windows and a Microsoft CA

You will notice that I have repeated some of the steps below from my previous article on ESXi SSL Certificate replacement. This is intentional so that you have all the necessary steps in one place and don’t have to switch back and forth between multiple articles (like I had to do).

You could execute a similar process to the one I’m about to describe using an OpenSSL or Public CA and using the Unix/Linux version of OpenSSL, however this is how I did it successfully in my lab and with my customer. As mentioned in the vSphere 5 Security Guide VMware uses X.509 v3 SSL certificates (base-64 encoded) for encrypting traffic between various components. If you CA has been set to support only SHA512 hash that is fine, it will work, although the VMware documentation doesn’t mention it. The three key files for an vCenter Server are rui.crt, rui.key and rui.pfx.

In order to generate the certificates you’ll need to get a copy of OpenSSL x86 v0.98r or higher, and have access to a Microsoft CA (2003 or higher). The certificates will use a clone of a standard web server request template with Subject Alternative Name added, for my lab I modified the default Web Server Certificate Template to accept up to 15 years for certificates. On the system where you will generate the certificate signing request rui.csr) you will need to ensure you have Microsoft Visual C++ 2008 Redistributable Package (x86) before installing OpenSSL. For the purposes of this process you will use the Microsoft CA Web Pages to submit the certificate request and download the resulting base-64 encoded certificate. You can use the certreq command if you wish also (not covered here). Before applying the certificates to your environment you should ensure that your clients and vCenter server trust your CA, if it’s an AD integrated CA this should be automated, else you may have to pre-trust the Root or Intermediary CA  by loading the CA public cert into your clients and vCenter server (not covered in this process).

Don’t forget to test access to all the management tools you use in your environment once the vCenter Certificate is updated. You will likely need to update their connections to vCenter as they will still hold the old SSL thumbprint.

Prerequsites:

Microsoft CA (2003 or above, with Web Server Template with Subject Alternative Name included and configured to your liking)
Microsoft Visual C++ 2008 Redistributable Package (x86) on the system where you will generate the certificate signing request (CSR)
OpenSSL 0.98r or above on the system you will use to generate the CSR
vCenter 5.0

Process Step by Step:

1. Before you start this process you should log into vCenter Server and check that all the services linked with Web Services are working, such as Hardware Status Tab, vCenter Service Status, and also Profile Driven Storage. These are the areas that are very likely to get broken if the process is not followed correctly.
2. After having installed Microsoft Visual C++ 2008 Redistributable Package (x86) and Open SSL 0.98r or later on a management system (vCenter or other system, not the CA) open a command prompt (As Administrator if on Windows 2008) and change to the OpenSSL\bin folder. Use the same command prompt opened As Administrator for all the OpenSSL actions in this list.
3. Edit the openssl.cfg file and ensure it looks similar to the one included at the bottom of this article but with your organization specific information, save the configuration.
4. Execute the following command – openssl req -new -nodes -out rui.csr -keyout rui.key -config openssl.cfg. Note: there will be no prompts as all the information is contained within the configuration file. This is a bit different than when generating the CSR for the ESXi hosts, but that is because there are more of them, and you may not want to have to generate entire config files for each host. For ESXi hosts it is much easier to just hit enter a few times and then specify a common name (fqdn) and then more on. However for vCenter and Update Manager it is better to have everything in the config file, especially as you will likely be specifying multiple Subject Alternative Names (SAN’s – not to be confused with storage area networks).
5. Copy or submit rui.csr to your CA, submit an advanced certificate reqeust, using the Web Server template that you modified, and download the base-64 encoded certificate to the system with OpenSSL that was used to generate the CSR (Screenshots of this available here: How to use CA certificate to replace VMware certificate on ESX(i) 4 and vCenter or here: vSphere 5 Certificates – Replacing the Default vCenter 5 Server Certificate.)
6. Execute the following command – openssl pkcs12 -export -in rui.crt -inkey rui.key -name “rui” -passout pass:testpassword -out rui.pfx. This will create a rui.pfx file which we will now verify.
7. Execute the following command – openssl pkcs12 -in rui.pfx -info. When prompted enter the password testpassword. You should see an base64 encoded string or characters displayed on the screen and information about the file. Note you will be asked to enter the password twice when it is displaying the private key.
8. Create a folder on the system used to generate the CSR to back up the existing VMware default certificates that are on the vCenter Server, a separate directory will be needed for vCenter Server, Inventory Service, and any other services (such as vSphere Web Client) running on the vCenter Server.
9. Copy the existing rui.crt, rui.key and rui.pfx files from the vCenter and Inventory Service SSL folders to the backup folders you just created. You should also back these files, and the new certificate files up to a safe location. The location by default for vCenter this is C:\ProgramData\VMware\VMware VirtualCenter\SSL and for the Inventory Service SSL certificate is C:\Program Files\VMware\Infrastructure\Inventory Service\ssl.
10. Copy the new rui.crt, rui.key, and rui.pfx files that were generated to the target host in the vCenter Server and Inventory Service SSL locations.
11. You will use the vCenter Server Managed Object Browser to load the new SSL Certificates into memory. Ensure that you have not yet disabled the MoB as part of your hardening before you have successfully changed the certificates. To access the MoB browse to the following location from the vCenter Server – https://localhost/mob/?moid=vpxd-securitymanager&vmodl=1, when prompted enter a vCenter Administrator username and password.
12. The Managed Object Type for the vpx.SecurityManager will load, Click on reloadSslCertificate
13. Click on Invoke Method
14. The new SSL certificates will be loaded into memory and you will see the following if successful – Method Invocation Result: void.
15. If you do not already have an RDP or Console session open on your vCenter Server please open one now. Log in as an administrator and open a command prompt As Administrator.
16. Change directory to C:\Program Files\VMware\Infrastructure\VirtualCenter Server\
17. Execute the following command and when prompted enter the database password - vpxd -p. This will reset the db password after the new certificate, which will allow all the web services to access it. You should enter the existing password, not a new password at this point.
18. Stop and then restart the VMware VirtualCenter Server, which will in turn restart vCenter Management Web Services, Inventory, and Profile Driven Storage Services. You may have to reconnect all your hosts if you are doing vCenter SSL certs before the host certs. In my environment I did the hosts first and did not have to reconnect them when changing the vCenter SSL certs as a result.
19. After the initial restart you may notice that after 5 minutes or so the Profile Driven Storage Service has stopped (I did). At that point you should restart it again and it should remain running.
20. Log into vCenter Server and verify that all Host Status Tab’s are working, the vCenter Service Status is functioning and all services are running correctly, and that Profile Driven Storage configuration is accessible and working.

Now that you’ve updated the vCenter Server and also the Inventory Service Certificates you may need to also update your vSphere Web Client Certificate if it is also on your vCenter Server. The location for the vSphere Web Client SSL Certificate is C:\Program Files\VMware\Infrastructure\vSphere Web Client\DMServer\config\ssl by default. Once you have updated the vSphere Web Client SSL Certificate and restarted the services you will then need to browse to the vSphere Web Client Admin App and re-register vCenter Server to ensure that it is registered with the correct thumbprint. Browse to https://localhost:9443/admin-app.

Up to step 10 the process is almost exactly the same for Update Manager. For Update Manager however you need to copy the new certificate files into <Update Manager Installation Directory>\SSL, after taking a backup of course. The default locations are as follows:

• The default path in 64-bit Windows is C:\Program Files (x86)\VMware\Infrastructure\Update Manager\SSL
• The default path in 32-bit Windows is C:\Program Files\VMware\Infrastructure\Update Manager\SSL

Once the new certificate files are copied into the correct location you need to do the following on the Update Manager Server:

1. If you do not already have an RDP or console session open on the Update Manager Server do that now and log in as an administrator.
2. Stop the Update Manager Services
3. Change directory to the Update Manager Installation Directory, by default as per above minus the SSL part.
4. Double click on VMwareUpdateManagerUtility.exe to execute the file.
5. Log in by using the administrator credentials and the IP address or host name of vCenter Server system.
6. In the Options pane of the Update Manager Utility, click SSL Certificate.
7. In the Configurations pane, select Followed and verified the steps and click Apply.
8. Once the operation completes start the VMware vCenter Update Manager service.

Please let me know if you have any trouble with the above process, and also if it works for you, your comments and feedback are appreciated.

Example OpenSSL Configuration file (openssl.cfg) without most of the normal comments and white space that is included:

# vCenter OpenSSL example configuration file start.
HOME            = .
RANDFILE        = $ENV::HOME/.rnd
oid_section        = new_oids

[ new_oids ]

[ ca ]
default_ca        = CA_default        # The default ca section

[ CA_default ]

dir                = ./demoCA        # Where everything is kept
certs                = $dir/certs        # Where the issued certs are kept
crl_dir            = $dir/crl        # Where the issued crl are kept
database            = $dir/index.txt    # database index file.
new_certs_dir        = $dir/newcerts        # default place for new certs.
certificate            = $dir/cacert.pem     # The CA certificate
serial            = $dir/serial         # The current serial number
crlnumber            = $dir/crlnumber    # the current crl number  must be commented out to leave a V1 CRL
crl                = $dir/crl.pem         # The current CRL
private_key        = $dir/private/cakey.pem# The private key
RANDFILE        = $dir/private/.rand    # private random number file
x509_extensions    = usr_cert        # The extentions to add to the cert
name_opt         = ca_default        # Subject Name options
cert_opt             = ca_default        # Certificate field options
default_days        = 5475            # how long to certify for
default_crl_days    = 30            # how long before next CRL
default_md        = sha512        # which md to use.
preserve            = no            # keep passed DN ordering
policy            = policy_match

[ policy_match ]
countryName            = match
stateOrProvinceName    = match
organizationName        = match
organizationalUnitName    = optional
commonName            = supplied
emailAddress            = optional

[ policy_anything ]
countryName            = optional
stateOrProvinceName    = optional
localityName            = optional
organizationName        = optional
organizationalUnitName    = optional
commonName            = supplied
emailAddress            = optional

[ req ]
default_bits            = 2048
default_keyfile             = privkey.pem
distinguished_name        = req_distinguished_name
attributes                = req_attributes
x509_extensions        = v3_ca    # The extentions to add to the self signed cert
input_password         = testpassword
output_password         = testpassword
encrypt_key             = no
prompt                = no
string_mask             = nombstr
req_extensions         = v3_req     # The extensions to add to a certificate request

[ req_distinguished_name ] # change these settings for your environment
countryName                = NZ
stateOrProvinceName        = Auckland
localityName                = Auckland
0.organizationName            = IT Solutions 2000 Ltd
organizationalUnitName        = IT
commonName                = vc.homedns.org
emailAddress                = admin@yourdomain.com

[ req_attributes ]

[ usr_cert ]

basicConstraints        =CA:FALSE
nsComment            = “OpenSSL Generated Certificate”
subjectKeyIdentifier        =hash
authorityKeyIdentifier    =keyid,issuer

[ v3_req ]
basicConstraints         = CA:FALSE
keyUsage                 = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage         = serverAuth, clientAuth
subjectAltName         = DNS:vc.homedns.org, DNS:vc41.homedns.org, DNS:vc41 #examples only

[ v3_ca ]
subjectKeyIdentifier        =hash
authorityKeyIdentifier    =keyid:always,issuer:always
basicConstraints         = CA:true

[ crl_ext ]
authorityKeyIdentifier    =keyid:always,issuer:always

[ proxy_cert_ext ]
basicConstraints        =CA:FALSE
nsComment            = “OpenSSL Generated Certificate”
subjectKeyIdentifier        =hash
authorityKeyIdentifier    =keyid,issuer:always
proxyCertInfo            =critical,language:id-ppl-anyLanguage,pathlen:3,policy:foo

# vCenter OpenSSL example configuration file end.

# vCenter Update Manager OpenSSL example configuration file Start.

HOME            = .
RANDFILE        = $ENV::HOME/.rnd
oid_section        = new_oids

[ new_oids ]

[ ca ]
default_ca        = CA_default        # The default ca section

[ CA_default ]

dir                = ./demoCA        # Where everything is kept
certs                = $dir/certs        # Where the issued certs are kept
crl_dir            = $dir/crl        # Where the issued crl are kept
database            = $dir/index.txt    # database index file.
new_certs_dir        = $dir/newcerts        # default place for new certs.
certificate            = $dir/cacert.pem     # The CA certificate
serial            = $dir/serial         # The current serial number
crlnumber            = $dir/crlnumber    # the current crl number must be commented out to leave a V1 CRL
crl                = $dir/crl.pem         # The current CRL
private_key        = $dir/private/cakey.pem# The private key
RANDFILE        = $dir/private/.rand    # private random number file
x509_extensions    = usr_cert        # The extentions to add to the cert
name_opt         = ca_default        # Subject Name options
cert_opt             = ca_default        # Certificate field options
default_days        = 5475            # how long to certify for
default_crl_days    = 30            # how long before next CRL
default_md        = sha512        # which md to use.
preserve            = no            # keep passed DN ordering
policy            = policy_match

[ policy_match ]
countryName            = match
stateOrProvinceName    = match
organizationName        = match
organizationalUnitName    = optional
commonName            = supplied
emailAddress            = optional

[ policy_anything ]
countryName            = optional
stateOrProvinceName    = optional
localityName            = optional
organizationName        = optional
organizationalUnitName    = optional
commonName            = supplied
emailAddress            = optional

[ req ]
default_bits        = 2048
default_keyfile         = privkey.pem
distinguished_name    = req_distinguished_name
attributes            = req_attributes
x509_extensions    = v3_ca    # The extentions to add to the self signed cert
input_password     = testpassword
output_password     = testpassword
encrypt_key         = no
prompt             = no
string_mask         = nombstr

req_extensions = v3_req     # The extensions to add to a certificate request

[ req_distinguished_name ] # change these settings for your environment
countryName            = NZ
stateOrProvinceName    = Auckland
localityName            = Auckland
0.organizationName        = IT Solutions 2000 Ltd
organizationalUnitName    = IT
commonName            = updmgr.homedns.org
emailAddress            = admin@corp.it-solutions.homedns.org

[ req_attributes ]

[ usr_cert ]
basicConstraints        =CA:FALSE
nsComment            = “OpenSSL Generated Certificate”
subjectKeyIdentifier        =hash
authorityKeyIdentifier    =keyid,issuer

[ v3_req ]
basicConstraints         = CA:FALSE
keyUsage             = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage         = serverAuth, clientAuth
subjectAltName        = DNS:updmgr.homedns.org, DNS:updmgr # examples only

[ v3_ca ]
subjectKeyIdentifier        =hash
authorityKeyIdentifier    =keyid:always,issuer:always
basicConstraints         = CA:true

[ crl_ext ]
authorityKeyIdentifier    =keyid:always,issuer:always

[ proxy_cert_ext ]
basicConstraints        =CA:FALSE
nsComment            = “OpenSSL Generated Certificate”
subjectKeyIdentifier        =hash
authorityKeyIdentifier    =keyid,issuer:always
proxyCertInfo            =critical,language:id-ppl-anyLanguage,pathlen:3,policy:foo

# vCenter Update Manager OpenSSL example configuration file End.

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +.

With vSphere 4.x if you enabled Lockdown Mode through vCenter it couldn’t be disabled through the host, even though an administrator could still log into the DCUI. The only way to have the ability to disable Lockdown Mode through the host was to have enabled it through the host in the first place. Unfortunately enabling Lockdown Mode through the host has the effect of removing any locally defined users and groups.

This original behavior of Lockdown Mode could be a pain for an administrator that had lost access to his/her vCenter, and perhaps it was running on the host that was in Lockdown Mode.  At that point it was almost as good as Total Lockdown Mode (No DCUI Access), and could have resulted in a host rebuild, except for the fact that you could always reboot the host and hope HA recovered the vCenter server (no good if all the servers were down, or HA can’t restart vCenter). This was good for security, but not so good if you lost your vCenter and didn’t have any way of getting it back (I learned this the hard way, but fortunately found a way out of it). When I came across this situation in the real world I was able to mount the vCenter Server storage and register the vCenter into another host, power it on, and then as soon as I had access to it, disable lockdown mode till the incidents were resolved. This was only possible thanks to iSCSI storage, as I had to bring it up at the DR site.

There has been a change to the way Lockdown Mode works recently, and it’s important to understand it, as it’s likely most people will now want to use it. Since the latest updates in vSphere 4.1 and also in vSphere 5, regardless of where Lockdown Mode is enabled, it can be disabled for troubleshooting purposes from DCUI. This is an emergency stop gap measure in case you loose access to vCenter. This would require physical access to the host or access via the hosts remote management card, in addition to knowing the username and password to access the DCUI.This is a great enhancement for the majority of customer environments and will allow a greater level of security without hindering troubleshooting when things go wrong. However there will be some environments where Total Lockdown Mode is now more appropriate. The old rule of if you enable Lockdown Mode through the DCUI you loose the host users and groups that were defined, so it’s always best to enable it in vCenter.

Don’t get caught out if you see the Change Lockdown Mode settings grayed out in the DCUI for a host if it’s not connected to vCenter (disconnecting a host from vCenter has this effect). This is expected and by design. Lockdown Mode is only applicable to hosts that are connected to vCenter.

If you want to allow access to the ESXi hosts directly from a management jump box, then using SSH keys would be the way to go. This access method should only be used for troubleshooting however. For normal day to day administration I would strongly recommend that vCenter and the vMA are used to run the RCLI commands against the hosts. This is the easiest way to administer the hosts will keeping good audit records of activities. Although if the hosts are AD integrated, and the logs are going to syslog you will also get a record of every shell command in the syslogs executed by every user.

So based on this do you think Locked Down Mode is Really Locked Down enough? It would be good to get your input to this discussion.

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +.

General Information on X.509 Certificates

For anyone that doesn’t know what an X.509 certificate is here are a couple of links that will explain it. The first one is a good human readable explanation and the second is the actual specification published by the Internet Engineering Taskforce (IETF).

Wikipedia.org – x.509

IETF RFC 3280 – X.509

Each component in your vSphere Infrastructure uses these X.509 SSL certificates for secure encrypted communications. Each SSL certificate is uniquely generated for each component and ties to the FQDN of the component. So this means every ESXi 5 server has a certificate generated for it based on it’s unique FQDN, as does vCenter, vCenter Update Manager, SRM, vShield Manager and any other components you may be using. This ensures non-repudiation. That is, every system knows that is communicating with the other system that it expects, and it’s not an imposter. This means you can’t just take one cert generated for vCenter for example and apply it to all of your hosts. I have not tested using wildcard certificates (*.domain) with vSphere 5, but in earlier versions some components didn’t support them. From a security standpoint it is much better to have a single SSL cert tied to a single host by FQDN.

VMware CA SSL Certificate Resources

While I was working through generating and applying the certificates in the environment I was working in over the last week I ran through all of these resources below. All have some good information. But none is a complete end to end guide on how to generate and apply the certificates that will work reliably with vSphere 5 (which is the reason I had to review them all). The reason why I went through some of the older material on this is because it is largely still relevant. But there are some subtle changes to the way that vSphere 5 works that you need to know about to be successful. You will also notice as you read through all of these documents and kb articles that there is a lack  of consistency.

vi_vcenter_certificates.pdf

How to use CA certificate to replace VMware certificate on ESX(i) 4 and vCenter

After upgrading to vSphere 5, you see the HA error: vSphere HA Cannot be configured on this host because its SSL thumbprint has not been verified

Configuring HA after upgrading to vCenter Server 5.0 fails with the error: Cannot complete the configuration of the vSphere HA agent on the host. Misconfiguration in the host setup

vSphere 5 Security Guide

Replacing vCenter Server 4.1 Certificates

Generating Domain Root CA signed certificates for vCenter Server

The Trouble with CA SSL Certificates and vCenter 5

Import an OpenSSL CSR into a Windows CA

Now for the Trouble

You’ll notice in the above list of resources are a couple of VMware KB articles referring to issued with the new VMware HA after you change the SSL certificates for CA signed certificates, or after an upgrade where it was previously done. This is the problem that I ran into both in my lab and also in my customers environment (both are new builds with CA SSL certificates). However following the steps in the KB was not successful.

The reason these errors occur in the first place is that FDM, which is the New VMware HA, enforces SSL certificate verification for communication whenever a host is configured for VMware HA. So you need to make sure that you also have vCenter set to verify host certificates (vSphere Client, Menu Bar – Administration > vCenter Server Settings > SSL Settings > tick vCenter requires verified host SSL certificates, which is the default setting), otherwise you won’t be able to use HA. This is fantastic for security, but unfortunately there is a bug (is expected to be fixed in vSphere 5.0 U1) that means the new thumbprints on hosts that have had their SSL certificates changed don’t end up in the vCenter database (not good). As a result when you try and configure VMware HA after an upgrade where the certs have been changed, or straight after you’ve changed them in a new environment, VMware HA configuration will fail.

You might see an error message such as this:

And also see something like this, an HA election that never ends:

You might also see this in your fdm.log in /var/log on your ESXi Host (related to the above picture):

Feb  3 23:38:37 vmserver12 Fdm: [7D620B90 info 'Cluster' opID=SWI-6cc6b9b8] Change state to Startup:0
Feb  3 23:38:38 vmserver12 Fdm: [7D620B90 info 'Cluster' opID=SWI-6cc6b9b8] Change state to SlaveConnecting:146874673025
Feb  3 23:38:38 vmserver12 Fdm: [7D620B90 info 'Election' opID=SWI-6cc6b9b8] Slave to host @ 192.168.3.222
Feb  3 23:38:42 vmserver12 Fdm: [7D59EB90 info 'Cluster' opID=SWI-f5f44234] [ClusterManagerImpl::MainLoop] curState 4 lastState 3
Feb  3 23:38:44 vmserver12 Fdm: [7D620B90 info 'Cluster' opID=SWI-6cc6b9b8] Change state to Slave:146874673025
Feb  3 23:38:56 vmserver12 Fdm: [7D7E7B90 info 'Election'] MasterShutdown
Feb  3 23:38:59 vmserver12 Fdm: [7D6E3B90 info 'Message'] Destroying connection
Feb  3 23:39:00 vmserver12 Fdm: [7D620B90 info 'Cluster' opID=SWI-6cc6b9b8] Change state to SlaveConnecting:146874673025
Feb  3 23:39:04 vmserver12 Fdm: [7D59EB90 info 'Cluster' opID=SWI-f5f44234] [ClusterManagerImpl::MainLoop] curState 3 lastState 1
Feb  3 23:39:04 vmserver12 Fdm: [7D59EB90 info 'Cluster' opID=SWI-f5f44234] [ClusterManagerImpl::MainLoop] curState 4 lastState 3
Feb  3 23:39:24 vmserver12 Fdm: [7D5DFB90 warning 'Libs' opID=SWI-f67a1d5c] SSL_VerifyX509: Certificate verification is disabled, so connection will proceed despite the error
Feb  3 23:39:24 vmserver12 Fdm: [7D5DFB90 warning 'Libs' opID=SWI-f67a1d5c] SSL_VerifyX509: Certificate verification is disabled, so connection will proceed despite the error
Feb  3 23:39:44 vmserver12 Fdm: [7D620B90 info 'Election' opID=SWI-6cc6b9b8] Slave to host @ 192.168.3.222
Feb  3 23:39:46 vmserver12 Fdm: [7D6E3B90 warning 'Libs' opID=SWI-a257b9a0] SSL_VerifyX509: Certificate verification is disabled, so connection will proceed despite the error
Feb  3 23:39:58 vmserver12 Fdm: [7D620B90 info 'Election' opID=SWI-6cc6b9b8] Slave to host @ 192.168.3.222
Feb  3 23:40:13 vmserver12 Fdm: [7D620B90 info 'Cluster' opID=SWI-6cc6b9b8] Change state to Startup:0
Feb  3 23:40:21 vmserver12 Fdm: [7D620B90 info 'Cluster' opID=SWI-6cc6b9b8] Change state to Startup:0
Feb  3 23:40:38 vmserver12 Fdm: [7D59EB90 verbose 'Cluster' opID=SWI-f5f44234] [ClusterManagerImpl::CheckElectionState] Transitioned from Startup to SlaveConnecting
Feb  3 23:40:40 vmserver12 Fdm: [7D661B90 warning 'Libs' opID=SWI-15378378] SSL_VerifyX509: Certificate verification is disabled, so connection will proceed despite the error
Feb  3 23:40:40 vmserver12 Fdm: [7D661B90 warning 'Libs' opID=SWI-15378378] SSL_VerifyX509: Certificate verification is disabled, so connection will proceed despite the error
Feb  3 23:40:44 vmserver12 Fdm: [7D620B90 info 'Election' opID=SWI-6cc6b9b8] [ClusterElection::ChangeState] SlaveConnecting => Slave : SlaveConnectingStateFunc
Feb  3 23:40:50 vmserver12 Fdm: [7D765B90 warning 'Libs' opID=SWI-16099a2a] SSL_VerifyX509: Certificate verification is disabled, so connection will proceed despite the error
Feb  3 23:40:54 vmserver12 Fdm: [7D6A2B90 info 'Election'] MasterShutdown

Now for the Fix

To resolve this situation you need to add one additional step to the process that is outlined in After upgrading to vSphere 5, you see the HA error: vSphere HA Cannot be configured on this host because its SSL thumbprint has not been verified. The step is this:

After changing the certificates, restarting the management agents on the host, and existing maintenance mode, wait for HA to configure and fail. Once the exit maintenance mode task is completed disconnect and reconnect the host to vCenter.

Now you can use either of the methods mentioned in KB 2006210 to fix the SSL certificate thumbprint problem. My preference is to use the pearl script via the vSphere API as this doesn’t require vCenter to be shut down. Once the fix has been applied as per the KB you will once again need to reconfigure VMware HA on the host. You will now notice that it is functioning correctly. Now for the step by step process I used.

Step by Step ESXi Host SSL Certificate Replacement using Windows and a Microsoft CA

You could execute a similar process to the one I’m about to describe using an OpenSSL or Public CA and using the Unix/Linux version of OpenSSL, however this is how I did it successfully in my lab and with my customer. As mentioned in the vSphere 5 Security Guide VMware uses X.509 v3 SSL certificates (base-64 encoded) for encrypting traffic between various components. If you CA has been set to support only SHA512 hash that is fine, it will work, although the VMware documentation doesn’t mention it. The two key files for an ESXi host are rui.crt and rui.key.

In order to generate the certificates you’ll need to get a copy of OpenSSL x86 v0.98r or higher, and have access to a Microsoft CA (2000 or higher). The certificates will use a standard web server request template. On the system where you will generate the certificate signing request (rui.csr) you will need to ensure you have Microsoft Visual C++ 2008 Redistributable Package (x86) before installing OpenSSL. For the purposes of this process you will use the Microsoft CA Web Pages to submit the certificate request and download the resulting base-64 encoded certificate. You can use the certreq command if you wish also (not covered here). Ensure you have a vSphere Management Appliance v5 (vMA) deployed in your environment, you will use this to execute the HostReconnet.pl script to save you having to shut down vCenter during the process (hopefully won’t be needed when vSphere 5.0 U1 is available). Before applying the certificates to your environment you should ensure that your clients and vCenter server trust your CA, if it’s an AD integrated CA this should be automated, else you may have to pre-trust the Root or Intermediary CA  by loading the CA public cert into your clients and vCenter server (not covered in this process).

Prerequsites:

Microsoft CA (2000 or above, with Web Server Template configured to your liking)
Microsoft Visual C++ 2008 Redistributable Package (x86) on the system where you will generate the certificate signing request (CSR)
OpenSSL 0.98r or above on the system you will use to generate the CSR
vSphere Management Assistant v5 (vMA)
FinalHostReconnect.rar, which contains HostReconnect.pl and can be obtained from VMware KB 2006210
Putty or other SSH client
WinSCP or other SFTP / SCP client
vCenter 5.0
ESXi 5.0
Assumes that the ESXi 5.0 hosts are in a cluster with VMware HA enabled.

Process Step by Step:

1. After having installed Microsoft Visual C++ 2008 Redistributable Package (x86) and Open SSL 0.98r or later on a management system (vCenter or other system, not the CA) open a command prompt and change to the OpenSSL\bin folder.
2. Edit the openssl.cfg file and ensure it looks similar to the one included at the bottom of this article but with your organization specific information, save the configuration.
3. Execute the following command – openssl req -new -nodes -out rui.csr -keyout rui.key -config openssl.cfg.
4. If you have specified all the relevant organization information in the OpenSSL configuration you will only have to specify the Common Name, which will be the FQDN of your ESXi host, and enter twice (i.e. blank/no password) when it asks you for a password at the end.
5. Copy or submit rui.csr to your CA, using the Web Server template, and download the base-64 encoded certificate to the system with OpenSSL that was used to generate the CSR (Screenshots of this available here: How to use CA certificate to replace VMware certificate on ESX(i) 4 and vCenter).
6. Create a folder on the system used to generate the CSR to back up the existing VMware default certificates that are on the host.
7. Enable SSH on the target host, ensure lockdown mode is disabled, and then put it into maintenance mode.
8. Using WinSCP or other SFTP/SCP client change directory on the target host to /etc/vmware/ssl and copy the rui.crt and rui.key files off  the host to your backup folder that you created in step 6.
9. Delete rui.crt and rui.key from the target host.
10. Copy the new rui.crt and rui.key files that were generated to the target host in /etc/vmware/ssl, be sure to use Text Mode or ASCII Mode transfer, otherwise you will have problems with special characters (^M) ending up in the certificate file and the process will fail.
11. Open up a console through the remote management card or KVM to the target host and log in as root to the Direct Console User Interface (DCUI – F2 on the console screen).
12. Scroll down the screen till you reach Troubleshooting Options, then press enter.
13. Scroll down to Restart Management Agents, then press enter.
14. Press F11 to restart the management agents (vpxa etc).
15. After the management agents are restarted press escape a couple of times till you log out of DCUI.
16. Ensure that you have copied the HostReconnect.pl script to your vMA v5, you will need it soon.
17. Take the target host out of maintenance mode, and wait for HA to reconfigure and fail (either time out, task completes and HA continues to say election).
18. Disconnect and then reconnect the host (this is currently the missing step from KB 2006210).
19. Once the host is connected and HA agent reconfigured you need to log into your vMA as vi-admin and change directory to where you copied HostReconnect.pl.
20. If this is the first time running HostReconnect.pl execute chmod u+x on HostReconnect.pl to ensure that you can run the command.
21. Execute HostReconnect.pl –server <vcenter server fqdn>, enter username and password of a vCenter administrator when prompted.
22. Monitor the output. You will notice that each host has been reconnected in the vCenter Tasks window. This script reconnects the hosts using their actual thumbprint and updates the expected thumbprint in the vCenter database. Without running this command, or stopping vCenter processes and manually editing the database, the thumbprints will not match and the configuration of HA will fail (as per KB 2006210).
23. Reconfigure HA on the target host, you should notice that the it works successfully and the host is back to normal.
24. Repeat the above steps for subsequent ESXi hosts.

Please let me know if you have any trouble with the above process, and also if it works for you, your comments and feedback are appreciated. Steps 16, 18 – 23 will hopefully not be needed when vSphere 5.0 U1 is available. I will write about it again once I’ve tested it.

Example OpenSSL Configuration file (openssl.cfg) without most of the normal comments and white space that is included:

#OpenSSL Configuration Start

HOME            = .
RANDFILE        = $ENV::HOME/.rnd
oid_section        = new_oids

[ new_oids ]

####################################################################
[ ca ]
default_ca    = CA_default        # The default ca section

####################################################################
[ CA_default ]
dir        = ./demoCA        # Where everything is kept
certs        = $dir/certs        # Where the issued certs are kept
crl_dir        = $dir/crl        # Where the issued crl are kept
database    = $dir/index.txt    # database index file.
#unique_subject    = no            # Set to ‘no’ to allow creation of
# several ctificates with same subject.
new_certs_dir    = $dir/newcerts        # default place for new certs.
certificate    = $dir/cacert.pem     # The CA certificate
serial        = $dir/serial         # The current serial number
crlnumber    = $dir/crlnumber    # the current crl number
# must be commented out to leave a V1 CRL
crl        = $dir/crl.pem         # The current CRL
private_key    = $dir/private/cakey.pem# The private key
RANDFILE    = $dir/private/.rand    # private random number file
x509_extensions    = usr_cert        # The extentions to add to the cert
name_opt     = ca_default        # Subject Name options
cert_opt     = ca_default        # Certificate field options
default_days    = 5475            # how long to certify for (e.g. 15 years)
default_crl_days= 30            # how long before next CRL
default_md    = sha512            # which md to use.
preserve    = no            # keep passed DN ordering
policy        = policy_match

# For the CA policy
[ policy_match ]
countryName        = match
stateOrProvinceName    = match
organizationName    = match
organizationalUnitName    = optional
commonName        = supplied
emailAddress        = optional

[ policy_anything ]
countryName        = optional
stateOrProvinceName    = optional
localityName        = optional
organizationName    = optional
organizationalUnitName    = optional
commonName        = supplied
emailAddress        = optional
[ req ]
default_bits        = 2048
default_keyfile     = privkey.pem
distinguished_name    = req_distinguished_name
attributes        = req_attributes
x509_extensions    = v3_ca    # The extentions to add to the self signed cert
input_password = testpassword
output_password = testpassword
string_mask = nombstr

[ req_distinguished_name ]
countryName            = Country Name (2 letter code)
countryName_default        = NZ
countryName_min            = 2
countryName_max            = 2
stateOrProvinceName        = State or Province Name (full name)
stateOrProvinceName_default    = Auckland
localityName            = Locality Name (eg, city)
localityName_default        = Auckland
0.organizationName        = Organization Name (eg, company)
0.organizationName_default    = IT Solutions 2000 Ltd
organizationalUnitName        = Organizational Unit Name (eg, section)
organizationalUnitName_default    = IT
commonName            = Common Name (e.g. server FQDN or YOUR name)
commonName_max            = 64
emailAddress            = Email Address
emailAddress_max        = 64
emailAddress_default        = admin@yourdomain.com

[ req_attributes ]
challengePassword        = A challenge password
challengePassword_min        = 4
challengePassword_max        = 20
unstructuredName        = An optional company name

[ usr_cert ]
basicConstraints=CA:FALSE
nsComment            = “OpenSSL Generated Certificate”
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage                 = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage         = serverAuth, clientAuth

[ v3_ca ]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer:always
basicConstraints = CA:true

[ crl_ext ]
authorityKeyIdentifier=keyid:always,issuer:always

[ proxy_cert_ext ]
basicConstraints=CA:FALSE
nsComment            = “OpenSSL Generated Certificate”
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer:always
proxyCertInfo=critical,language:id-ppl-anyLanguage,pathlen:3,policy:foo

#OpenSSL Configuration End

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +.