vSphere Web Client – Long White Virtual Cloudsu by

Virtual Infrastructure Navigator breaks when vCenter SSL Cert Changed

Michael Webster — Thu, 16 Feb 2012 01:06:18 +0000

Like a lot of people I was quick to download and implement VMware vCenter Operations Manager 5 Enterprise when it became available. One of the great tools that is included in the suite is Virtual Infrastructure Navigator (VIN), which will discover and map all the dependencies and also DR protection status of VM’s in a linked mode group. However there is a bit of a gotcha if you want to use VIN and you also want to change the SSL Certs in vCenter and/or vSphere Web Client.

Simply put, if you install VIN prior to changing your SSL certs, it will cease to function and be completely broken. It will require that you delete the VIN instance and redeploy from scratch. Fortunately the rediscovery afterwards will fairly quickly get the inventory service mapping and dependencies back. But this is not great from an end user experience perspective. There is also no documented way to change the default self signed SSL cert on VIN itself. Given that VIN is a great tool for a secure environment to identify what services are where and connected to what I’m hoping it will work better in the future when SSL certificates are updated and that there is an easy process provided to update the VIN SSL cert.

If you want a way to fully manage the certificate lifecycle and replace certs automatically then you’ll want to check out vCert Manager – Changing VMware SSL Certs Made Easy. This will completely automate the SSL certificate process in vSphere environments.

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2012 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

vCenter Server Virtual Appliance – Changing SSL Certs Made Easy

Michael Webster — Sun, 12 Feb 2012 14:18:06 +0000

I’ve been updating my vCenter and ESXi certificates recently and I ran into one particular system so far that had absolutely no documentation or KB articles to help with changing default SSL certificates for CA signed ones. The system was my vCenter Server Virtual Appliance. You might remember that I wrote about this as a means of using it as the vSphere Web Client without needing an additional Microsoft Windows License and then I used it with a load balancer to Increase vSphere Web Client Availability and Scalability. But a lack of documentation wasn’t going to stop me. Being a SLES based virtual appliance though meant things were quite different when changing the certs. If you want to save yourself a lot of time changing the SSL Certificates for the vCenter Server Virtual Appliance then read on.

I tried to find information on changing the SSL certificates for the vCenter Server Virtual Appliance (VCVA) via Google and also on the VMware web site. The only information I was able to find that was in the correct context was the following:

vCenter Server Appliance: Where Do I Upload SSL Certificate on the VMware Communities Site

vSphere 5 vCenter Server Virtual Appliance Quick-Start Guide

Automating vCenter Server Virtual Appliance Configurations by William Lam

vCenter Management Appliance Certificates by Doug Baer

Now before we begin, because the vCenter Server Virtual Appliance is Suse Linux Enterprise Server based you will have to be used to a Linux command line, using scp, and generally navigating around in order to successfully change your certificates. All operations will be done as root. The default password is vmware. Like in my previous articles regarding changing SSL Certificates I have included an example OpenSSL configuration file that you can use to generate your certificates.

The following directories on the VCVA contain SSL certificates in one form or another:

/opt/vmware/etc/lighttpd/

/etc/vmware-vpx/ssl

/usr/lib/vmware-vpx/inventoryservice/ssl

/usr/lib/vmware-vsphere-client/server/config

I will go through what needs to go where after I’ve given you what you need to create the certificates.

Step-by-Step Process for Changing SSL Certificates on VCVA

You could execute a similar process to the one I’m about to describe using an OpenSSL or Public CA and using the Unix/Linux version of OpenSSL, however this is how I did it successfully in my lab and with my customer. As mentioned in the vSphere 5 Security Guide VMware uses X.509 v3 SSL certificates (base-64 encoded) for encrypting traffic between various components. If you CA has been set to support only SHA512 hash that is fine, it will work. The three key files for an VCVA are rui.crt, rui.key and rui.pfx.

In order to generate the certificates you’ll need to get a copy of OpenSSL x86 v0.98r or higher, and have access to a Microsoft CA (2003 or higher). The certificates will use a clone of a standard web server request template with Subject Alternative Name added, for my lab I modified the default Web Server Certificate Template to accept up to 15 years for certificates. On the system where you will generate the certificate signing request rui.csr) you will need to ensure you have Microsoft Visual C++ 2008 Redistributable Package (x86) before installing OpenSSL. For the purposes of this process you will use the Microsoft CA Web Pages to submit the certificate request and download the resulting base-64 encoded certificate. You can use the certreq command if you wish also (not covered here). Before applying the certificates to your environment you should ensure that your clients and vCenter server trust your CA, if it’s an AD integrated CA this should be automated (if using Internet Explorer), else you may have to pre-trust the Root or Intermediary CA by loading the CA public cert into your clients and vCenter server (not covered in this process).

Note: If you are using a keypair provided by a commercial CA or already have a key and certificate generated outside of the VCVA you will need to import the keypair into the keystore to cover steps 21 – 23. Example instructions of how to do this you can follow Importing private keys into a Java keystore using keytool by Graham Leggett.

Prerequsites:

Microsoft CA (2003 or above, with Web Server Template with Subject Alternative Name included and configured to your liking).
Microsoft Visual C++ 2008 Redistributable Package (x86) on the system where you will generate the certificate signing request (CSR).
OpenSSL 0.98r or above on the system you will use to generate the CSR vCenter 5.0.
A VCVA Deployed and configured the way you like it.
A backup or snapshot of your VCVA prior to beginning this process would be recommended.

Process Step by Step:

Before you start this process you should log into vCenter Server and check that all the services linked with Web Services are working, such as Hardware Status Tab, vCenter Service Status, and also Profile Driven Storage. These are the areas that are very likely to get broken if the process is not followed correctly.
After having installed Microsoft Visual C++ 2008 Redistributable Package (x86) and Open SSL 0.98r or later on a management system (vCenter or other system, not the CA) open a command prompt (As Administrator if on Windows 2008) and change to the OpenSSL\bin folder. Use the same command prompt opened As Administrator for all the OpenSSL actions in this list.
Edit the openssl.cfg file and ensure it looks similar to the one included at the bottom of this article but with your organization specific information, save the configuration.
Execute the following command – openssl req -new -nodes -out rui.csr -keyout rui.key -config openssl.cfg. Note: there will be no prompts as all the information is contained within the configuration file. This is a bit different than when generating the CSR for the ESXi hosts, but that is because there are more of them, and you may not want to have to generate entire config files for each host. For ESXi hosts it is much easier to just hit enter a few times and then specify a common name (fqdn) and then more on. However for vCenter and Update Manager it is better to have everything in the config file, especially as you will likely be specifying multiple Subject Alternative Names (SAN’s – not to be confused with storage area networks).
Copy or submit rui.csr to your CA, submit an advanced certificate reqeust, using the Web Server template that you modified, and download the base-64 encoded certificate to the system with OpenSSL that was used to generate the CSR (Screenshots of this available here: How to use CA certificate to replace VMware certificate on ESX(i) 4 and vCenter or here: vSphere 5 Certificates – Replacing the Default vCenter 5 Server Certificate.)
Download your root CA certificate and save it as root.cer, copy it to the same location where you generated the CSR.
Execute the following command – openssl pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx. This will create a rui.pfx file which we will now verify.
Execute the following command – openssl pkcs12 -in rui.pfx -info. When prompted enter the password testpassword. You should see an base64 encoded string or characters displayed on the screen and information about the file. Note you will be asked to enter the password twice when it is displaying the private key.
Log into the VCVA through the console or via SSH as root and create a directory called /root/certs where you will copy the new certificate files and root CA files.
Using WinSCP or another SCP/SFTP client tool copy rui.crt, rui.key, rui.pfx and your root CA cert (root.cer) to /root/certs on the VCVA.
On the VCVA console (or SSH login) as root execute cd /root/certs.
Execute mv /opt/vmware/etc/lighttpd/server.pem /opt/vmware/etc/lighttpd/server.pem.bak, this will backup the lighthttpd cert.
Execute mkdir /etc/vmware-vpx/ssl/backup.
Execute mv /etc/vmware-vpx/ssl/rui.* /etc/vmware-vpx/ssl/backup, this will backup the vCenter Server SSL certs.
Execute cat rui.key rui.crt >/opt/vmware/etc/lighttpd/server.pem, this will create the new lighthttpd cert.
Execute cp rui.* /etc/vmware-vpx/ssl, this will update the vCenter Server SSL certs.
Execute mkdir /usr/lib/vmware-vpx/inventoryservice/ssl/backup.
Execute mv /usr/lib/vmware-vsphere-client/server/config/keystore /usr/lib/vmware-vsphere-client/server/config/keystore.bak, this will backup the keystore used by the vSphere Web Client.
Execute mv /usr/lib/vmware-vpx/inventoryservice/ssl/rui.* /usr/lib/vmware-vpx/inventoryservice/ssl/backup, this will backukp the Inventory Service SSL certs.
Execute cp rui.* /usr/lib/vmware-vpx/inventoryservice/ssl, this will update the Inventory Service SSL certs.
Execute /usr/lib/vmware-vpx/jre/bin/keytool -keystore /usr/lib/vmware-vsphere-client/server/config/keystore -storetype JCEKS -storepass testpassword -genkey -keyalg rsa -alias s2dmk.
When prompted enter the information corresponding to your environment, where the first and last name is the fqdn of the VCVA. An example is as follows:
What is your first and last name? [Unknown]:vcva.example.com
What is the name of your organizational unit? [Unknown]:Engineering
What is the name of your organization? [Unknown]:Example Corporation
What is the name of your City or Locality? [Unknown]:Palo Alto
What is the name of your State or Province? [Unknown]:California
What is the two-letter country code for this unit? [Unknown]:US
Is CN=vcva.example.com, OU=Engineering, O=”Example Corporation”, L=”Palo Alto”, ST=California, C=US correct?[no]:yes
Enter key password for (RETURN if same as keystore password):
Execute /usr/lib/vmware-vpx/jre/bin/keytool -keystore /usr/lib/vmware-vsphere-client/server/config/keystore -storetype JCEKS -storepass testpassword -certreq -alias s2dmk – file s2dmk.csr.
Execute /usr/lib/vmware-vpx/jre/bin/keytool -storetype JCEKS -storepass testpassword -keystore /usr/lib/vmware-vsphere-client/server/config/keystore -import -alias root -file root.cer.
Using WinSCP or SCP or another SFTP client copy /root/certs/s2dmk.csr to your CA and submit an appropriate certificate request and download a base64 encoded certificate, save it as s2dmk.crt.
Using WinSCP copy s2dmk.crt to /root/certs on your VCVA system. WARNING: Please make sure you use the Text transfer method else your certificates will be corrupted with additional ASCII characters (^M).
Execute /usr/lib/vmware-vpx/jre/bin/keytool -keystore /usr/lib/vmware-vsphere-client/server/config/keystore -storetype JCEKS -storepass testpassword -import -alias s2dmk -file s2dmk.crt.
Execute /usr/lib/vmware-vpx/jre/bin/keytool -keystore /usr/lib/vmware-vsphere-client/server/config/keystore -storetype JCEKS -storepass testpassword -list to verify that the s2dmk and root certificate have been successfully loaded.
Reboot the VCVA by executing the command reboot, or using the vSphere Client to restart the VCVA guest OS.
Log back into the VCVA as root using the console or thruogh SSH, as we now need to re-register the vCenter Server with the vSphere Web Client to update the SSL certificate thumbprint.
Execute the following command to unregister the local vCenter system from the vSphere Web Client: /usr/lib/vmware-vsphere-client/scripts/admin-cmd.sh unregister https://localhost:9443/vsphere-client localhost root .
Now we will execute the command to re-register the local vCenter server again with this vSphere Web Client: /usr/lib/vmware-vsphere-client/scripts/admin-cmd.sh register https://localhost:9443/vsphere-client localhost root .
Type CRTL-D to log out of the VCVA.
When the VCVA has finished restarting you can log into it using the vSphere Web Client, or vSphere Client and check that the certificates are correct. Provided your client trusts your root CA you should not be given a warning message.

Based on William Lam’s comments below it is possible to simplify the above process as follows:

Step 15 and 16 can be replaced with:
vpxd_servicecfg service stop
vpxd_servicecfg certificate change rui.crt rui.key

I would recommend that you check out his post on Automating vCenter Server Virtual Appliance Configurations.

Please let me know if you have any trouble with the above process, and also if it works for you, your comments and feedback are appreciated.

Example OpenSSL Configuration file (openssl.cfg) without most of the normal comments and white space that is included:

# Example vSphere Web Client OpenSSL Configuration File HOME = . RANDFILE = $ENV::HOME/.rnd oid_section = new_oids

[ new_oids ]

[ ca ] default_ca = CA_default # The default ca section

[ CA_default ]

dir = ./demoCA # Where everything is kept certs = $dir/certs # Where the issued certs are kept crl_dir = $dir/crl # Where the issued crl are kept database = $dir/index.txt # database index file. new_certs_dir = $dir/newcerts # default place for new certs. certificate = $dir/cacert.pem # The CA certificate serial = $dir/serial # The current serial number crlnumber = $dir/crlnumber # the current crl number # must be commented out to leave a V1 CRL crl = $dir/crl.pem # The current CRL private_key = $dir/private/cakey.pem# The private key RANDFILE = $dir/private/.rand # private random number file x509_extensions = usr_cert # The extentions to add to the cert name_opt = ca_default # Subject Name options cert_opt = ca_default # Certificate field options default_days = 5475 # how long to certify for default_crl_days= 30 # how long before next CRL default_md = sha512 # which md to use. preserve = no # keep passed DN ordering policy = policy_match

[ policy_match ] countryName = match stateOrProvinceName = match organizationName = match organizationalUnitName = optional commonName = supplied emailAddress = optional

[ policy_anything ] countryName = optional stateOrProvinceName = optional localityName = optional organizationName = optional organizationalUnitName = optional commonName = supplied emailAddress = optional

[ req ] default_bits = 2048 default_keyfile = privkey.pem distinguished_name = req_distinguished_name attributes = req_attributes x509_extensions = v3_ca # The extentions to add to the self signed cert

input_password = testpassword output_password = testpassword encrypt_key = no prompt = no string_mask = nombstr

req_extensions = v3_req # The extensions to add to a certificate request

[ req_distinguished_name ] countryName = NZ

stateOrProvinceName = Auckland

localityName = Auckland

0.organizationName = IT Solutions 2000 Ltd

organizationalUnitName = IT

commonName = vsphere-web.homedns.org

emailAddress = admin@homedns.org

[ req_attributes ]

[ usr_cert ]

basicConstraints=CA:FALSE nsComment = “OpenSSL Generated Certificate” subjectKeyIdentifier=hash authorityKeyIdentifier=keyid,issuer

[ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment extendedKeyUsage = serverAuth, clientAuth subjectAltName = DNS:vsphere-web.homedns.org, DNS:vcva.homedns.org, DNS:vcweb.homedns.org, DNS:vsphere-web, DNS:vcva, DNS:vcweb

[ v3_ca ] subjectKeyIdentifier=hash authorityKeyIdentifier=keyid:always,issuer:always basicConstraints = CA:true

[ crl_ext ] authorityKeyIdentifier=keyid:always,issuer:always

[ proxy_cert_ext ] basicConstraints=CA:FALSE nsComment = “OpenSSL Generated Certificate” subjectKeyIdentifier=hash authorityKeyIdentifier=keyid,issuer:always proxyCertInfo=critical,language:id-ppl-anyLanguage,pathlen:3,policy:foo

# End of vSphere Web Client OpenSSL Configuration File

vSphere Web Client SSL Cert not updated after vCenter SSL Cert Changed

Michael Webster — Fri, 10 Feb 2012 11:35:55 +0000

I’ve had a few people ask me over the last couple of days why their vSphere Web Client SSL certificates are not being updated when they change the vCenter SSL Certificate as per my article The Trouble with CA SSL Certificates and vCenter 5. The normal reason for this is that the vSphere Web Client, when installed on the vCenter Server, stores it’s SSL certificates in a completely different location to that of vCenter Server. I’ve also since found out since publishing my other articles that this is true for the Inventory Service also. Why both of these services when installed on the vCenter Server don’t leverage the same SSL certificate location I’m not sure. My previous article has now been updated to include the replacement of the SSL cert for the Inventory Service, and also mentions the vSphere Web Client when installed on the same system as vCenter.

The default location for the vSphere Web Client certificates is C:\Program Files\VMware\Infrastructure\vSphere Web Client\DMServer\config\ssl. You will need to restart the vSphere Web Client, or reboot the vCenter Server to load the new certificates into memory. You will then be able to log into the vSphere Web Client to test that it is still functioning.

Updated: In most cases you should use the default keystore password of testpassword for your pfx file. If you chose to use a custom password for your keystore you will need to update the tomcat configuration files with the new password. Please refer to VMware KB 1013472 – vCenter Server Service Status plug-in cannot be enabled. However this doesn’t offer any additional security as the keystore password is still stored in clear text in these configuration files either way. Access to the filesystem locations of the important certificate and configuration files should be locked down to prevent any unauthorized access. Thanks to Dan Corrigan for raising this in the comments below.

You may notice that when you change the vCenter SSL Certificate that vSphere Web Client will pop up a warning box the when you attempt to log in. The warning box will say that secure communication can’t be verified. This is due to the thumbprint of the vCenter Server SSL Certificate being different to what vSphere Web Client recognized when it was registered with vCenter. If you click install the certificate and ignore to continue it will not prompt you again on this system. You will need to unregister the vCenter system on the vSphere Web client using the admin-app url, and then re-register it again. To do this you will need to log into the vSphere Web Client system using RDP (Assumes Windows Version), then opening https://localhost:9443/admin-app in a web browser. Once the vCenter System is registered with the new thumbprint the warning dialog box should not be displayed again.

WARNING: Under normal circumstances you should not blindly ignore these types of warning messages and should not automatically just install certs and ignore to continue. You need to institutionalize Standard Operating Procedures that question every time a warning dialog such as this is present and you must verify the authenticity of the certificate. Here is an example of the warning box with the vCenter Server and SHA1 thumbprint obscured.

So you don’t have to jump back to my previous article just to find the default location for the Inventory Service SSL Certs it is C:\Program Files\VMware\Infrastructure\Inventory Service\ssl.