76 Responses

  1. The Trouble with CA SSL Certificates and ESXi 5 « Long White Virtual Clouds

    […] The Trouble with CA SSL Certificates and vCenter 5 […]

  2. vSphere Web Client SSL Cert not updated after vCenter SSL Cert Changed « Long White Virtual Clouds

    […] certificates are not being updated when they change the vCenter SSL Certificate as per my article The Trouble with CA SSL Certificates and vCenter 5.  The normal reason for this is that the vSphere Web Client, when installed on the vCenter Server, […]

  3. Whysyn
    Whysyn at |

    Any thoughts on getting the name used by Update Manager to update? I installed a commercial cert using a lot of information on your site (THANK YOU!). Now I get an error that the certificate name mismatches when installing the Update Manager plugin.

    Reply
    1. @vcdxnz001
      @vcdxnz001 at |

      Hi Whysyn, Which host is your Update Manager installed on? The cert used for Update Manager needs to have the correct FQDN, else it will report a mismatch. It's also possible that update manager is registered with vCenter using a short name or IP, what did you enter when you installed it? In the case of short name it may be possible to use the Subject Alternative Name in the CSR to get the cert working. But this is a bit inconvenient if you have public CA certificates given the costs.

      Reply
      1. Whysyn
        Whysyn at |

        Update Manager is installed on the vCenter server.

        Update manager is registered with the internal hostname… during installation they present a dialog that only allows choosing an IP address, short name, or full hostname – they don't permit entering an arbitrary hostname, which is what I need since my internal hostname (vc.dbsvi.local) is different from my public domain name.

        The cert has the proper, publicly accessible hostname. My question is how to change VUM's registered hostname.

      2. @vcdxnz001
        @vcdxnz001 at |

        There is no easy answer there. You will likely have to change your certificate to the internal hostname / FQDN, not the external / public one. Is your vCenter actually publicly available?

      3. Whysyn
        Whysyn at |

        Yes it is, we are a hosting provider. Was just hoping for an easy answer before getting on with VMWare support. Thanks again.

      4. @vcdxnz001
        @vcdxnz001 at |

        Ah, well in that case there may be another answer. Register the vCenter in the name of the external DNS and then make sure that the internal server can access vCenter using the external DNS name. Then access to vCenter is via external DNS name even for internal. You can use split horizon DNS to present different IP based on internal/external source.

  4. Changing vCenter Heartbeat to CA SSL Certificates « Long White Virtual Clouds

    […] a previous post, The Trouble with CA SSL Certificates and vCenter 5, I reported that there isn’t a supported way to change out the self-signed SSL certificates […]

  5. Behruz
    Behruz at |

    Hey Michael,

    which version of OpenSLL you are using? i have version OpenSSL 1.0.0g 64bit for Windows and original config file is much bigger than you have posted even after removing # dash sections. can i replace my cfg file with one you have posted here?

    Reply
  6. Behruz
    Behruz at |

    i have missed the part where you talks about openssl version. i will try to use your openssl config file on windows and will let you know results

    thank you

    Reply
  7. Behruz
    Behruz at |

    i have followed your instruction and everything is working like charm excpet VMware vSphere Profile-Driven Storage Service whihc still fails after resart with "The VMware vSphere Profile-Driven Storage Service service terminated with service-specific error Incorrect function.." error. any suggestions?

    Reply
  8. Behruz
    Behruz at |

    15 hours I'm stuck in front of my pc with this problem about certificates for vSphere vCenter 5 and I'm wrong in my previous post, it's not working as I expected.

    1. The VMware vSphere Profile-Driven Storage Service doesn't want to start

    2. vCenter Service Status shows Cannot access the health service

    3. Web Client still gives the same certificate warning but IE show that certificate is ok

    I followed yours and Julian Woods's instruction 5 times, no luck. VMware vSphere Profile-Driven Storage stops after 30 or so and health status shows error.

    I have removed Web Client thinking it uses port which suppose to be used by SPS, SMS- not helped. I have generated few more certs and it also doesn't help. After digging in tomcat logs I have added localhost to the new certificate in SAN field thinking it will help because tomcat uses localhost in it's extension config file

    C:Program FilesVMwareInfrastructureVirtualCenter Serverextensions sms and sps

    Then I have tried ports change and changed http: to https or added 80 or 81, no luck all this doesn't help. At the moment I'm installing original ssl certificates everything coming back and running from first try. Digging bring me to the KB from kb.vmware.com/kb/2007824 what i'm doing wrong with certificates?

    Reply
  9. @vcdxnz001
    @vcdxnz001 at |

    Hi Behruz, That is a common errror, as is the vCenter Service Status and the Host Status tabs not working if the certificates have been generated incorrectly or applied incorrectly. I've attempted to contact you offline regarding this and when we have worked out what has gone wrong we can post an update. Do you have a backup of all your certs?

    Reply
  10. Updating CA SSL Certificates in vSphere 5 « Long White Virtual Clouds

    […] The Trouble with CA SSL Certificates and vCenter 5 […]

  11. Dredd
    Dredd at |

    To minimise the impact of changing VC certs, you can re-use the existing RSA key when creating the new certificate. This can be done with something like "openssl req … -key rui.key …".where the existing rui.key file is used to create the CSR.

    The advantage of doing this with the VC is that you will not have to reconnect the ESX hosts manually after replacing the VC cert, if the key is the same on the old and new certs. Nor do you need to re-encrypt any database passwords or passwords in customzation specs.

    Reply
    1. @vcdxnz001
      @vcdxnz001 at |

      Thanks Dredd, good tip. You also don't need to reconnect the hosts if the hosts are changed to CA SSL certs prior to vCenter. Then it doesn't matter if you change the vCenter key or not you will not have to reconnect the hosts. I think most customers that are changing out the certs want to start a fresh though so will most likely be changing their keys. Most of the customers I deal with don't accept the default keys as they view them as too weak.

      Reply
  12. Why change VMware default self-signed SSL certs? « Long White Virtual Clouds

    […] articles now on how to change the self-signed SSL certs in a few of the VMware components, such as vCenter Server 5, vSphere Web Client, and ESXi 5 Hosts. All without any discussion about why you would want to do it […]

  13. Paul
    Paul at |

    Great post 🙂

    I can get all this to work fine, bar the fact I'm using a commercial CA and can’t seem to get the certs to chain properly, is this expected behaviour?

    Reply
    1. @vcdxnz001
      @vcdxnz001 at |

      Normally the full chain will be in the certificate. You may want to check with your CA if the full chain is not included. Make sure the vCenter trusts the CA by installing the Root CA Cert if necessary.

      Reply
  14. @vcdxnz001
    @vcdxnz001 at |

    Hi Behruz, it appears that your certificates do not contain all the mandatory attributes (based on the review of the supplied certs). Either the CA Certificate Template or the OpenSSL File is incorrect. Let me know how you get on when you do get time to retry the process.

    Reply
  15. Chris
    Chris at |

    How can I ensure that my certificate is correct? I followed your instructions but the web services won't start..

    Reply
  16. @vcdxnz001
    @vcdxnz001 at |

    Hi Chris, The easy way to verify your cert is by double clicking on the .crt file and looking through all the attributes. Compare the attributes listed in the cert with those that I have documented as required. The most likely reason why the web services won't start is that either the cert is not correct (would also normally cause vCenter not to start), or the database password hasn't been reset correctly using vpxd -p. When you issue the vpxd -p command you need to use the same database password as it was before. You should not specify testpassword here, unless your database password also just happens to be test password.

    Reply
  17. Johan
    Johan at |

    If you have ever configured certificates for SRM you will know that it uses Subject Alternative Name, but in the SRM case it is normally set to “SRM” and must be the same on both side <— Nah it should be the FQDN name the CN should be the same at both sides

    Reply
    1. @vcdxnz001
      @vcdxnz001 at |

      Good Spotting Johan. I have updated the wording of the article, where I had got it around the wrong way with regard to SRM.

      Reply
  18. Bill Scheirer
    Bill Scheirer at |

    Using a GoDaddy Wild Card Cert I recieve the following error when attempting to click the "Invoke Method" button:

    Method Invocation Result: vpx.fault.SecurityConfigFault

    dynamicType: String Unset

    dynamicProperty: vmodl.DynamicProperty[] Unset

    faultCause: vmodl.MethodFault Unset

    faultMessage: vmodl.LocalizableMessage[] Unset

    I have removed the rui.* files from both locations and replaced them with the .crt / .pfx / .key files I received from GoDaddy.

    Any insight? Thanks for the great guide

    Reply
    1. @vcdxnz001
      @vcdxnz001 at |

      Hi Bill, It's most likely some required attributes are missing or one or more of the files is corrupted. Have you verified the .pfx file? Have you checked all the attributes of the crt file?

      Reply
      1. Bill Scheirer
        Bill Scheirer at |

        Which are the "Required Attributes" ? and I have used this cert and .pfx file previously on other (non esxi / vcenter) servers without issue.

      2. @vcdxnz001
        @vcdxnz001 at |

        All of the required attributes are covered in the article. So verify that all the required attributes are present in the certificate files and that the pfx file is correct with the correct alias. It will not work without the correct alias of rui.

  19. Paul Howard
    Paul Howard at |

    I had another stab of this today, no joy i just cant get it to chain rapidssl certs or digicert for that matter, vCentre just wont play ball, anyone had any success with a comercial CA?

    Reply
  20. Nicholas
    Nicholas at |

    Hi Paul Just saw your post, I'm having issues getting a commercial CA to work on vCenter 5. Please see my post here and get in contact with me. http://communities.vmware.com/message/2041504.

    I have an open SR with VMware, MAN! they have no idea what they are doing. We need more people to open up an SR so they can fix the issue. So just reiterate when you replace the default SSL on VC with a commercially sign SSL vCenter will not load the certificate chain even if you have installed the intermediate CA into the trusted store. Another thing to try, when you compile the PFX file using openssl use the -certfile switch to import the Intermediate CA certificates provided by your CA "openssl pkcs12 -export -in rui.crt -inkey rui.key -certfile CACert.crt -name rui -passout pass:testpassword -out rui.pfx" now that PFX will have the chain. However vCenter still does not load the chain, BUT if you browse to https://vcFQDN:8443 this will load the tomcat page and the SSL works perfectly, I suggest to use and online ssl checker to make sure.

    Reply
    1. @vcdxnz001
      @vcdxnz001 at |

      Hi Nicholas, Thanks for your comment and additional information and comment. I agree SR's need to be raised for this and a proper fix provided. But vCenter not loading the chain isn't an issue per se for vCenter (I'm not saying it's not an issue, just vCenter should still work). Tomcat and IE/Windows will check/load the chain however. If your CA isn't in the Microsoft root list you will need to load the CA certs into the system also.

      Reply
  21. Nicholas
    Nicholas at |

    I agree, its not a show stopper and vCenter still functions correctly. I actually have actually made some progress with VMware Support, the engineer assigned to me case has been much more responsive and has acknowledged that there is an issue with vCenter 5 loading the commercially signed CA chains.

    Have a look at this documentation, http://www.vmware.com/files/pdf/techpaper//vsp_41

    I know its for vCenter 4.1 but the same principle still applys, in particular note these:

    Page 2 – "Certificates signed by a commercial certificate authority, such as Entrust or Verisign, are pre‐trusted on the

    Windows operating system."

    Page 6 – "VMware recommends that you replace default certificates with those signed by a commercial certificate

    authority."

    Reply
  22. James
    James at |

    I'm having the exact same problem with vCenter 5 U1. We Installed a GoDaddy Web server SSL, I also updated the ROOT CAs on the server and vCenter does not load the chain, I also tried Nicholas suggestion with the :8443, that did not work either at first, but then I recreated the pfx file with the certificate chain and browsing to the vCenter :8443 works perfectly with the entire chain. Definitely a problem with vCenter I have had this working in prior versions.

    Reply
    1. @vcdxnz001
      @vcdxnz001 at |

      I don't think it's a vCenter only problem as even without vCenter in the picture the chain is not visible on a standard windows system. For Thawte I've verified it's definitely in the trusted list of CA's in Windows by default, at least their high trust version is. I've not had a chance to verify why the chain is not visible in the base CRT file when the PFX isn't being used. Tomcat is using the PFX, so the main difference is that it has access to both the cert and also the keys. For cases where the environment is not exposed to the internet then an internal enterprise CA is normally the best and easiest option. For systems exposed to the internet, such as vCD / vC in public cloud, then a commercial public CA is normally the best option (doesn't help you guys). I'll see if I can make contact with VMware Support and determine why the public CA's are causing these symptoms. I know with GoDaddy their certs do not contain the required attributes, so that could be part of the problem. You would have to pick the correct template for the request and make sure your request contained the required attributes.

      Reply
  23. Nicholas
    Nicholas at |

    @vcdxnz0 Thanks for your comments, I believe somehow we will get to the bottom of this. Just so we know what I have tried, here is a list:

    1. Created CSR as per VMware vCenter 5 Documentation, Signed by Thawte using SSL123 certificate (domain validation only SSL) installed certificate as per VMware documentation.

    2. Created the CSR using methods above and methods explained in other blogs, where modifying the openssl.cfg, then every other steps described in VMware documentation.

    3. Also tried and SSL signed by Thawte, using Web Server Certificate, which validates company and other attributes.

    4. When updating the SSL in the vCenter directory, The Tomcat Web UI at port 8443 uses that certificate and requires the Intermediate CA's to be loaded into the PFX file to complete the chain. I was able to have the Tomcat Web UI load the certificate with the fully trusted chain, when I import the Intermediate CAs into the PFX using the -certfile switch on OpenSSL.

    5. VMware have also tested this in lab and was also not able to load the chanter on vCenter.

    If anyone out there believes they have successfully replaced the SSL with a commercially sign certificate on a public accessible vCenter server, can you please test it using https://search.thawte.com/support/ssl-digital-cer… or https://ssl-tools.verisign.com This will ensure that in fact that it is fully public trusted by the browser not just pre-trusted manually?

    Reply
    1. @vcdxnz001
      @vcdxnz001 at |

      Hi Nicholas, Just wanted to let you know that I'm working with Thawte and VMware on this problem and we will get too the bottom of it and publish the resolution.

      Reply
  24. conraddel
    conraddel at |

    There seems to be a bit of a devil in the detail here:

    We are busy implementing a PCI compliant environment for our customer. They have a Windows 2003 Std Edition CA in the environment, we have duplicated the Web Server Template. But when doing step 5, we are unable to select the duplicated customised web server template……

    So we went ahead and issued it with the standard web server template, but now the generated SSL certificate is missing some detail when we compare it with the VMware default one:

    Key usage:

    VMware default = Digital Signature, Key Encipherment, Data Encipherment (b0)

    Generated one = Digital Signature, Key Encipherment (a0)

    Enhanced Key usage

    VMware default = Server Authentication (1.3.6.1.5.5.7.3.1) Client Authentication (1.3.6.1.5.5.7.3.2)

    Generated one = Server Authentication (1.3.6.1.5.5.7.3.1)

    Any idea how to get around this problem ?

    Reply
    1. @vcdxnz001
      @vcdxnz001 at |

      If you go into the Certificate Authority Manager do you see the template listed under certificate templates? You will need to go into the certtmpl mmc snap-in to be able to manage the templates and configure the necessary attributes on the certificate templates. You also need to make sure that your CSR's contain all the necessary attributes that you need.

      Reply
  25. conraddel
    conraddel at |

    Hi there

    Thank you for your reply. We used the openssl.cfg config file as you supplied, and just modifed the:

    [ req_distinguished_name ] and subjectAltName

    sections as appropriate

    Yes when we launch certtmpl.mmc we do see our modfied template, however when we launch https://localhost/certsrv and we select request a certificate, advanced certificate request etc. we do not see the modified template in the "certificate template" dropdown.

    Reply
  26. Doron Offir
    Doron Offir at |

    Thank you so much, for me it helped beautifully, I wasted two days trying to convince the VC 5 and its services to work with the VC 4 certificate, the VC was working, but no the services, and this article helped to pinpoint the problem, many thanks!

    Reply
  27. Ross
    Ross at |

    Great article and I got there in the end. What wasn't spelt out was how to create the necessasry duplicate template so I'll list the key steps that I used on a Windows 2008 R2 Certificate Authority server:

    On the CA server launch Administrative Tools | Certificate Authority

    R-click certificate templates and choose Manage

    R-click Web Server and choose Duplicate Template

    Choose Windows Server 2003 Enterprise (this seemed to be important as by choosing the 2008 option I couldn’t get the new template appearing in the drop down list for certificate templates using the web based certificate request process.

    Change the Template Display Name to whatever e.g. vCenter Template

    Set the Validity Period to whatever you require e.g. 5 years

    On the Request Handling tab check the box for: Allow private key to be exported

    On the Subject Name tab ensure that “Supply in the request” is selected

    On the Extensions tab edit Application Policies and add Client Authentication to the existing Server Authentication

    Edit the Key Usage extension so that the Digital Signature box is checked, the Signature is proof of origin is checked, Allow key exchange only with key encryption is selected and Allow encryption of user data is checked.

    Close the managing templates window

    R-click Certificate Templates and choose New | Certificate Template to Issue and select the newly created template.

    Restart the Active Directory Certificate Services service.

    Reply
    1. @vcdxnz001
      @vcdxnz001 at |

      Hi Ross, Thanks for going to the trouble to add in the information regarding creating the necessary template. That is greatly appreciated.

      Reply
  28. Edward Beheler
    Edward Beheler at |

    Step 5 can be done from the command prompt rather than going through the certificate services website… Something like:

    certreq -submit -attrib "CertificateTemplate:VMwareCert" rui.csr

    which will return something like:

    RequestId: 123

    and then

    certreq -retrieve 123

    Reply
  29. Eugene
    Eugene at |

    In step 17 you say "You should enter the existing database password, not a new password at this point." What about if you are using SQL 2008 Express for your DB?

    Reply
    1. @vcdxnz001
      @vcdxnz001 at |

      Hi Eugene, SQL Express is not recommended for production environments, so I'm not really as familiar with replace certs in an environment using an Express DB. I would think that it would normally be the password of your windows user account or service account for vCenter. Unless you're using Local System. If you're using local system the whole username and password doesn't really come into play. Maybe this should be the topic for another blog post, and also something I have to consider for a new project that I'm working on. Thanks for your comment.

      Reply
  30. DavidS
    DavidS at |

    A great article I find After doing all this from the VMWare KBs (Not actually too bad, the v5 KBs are much better).

    One thing that I still need to figure out though is for the vCenter Web UI. Every user/pc that connects has to save/ignore the untrusted thumbprint for the vCenter server.

    The thumbprint and CN are correct. I am thinking I need to configure the webui's trusted CA store to include the CA I am using to sign my certs. Does anyone have some tips on this?

    Reply
    1. @vcdxnz001
      @vcdxnz001 at |

      Hi David, glad you found this useful. I spent a bit of time working with VMware to get the KB's updated, it's a work in progress. If you have further feedback on them please let me know and use the forms on the VMware KB also to submit the feedback. I have covered updating the SSL certs for the vSphere Web Client in one of my other articles – http://longwhiteclouds.com/2012/02/10/vsphere-web…. Also the info for changing the certs on the vCenter Server Virtual appliance are here – http://longwhiteclouds.com/2012/02/13/vcenter-ser…. Everything is listed on this page – http://longwhiteclouds.com/2012/02/24/updating-ca….

      Reply
      1. DavidS
        DavidS at |

        Got it.

        https://localhost:9443/admin-app << Re-register vcenter servers.

        Thanks!

  31. James Livingston
    James Livingston at |

    I found some information about using keytool to import the root certificate into the java trust keystore. But it applied to vCenter Orchestrator. However I used C:Program FilesVMWareInfrastructurejrebinkeytool.exe -import –alias root –keystore C:ProgramDataVMWareVirtual CenterSSLsms.truststore –trustcacerts c:temphpcass_ns.crt followed by the cert store password. It worked I was able to get all the vSphere services running and the pesky register.bat finally worked for the Inventory.

    Reply
    1. @vcdxnz001
      @vcdxnz001 at |

      Great news James. Thanks for posting a comment back here. This is very valuable information for everyone.

      Reply
  32. K. Zachary Abbott
    K. Zachary Abbott at |

    I am having the same issue as Bill Scrhrier had:

    When I go to invoke method I get:

    Method Invocation Result: vpx.fault.SecurityConfigFault

    dynamicType: String Unset

    dynamicProperty: vmodl.DynamicProperty[] Unset

    faultCause: vmodl.MethodFault Unset

    faultMessage: vmodl.LocalizableMessage[] Unset

    The only change I have made to your template for the OpenSSL.cfg is to change the subjectAltName as follows (recommnded here: http://lanestechblog.blogspot.com/2009/04/creatin

    subjectAltName = @alt_names

    [alt_names]

    DNS.1 = vdi-vc1.mydomain.com

    DNS.2 = vdi-vc1.mydomain.com

    DNS.3 = vdi-vc1

    rui file ls named rui.pfx

    Apache works just fine – not sure what vCenter isn't liking. Would be grateful for any assistance you might be able to offer – happy to take it "offline."

    Reply
    1. K. Zachary Abbott
      K. Zachary Abbott at |

      Quick update on my situation – just had my security folks issue me an in-house certificate using the same rui.csr, and viola, no problems….Only major difference – no intermediate CA, but then I can't see that being the issue. Maybe the format of the rui.crt that is issued? will try to compare the two and see if there's anything helpful….

      Reply
    2. K. Zachary Abbott
      K. Zachary Abbott at |

      And…Issue Resolved! If anyone else has this issue:

      1) make sure you do *not* change the password in the openssl.cfg file. I found in one of the VMware KBs that this password *has* to be "testpassword"

      2) if you use a commercial CA, make sure you download the cert in x509/PEM format

      Those were the last two changes I made, and everything is working great now.

      Michael – I'm not sure I said before, but thank you for putting this all together!

      Reply
      1. @vcdxnz001
        @vcdxnz001 at |

        Thanks for your comment. I've added some additional emphasis in my article to highlight these two critical factors. I'm in the process of developing a solution that will automate the process of changing SSL Certs in vSphere components. I will be blogging about this and where people can sign up for the early adopter program.

  33. stine
    stine at |

    Thanks for your instructions and also for your (and others) efforts getting VMware to implement and document this better.

    Regarding the 'testpassword' password, is the reason not to change it because its hardcoded in one or more of the applications or that you simply have to locate every .XML file and udpate them all?

    Unfortunatly, I'll probably re-image my vcenter server and install vcenter from scratch since I originally couldn't bring myself to use testpassword…

    Reply
    1. @vcdxnz001
      @vcdxnz001 at |

      There are a number of XML files to change it in and testpassword is the only officially supported password. You should maintain a supported config.

      To secure your certs from compromise you need to ensure the vCenter OS is secured, and the folders are locked down to only the vCenter service account. No matter what you change it to it will still be in clear text. This is why the OS and file system security is so important.

      Reply
  34. Henrik
    Henrik at |

    Am i the only one seeing the config example is duplicated/repeated?

    Reply
    1. @vcdxnz001
      @vcdxnz001 at |

      Hi Henrik, one of the config examples if for vCenter and the other config example if for Update Manager. Some of the config is similar and some of it is different.

      Reply
  35. Derek Seaman
    Derek Seaman at |

    What key usage values are required for vSphere 5.1? I looked at a variety of the 5.1 VMware self-signed SSL certs and none of them have nonrepudiation, but some do have Data Encipherment.

    Reply
    1. @vcdxnz001
      @vcdxnz001 at |

      Hi Derek, The required attributes are covered in the product documentation. There is a new replacing certs document – Replacing SSL Certificates in vCenter 5.1 and ESXi 5.1.

      Reply
      1. Derek Seaman
        Derek Seaman at |

        Mike that document is very inadequate and poorly written. I've been using that document and nowhere does it use the word "dataEncipherment", yet looking at some of the VMware self-signed certs for 5.1 that key usage is clearly there.

        The default MS web server certificate template does NOT have "client authentication" enhanced key usage, yet the guide shows that extendedkeyusage attribute yet. Looking at the self-signed VMware certs I didn't see any with enhanced key usage for client authentication, only server.

        There seems to be inconsistencies between the self-signed VMware cert key properties and the vSphere 5.1 certificate guide. Not to mention the guide suggest using the MS web server template, which doesn't support client authentication usage.

        I'll create a custom template in the MS CA for the extended attributes that I see in the self-signed certs, but aren't at all mentioned in the VMware guide to see if that helps resolve some major cert problems with 5.1.

      2. @vcdxnz001
        @vcdxnz001 at |

        Hi Derek,

        I agree there are some gaps. My plan is to use the same templates and key usage attributes as I did in 5.0. The template is based off the web server template but isn't exactly the web server template as you would have needed to modify it to include the required attributes. Feel free to submit feedback directly to VMware – docfeedback at VMware dot com also. I haven't looked at the default self-signed certs yet, so your feedback is valuable on that. Also if you think Windows is hard you should try the VCVA. I'm working with some guys on that right now and it is incredibly difficult and not at all well documented.

  36. David Dionne
    David Dionne at |

    Great article, did you ever get a chance to test Heartbeat? I'm struggling with the secondary server. The vcenter service fails to start on the secondary server when I shutdown the primary. I'm getting the following ssl error in the vpxd log file:

    2013-01-22T08:13:25.793-06:00 [03984 error 'Default'] Found dangling SSL error: [0] error:00000001:lib(0):func(0):reason(1)

    2013-01-22T08:13:25.793-06:00 [03984 error 'Default'] Found dangling SSL error: [1] error:00000001:lib(0):func(0):reason(1)

    2013-01-22T08:13:25.793-06:00 [03984 error '[SSO][SsoFactory_CreateFacade]'] Unable to create SSO facade: vmodl.fault.SystemError.

    2013-01-22T08:13:25.793-06:00 [03984 error 'vpxdvpxdMain'] [Vpxd::ServerApp::Init] Init failed: Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)

    –> Backtrace:

    –> backtrace[00] rip 000000018018a8ca

    –> backtrace[01] rip 0000000180102f28

    –> backtrace[02] rip 000000018010423e

    –> backtrace[03] rip 000000018008e00b

    –> backtrace[04] rip 0000000000495c2c

    –> backtrace[05] rip 00000000004b6512

    –> backtrace[06] rip 0000000140040701

    –> backtrace[07] rip 000000014003a51c

    –> backtrace[08] rip 000000014025c92b

    –> backtrace[09] rip 000007feff87a82d

    –> backtrace[10] rip 000000007731652d

    –> backtrace[11] rip 00000000778bc521

    –>

    2013-01-22T08:13:25.793-06:00 [03984 warning 'VpxProfiler'] ServerApp::Init [TotalTime] took 3682 ms

    2013-01-22T08:13:25.793-06:00 [03984 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down…

    Reply
  37. Jeremy Hagan
    Jeremy Hagan at |

    Thanks for this article. It was easy to follow and everything worked. Here are a couple of tips to add. Make sure your Subject name is the correct case. The SSL certificate must have the same case for the hostname and domain as the host reports when running the hostname or ipconfig /all. This is due to the fact the the vSphere client will still report the certificate as untrusted if the case doesn't match. I ended up putting upper case in the Subject name and including a lower-case FQDN as an alternate. See this article (about SRM, but it still applies):
    http://kb.vmware.com/kb/1008390

    Secondly, since I had to replace the certificate again with one that had correct case I was a bit hasty in revoking the old certificate. By the time I got around to trying to do the "reloadSslCertificate" step for the Inventory service, my SQL server had already received the CRL and would no longer authenticate the vCenter service. Eventually I reset the password again (as explained here :http://kb.vmware.com/kb/1003070) and entered the password of the service account. After that I was able to perform the "reloadSslCertificate" step for the Inventory service.

    Reply
  38. Artur Krzywdzinski (
    Artur Krzywdzinski ( at |

    Any solution guys – Unable to create SSO facade: vmodl.fault.SystemError – I had similar problem after V2V of vCenter server

    Reply
  39. Robert
    Robert at |

    In my environment, having nonRepudiation in the keyUsage attribute led to issues. You'll notice the self-signed certs that come by default don't have nonRepudiation set. Once I removed nonRepudiation from the attributes in your example config, the certificates worked fine.

    Reply
  40. techstarts
    techstarts at |

    Hi Mike,

    can you help me as to what "Request Hash" one should select. I see you are using SHA512. But by default webserver template uses SHA1. When I use default web server template I get a warning "**** signature uses weak one way hash (SHA-1). In secure environment it is recommended to use SHA2-256 or a stronger hash algorithm" can you please provide your suggestions"

    Reply
  41. techstarts
    techstarts at |

    Thanks Michael…

    Reply
  42. John Ball
    John Ball at |

    This might be a little late (posting 7 months later) but I recently had this problem when I conducted a test-fail of my vCenter Heartbeat cluster. For the life of me I couldn't get the vCenter Server service started and my logs had the same entries.

    I started going over the VMware services in my services.msc window and noticed that Vcenter SSO was configure to start up using the "local system" account whereas my other services were configured to startup using a special vCenter service account.

    I changed my SSO service to use the same account as the rest of my VMware services (special domain account), restarted the SSO service so the login would take effect, and voila. vCenter Heartbeat started all services immediately.

    Don't forget to make the service login change on your second heartbeat server too!

    Reply
  43. ispcolohost
    ispcolohost at |

    Just wanted to comment that as best I can tell, it is impossible to set up SSL certs in vCenter if they are signed by a public authority. There are two issues:

    1) The Subject Alternate Name field of your CSR, as generated by the SSL Certificate Automation Tool or you manually using openssl, should have a SAN of the FQDN of your server, short name of your server and the IP of your server. Any public CA will delete the short name from the SAN because it could be used on any internal network anywhere to secure that same name. If the IP is IPv4 and from private address space, they will also delete it, for the same reason. I’m guessing most, if not all, vCenter networks are not deployed using public IP’s, but you still can’t get the short name of your server installed.

    2) More importantly than the above, I have not found a public CA that does not strip, or strip and replace, the Organizational Unit field from the resulting cert. The OU field is the one that is required to be unique for each vCenter service (vcenter, vcenter web, inventory, SSO, etc). The Automation Tool generates the OU with a prefix of the service name, and then the server name, such as Hosting-vCenterSSO-server1. With that field not present, or replaced by something stupid like this:

    OU=GT44865949, OU=See http://www.geotrust.com/resources/cps (c)15, OU=Domain Control Validated – QuickSSL(R),

    then you have no way to get past the step of “Update vCenter Server trust to Inventory Service”, because it will fail with “Cannot register vCenter with Inventory Service: 7” which is defined as “This issue can occur if the certificates were generated with a misconfigured subject template on the certificate authority or the certificate is missing expected OU information.” per KB 2102685.

    Without the unique OU that vCenter needs, you’re dead in the water. I have not found any public CA that does not strip those fields so far, but then you still have the SAN field issue regardless.

    It appears the only way to install certs on vCenter is to use self signed or internal CA-signed (MS for example) certs so you can have the requisite SAN and OU fields present in the issued certificate.

    Reply

Leave a Reply