We had a customer call us saying that the Hardware Status tab was not working on either of their vCenter environments. We asked them when it last worked and they responded (as expected) with “Can’t remember.”
This led us on a bit of a wild goose chase, we tried all the standard troubleshooting steps, restarting services, rebooting vCenter servers, checking the firewalls (as mentioned in vmware KB1013080). But we kept on getting the error.
Upon checking the vws.log file (\ProgramData\VMware\VMware VirtualCenter\Logs) we had the following entries:
com.vmware.vim.health.impl.ComponentSpec] Unable to retrieve health for com.vmware.vim.vsm from https://localhost:443/vsm/health.xml
[2014-12-16 10:54:20,399 Thread-26 ERROR com.vmware.vim.health.impl.ComponentSpec] Unable to retrieve health for com.vmware.vim.vsm from any of its health URLs
[2014-12-16 10:54:20,399 Thread-26 ERROR com.vmware.vim.health.impl.XmlUtil] Error retrieving health from url: https://localhost:443/eam/eamService-web/health.xml
[2014-12-16 10:54:40,247 Thread-47 ERROR com.vmware.vim.vimclient.VimClientFactory] Failed login to VC with NoClientCertificate exception
com.vmware.vim.binding.vim.fault.NoClientCertificate:
inherited from com.vmware.vim.binding.vim.fault.VimFault:
inherited from com.vmware.vim.binding.vim.fault.NoClientCertificate: Client connected without supplying a certificate.
and
[2014-12-16 10:54:16,212 Thread-48 ERROR com.vmware.vim.cimmonitor.qs.provider.impl.QsHelperImpl] Vim configuration exception occurred while creating a provider client
com.vmware.vim.vimclient.exception.VimConfigException: VC client creation failed, retries in progress
at com.vmware.vim.vimclient.VimClientFactory.getVmomiClient(VimClientFactory.java:122)
at com.vmware.vim.cimmonitor.qs.provider.impl.QsHelperImpl.createProviderClient(QsHelperImpl.java:428)
at com.vmware.vim.cimmonitor.qs.provider.impl.QsHelperImpl.access$000(QsHelperImpl.java:63)
at com.vmware.vim.cimmonitor.qs.provider.impl.QsHelperImpl$1.run(QsHelperImpl.java:296)
at java.lang.Thread.run(Thread.java:662)
[2014-12-16 10:54:16,212 Thread-48 ERROR com.vmware.vim.cimmonitor.qs.provider.impl.QsHelperImpl] Refreshing provider client failed, retrying
We started to get the impression that something was wrong with certificates…
Luckily we stumbled across the following vmware KB
And as luck would have had it, the customer had recently upgraded to vSphere 5.0!
We ran through the steps in the KB, here is how we ran through the steps (PLEASE NOTE THAT THE CUSTOMER USED SELF SIGNED CERTIFICATES, DO NOT DO THIS IF YOU USE A ROOT CA)
Take a backup of the following files – rui.key, rui.crt. and rui.pfx located in C:\ProgramData\VMware\VMware VirtualCenter\SSL
- Downloaded the Shining Light Productions installer for OpenSSL x86 selecting the current version of 0.98, available at Shining Light Productions. This is a software developed from the OpenSSL Project.
- Take a backup of the cfgfile. By default, this file is located at the c:\OpenSSL-Win32\bin directory.
- Delete the contents of the file and replace with:
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS:<Host Name>, IP:<IP Address>, DNS:<FQDN>
[ req_distinguished_name ]
countryName =
stateOrProvinceName =
localityName =
0.organizationName =
commonName = <FQDN>
Fill in the country names in accordance with your location.
Save the file in its location open up the file location in command prompt (Admin)
Generate the new rui.key and rui.crt files (This will take the settings from the new .cfg you just created). Use the following command exactly –
openssl req -nodes -new -x509 -keyout rui.key -out rui.crt -days 3650 -config openssl.cfg
Generate the new rui.pfx file by running the following command
openssl pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx
NOTE DO NOT CHANGE THE PASSWORD FROM “testpassword”
Copy the newly generated files into C:\ProgramData\VMware\VMware VirtualCenter\SSL and overwrite your old files.
Open a command prompt in C:\Program Files\VMware\Infrastructure\VirtualCenter Server and run the command
vpxd.exe -p
This will prompt you to reset the DB password, however you can simply use the same password you are currently using for this to work correctly.
Once this has been done, stop and restart these services:
- VMware VirtualCenter Server
- VMware VirtualCenter Management Webservices
- vCenter Inventory Service
- VMware vSphere Profile-Driven Storage Service
After the above steps you will log into vCenter and see that all your VMs and Hosts are disconnected. Fear not, you should simply be able to hit “Reconnect” and bring everything back into the VC inventory. A lot of VMs will show in a powered off state and display a number of errors. Give them time and they will come around and should work with no issues. We did see a couple of HA agent errors but no servers were rebooted as a result.
Once this has been done, you should be able to go to your Hardware Status tab and see that it is now working, as well as the vCenter Service Status page!
Virtual Geezer 