vCNS/vShield Edge – Communication Agent not Connected to vCenter Server

After an upgrade of our vShield Manager and vCenter, I found that the Edge Appliances were not responding to Upgrade commands, and timing out with the error “Communication Agent not Connected to vCenter Server”.

This was after an upgrade of 5.1 to 5.5 using vCenter Server Appliances.

I found that in order to re-establish this connection I had to force a reconnect to the vCenter Server from the vShield Manager Console. Go to Settings & Reports and go to the Configuration tab, then select Edit on the vCenter Server section as below.

vCNSEdit

Then you need to enter admin credentials for vCenter and hit OK.

vCNSCreds

This then asked to re-accept the SHA SSH Thumbprint (I suspect because you have to deploy a new Appliance for the VCSA upgrade) and then it will connect successfully.

Once this was done everything was back to normal working order!

It’s always the simple things that catch you out…

vCloud Director VMs Showing as Unavailable

We hit a small problem where all our vCloud VMs were showing as Unavailable. We couldn’t perform any tasks or get to the console for any of them which was frustrating. Luckily there was a quick an easy fix.

After reading a vmware community post https://communities.vmware.com/thread/495532 “CFLSADASDSDSAD” mentions the following “We have had this issue a few times. In our case VMware support had us clear out the inventory information that vCD keeps in its database that it gets from vCenter. It seems this can get corrupted or out of sync but by clearing the tables in the database forces a complete re-sync”

SO… In order to do this you will to do the following:

You need to stop all of your vCD Cells for your vCD instance, to do this log into each cell and run the following command to stop the VCD Services – service vmware-vcd stop

Go to your SQL Server containing the vCD DB and execute the following statement

delete from cluster_compute_resource_inv;
delete from compute_resource_inv;
delete from custom_field_manager_inv;
delete from datacenter_inv;
delete from datacenter_network_inv;
delete from datastore_inv;
delete from datastore_profile_inv;
delete from dv_portgroup_inv;
delete from dv_switch_inv;
delete from folder_inv;
delete from managed_server_datastore_inv;
delete from managed_server_inv;
delete from managed_server_network_inv;
delete from network_inv;
delete from resource_pool_inv;
delete from storage_profile_inv;
delete from storage_pod_inv;
delete from task_inv;
delete from vm_inv;
delete from property_map;

Once this has been done, resume the Service on ONE of your Cells (service vmware-vcd start) and leave it for 5 minutes.

Log into vCloud Director and confirm the state of the VMs. In our case we had to perform a reconnect of the vCenter (Administration -> vCenters -> Right-Click vCenter and Reconnect). Once we did this, a re-sync was performed and hey-presto! All VMs looked perfect with accessible consoles!

Finish off by powering on the remaining vCloud Director Cells.

Thank you vmware communities!

VM Console will not “Fit to Guest”

A simple one, using a console on a newly built Windows 8 VM, noticed the Fit To Guest function under view wasn’t working. The VM resolution looked to be fixed. Checked the Video Memory allocated to the VM and saw it was only 8MB! Powered down the VM and increased this to 16MB.

Capture

Hey Presto! The VMs Max resolution could be increased. The VM was provisioned as HW Version 8, and although the limit of video memory for this is 128MB, it seems to provision it with the bare minimum.

I hope this helps save some frustration!

Hardware Status Tab and vCenter Service Status Not Working after Upgrade to vSphere 5.0

We had a customer call us saying that the Hardware Status tab was not working on either of their vCenter environments. We asked them when it last worked and they responded (as expected) with “Can’t remember.”

This led us on a bit of a wild goose chase, we tried all the standard troubleshooting steps, restarting services, rebooting vCenter servers, checking the firewalls (as mentioned in vmware KB1013080). But we kept on getting the error.

Upon checking the vws.log file (\ProgramData\VMware\VMware VirtualCenter\Logs) we had the following entries:

com.vmware.vim.health.impl.ComponentSpec] Unable to retrieve health for com.vmware.vim.vsm from https://localhost:443/vsm/health.xml
[2014-12-16 10:54:20,399 Thread-26 ERROR com.vmware.vim.health.impl.ComponentSpec] Unable to retrieve health for com.vmware.vim.vsm from any of its health URLs
[2014-12-16 10:54:20,399 Thread-26 ERROR com.vmware.vim.health.impl.XmlUtil] Error retrieving health from url: https://localhost:443/eam/eamService-web/health.xml

[2014-12-16 10:54:40,247 Thread-47 ERROR com.vmware.vim.vimclient.VimClientFactory] Failed login to VC with NoClientCertificate exception

com.vmware.vim.binding.vim.fault.NoClientCertificate:

inherited from com.vmware.vim.binding.vim.fault.VimFault: 

inherited from com.vmware.vim.binding.vim.fault.NoClientCertificate: Client connected without supplying a certificate.

and

[2014-12-16 10:54:16,212 Thread-48 ERROR com.vmware.vim.cimmonitor.qs.provider.impl.QsHelperImpl] Vim configuration exception occurred while creating a provider client
com.vmware.vim.vimclient.exception.VimConfigException: VC client creation failed, retries in progress
at com.vmware.vim.vimclient.VimClientFactory.getVmomiClient(VimClientFactory.java:122)
at com.vmware.vim.cimmonitor.qs.provider.impl.QsHelperImpl.createProviderClient(QsHelperImpl.java:428)
at com.vmware.vim.cimmonitor.qs.provider.impl.QsHelperImpl.access$000(QsHelperImpl.java:63)
at com.vmware.vim.cimmonitor.qs.provider.impl.QsHelperImpl$1.run(QsHelperImpl.java:296)
at java.lang.Thread.run(Thread.java:662)
[2014-12-16 10:54:16,212 Thread-48 ERROR com.vmware.vim.cimmonitor.qs.provider.impl.QsHelperImpl] Refreshing provider client failed, retrying

We started to get the impression that something was wrong with certificates…

Luckily we stumbled across the following vmware KB

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2007824

And as luck would have had it, the customer had recently upgraded to vSphere 5.0!

We ran through the steps in the KB, here is how we ran through the steps (PLEASE NOTE THAT THE CUSTOMER USED SELF SIGNED CERTIFICATES, DO NOT DO THIS IF YOU USE A ROOT CA)

Take a backup of the following files – rui.key, rui.crt. and rui.pfx located in C:\ProgramData\VMware\VMware VirtualCenter\SSL

  • Downloaded the Shining Light Productions installer for OpenSSL x86 selecting the current version of 0.98, available at Shining Light Productions. This is a software developed from the OpenSSL Project.
  • Take a backup of the cfgfile. By default, this file is located at the c:\OpenSSL-Win32\bin directory.
  • Delete the contents of the file and replace with:

default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS:<Host Name>, IP:<IP Address>, DNS:<FQDN>

[ req_distinguished_name ]
countryName = 
stateOrProvinceName = 
localityName = 
0.organizationName = 
commonName = <FQDN>

Fill in the country names in accordance with your location.

Save the file in its location open up the file location in command prompt (Admin)

Generate the new rui.key and rui.crt files (This will take the settings from the new .cfg you just created). Use the following command exactly –

openssl req -nodes -new -x509 -keyout rui.key -out rui.crt -days 3650 -config openssl.cfg

Generate the new rui.pfx file by running the following command

openssl pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx

NOTE DO NOT CHANGE THE PASSWORD FROM “testpassword”

Copy the newly generated files into C:\ProgramData\VMware\VMware VirtualCenter\SSL and overwrite your old files.

Open a command prompt in C:\Program Files\VMware\Infrastructure\VirtualCenter Server and run the command

vpxd.exe -p

This will prompt you to reset the DB password, however you can simply use the same password you are currently using for this to work correctly.

Once this has been done, stop and restart these services:

  • VMware VirtualCenter Server
  • VMware VirtualCenter Management Webservices
  • vCenter Inventory Service
  • VMware vSphere Profile-Driven Storage Service

After the above steps you will log into vCenter and see that all your VMs and Hosts are disconnected. Fear not, you should simply be able to hit “Reconnect” and bring everything back into the VC inventory. A lot of VMs will show in a powered off state and display a number of errors. Give them time and they will come around and should work with no issues. We did see a couple of HA agent errors but no servers were rebooted as a result.

Once this has been done, you should be able to go to your Hardware Status tab and see that it is now working, as well as the vCenter Service Status page!

Zerto Cloned/Replicated Server Fails to Boot Up

We had a problem where we were performing a test failover of a VM using Zerto Virtual Replication. Upon failing the VM over to the DR site, we noticed that the VM would not boot into its OS. This was a Windows Server 2000 machine which was originally P2V’d. The screen would just be black with a cursor up in the top left corner.

Initially we believed that the disk could have suffered some form of corruption during replication, however this was disproved by adding the disk to another server and being able to browse the contents with no issues. We tried adding the disk to a newly built VM but this would also not pick up the boot media.

Eventually we found the cause to be a mismatch in the vmdk geometry sizes as reported in the .vmdk descriptor file (not the flat file.) See below for the mismatches:

BootdiskVMDKDescriptor

Once we changed the DR .vmdk file values to match those at Production, we found that the VM could boot! This fix usually will only apply to the boot disk – however we have not fully recorded the impact of having mismatching geometry sizes and if this affects performance. In future we may conduct such a test.

Welcome!

Hello and Welcome to our blog on tips and fixes that we have found particularly helpful (and in some cases un-googleable)

We work primarily with vmware, NetApp, Cisco, Microsoft and Zerto, but there may be some odd bits floating here and there.

Please feel free to comment or contribute to any of our posts, or get in touch with us.

We hope that our posts can be of some help to you!

Virtual Geezer