Skip to content
 

vCenter Error: Cannot synchronize host. A general system error occurred.

I came across an interesting issue today with vCenter 5.0 (ESXi build 5.0.0 515841). For some reason one of my hosts was in a disconnected state, and the following error was being displayed in the vSphere console:

A more complete error description is below:

Cannot synchronize host HOSTNAME.
A general system error occurred: Unexpected exception reading HTTP response body:
class Vmacore::Http::TruncatedResponseException(While determining chunk size, truncated HTTP response.)
with trace:
backtrace[00] rip 000000018013d40a (no symbol)
backtrace[01] rip 00000001800ffa38 (no symbol)
backtrace[02] rip 00000001800fffee (no symbol)
backtrace[03] rip 000000018008794b (no symbol)
backtrace[04] rip 000000018003ed6f (no symbol)
backtrace[05] rip 000000018004349e (no symbol)
backtrace[06] rip 0000000000422989 (no symbol)
backtrace[07] rip 00000000003f35d7 (no symbol)
backtrace[08] rip 00000000003f3bd9 (no symbol)
backtrace[09] rip 000000013feef6af (no symbol)
backtrace[10] rip 000000013fef702a (no symbol)
backtrace[11] rip 000000013fef7b10 (no symbol)
backtrace[12] rip 000000013feef827 (no symbol)
backtrace[13] rip 000000013fef87c4 (no symbol)
backtrace[14] rip 000000000036263c (no symbol)
backtrace[15] rip 000000000045a2c7 (no symbol)
backtrace[16] rip 000000013f9b7d20 (no symbol)
backtrace[17] rip 000000013f9b890c (no symbol)
backtrace[18] rip 0000000180153dad (no symbol)
backtrace[19] rip 00000001801552d4 (no symbol)
backtrace[20] rip 000000018014dc65 (no symbol)
backtrace[21] rip 0000000073442fdf (no symbol)
backtrace[22] rip 0000000073443080 (no symbol)
backtrace[23] rip 0000000077a8652d (no symbol)
backtrace[24] rip 0000000077cbc521 (no symbol)
while parsing serialized value of type string at line 7, column 2272735
while parsing property "diskMode" of static type string
while parsing serialized DataObject of type ...[SNIP]…

It goes on for a while; another 700 lines or so!

Note that all VM’s on this host were still happily running and I could connect to the host directly with the vSphere Client and interact with the VM’s. It was just a case of problems communicating between the ESXi host and the vCenter server.

The solution to this issue is below.

SSH onto the host in question and rename/delete the /etc/vmware/vpxa/vpxa.cfg file:

~ # cd /etc/vmware/vpxa
/etc/vmware/vpxa # ls
dasConfig.xml vpxa.cfg
/etc/vmware/vpxa # mv vpxa.cfg vpxa.cfg.old

Once done, back in the vSphere Console, right click the host in question and select “Connect”. The host should re-connect successfully.

Note, there may be other ways to delete this file, however I enable SSH on all my ESXi hosts by default just in case!

13 Comments

  1. Mike Payne says:

    I have the same problem so I followed your steps and was able to bring the server back online inside of vCenter but then it went offline again. I tried the steps again and it still will not come online. Do you have any suggestions? Should I remove the host from vCenter then re-add it?

    Thanks,
    Mike

    • adam says:

      Is vCenter presenting the same error as listed above, or is it showing something different?

      Yes, one thing you can always try is to remove and re-add the server to vCenter. I’d suggest removing from vCenter, deleting the vpxa.cfg file and then re-adding. Note that when you do this any VM’s running on your host will be removed from their current folders/resource groups and need to be re-added/re-organized after joining vCenter again.

      Let me know how you go.

      • Mike Payne says:

        Thanks for the suggestions. I tried making the changes and I’m still getting the same error. I’m able to manage the server through vSphere client, but I’m not able to get it back added to the vCenter

  2. vinzcenzo says:

    Hi There,

    I Have the same issue and like Mike Payne, I’m still not able to reconnect the host to the vCenter.

    Any suggestion?

    Bests

    Vincent

  3. Mike G says:

    I had the same problem as you guys. I ended up calling tech support and this is how we got it resolved.
    1st try bouncing services (this will not effect the still-running VMs) like this:
    -SSH in to the disconnected host
    -type: services.sh restart

    If this does not resolve, see if there is a group of failed backups by typing:
    ls -l /vmfs/volumes/4*/*deltea* | less
    If there are lots of entries this is probably the problem (for us there were 200+ for one VM)
    we were then able to fix the problem by:
    -create a snapshot of the problem VM using the GUI Snapshot tool
    -open snapshot manager and ‘delete all’
    -Note it took about 2 hours to complete deleting all snapshots . You might think it is hung but it is not. If you are not sure, here is an article how to monitor a snapshot commit / delete VMW KB 1007566.

    Hope this Helps

  4. Mike G says:

    typo on above command should be:
    ls -l /vmfs/volumes/4*/*delta* | less

    • Shimon says:

      Hey
      This syntax doesnt work for me (on esxi 5.0.1)
      what does work is : * make sure that you have enough lines of buffer on Putty)

      cd /vmfs/volumes
      find . -name ‘*delta’

      which give a long list of all the delta (Snapshots) files

      what has been found with vmware support is that if there are more than 32 snapshot files, it can cause the host to disconnect and not function and result the error message

      i found out the our NetBackup agent create endless snapshots files for backup and not cleaning up well the trash.

      what is need to be done is
      1.to remove the VM from invntory
      2. load it up on a different seperate host
      3. convert or clone the VM to a new one so it will merge all the snapshots together.
      4. re-mount to the cluster
      hope it helps.

  5. Aaron says:

    Mike G’s fix worked for me too. I had a VM with ~230 phantom snapshots. Thanks Mike!

  6. Pete says:

    This happened to me yesterday. Turned out there was a VM that had 200+ orphaned snapshots (didn’t show up in the snapshot manager). I consolidated that VM, then re-added the host to the cluster, and all was good.

    As for what was creating the orphaned snapshots, I’m not entirely sure. I suspect VDR.

  7. Mike G says:

    Just to close the loop on this, I found the cause of the excessive snapshots to be backup software quiescing. I tried with VDR and NetBackup, they both seemed to cause the problem. I disabled quiescing in NetBackup and have not lost connection to the ESX server for about 3 weeks.
    So in conclusion I believe something about that specific VM is not playing nice with snapshot quiescing. The only thing different about that VM is it is on a different VLAN and a different AD domain.

    • Adam says:

      Yes, this is quite a common issue with vSphere, so much so that there are VMware KB’s (eg: http://communities.vmware.com/docs/DOC-11987). Additionally, if you look at the VMware VDR, there are some fairly strict requirements for quiesced snapshots. I do a lot of VMware consulting and one that I come across fairly regularly is windows VM’s using the Microsoft iSCSI adapter to attach additional LUN’s internally within the VM. This is not supported and will cause the VMware VSS provider to fail, in turn causing your VM to fail it’s quiesced snapshot. A work around for this is to use a Virtual RDM lun through VMware instead. These can be snapshot and are supported by VMware VSS.

  8. RobertoB says:

    You gave me a precious help.
    I tried many times to restart management agents and to reconnect host to vcenter without success.
    Then I found phantom snapshots with your help.

    Thank you.

Leave a Reply