Skip to content

vCenter Error: Cannot synchronize host. A general system error occurred.

I came across an interesting issue today with vCenter 5.0 (ESXi build 5.0.0 515841). For some reason one of my hosts was in a disconnected state, and the following error was being displayed in the vSphere console:

A more complete error description is below:

Cannot synchronize host HOSTNAME.
A general system error occurred: Unexpected exception reading HTTP response body:
class Vmacore::Http::TruncatedResponseException(While determining chunk size, truncated HTTP response.)
with trace:
backtrace[00] rip 000000018013d40a (no symbol)
backtrace[01] rip 00000001800ffa38 (no symbol)
backtrace[02] rip 00000001800fffee (no symbol)
backtrace[03] rip 000000018008794b (no symbol)
backtrace[04] rip 000000018003ed6f (no symbol)
backtrace[05] rip 000000018004349e (no symbol)
backtrace[06] rip 0000000000422989 (no symbol)
backtrace[07] rip 00000000003f35d7 (no symbol)
backtrace[08] rip 00000000003f3bd9 (no symbol)
backtrace[09] rip 000000013feef6af (no symbol)
backtrace[10] rip 000000013fef702a (no symbol)
backtrace[11] rip 000000013fef7b10 (no symbol)
backtrace[12] rip 000000013feef827 (no symbol)
backtrace[13] rip 000000013fef87c4 (no symbol)
backtrace[14] rip 000000000036263c (no symbol)
backtrace[15] rip 000000000045a2c7 (no symbol)
backtrace[16] rip 000000013f9b7d20 (no symbol)
backtrace[17] rip 000000013f9b890c (no symbol)
backtrace[18] rip 0000000180153dad (no symbol)
backtrace[19] rip 00000001801552d4 (no symbol)
backtrace[20] rip 000000018014dc65 (no symbol)
backtrace[21] rip 0000000073442fdf (no symbol)
backtrace[22] rip 0000000073443080 (no symbol)
backtrace[23] rip 0000000077a8652d (no symbol)
backtrace[24] rip 0000000077cbc521 (no symbol)
while parsing serialized value of type string at line 7, column 2272735
while parsing property "diskMode" of static type string
while parsing serialized DataObject of type ...[SNIP]…

It goes on for a while; another 700 lines or so!

Note that all VM’s on this host were still happily running and I could connect to the host directly with the vSphere Client and interact with the VM’s. It was just a case of problems communicating between the ESXi host and the vCenter server.

The solution to this issue is below.

SSH onto the host in question and rename/delete the /etc/vmware/vpxa/vpxa.cfg file:

~ # cd /etc/vmware/vpxa
/etc/vmware/vpxa # ls
dasConfig.xml vpxa.cfg
/etc/vmware/vpxa # mv vpxa.cfg vpxa.cfg.old

Once done, back in the vSphere Console, right click the host in question and select “Connect”. The host should re-connect successfully.

Note, there may be other ways to delete this file, however I enable SSH on all my ESXi hosts by default just in case!

Compile Git on OpenIndiana or Solaris

Dumping this here for my future reference.  If you find it useful, great!  In order to compile Git 1.7.7.1 on OpenIndiana oi_151a or Solaris Express 11 snv_151 perform the following:

Install some necessary packages:

# pkg install gcc-dev
# pkg install SUNWscp

Next, download and extract the latest git source archive, in this case 1.7.7.1 from http://www.git-scm.com:

# cd /usr/src
# curl http://git-core.googlecode.com/files/git-1.7.7.1.tar.gz | tar xvz

Next, compile and install the package:

# cd /usr/src/git-1.7.7.1
# PATH=/usr/sfw/bin:/usr/bin:/usr/sbin ; export PATH
# gmake prefix=/usr all
# gmake prefix=/usr install

Okay, so we’re now installed.  If you need the documentation as well, the easiest way to do this is to clone the official Git git repository and copy the MAN files into the /usr/share/man directory:

# git clone git://git.kernel.org/pub/scm/git/git.git
# git archive --format=tar origin/man | sudo tar -x -C /usr/share/man/ -vf -

That’s it, we’re done!

Ubuntu open-vm tools on VMware

Just a quick one today.  I have been using Ubuntu Server as a staple for linux virtual machines on VMware as it is stable, and fast.  I don’t like using the VMware based vmware tools, as I find it can break things on occasion.  Open-VM Tools is a good substitute as it is based on the opensource offerings from VMware.  It is a simple install using apt-get:

apt-get install --no-install-recommends linux-headers-virtual open-vm-dkms open-vm-tools

The “–no-install-recommends” disables the selection of all the X components, which will essentially install the whole gnome desktop environment if it’s missing on an Ubuntu server; something we don’t want on a server!

This package provides the vmxnet hardware, as well as the majority of the other VMware Tools functionality (Shutdown, Restart etc).

Patching ESXi through SSH

Sometimes it might be necessary to patch an ESXi host through SSH for various reasons, some of which might include:

  1. You are using the free edition of ESXi and don’t want to download the vSphere CLI package
  2. You need to update a remote ESXi host and only have access to SSH through VPN
  3. The host you want to patch is running a virtual vCenter server that contains your Update Manager

For me, I recently ran into a combination of all 3, whereby the server was the only server I had access to in a remote data center.  It was running a virtual “management” machine that simply had the vSphere client.  This allowed me to download the patch and upload to the ESXi host.  Finally, from across the VPN, I SSH’d in and manually updated the server.  Let me detail this below.

Step 1: Enable SSH on ESXi host

By default, SSH is disabled on ESXi by design, as it is considered by VMware to be a security vulnerability and only to be used for troubleshooting.  In ESXi 4.0, enabling SSH had to be done via a secret “support mode” key combination on the console.  Thankfully it’s a lot easier now and can be done either via the console, or via the vSphere Client.  In this case, I will enable via the vSphere Client:

  1. Launch the vSphere Client
  2. Select the host in the tree-view on the left hand side
  3. Select the “Configuration” tab
  4.  Under the “Software” menu, select “Security Profile”
  5. Click the “Properties” link which will open up the “Services Properties” window (see below)
  6.  Select the “Remote Tech Support (SSH)” service and then click the “Options” button
  7. Set the “Startup Policy” to “Start automatically” and click “Start”
  8. Click “OK” and “OK” again

SSH will now be started.

Step 2: Download patch to an available datastore

You can find the latest patches at VMware’s Patch Database.  Here you can search and download the latest patches available for any ESX/ESXi version from 3.0.3 and later.  In this case we’re going to be patching an ESXi 4.1 host.

  1. Using your web browser download the patch file from the VMware Patch Database.  In this case, the file is “ESXi410-201107001.zip
  2. In the vSphere Console, right click on one of the datastores available on the ESXi host, and click “Browse Datastore”
  3. In the “Datastore Browser” select the folder you want to upload the patch to.  I usually create a “Patches” folder
  4. Click the “Upload” button and select “Upload file”
  5. Locate your file and select “Open”

Your file will now be uploaded to the datastore

Step 3: Place the ESXi host into Maintenance mode

In order to install any patch, an ESXi host must be in maintenance mode.

  1. Power down or migrate all virtual machines.  If this is a standalone host (as was in my case), your only option is to power down all vm’s.
  2. Right click on the host in the vSphere Console and select “Enter Maintenance Mode”

You can also enter maintenance mode via SSH using the following command:

vim-cmd /hostsvc/maintenance_mode_enter

Step 4: Apply patch via SSH

SSH to your ESXi host using your favourite SSH client.  In my case I use PuTTY. From here, browse to the directory where your patch is located, and run the esxupdate utility.a

  1. SSH to the ESXi host
  2. Browse to the datastore location where your patch file is.  If your datastore is called “datastore1″ it should be located at /vmfs/volumes/datastore1
  3. run the esxupdate command with your filename (see below)
  4. Reboot the host
  5. Exit maintenance mode through the vSphere Console

The command to update is as follows:

esxupdate --bundle=ESXi410-201107001.zip update

The output should look something like this:

/vmfs/volumes/4be1f209-78ae1314-33c5-001b213d3c53/Patches # esxupdate --bundle=ESXi410-201107001.zip update
Unpacking deb_vmware-esx-tools-light_4.1.0-1.8.433742                         ################# [100%]
Unpacking deb_vmware-esx-firmware_4.1.0-1.8.433742                            ################# [100%]
Removing packages :vmware-esx-tools-light                                     ################# [100%]
Installing packages :deb_vmware-esx-firmware_4.1.0-1.8.433742                 ################# [100%]
Installing packages :deb_vmware-esx-tools-light_4.1.0-1.8.433742              ################# [100%]

The update completed successfully, but the system needs to be rebooted for the
changes to be effective.

Once the patch has completed installing execute the reboot command to reboot the host.

There you have it.  Your host will now be updated.  You can verify by checking the build number on the console, or the vSphere Console.

Goals for 2011

For the past few years I have set “open ended” goals and targets for myself to achieve during the course of the year.  I start each new year full of energy and ideas on what I should be doing as far as career, certification, and my personal life.  However I am a world class procrastinator and as such, am annually disappointed by my lack of achievement of these goals (a procrastinator such as myself will put off being disappointed until it’s absolutely necessary, such as a new year’s resolution list!)

As such, I’ve created a new heading on my blog to track my Goals for 2011 (albeit belated!).  Thanks to Aaron for the idea over on his blog.

Take a look at my Goals for 2011.  I’ll update as I (hopefully) complete them.

New hosting

After being plagued with downtime issues with arkf.net, I’ve moved the blog off of my home cable connection onto a dedicated hosting provider.

Originally, I was hosting the website on an Ubuntu 10.04 based virtual machine in my home VMware lab.  The problem with this setup is, well, it’s a lab.  Accordingly, I break it regularly, sometime intentionally, sometimes not, making it a less than ideal environment for a blog with some semi-regular readers.  In addition to the stability issues, my home cable connection’s upload speed is slow (500kbit) which is too slow for hosting a blog like this.

Fingers crossed everything will be okay going forward!  Now I just need to learn to post more regularly…

Also, I recently bid on and won the “arkf.com” domain.   For now I have set this up to forward to www.arkf.net. An interesting trip into the back water world of domain auctioning.

Merry Christmas all!

Merry Christmas everyone, see you in the new year!

And now for some Techie Xmas humor:

And as a side note, arkf.net is now 1 year old!

ZFS SAN: Growing pains

As I mentioned previously, I’m outgrowing my existing OpenFiler SAN which I’ve been using for several years now.  It has a capacity of ~3.5TB and is used as an iSCSI target for my VMware Lab, as well as general file storage.  It has been plagued with consistency problems as growing pains, requiring major “fork lift” upgrades with substantial down time any time I want to expand the RAID array or install updates.  With the holiday season rapidly approaching, and some quality time off coming my way, I think it’s time to work on a new SAN and retire the old one to a backup and non-”system critical” role.

Below is a list of the primary issues I have with the existing system, and issues that I would ideally like to be able to resolve:

  1. Running out of storage with no room for additional disks
  2. Inflexible Linux software RAID-5 configuration; difficult to modify/grow an existing RAID array
  3. Problems with snapshotting with present configuration (partly my fault, but also poorly implemented in OpenFiler)
  4. The system is LOUD so it must be continuously powered down for the sake of sanity
  5. No updates from the OpenFiler project in years, looks like the project is dead
  6. Issues with data integrity.  I have had issues with corrupt data when the power goes out.
  7. Inability to verify data integrity without using something like MD5 hashes
  8. No free PCIe slots.  I have a Fiber Channel HBA I would like to play with

This obviously leads into my next question; namely what do I want from new new system?

  1. Plenty of drive expandability and room to grow in the future
  2. Ability to upgrade existing drives/volumes when larger disks become available
  3. Ability to verify data on disk and prevent “bit rot”
  4. Painless snapshotting
  5. A quiet system.  This would allow the system to remain powered on 24/7
  6. Email notifications for failed disks, disk integrity issues, and power outage notifications
  7. Data redundancy, with “zero” downtime
  8. Upgradability, with “zero” downtime
  9. Plenty of hardware upgradability
  10. Service my needs for the next 4-5 years (base hardware that is)

Quite an extensive list indeed.  While some of the items might appear minor (eg: noise), this system will live in the basement and shouldn’t cause auditory unpleasantness.

I will be using some of the hardware from the existing SAN for the time being to allow me to save some money in the short term.  In particular I will be using the systemboard, RAM, and CPU.  The existing Promise RAID cards will remain in the old SAN and I will be using a new LSI SAS 6.0G/s HBA (non-raid) for the new system.  More on this later.

OpenFiler is a linux based storage appliance and has served me well.  However with lack of updates for years and issues with expandability and data integrity, I will be replacing it in favor of Solaris combined with ZFS as my Storage OS.  In the next post I will talk a bit about ZFS and why it’s such a great fit for my requirements.

Lack of posts

G’day All.

Apologies for the lack of poting recently.  I started a new job recently and have been very busy getting up to speed there.  I’m still working in a Systems Engineering/Architecting role, however I have moved away from an AD/Exchange/Citrix focus and into a storage and virtualization focus; right up my alley!

I’m working on a few personal projects and related blog posts at the moment.  While my focus remains on Vyatta and VMware, I’ll be doing some “personal projects” including upgrading my aging SAN at home.  Presently I have a 2U TYAN server with a total capacity of ~3.5TB (6 drive trays), and running OpenFiler, a linux based iSCSI/NFS/SMB appliance distribution.  Overall I have been quite happy with OpenFiler, but recently I’ve had varying issues with the setup, including running out of space for VMware.  My plan is to install a new server using a 4U Norco 24 drive tray server case, and install Solaris with ZFS and COMSTAR being used for storage.  The existing TYAN server will be retired to a simple “backup server” role.

More on this soon, I promise!

Vyatta OpenVPN Site-to-Site from Behind NAT or Firewall

Vyatta allows several methods for establishing site-to-site VPNs, namely IPSec and OpenVPN.  IPSec is probably the most efficient of the two protocols as it operates at Layer 2 however in most cases it requires both endpoints to be able to see each other.  What about establishing a vpn connection between two hosts, where one host is hidden behind NAT or a firewall?  OpenVPN can accomplish this with ease, at the small performance cost of being a Layer 7 protocol.

Vyatta allows you to configure OpenVPN in two ways; Remote Access and Site-to-Site (there is actually a third option, but I believe at writing this is a subscription only method).  Remote Access is designed for “Road Warrior” setup’s where a remote PC user needs to access a network, and operates in a one-to-many topology.  It only allows routing of subnets on the server side.  Site-to-Site, as the name suggests, allows you to connect 2 remote routers together, and allows routing of subnets on both ends.

OpenVPN uses two different forms of authentication; pre-shared key(PSK), or TLS mode using certificates (Public Key Infrastructure or PKI).  I tend to prefer using TLS mode but this requires additional steps which I will cover in my next article.  Here we will use a pre-shared key.

Configuration Lab

The following is a complete lab that will walk you through configuring an OpenVPN Site-to-Site VPN using two Vyatta routers (router-1 and router-2).  We will also configure a third vyatta router (nat-fw) to act as the NAT Firewall.  I suggest using a VMware Workstation team, or an ESXi server to create your topology.

Above is the topology we will be using.  Router-1 is “Internet” facing and Router-2 is sitting behind a NAT firewall that allows outbound connections.

NOTE: I will outline below all configuration required to get this lab up and running.  If you are just interested in the OpenVPN stuff, skip to the “Generating Pre-Shared Key” section.

Lab Pre-configuration

The following commands should be run on each respective router in your lab prior to proceeding with any of the OpenVPN configuration.

Router-1

set system hostname router-1
set interfaces ethernet eth0 address 10.0.0.1/24
set interfaces ethernet eth0 description "Router-1 WAN"
set interfaces ethernet eth1 address 192.168.100.1/24
set interfaces ethernet eth1 description "Router-1 LAN"
set service nat rule 10 outbound-interface eth0
set service nat rule 10 source address 192.168.100.1/24
set service nat rule 10 type masquerade
commit

Router-2

set system hostname router-2
set interfaces ethernet eth0 address 192.168.15.22/24
set interfaces ethernet eth0 description "Router-2 WAN"
set interfaces ethernet eth1 address 192.168.200.1/24
set interfaces ethernet eth1 description "Router-2 LAN"
set service nat rule 10 outbound-interface eth0
set service nat rule 10 source address 192.168.200.1/24
set service nat rule 10 type masquerade
commit

NAT Firewall

set system hostname nat-fw
set interfaces ethernet eth0 address 10.0.0.2/24
set interfaces ethernet eth0 description "NAT-FW WAN"
set interfaces ethernet eth1 address 192.168.15.1/24
set interfaces ethernet eth1 description "NAT-FW LAN"
set service nat rule 10 outbound-interface eth0
set service nat rule 10 source address 192.168.15.1/24
set service nat rule 10 type masquerade
commit

If everything is configured correctly, router-2 should be able to ping router-1′s WAN interface (10.0.0.1).

Generating Pre-Shared Key

Now that we have our lab prepared, the first things first we need to do is generate our pre-shared key:

vyatta@router-1:~$ vpn openvpn-key generate /etc/openvpn/key.psk
vyatta@router-1:~$ ls -l /etc/openvpn/key.psk
-rw------- 1 root root 636 Oct 17 19:05 /etc/openvpn/key.psk

This generates a file (/etc/openvpn/key.psk) that both systems will use to verify each other’s identity prior to establishing the VPN connection.  Thus we need to copy it from router-1 to router-2 via secure means.  I would suggest using Secure Copy (SCP) from router-2.  In order to do this we need to configure router-1 with an SSH server.  Note that the file created is owned and readable by root only, so our ssh server will need the “allow root” option, and we will need to provide a password for the root account.  From configuration mode, execute the following commands:

set service ssh
set service ssh allow-root
set system login user root authentication plaintext-password [PASSWORD]
commit

Now from router-2 we need to copy our key.psk file from router-1 using SCP.  As the file needs to be owned by root, we need to sudo on router-2 prior to running SCP:

vyatta@router-2:~$ sudo -s
root@router-2:~# scp 10.0.0.1:/etc/openvpn/key.psk /etc/openvpn/key.psk
The authenticity of host '10.0.0.1 (10.0.0.1)' can't be established.
RSA key fingerprint is bb:0d:48:3d:13:61:71:60:d2:13:7b:c9:cc:a7:52:d9.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.0.1' (RSA) to the list of known hosts.
Welcome to Vyatta
root@10.0.0.1's password:
key.psk                                       100%  636     0.6KB/s   00:00
root@router-2:~# ls -l /etc/openvpn/key.psk
-rw------- 1 root root 636 Oct 17 19:15 /etc/openvpn/key.psk
That’s it!  We now have our pre-shared key on both routers and are ready to configure our site-to-site OpenVPN.  As mentioned earlier, I’ll do a follow-up post using TLS PKI certificates.

Router-1 Configuration

Run the following configuration commands on router-1:

set interfaces openvpn vtun0
set interfaces openvpn vtun0 mode site-to-site
set interfaces openvpn vtun0 local-address 172.16.100.1
set interfaces openvpn vtun0 remote-address 172.16.100.2
set interfaces openvpn vtun0 shared-secret-key-file /etc/openvpn/key.psk

Your configuration should look something like this:

vyatta@router-1# show interfaces openvpn
 vtun0 {
     local-address 172.16.100.1
     mode site-to-site
     remote-address 172.16.100.2
     shared-secret-key-file /etc/openvpn/key.psk
 }

One important thing to note is that I have not included a “remote-host” configuration item for router-1.  This is because router-1 cannot directly communicate with router-2, thus the “remote-host” entry would be useless.  Router-2 will be responsible for initiating the OpenVPN connection.

Router-2 Configuration

For router-2 we will have essentially the same configuration with local and remote addresses being reversed, but we will also use the previously mentioned remote-host entry. This attribute will accept either an IP address, or a DNS name (great for DynDNS configurations!). Run the following configuration commands on router-2:

set interfaces openvpn vtun0
set interfaces openvpn vtun0 mode site-to-site
set interfaces openvpn vtun0 local-address 172.16.100.2
set interfaces openvpn vtun0 remote-address 172.16.100.1
set interfaces openvpn vtun0 remote-host 10.0.0.1
set interfaces openvpn vtun0 shared-secret-key-file /etc/openvpn/key.psk

Your configuration should look something like this:

vyatta@router-2# show interfaces openvpn
 vtun0 {
     local-address 172.16.100.2
     mode site-to-site
     remote-address 172.16.100.1
     remote-host 10.0.0.1
     shared-secret-key-file /etc/openvpn/key.psk
 }

Once we enter “commit”, router-2 will automatically try to establish an OpenVPN connection with router-1.  If successful, you should be able to ping 172.16.100.1 (router-1′s vtun0 address) from router-2, and vice versa.

Static Routes

Finally, in order for our respective router’s LAN subnets to be able to communicate, we need to configure static routes on both routers, telling each where to find each other’s subnets:

Router-1

set protocols static route 192.168.200.0/24 next-hop 172.16.100.2
commit

Router-2

set protocols static route 192.168.100.0/24 next-hop 172.16.100.1
commit

You should now be able to ping 192.168.200.1 from Router-1, and 192.168.100.1 from Router-2.  Congratulations! You have successfully configured a Site-to-Site OpenVPN connection from behind a NAT firewall.

Troubleshooting

You can watch the VPN nodes establish connections, as well as view any errors or problems by using the “tail -f /var/log/messages” command.  For example, here is what a successful connection should look like:

vyatta@router-2:~$ tail -f /var/log/messages
Oct 17 19:50:20 vyatta openvpn[8610]: Data Channel MTU parms [ L:1544 D:1450 EF:44 EB:4 ET:0 EL:0 ]
Oct 17 19:50:20 vyatta openvpn[8610]: Local Options hash (VER=V4): '2e1fe9d6'
Oct 17 19:50:20 vyatta openvpn[8610]: Expected Remote Options hash (VER=V4): '57bb0ab0'
Oct 17 19:50:20 vyatta openvpn[8623]: Socket Buffers: R=[112640->131072] S=[112640->131072]
Oct 17 19:50:20 vyatta openvpn[8623]: UDPv4 link local (bound): [undef]
Oct 17 19:50:20 vyatta openvpn[8623]: UDPv4 link remote: [AF_INET]10.0.0.1:1194
Oct 17 19:50:24 vyatta openvpn[8623]: Peer Connection Initiated with [AF_INET]10.0.0.1:1194
Oct 17 19:50:25 vyatta openvpn[8623]: Initialization Sequence Completed

Stay tuned for my next article on configuring TLS certificate based authentication for OpenVPN, using the easyrsa scripts.

Also, shout-out to Roggy for the original client/server idea that led me down this path! Take a look at his original post here: http://roggyblog.blogspot.com/2010/03/managed-service-provider-using-vyatta.html.