Configuring SNMP on vSphere ESX 4.0 and Up

Since SNMP is embedded in the hostd on ESX 4.0 and up, configuration is somewhat different.  We had several hosts that were already configured but ran into some issues when we added a new ESX host to our Data Center.  The below commands are what we ultimately used (along with how we fixed the error).

First – Open ESX Firewall
esxcfg-firewall -e snmpd

Next start configuring SNMP.  Per the vSphere Admin Guides, these commands can be combined, but I broke them out for testing purposes.

vicfg-snmp.pl –server <server>–username <user> –password <pass> -p <port>
<server> – your ESX Server
<user> – a user with root access on your server (probably root in most cases)
<pass> – password for the user you supplied
<port> – if you are running a default SNMP install, this will be 161.  Some documentation will have you change this to other ports, make sure you open those ports on your firewall.

vicfg-snmp.pl –server <server>–username <user> –password <pass> -c <community>
<community> – your SNMP community string

vicfg-snmp.pl –server <server>–username <user> –password <pass> –enable
This turns on SNMP.  You should now be able to walk or model your host.

vicfg-snmp.pl –server <server>–username <user> –password <pass> -t <monitorserver>@<monitorport>/<community>
<monitorserver> – the server you send traps to.  This can be an IP or a host name.
<monitorport> – the port your monitor server listens for traps on. The default is usually 162.
<community> – this is the community string that your monitor server is configured to accept traps from.  It may not be the same as your normal community string.

If on any of the commands you get the error:

Failed : A general system error occurred: Load persistant store failed

This probably means your /etc/vmware/snmp.xml is corrupt or has a bad tag.  To fix this, do the following:

Rename /etc/vmware/snmp.xml to snmp.xml.old (or some other unique name)
mv snmp.xml snmp.xml.old

Now create a new snmp.xml
vi /etc/vmware/snmp.xml

Paste the following into the file.

<config>
<snmpSettings>
<communities>public</communities>
<enable>true</enable>
</snmpSettings>
</config>

Be sure that you don’t have any extra line breaks or white spaces before or after the text.

Save the file.
:wq

Restart VMware Management
service mgmt-vmware restart

After the service restarts, wait about 5 minutes for everything to come back online and run the commands again.

Posted in System Administration, VMware | Tagged , , | 1 Comment

VMware Virtual Disk Descriptor Problems

Upgrading applications and operating systems can be a real pain. On more occasions than I’d like to count, I have seen problems during these activities. The problems generallly fall into a few root causes the most common being: system errors, documentation, configuration anomylies and, dare I say it, Administrator mistakes. A solid Change and Release Management process can mitigate many of these issues, but that is another topic altogether.  One feature I really like about VMware is the ability to take a snapshot of a Guest OS prior to upgrades or to test a configuration change. The option of quickly rolling back to a known good state can really reduce risk and help alleviate those late nights rebuilding servers.  Despite the huge benefit snapshots bring to Administrators they can also be the source of problems, as I recently experienced.

Earlier this week, I was preparing to upgrade an application running on a virtual server. The application houses all of our Service Desk tickets as well as our Development Team’s Application requirements, tasks and test activities. The last time we upgraded this application was over 1 year ago. When we upgraded it last, we had created a snapshot of the VM prior to the upgrade, which was the same plan I would follow now.  As I prepared for this new upgrade, I shut down the server and attempted to create a snapshot.  The Infrastructure Client immediately through an unknown error.  I checked the Events associated with this server and saw that it had not been rebooted in nearly 3 months, which was good – because I knew that it *had* been rebooted.  I then checked the settings of the Guest VM to ensure that it was right, everything seemed good.  So knowing that the VM had been working and was now not – I took a pause to collect myself.

I opened the Snapshot Manager and, to my horror, saw that the snap shot from the previous upgrade was stacked under two other snapshots.  The Parent Virtual Disk was dated almost 2 years ago and the most recent snapshot was 13 months old.  Given that the Parent disk was 50GB, and the sum of the snapshots was almost 20GB my heart started to sink.  A quick browse of the datastore told me exactly what the problem was.  The virtual disk descriptor files were gone!  Without those files the ESX server would have no idea how to mount the snapshots or the virtual disk.  Losing the descriptor files of a single virtual disk is not a crisis, losing the the same files for snapshots does push that envelope.

I’ll jump to the end of the story first, because it took me nearly 24 hours to fix the problem.  I did fix it though, and got the server back up before production started on Monday.

Now that we have a happy ending, let’s talk about the technical stuff.  Read on!

Continue reading »

Posted in System Administration, VMware | Tagged , , | Leave a comment

SNMP Proxy on VMware

On ESX 3.5 setting up SNMP was as simple as configuring net-snmp and opening the firewall to allow traffic. With the introduction of ESX4, the VMWare SNMP information was embedded into the hostd. The SNMP embedded into the hostd only sends VMware traps and responds to requests for VMware specific requests. If you are interested in the OS vitals as well, you need to setup up net-snmp to proxy the SNMP information.

The monitoring solution we use, Zenoss, allows us to poll for VMware and Server Vitals. In fact, if we don’t poll for server vitals, Zenoss tends to throw errors. The end result was that I need to use the proxy setup to get the information I need. I read through VMware’s KB, Admin Guide and community posts while trying to understand how to set this up. Needless to say, there seems to be quite a bit of confusion and frustration around the process.

In order to set up the SNMP configuration on ESX4, you will need to download the VMware vSphere™ CLI application onto your workstation. As of today, the file can be found here: http://www.vmware.com/support/developer/vcli/
Continue reading »

Posted in System Administration, VMware | Leave a comment