I always enjoy going to the quarterly Omaha VMware User Group (VMUG) meetings if for no other reason that I never know who I am going to meet or what I am going to learn. This meeting was no exception. While more sparsely attended than the last meeting (~ 60 people attendance), the stories they shared illustrated to me that most organizations are still years away from fully virtualizing all servers in their data centers.
Overall I was impressed by the progress that attendees are making in virtualizing their environments though, considering this is a VMUG event, it only makes sense that these attendees would be the first ones on board with any new VMware release. It appeared the majority of them were using vSphere 4.0 and about 20% were already using vSphere 4.1 in some capacity in their environment.
The speed of adoption and who was adopting did catch me somewhat off guard. In talking to one IT director of a church in Lincoln, NE, who was in attendance, he said that adoption of server virtualization in non-profits has been extremely aggressive.
At a recent conference that he attended, fully 85% of those IT directors of churches in attendance at that conference had already virtualized their environments and the other 15% were planning do so. However he partly attributed the speed of adoption to the hefty discounts that server virtualization providers offer to 401(c)(3) organizations (about 70% off of the retail price.)
Yet it became clear in talking to the individuals in attendance that while most wanted to virtualize every system that they managed as quickly as they could, there are obstacles that will preclude most of these organizations from virtualizing a number of their applications for the foreseeable future.
- A network administrator for the city of Omaha lamented that he has an 11 year old computer system running a city payroll program and cannot get funds to upgrade it to a more current system. He found this particularly ironic in light of the fact that the city of Omaha just instituted a city wide 2% tax on all restaurant sales to cover a shortfall in the pension fund for firemen and policemen. However the computer system that spits out those pension checks is six years out of date with no funds currently budgeted for an upgrade.
- The IT director of the church in Lincoln, NE, tells me that the phone system his church uses still runs on DOS-based PC and its interface with the church switchboard is an 8-bit ISA card. Most of the parts for the PC are no longer available and while he has been finding them as needed on eBay, due to the age of the parts even eBay is drying up as a source.
- Video surveillance is becoming more critical for some in attendance but virtualizing video surveillance applications really isn’t practical due to the number of feeds coming in and the write traffic that these feeds can generate so it looks like this will remain a stand alone app for the time being.
Other reasons that organizations might delay server virtualization or at least that might give them some pause has to do with troubleshooting issues in VMware itself. The session I attended was presented by Nathan Small, a staff engineer for VMware who handles escalated support calls on the ones related to storage. Nathan’s name is one you might want to remember as he shared that the storage support team at VMware takes double the number of #1 priority calls of any other support team at VMware.
In listening to his presentation on VMware Advanced Root Cause Analysis, it is not surprising that his team gets so many calls related to storage. Most of his presentation focused on knowing what log files were important and then decoding the cryptic messages stored in them.
While many of these log files were located in the /var/log directory in vSphere, what error messages were stored in specific log files could vary according to what version of vSphere you were running. vSphere 4.1 introduced some new log files so iSCSI error messages that may have been written to the vmkernel.log file in vSphere 4.0 may now be written to the vmkisciid.log file in vSphere 4.1.
He also warned those companies that were running both vSphere and ESXi that the names of the error logs in those operating systems were not the same. While vSphere writes messages to the vmkernel.log file, in ESXi that file is simply called “messages.”
ESXi administrators will also want to think twice about rebooting a VMware ESXi file without first saving the logs. In the case of the vmkisciid.log file on the ESXi server, that file is cleared on an ESXi reboot. So while a reboot may solve the immediate problem, if one goes back to determine the root cause of the problem, it may be impossible to diagnose if the problem is somehow related to the iSCSI driver.
All in all, most of the users in attendance were glad they had virtualized their environment and could not envision going back to a physical environment. If anything, they were looking forward to the day where their 11 old computer payroll system and DOS-based phone systems with 8-bit ISA cards were a thing of the past. But as VMware’s Small illustrated, using VMware is not without its caveats especially when it comes to decoding the error messages in VMware’s log files.