Blumetech's Tech Blog

Great VMware Health Check Script

If you are looking for a great script to check the health of your VMWare environment, please take a look at William Lam's script at http://communities.vmware.com/docs/DOC-9842

I run his script weekly to generate a nice series of reports about the status of our VMware enviroment. I've been using the script for two years and since then it has helped identify many problems before they became outages. Some examples are

1. Old Snapshots
2. Low disk space on datastores
3. VMs misconfigured to use local datastores instead of fiber channel datastores
4. Misconfigued NTP settings on ESX Hosts
5. VMs with attached floppies and CDs preventing vMotion migrations
6. SCSI reservation locks on fiber channel LUNs
7. Datastores with too many or too few VMs
8. Misc. ESX errors
9. Misc. VMware vCenter Server errors
10. Out of Date VMWare Tools

All the information generated by the report, can be accessed independently. The script does a great job consolidating the information into one simple report. I highly recommend using it.

NAS Analysis – Windows 2008 vs Openfiler vs Nexenta vs Red Hat vs Windows with OpenText – Part 1

NFS based storage is gaining a strong following in VMware environments. NFS has a lot of big pluses for storage of VMDKs especially in environments that lack the money or expertise for a full fiber channel SAN but still want to leverage some great ESX features like vMotion and High Availability.

In the NAS arena, there are two broad categories of vendors

1.       The Giants – This is EMC, NetApp, and IBM. Prices are high but level of features, functionality, availability and support are also great.

2.       Everyone Else – These could be your traditional servers running an operating system that you probably already know, e.g. Windows or Linux. Price is fairly low and feature set is pretty broad. Or these could be dedicated NAS boxes. Very cheap with questionable performance. Drobo and Overland are good examples.

Many SMBs simply can’t afford the complexity and cost that are associated with “The Giants”

One fairly cheap and easy to understand NAS solution for SMBs is to simply buy a standard server, e.g. Dell, HP, Super Micro, install your OS of choice, and then finally export your NFS mount points.

Analyzing the features of the different OS 's is fairly straightforward and can be done simply by reviewing the online documentation and datasheets.

I wanted to understand if any of the operating systems performed better with NFS workloads. As a side note I was also curious if Windows 2008 would outperform a typical Linux distribution.

I've been spending my spare time over the last few days setting up the lab for testing. I've run into my first roadblock which was actually suprising. The first interation of my NAS server is Windows 2008 R2 running on a HP ML570 G4. I've exported an NFS mount and created a datastore on my ESX host. From there I could easily create VMDKs for my VMs on the NFS Storage. Here is a good article on setting up a Windows 2008 server for NFS and ESX http://ross-family.org/blog/2009/07/08/how-to-use-windows-2008-to-host-storage-for-esx/

What surprised me was how well Windows 2008 performed in the first round of tests. By doing simple file copies in and out of the VMDK I was able to saturate the GigE link on the NAS box. See screenshot below. Given the number of people online complaining about slow NFS performance in Windows, I was happily surprised by the results. I think if given the choice most SMBs would choose Windows over Linux for their server OS simple because it's an OS most SMBs are familar with. Let's see if future tests confirm these initial good performance numbers.

I am going to upgrade the link to 10Gig-E. Bonding the two NICs together into a single channel would not improve performance. Traffic is being generated for my test from a single ESX Host and hence all that traffic would go over a single 1Gig-E link. I could setup another ESX Host and test both simultaneously, but going to 10Gig-E, since it is available to me, is much easier.

Nasuni and Cloud Storage

I recently evaluated what looked like a great small business product from Nasuni. Cloud storage is a hot marketing term right now, but I haven't been able to find a good fit for your typical SMB. Nasuni looked like it might just work.

Nasuni goal is to provide a low cost NAS (Network Attached Storage) with unlimited storage. Their solution runs as a simple VM and can emulate a Windows File Server. Users simply map a drive and start dropping in files. On that VM is a local cache up to 512GB that stores files. As the cache fills up files are moved to from the cache and put onto storage array at a storage cloud provider. Most of the majors, e.g. Amazon S3, Nirvanix, Rackspace, etc. are support. If a file is need that isn't in cache, Nasuni connects to the cloud and downloads the file. Since you are effectively using cloud storage you get

1. Unlimited storage -- No need to buy any more disks. No need to pay for power, support costs, replacement costs, etc. for all those disks.
2. Redundancy  -- The cloud provider automatically replicates all the data across multiple datacenters. This data is highly available. Amazon boasts 99.99% available. (Don't confuse availability with durability. Availability is how likely your storage will be available. Durability is how likely your data will be lost)
3. Disaster Recovery -- If your primary site goes down, fire up a Nasuni box at your DR site and access all your data from the cloud. The only data you may have lost is what was in cache at the time of the disaster.

Nasuni also takes care of a view other critical items

1. Security -- All data stored in the cloud is encrypted. Even if someone had access to the encrypted data, the data is useless without you key stored locally on the Nasuni box
2. Backup -- Nasuni provides effectively unlimited snapshots so you can roll back files to previous versions at any time. You can do single file and whole volume restores

Unfortunately there are two big problems with the product

1. No Delete -- I was shocked by this. Apparently you can't permanently delete a file. If a user, deletes a file from the file system, it certainly disappears. However it is still stored in previous snapshots and hence isn't truly deleted. There are two ramifications to this 1. if you have a legal requirement to delete the file, then you will not be able to meet it and 2. you still pay for cloud storage even on files that you deleted.
2. Costs -- Nasuni itself is inexpensive. $3000 per year which includes your upload and download fees to Amazon. (Please note that once you go beyond a certain size which I believe is 16TB, the pricing model changes and Nasuni wants you to pay the upload\download fees.) The real cost is the storage cost...

Today I just purchased rather expensive SATA drives from a major storage vendor. I won't give the name but I can assure you they are in the big 3 and their drives are NOT cheap. Total cost was $2292/USABLE TB. We replace drives every 4 years on average. We pay roughly 20% of drive costs for a premium support contract which includes everything necessary to support an maintain these drives.

That usable storage number includes the RAID overhead (RAID 6), file system, etc. It represents the true usable storage to the end user. The cost doesn’t include the array head or if doing NAS the filer head. Although this cost is significant, I am ignoring it because I just as easily could offset that cost with buying drives from a cheaper storage vendor, e.g. Polycom, Nexsan, etc. Drives from those companies vary, but you can easily get under $1000/Usable TB 

You could account for the fact that Nasuni is “pay-as-you-use” model, whereas the SATA solution you must buy storage upfront. Here I’m assuming that SATA storage will be purchased as needed without a bulk up front purchase where the storage sits idle. Getting storage utilization rates over 70% is not atypical in most SMB storage environments

To continue the comparison with Nasuni, I can double the price by purchasing drives at a DR site. Of course there is the DR site costs to consider, but Nasuni isn’t going to obviate the need for a DR site. It only solves the problem of file storage. There are many other business applications, e.g. database and mail, that won’t be able to leverage Nasuni for DR and hence will need some sort of DR solution. Here I’m just assuming a DR site and replicating the storage over. Again the cost doesn’t include the cost of replication software, additional bandwidth, and all the other pieces for DR. But I could also buy cheaper drives at the DR site. I also have included the 20% premium support contract for the drives at the DR, which is something I probably wouldn't do for the DR site as a premium support contract is unnecessary. Next-business day would have been fine.

All in all, $5500/Usable TB ($2292 * 2 for DR * 1.2 for support contract) is at least a close approximation to the total 4 year drive cost. It provides a much faster IO solution than Nasuni as the SATA drives will always serve data faster than Nasuni cache misses, which need to download the data over the Internet using whatever bandwidth is available.

As of today, 1GB of cloud storage on Amazon S3, the cheapest major cloud storage provider, costs $.15/month.  Therefore over 4 Years the cost per Usable TB is ($.15/Month * 1024 GB/TB * 48 Months) $7200.

There’s a lot of margin between the local SATA drives and the Amazon S3 solution. One could add a lot of costs to the SATA solution for personnel, DR, power, cooling, etc. etc. and still not get to the $7200. And even if that new cost surpasses the cloud storage costs, the local SATA solution is still faster and we would then need to move to a cost/IO discussion.

Nasuni really looked like a great solution. The interface is easy to use and the setup was quick and simple. I love what they have done creating a hybrid solution between local and cloud storage. However the cloud storage pricing, makes it cheaper for most SMBs to simply buy a Windows server (or better yet a free open source NAS) and some low-end drives.

PowerShell Script to VM Snapshots Older than X Days

I frequently see clients forget to delete snapshots after they are created. Over time as more and more data is changed on the VM, the size of the snapshot keeps growing. This growing snapshot size creates several problems

1. VMs tend to run slower when using snapshots especially those VMs that are disk intensive. The snapshot image needs to be updated for each write IO. So if you have 4 snapshots of a VM, that leads to 4 writes in comparison to a single write to the primary VMDK file with no snapshots
2. When left unchecked, snapshots can grow to use up the entire datastore. When the datastore runs out of space, your VMs will likely all crash. At a minimum new VMs will not be able to start as space is required to create the VSWP file.
3. When creating linked snapshots, i.e. a snapshot of a snapshot, you can quickly run out of space even when removing the snapshot. (Note: This issue is resolved in ESX v4.1 and later) For example if you have 4 snapshots all linked together at 10GB each and try to delete all the snapshots, you will need signifcantly more than 40GB before all the snapshots are deleted. That is because each snapshot needs to be copied into the previous snapshot and all these copies need to finish before even a single snapshot is deleted. For more information http://www.yellow-bricks.com/2010/07/05/changes-to-snapshot-mechanism-delete-all/
4. There is a risk especially with large snapshots of data corruption. It is exceedingly rare but can occur anytime you have multiple linked snapshots and the link fails or when you run out of disk space.

To avoid all of these potential problems, I've written a simple script that I use to e-mail me when a snapshot is getting old and either 1. someone forgot to delete 2. is probably using up too much space or 3. is making the VM run slower than normal.

To run the full script you'll need Windows Powershell v1 or better, which comes with Windows 2008 and Windows 7, VMware CLI, and the Powershell Community Extensions. You only need PSCX if you plan on using the email functionality of the script.

Here is the Script

Get-VM | Get-Snapshot | Where { $_.Created -lt (Get-Date).AddDays(-15)} | FL | Send-SmtpMail -to [email protected] -Subject "Virtual Machines with Snapshots Older than 15 Days" -Body $_

The details...

• Get-VM -- Gets a list of VMs
• Get-Snapshot -- Gets a list of all the snapshots on the VMs if any snapshots exist
• Where { $_.Created -lt (Get-Date).AddDays(-15)} -- This is really the meat of the script. Here the where-object, a.k.a "where", selects only those snapshots that were created ($_.Created) more than 15 days ago ((Get-Date).AddDays(-15)). You are free to change the number of days.
• FL -- Format-List formats the result into an easy to read table
• Send-SmtpMail -- Sends me an e-mail message with the results. If the body of the message is blank, no snapshots are older than 15 days.

I then schedule a batchfile to run the job weekly. I use Automate, but any good job scheduling tool will work just as well, e.g. Windows Task Scheduler, cron, etc.

There's a lot that could be done to make this a little more sexy, for example, instead of using a date use a snapshot size, only send a mail when there are snapshots older than 15 days, and make the number of days a variable instead of hard coded.

Some important reminders on snapshots...Don't use snapshots as a means for long-term backup. VMware snapshots are designed to be deleted in a short time frame, i.e. a few days at most, and hence are not an appropriate solution for long term storage.

Also, snapshots are stored with the VMX file and not necessarily with the VMDK you are snapshotting. Too often I see someone look at a datastore that has plenty of free space, take a snapshot, and then run out of space on a different datastore. They never realized that the snapshot was being stored on a separate datastore with the VMX file.

Auto-Installation of Adobe Reader

Recently, Adobe has been keeping pace with Microsoft for most frequently patched product. It seems like Adobe Reader is coming out with a new version monthly. Some of the security holes with the product are severe. In the worse cases a simply opening of a malformed PDF can allow complete administrative control of the machine. The problem with PDFs get even worse when you consider:

1. The problems presists across all platforms. These flaws aren't just windows based but also apply to Macs as well.
2. Historically users have viewed PDFs as secure and will open PDFs from unknown sources simply because the user feels safe behind what they incorrectly think is a secure document format.

https://www.networkworld.com/news/2010/081910-patch-critical-security-flaws-in.html

http://www.networkworld.com/news/2010/100610-adobe-hits-reader-users-with.html?source=nww_rss

Here at iN DEMAND, we have Adobe Reader installed on every workstation. Every time Adobe comes out with a new release to patch some security flaw, we need to go around and update all the machines.

To speed up the proces we auto-deploy Adobe Reader. We use Script Logic's Desktop Authority to handle the running of the deployment command. Any means to remotely execute the installation will work...Powershell, Task Scheduler, GPO, or the ubiquitous login script.

The real trick to the deployment is the necessary command line switches required to install the program silently. This is what we use

AdbeRdr940_en_US.exe /sAll /rs /l /msi"/qb-! /norestart ALLUSERS=1 EULA_ACCEPT=YES SUPPRESS_APP_LAUNCH=YES

There are two groups of command line switches. The first set are for the iNOSSO extraction program. This program is used to extract the MSI file from the .EXE distribution file. The second set of switches, the ones after "/msi", are for the actual MSI install package.

You could conceivable download the EXE, extract it to the MSI, and then run just the MSI files during the auto-deployment. This will probably be faster for each deployment as the extraction step is skipped. However, I wanted the easiest possible administrative steps. The way I have it a help desk technician can download any new updates to a network share and then automatically update machines without doing any additional work.This is a very simple design that's difficult to break.

Background on the switches:

• /sAll - Silent Mode
• /rs - Supress reboot
• /l - Enable Error Logging
• /msi - Parameters for MSI
  
  • /qb-! - Silent Mode
  • /norestart - Supress reboot
  • ALLUSERS - Update All Users profile
  • EULA_ACCEPT - Acept license agreement
  • SUPPRESS_APP_LAUNCH - Do not launch Adobe Reader after the installation completes.

 

Microsoft Has Record Patch Tuesday

Yesterday marked a dubious record for Microsoft...49 security holes patched in a single month. There's no need to list all the fixes here. Some helpful links with background on the worse vulnerabilities and additional links to the source information at the Microsoft site...

http://www.networkworld.com/community/node/67336?hpg1=bn

http://www.networkworld.com/news/2010/101210-record-patch-tuesday-hits-older.html?source=nww_rss

Have no doubt that these vulnerabilities are very serious and effect even the latest and greatest software, e.g. Windows 7, Windows 2008, and IE v8. In the worse vulnerabilities, simply browsing a web page with malformed content is all that is required to allow full remote access to the machine. Expect widespread use of these vulnerabilities to spread viruses and malware.

I recommend applying the updates as soon as possible.

A Note on VMWare Thin Provisioning

In ESX v3.5 and VMWare for Windows, the default creation of a new hard disk would allocate the entire hard disk file immediately. If you created a 50GB C: drive, 50GB was immediately allocated from the datastore. This allocation occurred even if you only planned to put 10-20GB of real data into the virtual drive.

 

With ESX v4 you now have a new option, called Thin Provisioned. With this option you can provision a 50GB C: drive. Inside the VM, the VM “sees” a 50GB drive. However, the space allocated at the datastore level only reflects real data, i.e. the actual size of files stored on the virtual C: drive. For example, if you take this VM and install windows with roughly 10GB of files, only 10GB of data is used on the datastore. http://www.vmware.com/files/pdf/VMware-vStorage-Thin-Provisioning-DS-EN.pdf

 

The benefit is that you can now over provision your storage. Using the client below as an example. They only have 130GB for OS Storage. With 50GB C: drives the datastore could only hold 2 machines. Instead using the standard Windows install of 10GB and thin provisioning, you could now easily fit 5-10.

 

In the screenshot below I have provisioned a VM with 72GB, but only 9.6GB is being used on the datastore.

The downside is losing track of over provisioned machines could quickly catch you with no free space. Over provisioning only works if you don’t run ALL your virtual drives at near full capacity. If in the above example you take the 5-10 machines to the full 50GB you would end up using 250-500GB and hence run out of space.

Useful Remote Desktop Keyboard Commands

Here are some keyboard shortcuts that I have found very useful for remote desktop sessions

CTRL+ALT+Pause and CTRL+ALT+Break--Switch between full screen mode and a windowed remote desktop session.

ALT+INS--Similar to ALT+TAB, allows you to switch between applications in the remote desktop session

CTRL+ALT+END--Equivalent to sending CTRL+ALT+DELETE to the remote desktop. FYI--To send a CTRL+ALT+DELETE to a VM in a VM Console session press CTRL+ALT+INS

Why a Blog?

Why bother with a blog at all? There are a couple of reasons that I have for starting a blog:

1.       I feel that the small and medium sized business of less than 500 users is not in general getting the service they deserve from the IT services industry. My opinion is that the SMB market gets far too many “snake oil” salesmen and unskilled, unpolished engineers. Furthermore, the SMB market lacks the sophisticated IT skills to “see through” the FUD and get to real, effective, and efficient solution. Hopefully this blog will help SMBs make more intelligent and informed solutions.

2.       I’m constantly being asked to provide my opinion or advice on new technology and products. A blog is a great way to communicate that information.

3.       I’m constantly asked about the “right” way to design a solution. People frequently ask me to boil down a lengthy best practices document into something tangible for their particular environment.

4.       I love talking about technology whether it’s new products, solutions, or just technology in general.          

5.       I have a lot of information worth sharing with others. I think there is a sizable group of people interested with the information I’m providing

6.       A blog provides a digital archive of information and knowledge that I have accumulated, i.e. I can use the blog as a means to store old information that is valuable to me as well as others.

7.       It generates traffic for my wife’s company, Blumetech. She has a successful business providing IT services to small and medium sized businesses in NYC, Northern NJ, and Westchester. Helping her seems like a wise thing to do.

8.       A blog will help boost my professional career by reconfirming what’s already on my résumé.