Blumetech's Tech Blog

Veeam Backup to the Cloud

Veeam recently announced a new product that will allow SMBs to backup their vmware infrastructure to cloud based storage. This product offers a nice, clean way to combine vmware based hypervisor backup with a cloud based storage solution. 

As an added benefit the solution also support Amazon Glacier. At $.01/GB/Month SMBs can now easily replace their entire tape infrastructure. 

Combining the new Veeam Backup solution, Amazon Glacier, and Amazon EC2, SMBs have an extremely low cost, cold restore DR solution 

http://www.veeam.com/cloud-backup-vmware-hyper-v.html?ad=pr1

VMWare vSphere v5.1 Released

VMWare released version 5.1 of vSphere. It looks like a worthwhile upgrade.

Some of the improvements for SMBs:

1. Vmotion now no longer needs centralized storage. VMWare has combined storage vmotion and vmotion together so you can move VMs without any downtime between ESX Hosts with local datastores
2. Datastore replication is now included with vSphere so you can replicate datastores across the WAN without having to purchase SRM
3. VMWare tools upgrades after v5.1 no longer require a reboot. I find I am always several revisions behind in my vmware tools because of the reboot requirement. This new feature is a big plus.

See attached for What's New in v5.1 PDF Download Vmware-what-is-new-vsphere51

vSphere 5.0 Upgrade Status -- Rollout Script with vSphere Power CLI

VMWare released Update 1 for vSphere 5.0 this month. With that update and the success I've had with vSphere v5.0 on our test servers, I've decided to start the production rollout.

We have over 30 ESX Hosts. After two the rollout really started to get bogged down with lots of little configuration changes for each new host. I'm doing a fresh install of ESXi blowing away the old ESX install. I feel that this is a cleaner build. The problem with that route is now all the configuration for NTP servers, syslog, firewall ports, and the worse part...network configuration has to be done by hand.

The network configuration is the worse part. Lots of little peices need changes. About 30% of the upgrade time was spent just on the network configuration page. This would be a good time to upgrade to Enterprise Plus and get the host profiles features. Unfortunately that upgrade is way too much money.

Instead I wrote a script to do most of the configuration. I was surprised at how easy it was to write that I don't have much Power CLI expertise. It works very well. It took about 30 minutes to write and test. Running the script reduces my work from about 20 boring minutes per host to about 20 seconds. Definitely worth the time spent to write the script.

The script handles networking, syslog, ntp, and datastore configuration

Here it is...

$esx_host = "esx10.vcnyc.indemand.net"
$vmmotion_ip = "10.12.6.19"
$nfs_ip = "10.12.19.110"

$vswitch =  Get-VirtualSwitch -VMHost $esx_host -Name vSwitch0
New-VirtualPortGroup -Name VMMotion -VirtualSwitch $vswitch -VLanId 6
New-VMHostNetworkAdapter -VMHost $esx_host -PortGroup VMMotion -VirtualSwitch $vswitch -IP $vmmotion_ip -SubnetMask 255.255.255.0 -VMotionEnabled $true

#    Other policies http://www.vmware.com/support/developer/PowerCLI/PowerCLI41U1/html/LoadBalancingPolicy.html

New-VirtualSwitch -VMHost $esx_host -Name vSwitch1 -Nic vmnic2, vmnic3
New-VirtualSwitch -VMHost $esx_host -Name vSwitch2 -Nic vmnic4, vmnic5

$policy = Get-VirtualSwitch -VMHost (Get-VMHost $esx_host) -Name vSwitch1 | Get-NicTeamingPolicy
$policy | Set-NicTeamingPolicy -LoadBalancingPolicy LoadBalanceIP
$vswitch =  Get-VirtualSwitch -VMHost $esx_host -Name vSwitch1
New-VirtualPortGroup -Name NFS_Storage -VirtualSwitch $vswitch -VLanId 19
New-VMHostNetworkAdapter -VMHost $esx_host -PortGroup NFS_Storage -VirtualSwitch $vswitch -IP $nfs_ip -SubnetMask 255.255.255.0

$policy = Get-VirtualSwitch -VMHost (Get-VMHost $esx_host) -Name vSwitch2 | Get-NicTeamingPolicy
$policy | Set-NicTeamingPolicy -LoadBalancingPolicy LoadBalanceIP
$vswitch =  Get-VirtualSwitch -VMHost $esx_host -Name vSwitch2
New-VirtualPortGroup -Name VLAN001 -VirtualSwitch $vswitch
New-VirtualPortGroup -Name VLAN003 -VirtualSwitch $vswitch -VLanId 3
New-VirtualPortGroup -Name VLAN015 -VirtualSwitch $vswitch -VLanId 15
New-VirtualPortGroup -Name VLAN007 -VirtualSwitch $vswitch -VLanId 7
New-VirtualPortGroup -Name VLAN014 -VirtualSwitch $vswitch -VLanId 14
New-VirtualPortGroup -Name VLAN010 -VirtualSwitch $vswitch -VLanId 10

Add-VmHostNtpServer -NtpServer us.pool.ntp.org -VMHost $esx_host
$ntp = Get-VmHostService -VMhost $esx_host | Where {$_.Key -eq 'ntpd'}
Set-VMHostService -HostService $ntp -policy "automatic"
$ftpFirewallExceptions = Get-VMHostFirewallException -VMHost $esx_host | where {$_.Name.StartsWith('NTP Client')} 
$ftpFirewallExceptions | Set-VMHostFirewallException -Enabled $true

Set-VMHostSysLogServer -SysLogServer udp://vc1.vcnyc.indemand.net:514 -VMHost $esx_host
$ftpFirewallExceptions = Get-VMHostFirewallException -VMHost $esx_host | where {$_.Name.StartsWith('syslog')} 
$ftpFirewallExceptions | Set-VMHostFirewallException -Enabled $true

New-Datastore -VMHost $esx_host -Nfs -Name VM_VSWP01 -NfsHost 10.12.19.23 -Path /vol/VM_VSWP01
New-Datastore -VMHost $esx_host -Nfs -Name VM_OS01 -NfsHost 10.12.19.23 -Path /vol/VM_OS01
New-Datastore -VMHost $esx_host -Nfs -Name VM_DATA01 -NfsHost 10.12.19.23 -Path /vol/VM_DATA01
New-Datastore -VMHost $esx_host -Nfs -Name VM_DATA10K01 -NfsHost 10.12.19.23 -Path /vol/VM_DATA10K01
New-Datastore -VMHost $esx_host -Nfs -Name VM_VSWP02 -NfsHost 10.12.19.24 -Path /vol/VM_VSWP02
New-Datastore -VMHost $esx_host -Nfs -Name VM_OS02 -NfsHost 10.12.19.24 -Path /vol/VM_OS02
New-Datastore -VMHost $esx_host -Nfs -Name VM_DATA02 -NfsHost 10.12.19.24 -Path /vol/VM_DATA02

Preliminary Review on vSphere 5.0

I've started my research on our upgrade to vSphere v5.0. Right now we are running v4.1. My company is running about 200 VMs in 4 different physical locations. We have 15 ESX Hosts.

The plan has been for me to upgrade to v5.0 in the first quarter. Unfortunately after reviewing all the various upgrade guides, reading the documentation, and now about a quarter of the way through Master vSphere 5, I haven't been very impressed with the upgrade feature set.

Usually when I do a major upgrade I like to see at least one feature that really "WOWS" me. Something that makes it a real compelling reason to upgrade. I don't like to upgrade just for the sake of upgrading or to simply have something to do. For example, we are in the midsts of upgrading from Exchange 2007 to Exchange 2010. The feature worthwhile to upgrade was the improved web client. It really is a great client and is fairly uniform across all the major browsers. This was something the staff wanted for a long time. The upgrade from Windows XP to Windows 7 (Vista never happened) gave us Aero Glass and a much improved security model. Both features well worth the upgrade.

So what does v5 have? Well, you have to remember that we have an Enterprise license with VMware, so we get "only" the feature set included with Enterprise. If you aren't familar with VMware licensing you may be surprised to know that there is a license tier above Enterprise called Enterprise Plus. Many of the new features of v5 are only available in Enterprise Plus, which I feel is an extremely costly upgrade. It is not some nominal increase in cost from Enterprise to Enterprise Plus. So to be clear what is new in v5 Enterprise? Specifically what are the really important new features, The "WOW" features, in v5 Enterprise?

1. Larger VMs -- More RAM and More CPUs per VM -- Maximum number of vCPUs increases to 8 (Remember: This is the Enterprise license). This is probably our most important must have feature. Having virtualized 90% of our servers, we definitely have a few big boxes that could use 8x CPUs.
2. Bigger VMFS volumes -- This would have been nice in VMware v4.1, but now that we have converted everything to NFS storage for the VMDK files, this improvement really doesn't matter. NFS volumes are unlimited in size.
3. Storage Appliance and vCenter Appliance -- Although we don't need either, I see where a lot of SMBs could really use either solution.
4. Replication in SRM - Site Recovery Manager now includes data replication. SRM v5 requires vSphere v5 to get the replication peice. I have yet to find a truly great replication solution for DR for our sized company. SRM v5 gives me hope.
5. New License Model -- Not only do you need to pay licensing per socket, but you now need to pay based on how much vRAM is provisioned. This is a "WOW" that sucks feature. It's a wonderful example of a company just trying to vacuum up as much money as possible from clients. For years, I've been virtually handing ram out for free for new VMs. With the shared memory model and compression, the RAM I would provision was fairly cheap. Now I am faced with the herculean option of asking users for the RAM back or paying the new vRAM tax.

All the other major features...fairly useless to us

1. Auto-Deploy for ESX -- For 15 ESX Hosts, it will faster to deploy by hand
2. 3D Graphics for Windows Aero -- Meant for VDI deployments. VMware needs to solve the HD playback problem first.
3. Profile-Driven Storage, SIOC, NIOC, & Distributed Switch -- All look great but....I'm not a member of the elite Enterprise Plus club
4. New High Availability, New Web Client - meh ... The HA was never a major problem if you read the manual. When would I use the feature poor web client instead of vCenter Client?
5. ESXi only -- I love watching a company try to market removing a feature\option as a positive. Apple does it all the time to great success. VMware not as well. v5 can only be installed using ESXi. All my ESX hosts will need to be converted. No over the top install possible.

I'm probably going to push the upgrade to Q2-Q3. I'll probably wait for the first service pack and do something else instead.

NAS Analysis -- Solaris vs NetApp -- Part 4

I recently had an opportunity to compare Solaris and NetApp as a NAS Gateway. This is part of my series comparing various operating systems as potential NAS gateways to store VMDK files accessible via NFS. See my previous performance post

NetApp gateways are a lot more expensive then a comparable standard HP server running Solaris or any other major OS, e.g. Windows or Linux. I was curious if all that money paid for NetApp was actually worth it from a pure performance perspective. Did the NetApp performance really justify the price.

Previously Solaris posted the best NAS NFS gateway results. It would be the best operating system to compare to.I just so happen to have a NetApp with similar harware specs. Obviously its very difficult to compare a standard server from HP and the controllers from NetApp. Realizing that there are vast differences between the two is simply part of the test. You are paying for more with the NetApp.

However, one of the most important considerations in comparing an HP\Solaris solution with NetApp is making sure with use the same number of drives. Using different drive types, spindle speeds, and number of drives would skew the results. That is to say that the drive sub-system could easily make one system appear worse or better than it really was.

Luckily I got my hands on a NetApp system that almost exactly matched the Solaris\HP system in drive layout. They both had SAS 3gbps drives. They both had 15K drives. The drive counts for the vmdk storage, i.e. where the IO was going, were the same.Drive size was different with the Solaris\HP system using 72GB drives and NetApp using 600GB.

RAID configuration was different. The Solaris\HP solution had 2x RAID-1 drives from OS. NetApp has 3x RAID-DP for OS. The data volume for Solaris\HP was RAID-0 and NetApp was RAID-DP. This favors the Solaris\HP solution as RAID-0 is just a simple stripe with no read or write RAID penalty.

NetApp Specs

    FAS3160

    2x Controllers - 1 Active and 1 Passive

    3x 600GB SAS 15K RAID-DP for OS\Root Volume

    14x 72GB SAS 15K RAID-DP for Data Volume

    8GB RAM per Controller

    2GB NVRAM per Controller

 Test Results

The NetApp performance numbers across the board were excellent beating the Solaris\HP solution by wide margins in all categores. For some categories, NetApp out performed the competition by almost 4x. It's up to the reader to decide if the performance benefit is worth the additional price. However, it is nice to now that all that additional money did buy a substantial increase in performance.

NAS Analysis -- Windows 2008 vs Openfiler vs Nexenta vs Red Hat -- Part 3

This post is a continuation in a series of posts on comparing various operating systems as network attached storage solutions for VMDK files in a VMWare environment. Please read Part 1 and Part 2 in the series.

Test Setup
My test lab had three major components. The first was the NAS server itself. The second was the ESX host. The third was the VM running the test on the ESX Host

NAS Server

    HP ML570 G4

    4x Dual Core 3.2GHZ/800MHZ Processors

    8GB of RAM

    P400 Array Controller with 512MB of Cache

    2x 72GB 15K SAS Drives - RAID 1 - For Operating System

    14x 72GB 15K SAS Drives - RAID 0 - For Data

    2x 1Gig-E NICs - Management Only

    1x Intel 10Gig-E NIC - NFS Traffic Only

ESX Host   

    HP BL 460c G6

    8x Cores at 2.4GHZ

    90GB of RAM

    2x 10 Gig-E Dedicated NICs for NFS Traffic

    2x 10 Gig-E Dedicated NICs for VM Traffic

    2x 1 Gig-E Dedicated NICs for ESX Management Traffic

    2x 72GB 15K local drives for ESX Install

VM Guest

    Windows 2008 R2

    2x CPU

    4GB of RAM

    50GB C: VMDK Drive - VMDK drive stored on Fiber Channel Storage separate from NAS

    750GB E: VMDK Drive - Stored on NAS and accessed via NFS.

    Running IO Meter v2006.07.27

Note on JUMBO frames: Jumbo frames were enabled on the ESX host and switches. However, they were not enabled on the NAS Server.

Several different IOMeter Profiles were used to emulate various applications. Each of the worker processes had an I/O queue of 32, ensuring that IOMeter was keeping plenty of I/O ready to go. This workload at the level configured far exceeds what a typical server would present.

IOMeter was run in the VM. The VM had a C: drive hosted on a datastore unrelated to the test. The E: drive, which was 750GB, was created on NFS datastore hosted on the NAS gateway server. Each IO profile was tested 3 times for 15 minutes each. The three test results were then averaged together.

The IO profiles are summarized in the table below.

Some Notes on the NAS OS
For all operating systems, the default installation and setup were used. The standard patches and updates were applied. No other changes except where noted were made. No additional or upgraded drivers were installed.

Of course, each vendor has tweaks to improve performance. In order to keep the test process and time taken manageable, the defaults were used. Otherwise, significant research would be needed to understand the best options for each OS in order to maximize the performance. This in turn opens the door to two accusations: 1. That the proper procedure was not followed and hence invalidating the test and 2. That there was purposeful manipulation of the test setup to favor a particular vendor. By choosing the defaults, the test implicitly relies on the vendor to setup their operating system for the best performance, i.e. it's the vendor's responsibility to configure their own OS.

• Windows 2008 64-Bit R2
  • Standard list of patches and updates
  • No additional software
  • Services for NFS
  • Installed HP Support Pack. Performance was so horrible initially that it was believed that installing the latest HP Support Pack may resolve the problem. It didn't.
• OpenFiler 
  • v2.3 64-Bit
  • Completed standard system Update
  • Standard NFS Share options except turned on "no_root_squash"
  • Used XFS file system
• RedHat
  • Enterprise Linux v5
  • Used EXT3 file system
  • NFS SYNC Option on
• Nexenta
  • Used version 3 with latest updates
  • There were numerous problems with this OS and I opted to not use it in the test. Some of the problems were
    • No native support for Intel 10Gig-E nic and no simple way to add the driver
    • Random crashes in the middle of tests and at times just during boot up
  • Significant questions regarding the longevity of this product given Oracle's lack of support of its base OS, Open Solaris.
• Suse Linux Enterprise 11 SP1
  • Kernel version v2.6.32
  • Used EXT3 file system
• Solaris 10

Test Results




Test Conclusions
Windows NFS Performance was Horrible
It is stunning how poorly a major operating system like Windows 2008 did in the tests. It stands out simply for its poor performance. As an example the next best competitor in the File Server IOps category is 5x better. That can't be explained away by some simple tweaks. That level of poor results speaks to a fundamental problem with the OS running NFS.

Attempts were made to improve performance by tweaking settings, installing updated drivers from HP, searching for solutions online, etc. etc. None of these steps were required for any of the other operating systems. Regardless, all of this additional work was to no avail. Even with all the changes and updates Windows 2008 failed to produce any noticeable improvement.

As a result of these tests, Windows 2008 should not be used as an VMWare NFS datastore. Only for the smallest simplest workloads would this operating system work.

Wide Variance in Results
Excluding the Windows 2008 poor results, there was still a wide variance in results. There was an expectation going into the test that there would be only small differences in the results. Three of the five operating system all use a modern version of Linux. It seemed likely that the Linux versions would provide similar results.

However, there was a wide descrepency even amongst the Linux operating systems. In several cases, OpenFiler and Suse put up performance numbers that were DOUBLE RedHat's. A few percentage points would be reasonable. But double the performance strikes as an unusual result.

Again steps were taken to try to tweak RedHat to provide better performance. Again, none of the changes or updates had any effect.

Openfiler a Good Opensource Option
Openfiler had great results. Consistently one of the better Linux operating systems and for the Exchange and File Server tests beat out all the other Linux variants.

Openfiler and Suse were virtually tied with Openfiler taking Exchange and File Server and Suse taking Web Server and SQL

If you like open source software, Openfiler should be a strong contender for your NAS gateway.

Solaris Dominates
Solaris far and away crushed the competition. Solaris posted the best results in every test and in some cases beat the competition by a wide margin. If you don't mind using Solaris and dealing with Oracle, it's certainly the NAS gateway of choice.

Some of these great results were probably due to the ZFS file system. ZFS is simply an incredible file system, which I believe to currently be the best commercially available file system on the planet. It offers effectively unlimited volume sizes, deduplication, unlimited snapshots, no write penalty snapshots, mega sized read and write caches, etc. etc. It's too bad they don't offer a version that runs on Linux. It's better than anything Linux has including the upcoming btrfs

Expanding the C: Drive on Windows VMs

One tasks that frequently comes up in IT is adding more disk space to a server. In a Windows VM environment, for the most part, this is very easy. If your disks are Basic as opposed to Dynamic, simply add more space to the VMDK file via VMware and then use diskpart to extend the partition. If your disks are dynamic, create a new virtual disk in VMware and in Windows span onto this new disk.

Unfortunately, neither of these two scenarios work if you are trying to add space to the C: drive. Diskpart will not extend a partition that contains the boot volume, a.k.a. C:. If your C: drive is dynaimic, you cannot span the C: drive with multiple disks.

One solution assuming you are using Basic disks is to

1. Shutdown your VM
2. Remove the virtual disk from the VM configuration. (DON’T DELETE IT)
3. Add this virtual disk to another VM
4. Increase the virtual disk size using VMware
5. Use diskpart in the VM to extend the partition
6. Remove the virtual disk from the temporary VM
7. Add the VM back to the original VM and boot

There are a couple of problems with this method

1. It assumes you have a spare VM to work with
2. It assumes that you don’t mind making changes to the temporary VM even if the changes are minor

There is a better way to do the C: drive expansion that is a bit faster and sidesteps the problems listed above. Parted Magic is the secret sauce. Pmagic is an opensource, Linux distribution that includes many easy to use disk based utilities including gparted, a disk partition utility.

Requirements:

1. Basic Disks. (It may work with Dynamic. I have never tried)
2. A Backup of your data specifically the vmdk file you are modifying. I highly recommend using FastSCP from Veeam. This is a free tool that is easy to use and can make fast backups of VMDK files
3. Parted Magic Live CD

Steps:

1. Shutdown your VM
2. Backup your VMDK file. Do not rely on just a snapshot. Since we are making major changes to the partition table of the vmdk file a full backup is recommended. A snapshot will probably work but not recommended.
3. In VMware Infrastructure Client, increase the size of the virtual disk
4. Mount the Pated Magic Live CD in your VM
5. Boot the VM into the Pmagic OS
6. Start the Gparted Application. There should be an icon on the desktop. In the screenshot you can see the C: drive with the additional 40GB of unallocated space. The drive designation is in Linux parlance, i.e. it is now /dev/sda1 and not C:
7. 
8. Click Resize/Move
9. Simply drag the green box representing the first partition so that it fills the entire drive
10. 
     
    
11. Exit gparted and apply your changes. After churning for a few moments you should see a success screen
12. 
     
    
13. Shutdown the VM. The equivalent Windows “Start Button” is in the lower left hand corner
14. Unmount the Pmagic CD
15. Boot the VM. Windows will immediately complain that changes have been made to the system that require chkdsk to run. Let chkdsk run. The machine may reboot after this is complete.
16. Login and check that Windows sees the larger C: drive.

Performance Best Practices for VMware vSphere 4.1

VMware recently released a new version of their Performance Best Practices for VMware vSphere 4.1. If you have never read this technical paper before, I strongly recommend that you take the time to read it. It gives a great overview of the major technical features that effect performance in a VMware environment. The paper can be found here.

NAS Analysis – Windows 2008 vs Openfiler vs Nexenta vs Red Hat vs Windows with OpenText – Part 2

This post is a continuation in a series of posts on comparing various operating systems as network attached storage solutions for VMDK files in a VMWare environment. Please read Part 1 in the series.

Well, the wonderful performance I saw with Windows 2008 was short lived. Using Openfiler I was able to push roughly 3-4x the number of IOPS. These results were pervasive across different types of workloads, e.g. Exchange, SQL, Webserver, etc. I am now testing with RedHat Enterpise Linux and the results for certain tests are even better than Openfiler.

There have been delays with the testing because my NAS server kept randomly failing. After running 2-3 10 minute test runs using IOMeter the virtual disk in my VM would suddenly just stop responding. The problem sometimes occurred mid-test. I tried several diagnositics steps but the only thing that got the virtual disk working again was rebooting the NAS server. The problem was occurring both on RedHat and Nexenta. Openfiler and Windows2008 did not have the same problem.

I tracked the cause to this bug in the HP array firmware. After updating the firmware, the problem never reoccurred. Unfortunately now that I've changed the underlining hardware configuration, I feel that the results from OpenFiler and Windows2008 are now suspect. I will have to redo those tests.

Great VMware Health Check Script

If you are looking for a great script to check the health of your VMWare environment, please take a look at William Lam's script at http://communities.vmware.com/docs/DOC-9842

I run his script weekly to generate a nice series of reports about the status of our VMware enviroment. I've been using the script for two years and since then it has helped identify many problems before they became outages. Some examples are

1. Old Snapshots
2. Low disk space on datastores
3. VMs misconfigured to use local datastores instead of fiber channel datastores
4. Misconfigued NTP settings on ESX Hosts
5. VMs with attached floppies and CDs preventing vMotion migrations
6. SCSI reservation locks on fiber channel LUNs
7. Datastores with too many or too few VMs
8. Misc. ESX errors
9. Misc. VMware vCenter Server errors
10. Out of Date VMWare Tools

All the information generated by the report, can be accessed independently. The script does a great job consolidating the information into one simple report. I highly recommend using it.