Blumetech's Tech Blog

NAS Analysis -- Windows 2008 vs Openfiler vs Nexenta vs Red Hat -- Part 3

This post is a continuation in a series of posts on comparing various operating systems as network attached storage solutions for VMDK files in a VMWare environment. Please read Part 1 and Part 2 in the series.

Test Setup
My test lab had three major components. The first was the NAS server itself. The second was the ESX host. The third was the VM running the test on the ESX Host

NAS Server

    HP ML570 G4

    4x Dual Core 3.2GHZ/800MHZ Processors

    8GB of RAM

    P400 Array Controller with 512MB of Cache

    2x 72GB 15K SAS Drives - RAID 1 - For Operating System

    14x 72GB 15K SAS Drives - RAID 0 - For Data

    2x 1Gig-E NICs - Management Only

    1x Intel 10Gig-E NIC - NFS Traffic Only

ESX Host   

    HP BL 460c G6

    8x Cores at 2.4GHZ

    90GB of RAM

    2x 10 Gig-E Dedicated NICs for NFS Traffic

    2x 10 Gig-E Dedicated NICs for VM Traffic

    2x 1 Gig-E Dedicated NICs for ESX Management Traffic

    2x 72GB 15K local drives for ESX Install

VM Guest

    Windows 2008 R2

    2x CPU

    4GB of RAM

    50GB C: VMDK Drive - VMDK drive stored on Fiber Channel Storage separate from NAS

    750GB E: VMDK Drive - Stored on NAS and accessed via NFS.

    Running IO Meter v2006.07.27

Note on JUMBO frames: Jumbo frames were enabled on the ESX host and switches. However, they were not enabled on the NAS Server.

Several different IOMeter Profiles were used to emulate various applications. Each of the worker processes had an I/O queue of 32, ensuring that IOMeter was keeping plenty of I/O ready to go. This workload at the level configured far exceeds what a typical server would present.

IOMeter was run in the VM. The VM had a C: drive hosted on a datastore unrelated to the test. The E: drive, which was 750GB, was created on NFS datastore hosted on the NAS gateway server. Each IO profile was tested 3 times for 15 minutes each. The three test results were then averaged together.

The IO profiles are summarized in the table below.

Some Notes on the NAS OS
For all operating systems, the default installation and setup were used. The standard patches and updates were applied. No other changes except where noted were made. No additional or upgraded drivers were installed.

Of course, each vendor has tweaks to improve performance. In order to keep the test process and time taken manageable, the defaults were used. Otherwise, significant research would be needed to understand the best options for each OS in order to maximize the performance. This in turn opens the door to two accusations: 1. That the proper procedure was not followed and hence invalidating the test and 2. That there was purposeful manipulation of the test setup to favor a particular vendor. By choosing the defaults, the test implicitly relies on the vendor to setup their operating system for the best performance, i.e. it's the vendor's responsibility to configure their own OS.

• Windows 2008 64-Bit R2
  • Standard list of patches and updates
  • No additional software
  • Services for NFS
  • Installed HP Support Pack. Performance was so horrible initially that it was believed that installing the latest HP Support Pack may resolve the problem. It didn't.
• OpenFiler 
  • v2.3 64-Bit
  • Completed standard system Update
  • Standard NFS Share options except turned on "no_root_squash"
  • Used XFS file system
• RedHat
  • Enterprise Linux v5
  • Used EXT3 file system
  • NFS SYNC Option on
• Nexenta
  • Used version 3 with latest updates
  • There were numerous problems with this OS and I opted to not use it in the test. Some of the problems were
    • No native support for Intel 10Gig-E nic and no simple way to add the driver
    • Random crashes in the middle of tests and at times just during boot up
  • Significant questions regarding the longevity of this product given Oracle's lack of support of its base OS, Open Solaris.
• Suse Linux Enterprise 11 SP1
  • Kernel version v2.6.32
  • Used EXT3 file system
• Solaris 10

Test Results




Test Conclusions
Windows NFS Performance was Horrible
It is stunning how poorly a major operating system like Windows 2008 did in the tests. It stands out simply for its poor performance. As an example the next best competitor in the File Server IOps category is 5x better. That can't be explained away by some simple tweaks. That level of poor results speaks to a fundamental problem with the OS running NFS.

Attempts were made to improve performance by tweaking settings, installing updated drivers from HP, searching for solutions online, etc. etc. None of these steps were required for any of the other operating systems. Regardless, all of this additional work was to no avail. Even with all the changes and updates Windows 2008 failed to produce any noticeable improvement.

As a result of these tests, Windows 2008 should not be used as an VMWare NFS datastore. Only for the smallest simplest workloads would this operating system work.

Wide Variance in Results
Excluding the Windows 2008 poor results, there was still a wide variance in results. There was an expectation going into the test that there would be only small differences in the results. Three of the five operating system all use a modern version of Linux. It seemed likely that the Linux versions would provide similar results.

However, there was a wide descrepency even amongst the Linux operating systems. In several cases, OpenFiler and Suse put up performance numbers that were DOUBLE RedHat's. A few percentage points would be reasonable. But double the performance strikes as an unusual result.

Again steps were taken to try to tweak RedHat to provide better performance. Again, none of the changes or updates had any effect.

Openfiler a Good Opensource Option
Openfiler had great results. Consistently one of the better Linux operating systems and for the Exchange and File Server tests beat out all the other Linux variants.

Openfiler and Suse were virtually tied with Openfiler taking Exchange and File Server and Suse taking Web Server and SQL

If you like open source software, Openfiler should be a strong contender for your NAS gateway.

Solaris Dominates
Solaris far and away crushed the competition. Solaris posted the best results in every test and in some cases beat the competition by a wide margin. If you don't mind using Solaris and dealing with Oracle, it's certainly the NAS gateway of choice.

Some of these great results were probably due to the ZFS file system. ZFS is simply an incredible file system, which I believe to currently be the best commercially available file system on the planet. It offers effectively unlimited volume sizes, deduplication, unlimited snapshots, no write penalty snapshots, mega sized read and write caches, etc. etc. It's too bad they don't offer a version that runs on Linux. It's better than anything Linux has including the upcoming btrfs

Huge Patch Release for OS X

And I've been told MAC has fewer bugs than Windows...http://www.networkworld.com/news/2010/111110-apple-smashes-patch-record-with.html?source=nww_rss

Apple's Server Line Discontinued

Apple's xServ is dead. This like Apple's failed xRAID should really be no suprise.

1. No support for running their OS in a virtual environment.
2. OS that only runs on Apple specific hardware
3. OS just had a pretty face and none of the enterprise features needed in a datacenter.

It's one thing for Apple to convince the average, gullible, end-user that they have an OS that goes beyond just a pretty face. It's quite a more difficult challenge to convince highly technical people in the Datacenter, that OS X is ready for scalable, high-performance computing.

NAS Analysis – Windows 2008 vs Openfiler vs Nexenta vs Red Hat vs Windows with OpenText – Part 2

This post is a continuation in a series of posts on comparing various operating systems as network attached storage solutions for VMDK files in a VMWare environment. Please read Part 1 in the series.

Well, the wonderful performance I saw with Windows 2008 was short lived. Using Openfiler I was able to push roughly 3-4x the number of IOPS. These results were pervasive across different types of workloads, e.g. Exchange, SQL, Webserver, etc. I am now testing with RedHat Enterpise Linux and the results for certain tests are even better than Openfiler.

There have been delays with the testing because my NAS server kept randomly failing. After running 2-3 10 minute test runs using IOMeter the virtual disk in my VM would suddenly just stop responding. The problem sometimes occurred mid-test. I tried several diagnositics steps but the only thing that got the virtual disk working again was rebooting the NAS server. The problem was occurring both on RedHat and Nexenta. Openfiler and Windows2008 did not have the same problem.

I tracked the cause to this bug in the HP array firmware. After updating the firmware, the problem never reoccurred. Unfortunately now that I've changed the underlining hardware configuration, I feel that the results from OpenFiler and Windows2008 are now suspect. I will have to redo those tests.

Nasuni and Cloud Storage

I recently evaluated what looked like a great small business product from Nasuni. Cloud storage is a hot marketing term right now, but I haven't been able to find a good fit for your typical SMB. Nasuni looked like it might just work.

Nasuni goal is to provide a low cost NAS (Network Attached Storage) with unlimited storage. Their solution runs as a simple VM and can emulate a Windows File Server. Users simply map a drive and start dropping in files. On that VM is a local cache up to 512GB that stores files. As the cache fills up files are moved to from the cache and put onto storage array at a storage cloud provider. Most of the majors, e.g. Amazon S3, Nirvanix, Rackspace, etc. are support. If a file is need that isn't in cache, Nasuni connects to the cloud and downloads the file. Since you are effectively using cloud storage you get

1. Unlimited storage -- No need to buy any more disks. No need to pay for power, support costs, replacement costs, etc. for all those disks.
2. Redundancy  -- The cloud provider automatically replicates all the data across multiple datacenters. This data is highly available. Amazon boasts 99.99% available. (Don't confuse availability with durability. Availability is how likely your storage will be available. Durability is how likely your data will be lost)
3. Disaster Recovery -- If your primary site goes down, fire up a Nasuni box at your DR site and access all your data from the cloud. The only data you may have lost is what was in cache at the time of the disaster.

Nasuni also takes care of a view other critical items

1. Security -- All data stored in the cloud is encrypted. Even if someone had access to the encrypted data, the data is useless without you key stored locally on the Nasuni box
2. Backup -- Nasuni provides effectively unlimited snapshots so you can roll back files to previous versions at any time. You can do single file and whole volume restores

Unfortunately there are two big problems with the product

1. No Delete -- I was shocked by this. Apparently you can't permanently delete a file. If a user, deletes a file from the file system, it certainly disappears. However it is still stored in previous snapshots and hence isn't truly deleted. There are two ramifications to this 1. if you have a legal requirement to delete the file, then you will not be able to meet it and 2. you still pay for cloud storage even on files that you deleted.
2. Costs -- Nasuni itself is inexpensive. $3000 per year which includes your upload and download fees to Amazon. (Please note that once you go beyond a certain size which I believe is 16TB, the pricing model changes and Nasuni wants you to pay the upload\download fees.) The real cost is the storage cost...

Today I just purchased rather expensive SATA drives from a major storage vendor. I won't give the name but I can assure you they are in the big 3 and their drives are NOT cheap. Total cost was $2292/USABLE TB. We replace drives every 4 years on average. We pay roughly 20% of drive costs for a premium support contract which includes everything necessary to support an maintain these drives.

That usable storage number includes the RAID overhead (RAID 6), file system, etc. It represents the true usable storage to the end user. The cost doesn’t include the array head or if doing NAS the filer head. Although this cost is significant, I am ignoring it because I just as easily could offset that cost with buying drives from a cheaper storage vendor, e.g. Polycom, Nexsan, etc. Drives from those companies vary, but you can easily get under $1000/Usable TB 

You could account for the fact that Nasuni is “pay-as-you-use” model, whereas the SATA solution you must buy storage upfront. Here I’m assuming that SATA storage will be purchased as needed without a bulk up front purchase where the storage sits idle. Getting storage utilization rates over 70% is not atypical in most SMB storage environments

To continue the comparison with Nasuni, I can double the price by purchasing drives at a DR site. Of course there is the DR site costs to consider, but Nasuni isn’t going to obviate the need for a DR site. It only solves the problem of file storage. There are many other business applications, e.g. database and mail, that won’t be able to leverage Nasuni for DR and hence will need some sort of DR solution. Here I’m just assuming a DR site and replicating the storage over. Again the cost doesn’t include the cost of replication software, additional bandwidth, and all the other pieces for DR. But I could also buy cheaper drives at the DR site. I also have included the 20% premium support contract for the drives at the DR, which is something I probably wouldn't do for the DR site as a premium support contract is unnecessary. Next-business day would have been fine.

All in all, $5500/Usable TB ($2292 * 2 for DR * 1.2 for support contract) is at least a close approximation to the total 4 year drive cost. It provides a much faster IO solution than Nasuni as the SATA drives will always serve data faster than Nasuni cache misses, which need to download the data over the Internet using whatever bandwidth is available.

As of today, 1GB of cloud storage on Amazon S3, the cheapest major cloud storage provider, costs $.15/month.  Therefore over 4 Years the cost per Usable TB is ($.15/Month * 1024 GB/TB * 48 Months) $7200.

There’s a lot of margin between the local SATA drives and the Amazon S3 solution. One could add a lot of costs to the SATA solution for personnel, DR, power, cooling, etc. etc. and still not get to the $7200. And even if that new cost surpasses the cloud storage costs, the local SATA solution is still faster and we would then need to move to a cost/IO discussion.

Nasuni really looked like a great solution. The interface is easy to use and the setup was quick and simple. I love what they have done creating a hybrid solution between local and cloud storage. However the cloud storage pricing, makes it cheaper for most SMBs to simply buy a Windows server (or better yet a free open source NAS) and some low-end drives.

Auto-Installation of Adobe Reader

Recently, Adobe has been keeping pace with Microsoft for most frequently patched product. It seems like Adobe Reader is coming out with a new version monthly. Some of the security holes with the product are severe. In the worse cases a simply opening of a malformed PDF can allow complete administrative control of the machine. The problem with PDFs get even worse when you consider:

1. The problems presists across all platforms. These flaws aren't just windows based but also apply to Macs as well.
2. Historically users have viewed PDFs as secure and will open PDFs from unknown sources simply because the user feels safe behind what they incorrectly think is a secure document format.

https://www.networkworld.com/news/2010/081910-patch-critical-security-flaws-in.html

http://www.networkworld.com/news/2010/100610-adobe-hits-reader-users-with.html?source=nww_rss

Here at iN DEMAND, we have Adobe Reader installed on every workstation. Every time Adobe comes out with a new release to patch some security flaw, we need to go around and update all the machines.

To speed up the proces we auto-deploy Adobe Reader. We use Script Logic's Desktop Authority to handle the running of the deployment command. Any means to remotely execute the installation will work...Powershell, Task Scheduler, GPO, or the ubiquitous login script.

The real trick to the deployment is the necessary command line switches required to install the program silently. This is what we use

AdbeRdr940_en_US.exe /sAll /rs /l /msi"/qb-! /norestart ALLUSERS=1 EULA_ACCEPT=YES SUPPRESS_APP_LAUNCH=YES

There are two groups of command line switches. The first set are for the iNOSSO extraction program. This program is used to extract the MSI file from the .EXE distribution file. The second set of switches, the ones after "/msi", are for the actual MSI install package.

You could conceivable download the EXE, extract it to the MSI, and then run just the MSI files during the auto-deployment. This will probably be faster for each deployment as the extraction step is skipped. However, I wanted the easiest possible administrative steps. The way I have it a help desk technician can download any new updates to a network share and then automatically update machines without doing any additional work.This is a very simple design that's difficult to break.

Background on the switches:

• /sAll - Silent Mode
• /rs - Supress reboot
• /l - Enable Error Logging
• /msi - Parameters for MSI
  
  • /qb-! - Silent Mode
  • /norestart - Supress reboot
  • ALLUSERS - Update All Users profile
  • EULA_ACCEPT - Acept license agreement
  • SUPPRESS_APP_LAUNCH - Do not launch Adobe Reader after the installation completes.

 

Why a Blog?

Why bother with a blog at all? There are a couple of reasons that I have for starting a blog:

1.       I feel that the small and medium sized business of less than 500 users is not in general getting the service they deserve from the IT services industry. My opinion is that the SMB market gets far too many “snake oil” salesmen and unskilled, unpolished engineers. Furthermore, the SMB market lacks the sophisticated IT skills to “see through” the FUD and get to real, effective, and efficient solution. Hopefully this blog will help SMBs make more intelligent and informed solutions.

2.       I’m constantly being asked to provide my opinion or advice on new technology and products. A blog is a great way to communicate that information.

3.       I’m constantly asked about the “right” way to design a solution. People frequently ask me to boil down a lengthy best practices document into something tangible for their particular environment.

4.       I love talking about technology whether it’s new products, solutions, or just technology in general.          

5.       I have a lot of information worth sharing with others. I think there is a sizable group of people interested with the information I’m providing

6.       A blog provides a digital archive of information and knowledge that I have accumulated, i.e. I can use the blog as a means to store old information that is valuable to me as well as others.

7.       It generates traffic for my wife’s company, Blumetech. She has a successful business providing IT services to small and medium sized businesses in NYC, Northern NJ, and Westchester. Helping her seems like a wise thing to do.

8.       A blog will help boost my professional career by reconfirming what’s already on my résumé.