Playing around with VMware IO Analyzer

About a year ago I was at an EMC event in Orange County, I think it was one of the traveling roadshow type events and they had some hands on labs for a bunch of their different product lines. The labs were setup in VMware and at one of the labs had an IO tool that would let you create workloads and run them. At the end of the lab I saw a blurb about IO  Analyzer. Now I wasn’t familiar with the product so when I got back to the office, I tried googling for it, no luck. But I did take a picture of the screen (can’t find that now) and it had a contact email for the beta program for the IO Analyzer, so I shot a note over to see if I could get a copy of the program.

About a day later I got an email from Bobbie Morrison who is a Program Manager at VMWare with an invite to the Beta program. Schwing! I’m not sure many people were aware of the program. There was not a lot of feedback in the beta group page, but I did post a few questions about the results I was getting and got responses fairly quickly. Still, I had not seen anyone around the various VMware blogs post about it, so I quietly ran tests utilizing the beta and in general it gave me some excellent feedback about my storage performance and expectations.

Prior to this, I had been curious about the performance levels I could get from Virtual Machines. There was a pretty long standing thread in the VMTN where users were posting informal performance metrics, and up until that point I had been using the custom ICF scripts from that with a standard windows VM that I had built just for testing. The results were OK, but it would have been nice to have something pre-canned that would do this. So from my standpoint, the IO Analyzer was a really awesome tool.

Once I got the files it was pretty simple to configure and deploy. The steps are pretty simple. Import the OVF template, add a disk, setup the network, login and you’re off and running. After running a few test MaxIO/MaxThroughput tests I decided I wanted to hammer our storage to see what kind of performance I could get. My first tests were very similar in results to what I posted a while back.  I did notice that the results I got from when I first ran the tool were far better than results from the same batch of systems several months later. I’m not 100% sure as to why performance was about 20% lower, but I believe it might have to do with storage vmotion of the VM’s onto a data store with a far different configuration.

Enter the now.

So today I saw a tweet from Wade Homes (@wholmes) who is a VCDX and works as a Solution Architect at VMware mentioning that the VMware IO Analyzer was public release. So, I’ve spent the morning rebuilding 5 VIOA boxes and running some tests.

The setup is as follows:

  • 5 hosts:  3 IBM X3690 X5’s with dual 8core procs, 128GB RAM, Emulex 8GB FC with 10GbE, 2 IBM 3850M2 quad 4core procs with 128GB RAM, Emulex 8GB FC and 10GbE
  • 2 IBM 24-B (brocade silk worm 300) FC Switches zoned between all ESXi Hosts and the XIV FC SAN Array
  • 5 VM IO Analyzer virtual machines, one deployed to each host all running on the same VMFS datastore, each with its own 8GB disk volume
  • 1 XIV Gen2 full frame with 8 4GB FC connections to two fabrics on the IBM 24-B
  • 1 VMFS Datastore 1MB block size no SOIC configured
So that’s the setup, nothing too fancy. With the beta, I would have to run each of the VMIO systems from an individual portal, but the new version lets you control multiple analyzers from the same single page. So lets test this sucker out.
First round, Max IO with 5 workloads one on each host.
This is from the XIV GUI which shows the IOPS for each of the 5 VM’s running with an aggregate of around 59k IOPS for all 5 workloads.
For some independent confirmation, I looked at what Veeam Monitor was showing me. (Love me some Veeam)
 
So we can see that Veeam Monitor is showing roughly the same results as what the XIV TOP Tools GUI shows.  I can also look on the XIV management GUI to see the total IOPS being generated on the array during the test period:
And the results from the IO Analyzer (which shows the ESXTOP output). One thing I’ve noticed is that when I run multiple workloads from a single pane, that the results get skewed a bit in their placement. Not a big deal, but it does make reading the grouped chart a little messy.
Next, lets try a Max Throughput test
Same setup, but now we will run all 5 workloads with the MAX Throughput configuration.
1441 MBps! Groovy.
Veeam Monitor shows the same results.
As do the results from the XIV Top Tools and the IO Analyzer
Ok finally a mixed workload of 5 different workers:
I run multiple disparate workloads on our storage array, what I like most about the IO Analyzer is the ability to run multiple workloads to anticipate how the array will react in a given situation. I don’t run Video on Demand, but if I had to, what kind of results would I get and would the performance be sufficient? That’s the real value I see in using this tool, the ability to model and gauge expectations.
So for this last run, its 5 disparate workloads.
  •  OLTP @ 8K
  • Exchange 2007
  • SQL @ 64K
  • Web Server @ 8k
  • Video on Demand @ 8k
XIV GUI results:
Veeam (this is a 4 hour view)
IO Analyzer results:
Busy morning 🙂
So this is just a pretty hasty setup and test to show what you can do. I’ve noticed a few things, if I run multiple workloads from the same console, it will skew the results columns. The data is right, its just out of whack on the results display. The original instructions forgot the step of having to add an additional disk after you create the IO Analyzer VM (if you don’t do that, it will run off the VM’s memory and skew your results. So far, not a lot changed from the beta to final release product except the ability to save your configurations, which I think is nice.
I have a few ESXi hosts that are connected to Equallogic iSCSI arrays and I will run similar tests with that setup as well in a few days. If you have any questions give leave me a comment.
This entry was posted in Storage & Virtualization, VMWare, XIV Storage. Bookmark the permalink.

8 Responses to Playing around with VMware IO Analyzer

  1. PParkin says:

    Good post, interested in your Equallogic results we use a lot of EQL at CCS Media …

  2. David says:

    Good post, thanks!
    I’m also insteresed by the test with EqulLogic.

  3. Jinpyo Kim says:

    I confirmed misalignment issue in guest I/O summary page.

    Until we revised appliance, here is a quick fix for it.

    Please edit the following line in /var/www/results/index.php file.

    line # 226: $linenum++ –> move after following if(), move it to line #234.
    line # 227: if (($linenum % 10) == 0) { –> if( ($linenum !=0) && ($linenum % 9) ==0 ) {

    It will fix misalign problem.

  4. Pingback: Open Tabs 12/13

  5. Pingback: Thankfully the RAID saved us. » Blog Archive » 2012 vExpert Nominations are now open

  6. Pingback: Performance and bench-marking tools | Aussie Storage Blog

  7. Pingback: Performance and bench-marking tools | ServerGround.net

  8. SM says:

    Great post. So Veeam monitor is a good alternative then?

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.