Storage benchmarking is what I like to call a “dark art”. The intent of benchmarking is nobel, but the results are sometimes less than honest or more typically not applicable to your data center. From a manufacture’s perspective you want to show real-world results while showing your product in the best light possible as compared to the competition. From a customer’s perspective you want to know exactly how the system or component will work in your specific environment. It’s clear that testing every possible customer environment is impossible, so how do you find a standard way to test a product while giving meaningful results that will be indicative of real-world performance that the customer will see.

Let’s start with the tests that are out there: There’s the various flavors of SPC (Storage Performance Council), SPEC SFS (Standard Performance Evaluation Corporation), NetBench (old but still used for CIFS performance), Iometer, IOzone

And of course application specific benchmarks: From Microsoft you get SQLIO for SQL and LoadSim for Exchange. There’s various ways to test Oracle and other applications as well.

Once you agree on a test that you think will test the type of network or protocol access you have in your environment, now you have to start finding some results from the various vendors you are looking at evaluating. Some 3rd party test facilities such as Lionbridge/Veritest may have some information that they’ve been commissioned to test and report on. The SPC and SPEC websites will give you validated test results that may be helpful also.

With all of this information, you still won’t know how the product will work in your environment. Were the PC’s/workstations used for the tests representative of your environment in quantity and horsepower? Will your own hosts actually cause a bottleneck due to CPU, memory, interfaces, or applications? How about your network if your using a SAN or adding a NAS to your LAN? (Things outside the storage system are incredibly difficult to emulate.) If you are purchasing additional storage for a specific application, do the block sizes and randomness of the data used in the test match the application in question? And those are the easy questions!

In many cases the individual vendors can simulate your environment pretty closely and give you a feeling of what you can expect - but make sure they are using products and components you can actually buy and that are within your budget! Performance always comes at a price - don’t forget that.

Whether you’re measuring IOPS, throughput, response time… whether it’s for generic file services or for a specific application - the best and only way to really know how a unit will work in your environment is to try it. So if performance is a concern for you and if performance is at the top of the list for making a purchase decision - push your vendor for an evaluation unit so that you can trully assess the performance of the product for your environment.

There was a really good article on this topic on Infostor in December 2006. It’s long - but give it a read.

Let me know what you think.

Blog ya later!

Joe

This entry was posted on Wednesday, January 17th, 2007 and is filed under General, Platforms, Advisor - Joe Disher. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.