Network World posted an article last week called The High Price of RAID-5, written by Mike Karp.

First, I don’t know Mike. I’m sure he’s a great guy, smart as a tack, experienced in the storage field, and kind to animals. He wrote a good article but I do want to address some of his comments. I’ve reproduced it below in its entirety, with my snarky comments embedded.

There are many levels of IT hell. Surely, one of the worst of those involves coping with the looming torture of RAID 5.

Two sentences into the article and I’m already wincing. (I have to admit I also make outlandish claims just to get people’s attention, so I won’t get too righteous about this one. )

To setup a RAID-5, the IT admin should point to a handful of drives, or sections of drives, click “Make a RAID-5”, and then forget about it. Someday a light may come on, an alarm may sound, an e-mail may be sent, or a log entry may be made, indicating that a drive has failed. The IT admin gets up from his comfy chair, replaces the bad drive, and then goes back to playing Solitaire.

That doesn’t sound like torture.

RAID has been with us for more than 20 years, and during that time has saved the corporate bacon for many an admin. RAID is a splendid thing when it works but when it fails, depending on the circumstances, the result may be anything from an inconvenience to a disaster.

I’d suggest that when anything fails the results are often bad, but I’d certainly agree that RAID-5 is in that category of Things That Just Better Work.

Most of us are familiar with RAID 0 (data striping) and RAID 1 (mirroring). The first provides enhanced read performance but has no protection against data loss and is awkward to scale due to the need to re-stripe whenever capacity is added;

The re-striping as drives are added should be completely transparent to the user, the OS and the applications. Just like the RAID-5 setup above, the IT admin should just point to one or more new drives and then click “Add to RAID-0”. It should be that easy.

…the second protects data, but at the expense of reduced write performance and, of course, the necessity of buying twice as much disk capacity to support the mirrors.

RAID 0 and RAID 1 - sometimes used in the same RAID set (RAID 10) - have typically appealed to managers using less expensive systems.

FYI, RAID-10 is just as expensive as RAID-1. They both require twice the storage as RAID-0 or JBOD. RAID-10 is the most expensive RAID level.

The preferred alternative when it comes to first-tier storage with high performance disks has been RAID 5. But we have now arrived at the point where we ought to ask ourselves if RAID 5 continues to be a good choice for first tier data.

RAID 5 stripes data and distributes parity information (used for error correction) across all drives within the array. This yields most of the read performance advantages of RAID 0, but comes at the expense of slower writes due to the parity calculations that accompany each write operation. Like RAID 0, adding more disks to the array involves parity recalculation and imposes a significant performance penalty.

Short random writes to a RAID-5 are VERY slow, as Mike points out, but long sequential writes to a RAID-5 “should” be pretty fast, even approaching the performance of RAID-0. Folks rarely use RAID-5 for databases with a heavy write component. RAID-10 makes more sense. But RAID-5 is great for video servers or read-mostly storage.

Beyond that, with RAID 5 we find that one high-risk element is added. When a second drive fails, the consequence is always catastrophic data loss.

Huh? When a second disk fails in a RAID-1 you also get catastrophic data loss. If you want to survive a two disk failure then RAID-6 is the answer. (I’ve made a ton of posts on this topic and I recommend checking them out.)

As a result, replacing failed RAID 5 drives immediately is crucial and hot spares are often kept on hand so that rebuilds can begin right away. The expense of owning a spare Fibre Channel or SCSI drive, admittedly costly, is pretty easy to swallow compared to the potential disaster of losing access to high performance, online data.

Yep, agreed. But replacing a drive immediately is just as critical with a RAID-1 or RAID-10. Only RAID-6 will allow a slothful IT admin to wait weeks before replacing a drive.

Unfortunately, the need to re-stripe the parity imposes essentially the same calculation penalty on a system as would a simple expansion of the array.

First, re-stripe is often defined as reconstructing the parity on all the drives – also known by the clumsy name of “Verify with Fix”. A re-build is different. It is the reconstruction of bits on a failed drive, including both parity and data. I assume Mike is referring to a rebuild here. There is little reason to ever do a re-stripe of parity.

Second, re-building a degraded array is NOT the same as an array expansion. They both involve reading data and XOR’ing data, but the rebuild only involves a write to the replaced drive. The array expansion involves writing to all the drives. Reads typically come out of the drive’s cache without loss of rotation, but writes must go straight to disk since write cache is turned off in any serious storage system. Writes will miss rotations, killing performance, tying up the drives for ~10ms at a time, and therefore the time to do an expansion is often several times that of a rebuild.

While all that calculation is taking place, system performance degrades markedly, frequently degrading things so much that I/O performance during the rebuild renders the system unusable. If you’ve never been through a RAID 5-rebuild process while users continue to access the system, you can capture the flavor of it by using a demanding application at the same time as your system is running a backup.

I don’t disagree with this, but it’s not quite as bad as it sounds. First, if the disks were 100% consumed by OS and application IO before the failure, then the rebuild will definitely be felt by the user. But if there is enough dead time between IO bursts, such as at night, then a good RAID controller will try to schedule the rebuild between IOs to reduce any disruption to the user IO.

But with all that said, a RAID-5 rebuild is often noticeable. This is just one of the factors to be taken into account when selecting a RAID controller and RAID level. I recommend that you measure the performance level during degraded and rebuilding modes to make sure you’ve got enough horsepower to allow your business to continue running.

Have a good book handy.

I don’t recommend that you sit next to the array and watch it being rebuilt. This isn’t laundry. Once the failed drive is replaced your job is done. Check back later to make sure the rebuild completed, but never, ever just sit there and stare at your RAID controller. This is viewed as a sign of hostility.

Performance penalties during rebuilds have always been part of the price of using RAID 5, but we may be getting to the point where that price has gotten to be too high. With small disks, performance during rebuilds was certainly exasperating, but at least it was a relatively short-lived problem. As systems get larger however, the situation becomes radically different, and not in a good way.

The basic idea, I suppose, is that a user might have been able to stomach poor performance for a few hours with small drives, but not necessarily a full day with large drives. I can agree with that.

But you may want to just tune your RAID card to do the rebuild a little slower and affect performance less. The downside of this is that the array is in degraded mode longer, where a second drive failure before the rebuild is complete will cause you to lose data. But the chance of a second drive failure (due to MTBF) in this several hour to several day window is extremely low. If it’s an environmental failure, like high temp, then you’ll probably doomed anyway, regardless of the size of the rebuild window.

Next time, I’ll show you why.

Thanks, Mike. If you read this I hope you take it as constructive criticism. There was certainly no malice intended.

TT

This entry was posted on Tuesday, August 8th, 2006 and is filed under Storage Interconnects & RAID, Advisor - Tom. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.