Question to the Storage Advisors, from Chris: Need some quick insight if you have a moment. This is for an Oracle db server btw. We’re buying 2 cages that use a PCIe interface and 24 146G 15K drives. We’re going to use RAID 10. Each cage only holds 12 drives. We’ll be using 2 spares in each cage (since we have to have 1 spare and can’t use RAID 10 on 11 drives). So that’s a stripe of 5 disk groups. At 175 IOPs that’s approx 875 IOPs across the stripe (if I’m using the correct terminology). If we go higher than 875 IOPs we begin seeing higher queue lengths correct? Both cages are hooked to the same server thru the PCIe controller that supports up to 8Gb/s. This is for an Oracle db, and will host 4 dbs. 2 dbs are IO intensive, 2 are not. Right now we plan on separating the 2 IO intensive dbs to 1 cage each. So here is my question: Would it be better to use both cages per db? I don’t think you can stripe disks across cages(?), so would it be better to create two stripes and put datafiles from both dbs on both cages so that we effectively use 2 stripes of 5 disk groups? Does that make sense? (Love your blogs by the way!)

Chris, first, thanks for the love. We’re feeling you, man.

You had a lot of good questions, so let’s try to handle them one at a time.

RAID-10 drive count: It’s true that a typical RAID-10 has to use an even number of drive since the array is basically a bunch of two-drive RAID-10s striped together. Obviously you realize this, but here’s a picture for the noobs (I say affectionately):

However there is a different form of RAID-10 that does support an odd drive count, and the Adaptec name for that RAID level is RAID-1E. (I guess “E” means “extended” or “extra”, not “even”. RAID-1o for “odd” probably would have been a better name, but it’s hard to tell the difference in a zero and an “o”.) Here’s a picture of a RAID-1E disk layout.

So you should probably consider RAID-1E with one hot spare instead of RAID-10 with two hot spares. RAID-1E would give you one more drive’s worth of performance, with no real drop in reliability.

RAID-10 IOPS calculation: I assume you’re talking about each drive getting 175 IOPS, right? That’s about 2X high for a purely random workload, but if the access pattern has a mix of short seeks and sequential IO, then I suppose I could squint enough to see 175 IOPS. So with reads to a RAID-10, every drive in the array should be able to process a command – assuming that the queue depth from the host is large enough. For a 10 drive array you’d probably need around 20 concurrent host commands to be assured that each drive had a command to process. This means that a read-only access pattern to a 10-drive RAID-10 should be able to give you 1750 IOPS. But on writes, since each command has to go to two drives, that number is cut in half to 875 IOPS. If you’ve got a mixture of reads and writes, then the real IOPS is somewhere in between.

Database distribution: If I understand you correctly, you plan to create two separate RAID-10 arrays, one per cage. And you plan to let each RAID-10 array support one intensive and one not-intensive database. In general, that sounds like a good idea. That should keep the workload evenly distributed across the drives. However, that gets me to the next question:

RAID-10 spanning cages: The RAID controller should definitely support spanning arrays across cages. Some people like to keep an array within a cage simply for the sake of portability. For example, they might want to move one entire cage (along with the databases) to a new server. You obviously can’t do that if the array spans both cages. However, having one array has a lot of advantages. For one, you have only one drive letter to deal with. If you have two logical drives, then you’re guaranteed to someday have one drive almost full while the other has plenty of space. And at that point you’re basically screwed. So having just one large logical drive is always more flexible.

Another benefit of a larger RAID-10 is that the performance of a single database can be twice as great (assuming that there are enough host commands to keep it busy). That may not be clear, so look at it this way. Imagine having two databases. One is on one RAID-10 while the other is on the other RAID-10. If both databases are busy, then all the drives are busy, and performance is maxed out. But now imagine just one of those databases being busy. The other is completely idle. Now you have half of your disks just sitting idle contributing nothing to the performance of the system. At this point you’ll be wishing that the active database spanned all the drive, allowing performance to basically double. So when you said that you have four separate databases, think about when they’re actually be used. For example, are the intensive databases always intensive at the same time?

I hope this helps. If you have any questions, you know where to find me.

TT

P.S. With all of these drives, you should also consider RAID-6. There are PLENTY of posts in this blog about RAID-6. Also, you didn’t mention whether you’re using SATA, SAS, or SCSI. From the drive sizes, I assume you’re using SCSI or SAS. You should take a serious look at SATA with RAID-6. You’ll get a LOT more capacity for a fraction of the cost, and even higher reliability than SCSI. The downside is that short random writes will suffer. I can’t tell from your post what your read/write ratio looks like, but if it’s mostly reads then RAID-6 is your friend.

This entry was posted on Wednesday, March 21st, 2007 and is filed under Storage Interconnects & RAID, Advisor - Tom Treadway. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.