I will also point out that adding more drives to your stripe width is a two–edged sword; especially with today’s Disk capacities increasing, soon to be 1TB per disk; The more high-capacity drives you have in a RAID set, the more likely it is to get bit errors and or a drive failure that will require a RAID rebuild; and RAID re-build / reconstruction times of these drives can be very long. I refer to my astute colleague, Tom Treadway, who has written several blogs about how MTDDL decreases as drive count increases - mostly due to BER. Not to be redundant in this Blog entry, his articles can be found at our Blog page referencing several titles: NOT everybody loves SATA, Is RAID-6 made of wood?, Real-life RAID reliability; and lastly, RAID reliability calculations.

From a performance perspective, my testing has shown that a stripe width of 8 to 12 drives is usually maximum for most controllers or storage arrays. Beyond that, you are usually maxing out the interface to the host or the backend of the storage array. Moreover, the more RAID sets you have, the more isolated your failures will be to a single data set that won’t affect data on the other RAID sets. If you go larger, higher RAID levels, like RAID 6, 50 or 60 are more “data resilient ” choices.

There really aren’t many practical reasons to go with a stripe width much larger, unless you have a hugely performance dependent application like -for real-time data capture - holding temporary/transitory data meaning, you don’t plan on keeping the data there very long, at least not without backing it up or having a copy somewhere else.

SR

Dean, there are several ways to look at this issue. Some of the areas I point out below are pretty obvious, but for the sake of completeness I’ll cover it all:

Storage efficiency: As the disk count increases so does the storage efficiency. This is because there is one disk’s worth of redundancy (parity) per array. For example a 3-disk RAID-5 has one disk’s worth of parity and two disk’s worth of usable space, therefore the efficiency is 67%, i.e., 67% of the total disk space is available for user data. Likewise, a 10-disk RAID-5 has an efficiency of 90%. As a formula, it looks like this:

Efficiency = (DiskCount-1) / DiskCount

Degraded performance: A degraded RAID-5 is an array with a failed disk. If the user tries to read a block on the failed disk the RAID software will have to access all the other disks in the array to reconstruct that missing data. However if the user tries to read a block on one of the remaining good disks then nothing special happens. The data is simply read from the disk.

So let’s go back to the 3-disk example and assume a single failed disk. And let’s assume that the user is reading just one block, and let’s see how that read transforms to one or more disk accesses. If the user reads the two good disks then each read will be converted to just one disk read. However if the user reads the bad disk then that read will turn into two disk reads (from the good disks). So on average, three random disk reads (one per disk) will result in four disk IOs, or an increase in IOs of 33% from the optimal array case. Now let’s look at the 10-disk array. A read from nine of the disks will result in just one IO, however a read to the bad disk will result in nine IOs, i.e., one read from each of the remaining good disks. So ten random reads will result in 18 disk IOs, an increase of 80%. As a formula, it looks like this:

IOIncrease = ((Disk Count-1) + (DiskCount-1) – DiskCount ) / DiskCount

which reduces to:

IOIncrease = (DiskCount-2) / DiskCount

Now let’s look at writes - they’re a little more complicated. Each host write will typically result in four disk IOs – two reads and two writes. One read and write will be on a data disk while the other read and write will be on a parity disk. We’ll need to see what happens if any of those IOs go to a bad disk. If a read touches a bad data disk then all the other disks will have to be read, just as in the previous example. However the data disk won’t be written, obviously, because the disk is bad. Therefore the read becomes (DiskCount - 1) reads, and the write just goes away. The parity disk still has both the read and write. So the total number of IOs is (DiskCount - 1) + 2.

If a read touches a bad parity disk, then nothing special happens. There is no parity to update and therefore the write to the data disk is just a plain ol’ write. The total number of IOs will go from four to just one.

OK, to summarize, normally a write causes 4 IOs. However if the data disk is bad the total IOs will increase to (DiskCount + 1). Likewise, if the parity disk is bad the total IOs will decrease to 1.

If we go back to our example of ten IOs spread evenly (one per disk) across ten disks, you’d see that 8 host writes will result in 4 IOs, one host write will result in 11 IOs and one will result in 1 IO, for a total of 44 IOs. In an optimal array all 10 host writes would result in 4 IOs each, of a total of 40 IOs. That’s only an increase of 10%. The formula looks like this:

Increase = ((DiskCount-2)*4 + (DiskCount+1) + 1 – (DiskCount*4)) / (DiskCount*4)

which reduces to:

Increase = (DiskCount-6) / (DiskCount*4)

It’s interesting to note that if the disk count is exactly 6 then the increase is zero and the total IOs don’t change. If the disk count is less than 6 then the total IOs actually drop!

Also, you may have noticed that I conveniently left out the XOR time. In general, assuming that the controller doesn’t have a memory bottleneck or the XOR isn’t done in software, then the XOR time is relatively small compared to the disk time, so it can be just left out of the equation.

Rebuild time: [Note that this section was reworded on July 18, 2007. An observant reader had noticed that I was in the weeds. ] When a bad disk is replaced it is re-created by reading from all the other disks in the array. Luckily these reads can be issued in parallel and therefore the rebuild time does NOT increase linearly as the disk count increase. In other words, to rebuild a 3-disk array will require reading two entire disks and writing a third disk. Likewise rebuilding a 10-disk array will require reading nine entire disks and writing a tenth disk. The time to read two disks should be roughly the same time to read nine disks. There are some additional complications regarding XOR and potential limitations in the hardware, but they don’t have too much effect on the rebuild time and probably aren’t interesting enough to repeat here.

Second disk failure: There is a small chance that another disk will fail before the first one is replaced. The chance of an array failing is simply the chance of a disk failing multiplied by the number of disks in the array. Therefore the more disks in the array, the higher the likelihood of at least one disk in the array failing. BTW, the chance of a disk failing is inversely proportional to the MTBF (Mean Time Before Failure) of the disk. MTBF is a common way of indicating disk reliability. Here’s the simple formula:

ChanceOfArrayFailure = ChanceOfDiskFailure * DiskCount

Some folks like to correlate multiple failures, meaning that if one disk fails then there is a higher chance (such as 10X) that a second disk will fail – possibly due to power supply problems, overheating, or shared cable issues (as with parallel SCSI). But really these chances are extremely small and it’s really not worth going into the detail again. A more thorough review can be found here.

Bit error during rebuild: This is probably the biggest negative with extremely large RAID-5 arrays. Every sector on the disk has a very small chance of being unreadable, even using error correction codes (ECC). This is referred to as the Bit Errors Rate (BER). A typical low-end SATA disk will have an uncorrectable bit error for every 10^14 bits read and a typical high-end SAS disk will have an uncorrectable bit error every 10^15 bits. These seem like big numbers, but keep in mind that there are almost 10^13 bits in a 1TB disk. That means that you’re almost guaranteed to get a bit error if you read from ten SATA disks.

So what does it mean if you get a bit error? Basically it means that one entire 512 byte sector will be unreadable, further meaning that the corresponding failed sector can’t be rebuilt. The bottom line is that you’ve just lost data. A good RAID controller will mark that sector “offline”, allowing the OS to get an error which will cause the user to restore the corresponding file from backup. A bad RAID controller will ignore the error (causing hidden data corruption) or will abort the rebuild, leaving the user one disk failure away from total loss of all data.

Going back to our 10-disk example, let’s assume that we’re using 1TB SATA disks with a total of 8.8×10^12 bits and an error rate of 10^14. We’ll have to read nine disks to rebuild the bad disk, resulting in 7.9×10^13 total read bits. Divide that by the 10^14 error rate and you have a 79% chance of getting a bit error. In other words, you probably won’t be able to rebuild the array!

(Note that the situation is often not that horrific. A good RAID controller will perform a continuous background media check looking for bit errors before the disk fails. If one is found then it is repaired while the array is still optimal. It’s difficult to say how much that improves your chances of rebuilding, but it’s generally accepted that background media checks are a “good thing”.)

Here’s the resultant formula:

ChanceOfDataLoss = (DiskCount–1) * DiskCapacityInBits / BER

In addition to minimizing the chance of a BER by performing the background scan, it’s common to divide the RAID-5 into multiple RAID-5 arrays combined with striping – more commonly referred to as RAID-50. All of the equations above can be easily adapted to a RAID-50 configuration.

Dean, I hope that helps answer your question. I realize it was WAY more information than you were looking for, but I was on a roll.

TT

Here’s a list of the topics that can be found at the webinar website.

How to Map Applications to RAID Configurations - Learn how to match the read and write access patterns in your daily applications to the best RAID configuration for your business.

Maximize Data Protection for Cost-Effective SATA Drives - You don’t have to compromise reliability when using SATA. We’ll show you how to deploy RAID 6 to provide maximum fault tolerance.

RAID Error Handling - Learn how comprehensive error handling capabilities built into Adaptec RAID increase the effectiveness of data protection.

Success with Snapshot Backups - Gain a thorough understanding of how you can enhance data protection with the fast backup and instant data access of virtual point-in-time snapshots.

Growing Data Protection as Your Server Grows - Get tips on how to manage RAID and remove the complexity of capacity management as your storage grows.

Software vs. Hardware RAID - Learn the advantages and disadvantages of software RAID and hardware RAID.

If anyone has any questions just drop me a line and I’ll start a blog post.

Enjoy,
TT

Good question. For those not verse in the dark arts of cache write-back strategies, we’re talking about methods for protecting user data that has been written by the OS but hasn’t actually made it to the media yet. It’s common for a disk controller to improve write performance by accepting the data from the OS and saying that it’s been written to disk, when in reality it’s still in memory (or disk, as suggested by the poster’s question). This technique is referred to as “write-back” because the data is written in the background. The opposite of write-back is “write-through” where the controller really does write the data to disk before telling the OS that it’s finished.

[Note that controllers aren’t the only things that have a write-back cache - the OS and drives also have one. But to avoid complicating a somewhat simple question, let’s just ignore those other caches for now. The OS has ways of protecting itself, and drive caches should be disabled if a write-back controller is being used.]

It’s very important that this un-written controller data is protected because it’s the only copy of that data. The OS thinks that the data is written to disk and therefore purges it from memory or wherever it came from. If a power failure occurs before this write-back data is written to disk, then it’s permanently lost. With large caches we’re talking about 100’s of MBs of data. And it can even be worse than that because the missing writes could be to a file structure or database, resulting in massive corruption and loss of files that aren’t even being accessed. It can be a real mess. And the user won’t know about it until they read back corrupted data – which isn’t always obvious.

So, on to the question: What’s the best way to protect this unwritten data? The most common approach is to simply put a battery on the disk controller. If power is lost to the system, including to the drives, the controller memory will transition to battery-backed mode and preserve any write-back data that hadn’t made it to disk yet. The battery is typically selected to provide at least 72 hours of backup time – protecting data across a weekend.

An alternative method, as suggested by the poster, is to save this write-back data to disk. There are different ways to implement this, but the most common is to “simply” store the data in a transaction log on the disk. Now, note that the data is typically stored on the disk (in either method – battery or log) using some form of RAID, protecting against data loss due to a drive failure. RAID-5 is a pretty commonly selected RAID level, but has very poor random write performance – a problem which just happens to be greatly alleviated by some form of write-back cache. So for this example, let’s assume that RAID is being used. This means that the write-back data being logged to disk should also be protected from disk failure. The easiest way to do this is to simply write the log file to two disks. (Some users prefer RAID-6 which protects against two drive failures, in which case the transaction log should be written to three disks!)

OK, now let’s look at the pros and cons of the controller-based battery and disk-based log approaches.

Backup Protection Time: A battery has a limited storage time – around 72 hours as previously pointed out. However a transaction log on disk can last almost indefinitely, i.e., the lifetime of the drive. So here the advantage clearly goes to disk-based logs. (BTW, some folks are looking at ways to automatically move the controller cache data to a more permanent storage device, like CompactFlash, allowing controller backup times similar to transaction log backup times. So this will eventually become moot.)

Life Expectancy: Another issue with batteries is that they don’t last forever. They eventually degrade and fail, lasting maybe a few years before they need to be replaced. Drives obviously don’t have this issue.

Capacity: This one is really a nit, but I figured I’d list it to be complete. If a controller has 256MB of memory, for example, then the transaction log will require 2×256MB of disk space, or 512MB. With 1TB drives, this one is a big fat “don’t care”.

Cost: Batteries and the associated circuitry probably add about $100 to the user-cost of a controller, while 512MB of disk space for the log is practically free. $100 might be a big deal for a home user (who probably doesn’t need RAID or write-back cache anyway), but it’s just another nit for serious IT folks. Once you add up the price of the motherboard, OS, drives, etc., $100 is in the noise.

Performance: So far the advantage has clearly gone to the disk log, but performance is probably the most important factor when choosing a cache backup protection method. With a battery-backed controller there are no additional steps to protect the data in cache. “It just works.” Of course there is a lot of magic in the hardware design to make it “just work”, but that has no effect on the performance. On the other hand, with disk-based logs the data has to be written to two different disks. This will probably entail two seeks, assuming that the drives had been servicing requests in some other section of the media. And eventually, that logged data will have to be read back from disk and moved to the permanent location – causing two more seeks and reads. So now a single OS write will cause four additional IOs to the disks.

So how the heck do we figure out the performance hit due to these four IOs? Let’s try this crude method:

Assume that RAID-5 is being used. Therefore each random OS write will cause four disk IOs - two reads and two writes. With disk-based logging there are four additional IOs to log and “un-log” the data for a total of eight IOs. Using this approach we can see that disk logging has twice as many IOs as controller battery-protected cache, therefore you get about a 2X difference in performance. Of course real performance modeling will be more complex than this, so just squint at the numbers and figure that the difference is anywhere from 50% to 150%. That’s a big dang difference.

The bottom line is that most users that are concerned with performance aren’t concerned with saving $100, therefore battery-backed cache is clearly the winner.

Enjoy,
TT

John, first, are you willing to temporarily dump your data to tape in order to reconfigure the storage? This is probably a requirement because I’m not aware of any RAID controller that can combine multiple logical drives without losing data.

Next, are these SCSI arrays created by an external SCSI-to-SCSI controller that you plan to keep? If so, you need to make sure the logical drives are presented in a way that makes the drives look enough like a physical drive to be striped by your new PCI RAID card. You’re probably ok on this point. Also, are these logical drives presented by the SCSI-to-SCSI controller all the same size? If not, you’re going to have trouble striping them and using all of your capacity since each component of the RAID-0 has to be the same size. (In other words, the minor stripe size has to be the same on all drives.) You might end up creating one RAID-0 across all the logical drives, and then another across the “extra” sections of the logical drives, and so on. And of course you’ll have to have a RAID card that supports this; most of them don’t. (Hopefully I’m clear about this – if not, let me know and I can draw a picture.)

The easiest way to fix this is simply attach all your drives to a single PCI RAID card and create a single new array. Don’t bother trying to stripe the arrays that you currently have. Just start from scratch.

Feel free to drop more details about drive count and sizes and I can make a more concrete recommendation on how it can be configured. Also, have you already figured out what RAID type you need to use?

TT

John, good question. First, let’s talk about the RAID-0 and RAID-1 combination. You mention mirroring two 4-drive RAID-0’s. But the more typical approach is to stripe four 2-drive RAID-1’s. The end result as far as performance and capacity is the same, but the reliability and rebuild time is better with the latter - striped RAID-1’s. The reason is that if a single drive fails in the former mirrored RAID-0’s case then the entire set of 4 drives in the RAID-0 is typically taken off-line. This means that the array can tolerate at most one drive failure, but all four drives will have to be rebuilt. However with the latter striped RAID-1’s case, a single failed drive causes just one mirror pair to be rebuilt. In fact, you could actually have one drive in each RAID-1 fail and still have the array on-line.

So as far as your original question, it doesn’t actually matter which version of RAID-10 is being used. Both versions will provide the same performance assuming that no drives are failed, and also assuming that the queue depth from the host is large enough and random enough to cause all the drives to be accessed.

In your example the access pattern is mostly reads and mostly random (because it’s a database). Just for the sake of comparison, let’s say that it is 100% reads and 100% random. The end result is that all eight of the drives will see a command, or an IO. That means that if each drive can do 100 IOs per second, then the RAID-10 can do 800 IOPS total.

On a RAID-5 with 100% reads there are no RAID-5 calculations other than the block redirection due to the striping, which is almost identical to the redirection in RAID-10. So the end result is that all eight drives are used (since parity is distributed across all drives), and therefore RAID-5 will do the same 800 IOPS as RAID-10.

So, if you’re doing 100% reads, the RAID levels are identical in performance.

But your data base isn’t 100% reads, so let’s look at what happens with 100% writes. With RAID-10, each host access will be written to two drives, so the array performance will drop to one-half, or 400 IOPS. However with RAID-5 each host access is converted to four IOs (two reads and two writes), so the array performance will drop to one-quarter, or about 200 IOPS. (It’s a little more complicated than that, but this is a pretty good estimate.)

Now you’ll have to determine the real read/write ratio and calculate the harmonic mean to estimate the performance impact of these write commands. For example, if the ration was 50:50, then the RAID-10 would get around 533 IOPS and the RAID-5 would get around 320 IOPS.

The bottom-line is that the more writes you have the more RAID-5 will hurt performance. But if writes are rare, then the improved capacity of RAID-5 may warrant the slight performance hit.

Good luck!

TT

Chris, first, thanks for the love. We’re feeling you, man.

You had a lot of good questions, so let’s try to handle them one at a time.

RAID-10 drive count: It’s true that a typical RAID-10 has to use an even number of drive since the array is basically a bunch of two-drive RAID-10s striped together. Obviously you realize this, but here’s a picture for the noobs (I say affectionately):

However there is a different form of RAID-10 that does support an odd drive count, and the Adaptec name for that RAID level is RAID-1E. (I guess “E” means “extended” or “extra”, not “even”. RAID-1o for “odd” probably would have been a better name, but it’s hard to tell the difference in a zero and an “o”.) Here’s a picture of a RAID-1E disk layout.

So you should probably consider RAID-1E with one hot spare instead of RAID-10 with two hot spares. RAID-1E would give you one more drive’s worth of performance, with no real drop in reliability.

RAID-10 IOPS calculation: I assume you’re talking about each drive getting 175 IOPS, right? That’s about 2X high for a purely random workload, but if the access pattern has a mix of short seeks and sequential IO, then I suppose I could squint enough to see 175 IOPS. So with reads to a RAID-10, every drive in the array should be able to process a command – assuming that the queue depth from the host is large enough. For a 10 drive array you’d probably need around 20 concurrent host commands to be assured that each drive had a command to process. This means that a read-only access pattern to a 10-drive RAID-10 should be able to give you 1750 IOPS. But on writes, since each command has to go to two drives, that number is cut in half to 875 IOPS. If you’ve got a mixture of reads and writes, then the real IOPS is somewhere in between.

Database distribution: If I understand you correctly, you plan to create two separate RAID-10 arrays, one per cage. And you plan to let each RAID-10 array support one intensive and one not-intensive database. In general, that sounds like a good idea. That should keep the workload evenly distributed across the drives. However, that gets me to the next question:

RAID-10 spanning cages: The RAID controller should definitely support spanning arrays across cages. Some people like to keep an array within a cage simply for the sake of portability. For example, they might want to move one entire cage (along with the databases) to a new server. You obviously can’t do that if the array spans both cages. However, having one array has a lot of advantages. For one, you have only one drive letter to deal with. If you have two logical drives, then you’re guaranteed to someday have one drive almost full while the other has plenty of space. And at that point you’re basically screwed. So having just one large logical drive is always more flexible.

Another benefit of a larger RAID-10 is that the performance of a single database can be twice as great (assuming that there are enough host commands to keep it busy). That may not be clear, so look at it this way. Imagine having two databases. One is on one RAID-10 while the other is on the other RAID-10. If both databases are busy, then all the drives are busy, and performance is maxed out. But now imagine just one of those databases being busy. The other is completely idle. Now you have half of your disks just sitting idle contributing nothing to the performance of the system. At this point you’ll be wishing that the active database spanned all the drive, allowing performance to basically double. So when you said that you have four separate databases, think about when they’re actually be used. For example, are the intensive databases always intensive at the same time?

I hope this helps. If you have any questions, you know where to find me.

TT

P.S. With all of these drives, you should also consider RAID-6. There are PLENTY of posts in this blog about RAID-6. Also, you didn’t mention whether you’re using SATA, SAS, or SCSI. From the drive sizes, I assume you’re using SCSI or SAS. You should take a serious look at SATA with RAID-6. You’ll get a LOT more capacity for a fraction of the cost, and even higher reliability than SCSI. The downside is that short random writes will suffer. I can’t tell from your post what your read/write ratio looks like, but if it’s mostly reads then RAID-6 is your friend.

Neal, it really depends on the operating system and how the application is writing the file. Plus, besides the file write, there are plenty of other disk accesses such as directory, FAT or inode reads and writes.

But your question made me curious, so I did a quick experiment with SysInternals’ Disk Monitor utility which traps IO type (read vs write), transfer length, and disk number. Then I simply copied a 120KB file from one drive to another, and captured the IOs to the destination drive.

Using Windows XP Pro it was nice to see that the OS issued large writes - 64KB when possible. So the 120KB file was written using just two IOs. But I also saw about a dozen other short (16KB or less) reads and writes to other sections of the disk, which I assume are directory and FAT updates.

Do a search for SysInternals and you will find Disk Monitor. Microsoft bought them last year, but I believe their tools are still free.

Enjoy,
TT

Darwin, let’s break down the bottlenecks in this picture.

First, each drive is capable of about 100MB/s from the media. Some drives are faster and some are slower, but this is a good average number.

Next, each SAS 3Gb link is capable of 300MB/s bit rate. But once you apply overhead and encoding conversion, you’re down around 270MB/s - let’s say 250MB/s just to be safe. So with a 4x link, you can get close to 1000MB/s.

Now let’s look at your PCIe 8x bus. Each 2.5GHz PCIe link can do 250MB/s. I’ve seen efficiency estimates as low as 70%, so let’s say that each link can do 175MB/s. Therefore an 8x bus can do 1400MB/s. In fact PCIe is full duplex so each bus can actually be transferring in both directions for a total of 2800MB/s. But this is all moot. The SAS 4x bus is limited to 1000MB/s, so the PCIe 8x bus is overkill.

Lastly, let’s look at your 10GbE link. That link can probably burst at around 1GB/s, or 1000MB/s. Coincidently, that almost exactly matches the SAS 4x throughput that we calculated earlier.

So it would seem that your system is nicely balanced with 10 drives feeding a 10GbE link. But of course it’s never that easy.

First, can your disk controller hit the 1000MB/s mark? That answer will depend a lot on whether you’re using RAID for protection against drive failure, and furthermore what type of RAID and what access pattern you’re using. For example, if you’re doing random IO to the disks, then each drive is spending the majority of its time waiting for the head to seek or the media to rotate. Using the 71 IOPS from the previous post on SATA IOPS measurement, and assuming a 4KB transfer size, each drive is supplying less than 300KB/s. Therefore it would take over 3000 drives to hit the magic 1000MB/s mark! That’s not going to happen.

Next, what is the overhead of the drivers and firmware sitting between the OS and the drives? This number is probably around 10%, so it’s not too interesting. We can move on.

The next biggest problem, after drive access pattern, is “what is driving the GbE link”? Is it iSCSI with a TOE to offload the TCP/IP processing? You mention that this is a file server, so I assume that you’re just running NAS. If so, you’ve got a ton of overhead in processing filesystem metadata. You’re probably lucky to get 100-200MB/s.

So you can probably see that you asked a very complex question with lots of unknown variables. But hopefully all the simple factoids that I presented will help you find the bottlenecks in your system and tune the performance accordingly.

Good luck.

TT

Michael, IOPS are very dependent on the access pattern. For this reason it’s normal to measure the “worse case” IOPS which are acheived on short random IOs. The IOPS can be calculated by simply adding the time to do an average seek to the time for a half rotation. Note that the transfer rate doesn’t come into play because it’s so insignificant compared to the seek and rotation time.

For example, the 500GB Western Digital SE16 SATA drive (WD5000KS) has a rotational rate of 7200 RPM. By inverting that number you get one rotation per 8.4ms. Therefore a half rotation is 4.2ms. This is also noted on the spreadsheet as “Average Latency”.

Next, the average seek times for reads are listed as 8.9ms, while writes are 10.9ms. That difference may seem odd at first, but drives eek out a little extra performance on reads by attempting to read the data before the head has settled on the track (and that settling for writes takes ~2ms). So, let’s say that you have an even mix of reads and writes, and therefore we’ll assume the average seek time is 9.9ms.

Adding 4.2ms and 9.9ms gives has an average random IO latency of 14.1ms. Since the drive can process, i.e., access the media, with just one command at a time, you can invert 14.1ms to get 71 IOPS.

You can start playing games to increase the IOPS - such as shorting the seeks by accessing a smaller section of the disk, or making the IO sequential so that requests are serviced from the drive cache. With these tricks you can get the IOPS to exceed 100,000!

Now you can see that it’s difficult to measure and specify IOPS. The number is somewhere between 71 and 100K. For this reason it’s normal to use the worse case, random access pattern value of ~71.

TT