Question to the Storage Advisors, from Olaf: Why is such a large drive buffer/cache needed, considering controllers and the host have much larger caches anyway? And/or, why is read-ahead data not forwarded to the host immediately?

Olaf, the question of where to put the buffers has been around for years and years. As you point out, you have three general areas: the OS, the controller and the drive. We could actually break the OS into several areas, such as the application, the filesystem and the driver, but let’s just leave it simple for now and treat the OS as one unit.

First, let’s look at reads. Clearly the shortest latency for a cache hit on reads would be from the OS cache. Getting the same cache hit on the controller or drive would only add latency. So for this reason, controllers and drives typically don’t concentrate on caching read data. The OS cache is bigger and more efficient.

Next is the issue of read-aheads. You asked why we don’t just forward those to the OS immediately. Well, we could only do that if the OS asked for the data. And the game of speculatively reading data all the way from the drive to the OS that may or may not be used is very risky. It turns out that the least risky place to perform read-ahead is on the drive. While the media is rotating to find the requested data, the read-ahead data may simply pass underneath the drive head. There is no overhead to reading this data, other than finding a place in the drive’s buffer to store it. And after the read completes, if the drive doesn’t have another command to process, it might as well read in more data – especially if it detects that the last few commands have been sequential.

Lastly is the write-buffer. Allowing dirty writes to “build up” in a buffer serves two purposes. One is that multiple writes to the same blocks will be reduced to only one disk write, saving disk seeks and rotations. The other is that short writes may be combined into highly efficient long writes, again saving disk seeks and rotations. It would seem that putting this buffer in host memory would make the most sense because it reduces latency, just like reads. But that data is at risk. If you lose power, you lose your data. You have to make sure that the write-buffer memory has a battery in case of power failure. And you typically want the battery to last 24 hours, or maybe more if you want to make it through the weekend. [A UPS will only last a few minutes.] Since it’s impossible to put a battery on OS memory or drive memory, the only logical place to put it is on the controller.

In summary, you need a buffer in all three locations for different purposes. You can start combining them, but you will lose either performance or reliability.

TT

This entry was posted on Monday, November 20th, 2006 and is filed under Storage Interconnects & RAID, Advisor - Tom Treadway. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.