SAS initiator and target mix
Posted in Storage Interconnects & RAID, Advisor - Tom Treadway by Tom TreadwayQuestion to the Storage Advisors, from DavidL: Are there any constraints on the numbers of SAS initiators and targets you can mix & match?
David, in a SAS domain every device is treated the same whether it’s an initiator or a target. All devices require a SAS address. The maximum number of addresses supported in a typical domain is 128, however the specification does support over 16K addresses using fan-out expanders. The SAS protocol allows any number of these addresses to be initiators or targets. Of course, you need at least one of each type for a meaningful exchange of data.
Regardless of what the spec allows, multiple initiators sharing a set of drives is problematic because the host OSes behind those initiators typically don’t know about each other and any attempt at sharing data will quickly result in data corruption. Of course clustering is the method used for hosts to coordinate their accesses, but that comes with its own limitations as I mentioned in a previous post.
Additionally, due to constraints in memory size, maximum command queue depth, etc., HBA vendors will place limitations on the number of SAS devices they support. The first wave of products will typically support ~100 drives, but we should expect that count to grow as customers figure out new ways of using SAS to replace their FC infrastructure.
TT
March 14th, 2006 at 4:57 pm
I have been confounded by this particular question. If an HBA is limited to a 100 or so disk drives, can those disk exist in different Edge Domains?
So the typical SAS HBA has 4 or 8 ports - could it connect to 4 or 8 separate Edge Domains - as long as the total does not exceed the card’s limit? The SAS diagrams always show a host connected to the fan-out expander - which in turn connects multiple edge domains. What is limiting the host from seeing only a 100 disk in the total SAS network?
March 24th, 2006 at 12:17 pm
Joseph,
An HBA is allowed to straddle multiple SAS domains. As in your example, an HBA could have a 4X connector to one domain and another 4X connect to a different domain. Each of these domains can have >16K devices.
The limit of 100 disks is not a limit of expanders. It’s simply limited by the controller hardware. One reason is that each disk typically has a structure defining that disk, and that structure is usually kept on-chip for performance reasons. Also, each command to those disks has a structure. All this adds up to signficant memory. It appears that most of the chip vendors have chosen “~100″ as a big enough disk count.
But eventually we’ll see this change. As SAS becomes more prevalent, I’m sure high-end SAS controllers will start to support a LOT more than 100 drives.
TT