SAS fanout expanders
Posted in Storage Interconnects & RAID, Advisor - Tom Treadway by Tom TreadwayQuestion to the Storage Advisors, from anonymous: Is there a physical difference in edge and fanout expanders, or is it just a matter of where they exist in the SAS domain? In other words, if you surround an expander with edge expanders is that central expander considered a fanout?
To some extent, the location of the expander within the domain does have an effect on what it’s called. But there is an additional expectation on fanout expanders with regard to the addressing mode that they use. I addressed this in my post on directional SAS cables.
There are three types of addressing used in expanders. These addressing modes basically describe how commands and data are routed as they flow in one port and are redirected to another port. The following diagram shows an expander with all three modes.
Direct addressing is the simplest, with each expander port attaching directly to a device, like a drive or an initiator. So when an initiator tries to access address “a”, the expander will know from its boot scan which port is connected to address “a”.
Subtractive addressing is similar to the concept of a gateway on an Ethernet switch or router. If the expander gets an address that it doesn’t recognize, it simply passes the command or data off to the subtractive port. This port would be attached to another expander that would know how to get to that address. The subtractive port is often attached to the initiator.
Fanout expanders are typically defined by the presence of table based addressing and a high number of ports. This means that each port is connected, typically, to an edge expander that has multiple direct attached devices. Therefore the fanout expander has to keep a table, or list, of addresses for each of its ports.
Below is a system diagram using a small fanout expander connected to three small edge expanders.
I hope that answered your question.
TT
March 14th, 2006 at 10:22 am
Hi Tom,
Is there an e.t.a. on when seperate expanders (fanout or edge) are readily available ? So far I’ve only been able to find expanders from Rancho and those are “only” 12 ports (1×1p, 1×3p, 2×4p).
To get a bit of traction a 36 ports fanout expander would be nice. That would make multiple HBA’s within the same SAS domain a real posibility.
Regards,
Dennis
March 24th, 2006 at 12:22 pm
Dennis,
I believe Adaptec is shipping a 24-port expander in our external SAS JBOD. And I’ve heard that 36-port expanders are also shipping. These are both edge expanders, but fanout expanders will start shipping this summer.
BTW, are you talking about expanders in a box, with sheetmetal, power supply, etc? After re-reading your message, maybe that’s what you’re looking for. And if so, I 100% agree that it would be a great product. I’d also like to see a 1U rackmount version. As far as I know, nothing has been announced, but it’s pretty likely we’ll see something soon.
TT
March 30th, 2006 at 11:09 am
Tom,
Yes I’m talking about separate expanders in a 1U box preferably with Infiniband connections. This would be a real good replacement for local loop fiberchannel networks.
I’ve been round the internet a couple of times now. I’m still amazed that there is so little information out there about the real possiblities with SAS. For me it’s one of the bigger break throughs in IT tech in awhile. Mostly because that for once people actually thought about what the new standard should encompass. It’s a really great mix of technologies : more options and better options with regard to external storage, better use of existing tech with regard to SATA(II), good use of existing cabling standards, etc. etc.
Regards,
Dennis
April 3rd, 2006 at 12:40 pm
Dennis,
I couldn’t agree more. At the risk of incurring the wrath of the jack-booted FC thugs (just joshing guys!
), THIS is the SAN that people really need. I venture to guess that most of the FC SANs that people are using today are really just a small configuration of servers and storage connected by a switch. And it is all VERY expensive. And it’s really a misue of the word “SAN”.
Throw together a couple of inexpenive SAS RAID controllers, a few racks of inexpensive SATA drives, and an inexpensive SAS switch with zoning, and you’ve got a very high performance and very low cost solution.
I did a little snooping on the web, and found a few SAS switches from Rancho Technologies and Quanta Computers.
I bet the FC folks are starting to sweat.
TT
June 17th, 2006 at 1:43 pm
Dea Team,
Here is my question.
I have looked around to read up on SAS and expanders and how to connect devices together. I come from the SCSI age of addressing so I am about to start the purchase of SAS HDDs for my clients..My question had to do with addressing and how it is done on a) SAS adapter with 4 HDDs b) through expanders
Thanks in advance
June 22nd, 2006 at 12:14 am
To George’s questions, the SAS addresses of the drives are already set by the drive manufacturers, you can simply use a direct-attach connection topoogy (using 1x or 4x cables whichever connector is present on the HBA), everything should just start working.
To Expanders, pretty much the similar thing, pick a particular topology and connect the drives to the Expander Target ports, and connect the Expander to the HBA, through subtractive port(if Edge Exp) or an Initiator port (if Fanout). SAS addresses of the Expanders can be changed using their Serial Port or using their own Management Software, do it only if you really want to. To just use these hardware, just plug it in, it automatically works! Correct me if I am wrong, Tom!
Thanks -Ganesh
June 28th, 2006 at 8:59 am
Thanks, Ganesh. And if I could add to those good comments…
The SAS address is an 8-byte field that contains a unique company ID and a serial number. Assuming that a company never reuses a serial number, then the SAS address is guaranteed to be unique across multiple companies. As Ganesh said, this address is configured at the factory. And that applies to initiators (HBAs and RAID cards), expanders and drives.
These addresses, for example the drive addresses, remain constant whether they are directly attached to a RAID card or indirectly attached via an expander. The RAID card will use the same address regardless of how many expanders have to be traversed to get to the drive. And as I mentioned in the original post, the expander keeps tables of these addresses so it knows which port the command or data should be forwarded to.
Now, the dirty little secret is how those big ol’ 8-byte addresses are converted to something the OS, drivers and applications know about. A lot of software just knows about SCSI addressing, and is therefore limited to 16 IDs per bus. And of course with SAS point-to-point the concept of busses doesn’t apply, with these 8-byte addresses simply existing in a auto-detected domain of devices. So drivers, etc., end up jumping through hoops to “fake it”, and map SAS addresses to fake SCSI bus and ID numbers. And that’s fine as long as you just agree to look the other way and ignore the mapping. But don’t bother trying to figure it out because everyone does it differently.
TT