On Wed, May 26, 2010 at 6:09 PM, Giovanni Tirloni <gtirloni@xxxxxxxxxxxx> wrote:
> On Wed, May 26, 2010 at 9:22 PM, Brandon High <bhigh@xxxxxxxxxx> wrote:
>> I'd wager it's the PCIe x4. That's about 1000MB/s raw bandwidth, about
>> 800MB/s after overhead.
> Makes perfect sense. I was calculating the bottlenecks using the
> full-duplex bandwidth and it wasn't apparent the one-way bottleneck.
Actually both of you guys are wrong :-)
The Supermicro X8DTi mobo and LSISAS9211-4i HBA are both PCIe 2.0 compatible,
so the max theoretical PCIe x4 throughput is 4GB/s aggregate, or 2GB/s in each
direction, well above the 800MB/s bottleneck observed by Giovanni.
This bottleneck is actually caused by the backplane: Supermicro "E1" chassis
like Giovanni's (SC846E1) include port multipliers that degrade performance
by putting 6 disks behind a single 3Gbps link.
A single 3Gbps link provides in theory 300MB/s usable after 8b-10b encoding,
but practical throughput numbers are closer to 90% of this figure, or 270MB/s.
6 disks per link means that each disk gets allocated 270/6 = 45MB/s.
So with 18 disks striped, this gives a max usable throughput of 18*45 = 810MB/s,
which matches exactly what Giovanni observed. QED!
zfs-discuss mailing list