zfs-discuss@opensolaris.org
[Top] [All Lists]

Re: [zfs-discuss] Ideal SATA/SAS Controllers for ZFS

Subject: Re: [zfs-discuss] Ideal SATA/SAS Controllers for ZFS
From: Marc Bevand
Date: Wed, 26 May 2010 20:35:44 -0700
On Wed, May 26, 2010 at 6:09 PM, Giovanni Tirloni <gtirloni@xxxxxxxxxxxx> wrote:
> On Wed, May 26, 2010 at 9:22 PM, Brandon High <bhigh@xxxxxxxxxx> wrote:
>>
>> I'd wager it's the PCIe x4. That's about 1000MB/s raw bandwidth, about
>> 800MB/s after overhead.
>
> Makes perfect sense. I was calculating the bottlenecks using the
> full-duplex bandwidth and it wasn't apparent the one-way bottleneck.

Actually both of you guys are wrong :-)

The Supermicro X8DTi mobo and LSISAS9211-4i HBA are both PCIe 2.0 compatible,
so the max theoretical PCIe x4 throughput is 4GB/s aggregate, or 2GB/s in each
direction, well above the 800MB/s bottleneck observed by Giovanni.

This bottleneck is actually caused by the backplane: Supermicro "E1" chassis
like Giovanni's (SC846E1) include port multipliers that degrade performance
by putting 6 disks behind a single 3Gbps link.

A single 3Gbps link provides in theory 300MB/s usable after 8b-10b encoding,
but practical throughput numbers are closer to 90% of this figure, or 270MB/s.
6 disks per link means that each disk gets allocated 270/6 = 45MB/s.

So with 18 disks striped, this gives a max usable throughput of 18*45 = 810MB/s,
which matches exactly what Giovanni observed. QED!

-mrb
_______________________________________________
zfs-discuss mailing list
zfs-discuss@xxxxxxxxxxxxxxx
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

<Prev in Thread] Current Thread [Next in Thread>