It seems to me that there is some detailed information which would
be needed for a full analysis. So, to keep the ball rolling, I'll
Jason J. W. Williams wrote:
Been watching the stats on the array and the cache hits are < 3% on
these volumes. We're very write heavy, and rarely write similar enough
data twice. Having random oriented database data and
sequential-oriented database log data on the same volume groups, it
seems to me this was causing a lot of head repositioning.
In general, writes are buffered. For many database workloads, the
sequential log writes won't be write cache hits and will be coelesced.
There are several ways you could account for this, but suffice to say
that the read cache hit rate is more interesting, for performance
improvement opportunities. The random reads are often cache misses,
and adding prefetch is often a waste of resources -- the nature of the
For ZFS, all data writes should be sequential until you get near the
capacity of the volume when there will be a search for free blocks
which may be randomly dispersed. One way to look at this is that
for new and not-yet-filled volumes, ZFS will write sequentially,
unlike other file systems. Once you get filled, then ZFS will write
more like other file systems. Hence, your write performance for
ZFS may change over time, though this will be somewhat mitigated by
the RAID array write buffer cache.
By shutting down the slaves database servers we cut the latency
tremendously, which would seem to me to indicate a lot of contention.
But I'm trying to come up to speed on this, so I may be wrong.
This is likely.
Note that RAID controllers are really just servers which speak a
block-level protocol to other hosts. Some RAID controllers are
ZFS on a modern server can create a significant workload. This
can also clobber a RAID array. For example, by default, ZFS will
queue up to 35 iops per vdev before blocking. If you have one
RAID array which is connected to 4 hosts, each host having 5 vdevs,
then the RAID array would need to be able to handle 700 (35 * 4 * 5)
concurrent iops. There are RAID arrays, which will remain nameless,
that will not handle that workload very well. Under lab conditions
you should be able to empirically determine the knee in the response
time curve as you add workload.
To compound the problem, fibre channel has pitiful flow control.
Thus it may also be necessary to throttle the concurrent iops at the
source. I'm not sure what the current thinking is on tuning
vq_max_pending (35) for ZFS, you might search for it in the archives.
[the intent is to have no tunables, let the system figure out what to
"iostat -xtcnz 5" showed the latency dropped from 200 to 20 once we
cut the replication. Since the masters and slaves were using the same
the volume groups and RAID-Z was striping across all of them on both
the masters and slaves, I think this was a big problem.
It is hard for me to visualize your setup, but this is a tell-tale
sign that you've overrun the RAID box. Changing the volume partitioning
will likely help, perhaps tremendously.
zfs-discuss mailing list