On Sep 6, 2009, at 7:53 AM, Ross Walker wrote:
On Sun, Sep 6, 2009 at 9:15 AM, James Lever<j@xxxxxxxxxxxx> wrote:
I’m experiencing occasional slow responsiveness on an OpenSolaris
system typically noticed when running an ‘ls’ (no extra flags, so no
directory service lookups). There is a delay of between 2 and 30
but no correlation has been noticed with load on the server and the
return. This problem has only been noticed via NFS (v3. We are
to NFSv4 once the O_EXCL/mtime bug fix has been integrated -
snv_124). The problem has been observed both locally on the primary
filesystem, in an locally automounted reference (/home/foo) and
I'm confused. If "This problem has only been noticed via NFS (v3" then
how is it "observed locally?"
zpool is RAIDZ2 comprised of 10 * 15kRPM SAS drives behind an LSI
512MB BBWC exposed as RAID0 LUNs (Dell MD1000 behind PERC 6/E) with
each partitioned as 10GB slog and 36GB remainder as l2arc behind
1078 w/ 256MB BBWC (Dell R710 server with PERC 6/i).
The system is configured as an NFS (currently serving NFSv3), iSCSI
(COMSTAR) and CIFS (using the SUN SFW package running Samba 3.0.34)
authentication taking place from a remote openLDAP server.
Automount is in use both locally and remotely (linux clients).
/home/* is remounted from the zpool, remotely /home and another
(and children) are mounted using autofs. There was some suspicion
automount is the problem, but no definitive evidence as of yet.
The problem has definitely been observed with stats (of some form,
‘/usr/bin/ls’ output) both remotely, locally in /home/* and locally
/zpool/home/* (the true source location). There is a clear
recency of reads of the directories in question and reoccurrence of
fault in that one user has scripted a regular (15m/30m/hourly tests
‘ls’ of the filesystems of interested and this has reduced the
fault to have
minimal noted impact since starting down this path (just for
iostat(1m) is the program for troubleshooting performance issues
related to latency. It will show the latency of nfs mounts as well as
I have removed the l2arc(s) (cache devices) from the pool and the
behaviour has been observed. My suspicion here was that there was
occasional high synchronous load causing heavy writes to the slog
and when a stat was requested it may have been faulting from ARC to
prior to going to the primary data store. The slowness has been
since removing the extra cache devices.
Another thought I had was along the lines of fileystem caching and
writes causing read blocking. I have no evidence that this is the
some suggestions on list recently of limiting the ZFS memory usage
caching. Can anybody comment to the effectiveness of this (I have
write cache in front of the slog SSDs and 512MB in front of the
stat(2) doesn't write, so you can stop worrying about the slog.
My DTrace is very poor but I’m suspicious that this is the best way
cause this problem. If somebody has any code that may assist in
this problem and was able to share it would much appreciated.
Any other suggestions for how to identify this fault and work
would be greatly appreciated.
Rule out the network by looking at retransmissions and ioerrors
with netstat(1m) on both the client and server.
That behavior sounds a lot like a process has a memory leak and is
filling the VM. On Linux there is an OOM killer for these, but on
OpenSolaris, your the OOM killer.
See rcapd(1m), rcapadm(1m), and rcapstat(1m) along with the
"Physical Memory Control Using the Resource Capping Daemon"
in System Administration Guide: Solaris Containers-Resource
Management, and Solaris Zones
You have iSCSI, NFS, CIFS to choose from (most obvious), try
restarting them one at a time during down time and see if performance
improves after each restart to find the culprit.
zfs-discuss mailing list
zfs-discuss mailing list