|
|
On May 30, 2010, at 3:04 PM, Sandon Van Ness wrote:
> Basically for a few seconds at a time I can get very nice speeds through
> rsync (saturating a 1 gig link) which is around 112-113 megabytes/sec
> which is about as good as I can expect after overhead. The problem is
> that every 5 seconds when data is actually written to disks (physically
> looking at the disk LEDs I see the activity on the one sending data
> stall as the ZFS machine is showing disk activity (writes).
>
> Basically this problem is bringing down the average write speed from
> around 112-113 megabytes/sec to around 100 megabytes/sec (sometimes
> lower) and thus lowering speeds by 10% (or a bit more). I only really
> care as 10% can make a difference when you are copying terrabytes of
> data to the machine. Anyway here is what I am seeing on the linux
> machine that is sending to the ZFS machine:
>
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
> avgrq-sz avgqu-sz await svctm %util
> sdc3 17.00 0.00 496.00 0.00 112.00 0.00
> 462.45 0.42 0.84 0.16 7.90
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 6.34 0.00 3.80 0.00 0.00 89.86
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
> avgrq-sz avgqu-sz await svctm %util
> sdc3 1.00 0.00 584.00 0.00 111.75 0.00
> 391.88 0.43 0.74 0.12 7.10
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 6.84 0.00 5.44 0.00 0.00 87.72
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
> avgrq-sz avgqu-sz await svctm %util
> sdc3 4.00 0.00 557.00 0.00 112.00 0.00
> 411.81 0.43 0.77 0.13 7.10
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 4.64 0.00 2.51 0.00 0.00 92.86
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
> avgrq-sz avgqu-sz await svctm %util
> sdc3 1.98 0.00 104.95 0.00 23.76 0.00
> 463.70 0.09 0.84 0.17 1.78
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 7.11 0.00 5.08 0.00 0.00 87.82
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
> avgrq-sz avgqu-sz await svctm %util
> sdc3 9.00 0.00 538.00 0.00 112.69 0.00
> 428.97 0.39 0.72 0.14 7.50
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 7.46 0.00 5.56 0.00 0.00 86.98
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
> avgrq-sz avgqu-sz await svctm %util
> sdc3 8.00 0.00 524.00 0.00 112.00 0.00
> 437.74 0.38 0.72 0.13 6.90
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 7.98 0.00 5.96 0.00 0.00 86.06
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
> avgrq-sz avgqu-sz await svctm %util
> sdc3 1.00 0.00 493.00 0.00 111.29 0.00
> 462.31 0.39 0.80 0.16 7.90
>
>
> This is iostat -xm 1 sdc3
>
> Basically you can see its reading at around full gig speed and then
> reads drop down due to writes being stalled on the ZFS machine. These
> are 1 second averages.
>
> On the ZFS end with 10 second averages I see approximately 100 MB/sec:
>
>
> data 25.1G 18.1T 0 834 0 100M
> data 26.2G 18.1T 0 833 0 100M
> data 27.3G 18.1T 0 833 0 100M
>
>
> Change this to 1 second and I see:
> data 32.7G 18.1T 0 0 0 0
> data 32.7G 18.1T 0 2.86K 0 360M
> data 33.3G 18.1T 0 264 0 21.8M
> data 33.3G 18.1T 0 0 0 0
> data 33.3G 18.1T 0 0 0 0
> data 33.3G 18.1T 0 0 0 0
> data 33.3G 18.1T 0 2.94K 0 369M
> data 33.8G 18.1T 0 375 0 35.1M
> data 33.8G 18.1T 0 0 0 0
> data 33.8G 18.1T 0 0 0 0
> data 33.8G 18.1T 0 0 0 0
> data 33.8G 18.1T 0 2.90K 0 365M
> data 34.4G 18.1T 0 599 0 62.6M
> data 34.4G 18.1T 0 0 0 0
> data 34.4G 18.1T 0 0 0 0
> data 34.4G 18.1T 0 0 0 0
> data 34.4G 18.1T 0 2.10K 0 265M
> data 34.9G 18.1T 0 1.77K 0 211M
>
>
> I tried changing the txg sync time from 30 to 1 and that did make things
> more smooth but in general lowered speeds (down to 90 megabytes/sec or
> so). Actually writing files to the array I see well excess of 112
> megabytes/sec so I would think I should be able to get this to go at
> full gig speeds without the small stalls:
I have better luck tuning the zfs_txg_synctime_ms from 5000 to 1000 or less.
>
> root@opensolaris: 11:36 AM :/data# dd bs=1M count=100000 if=/dev/zero
> of=./100gb.bin
> 100000+0 records in
> 100000+0 records out
> 104857600000 bytes (105 GB) copied, 233.257 s, 450 MB/s
If you want to decouple the txg commit completely, then you might consider
using a buffer of some sort. I use mbuffer for pipes, but that may be tricky
to use in an rsync environment.
-- richard
--
Richard Elling
richard@xxxxxxxxxxx +1-760-896-4422
ZFS and NexentaStor training, Rotterdam, July 13-15, 2010
http://nexenta-rotterdam.eventbrite.com/
_______________________________________________
zfs-discuss mailing list
zfs-discuss@xxxxxxxxxxxxxxx
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|