> From: Asif Iqbal [mailto:vadud3@xxxxxxxxx]
> currently to speed up the zfs send| zfs recv I am using mbuffer. It
> moves the data
> lot faster than using netcat (or ssh) as the transport method
Yup, this works because network and disk latency can both be variable. So
without buffering, your data stream must instantaneously go the speed of
whichever is slower: The disk or the network.
But when you use buffering, you're able to go as fast as the network at all
times. You remove the effect of transient disk latency.
> that is why I thought may be transport it like axel does better than
> axel let you create multiple pipes, so you get the data multiple times
> than with wget.
If you're using axel to download something from the internet, the reason
it's faster than wget is because your data stream is competing against all
the other users of the internet, to get something from that server across
some WAN. Inherently, all the routers and servers on the internet will
treat each data stream fairly (except when explicitly configured to be
unfair.) So when you axel some file from the internet using multiple
threads, instead of wget'ing with a single thread, you're unfairly hogging
the server and WAN bandwidth between your site and the remote site. Slowing
down everyone else on the internet who are running with only 1 thread each.
Assuming your zfs send backup is going local, on a LAN, you almost certainly
do not want to do that.
If your zfs send is going across the WAN ... maybe you do want to
multithread the datastream. But you better ensure it's encrypted.
zfs-discuss mailing list