zfs-discuss@opensolaris.org
[Top] [All Lists]

Re: [zfs-discuss] HAMMER

Subject: Re: [zfs-discuss] HAMMER
From: "Dave Johnson"
Date: Wed, 17 Oct 2007 03:56:28 -0700
From: "Robert Milkowski" <rmilkowski@xxxxxxxxxxx>
> LDAP servers with several dozen millions accounts?
> Why? First you get about 2:1 compression ratio with lzjb, and you also
> get better performance.

a busy ldap server certainly seems a good fit for compression but when i 
said "large" i meant, as in bytes and numbers of files :)

seriously, is anyone out there using zfs for large "storage" servers?  you 
know, the same usage that 90% of the storage sold in the world is used for ? 
(yes, i pulled that figure out of my *ss ;)

are my concerns invalid with the current implementation of zfs with 
compression ?  is the compression so lightweight that it can be decompressed 
as fast as the disks can stream uncompressed backup data to tape while the 
server is still servicing clients ?  the days of "nightly" backups seem long 
gone in the space I've been working in the last several years... backups run 
almost 'round the clock it seems on our biggest systems (15-30Tb and 
150-300mil files , which may be small by the standard of others of you out 
there.)

what really got my eyes rolling about c9n and prompted my question was all 
this talk about gzip compression and other even heavierweight compression 
algor's.  lzjb is relatively lightweight but i could still see it being a 
bottleneck in a 'weekly full backups' scenario unless you had a very new 
system with kilowatts of cpu to spare.  gzip ? pulease.  bzip and lzma 
someone has *got* to be joking ?  i see these as ideal candiates for AVS 
scenarios where the aplication never requires full dumps to tape, but on a 
typical storage server ?  the compression would be ideal but would also make 
it impossible to backup in any reasonable "window".

back to my postulation, if it is correct, what about some NDMP interface to 
ZFS ?  it seems a more than natural candidate.  in this scenario, 
compression would be a boon since the blocks would already be in a 
compressed state.  I'd imagine this fitting into the 'zfs send' codebase 
somewhere.

thoughts (on either c9n and/or 'zfs send ndmp') ?

-=dave

----- Original Message ----- 
From: "Robert Milkowski" <rmilkowski@xxxxxxxxxxx>
To: "Dave Johnson" <dj4904@xxxxxxxxxxx>
Cc: "roland" <devzero@xxxxxx>; <zfs-discuss@xxxxxxxxxxxxxxx>
Sent: Wednesday, October 17, 2007 2:35 AM
Subject: Re[2]: [zfs-discuss] HAMMER


> Hello Dave,
>
> Tuesday, October 16, 2007, 9:17:30 PM, you wrote:
>
> DJ> you mean c9n ? ;)
>
> DJ> does anyone actually *use* compression ?  i'd like to see a poll on 
> how many
> DJ> people are using (or would use) compression on production systems that 
> are
> DJ> larger than your little department catch-all dumping ground server.  i 
> mean,
> DJ> unless you had some NDMP interface directly to ZFS, daily tape backups 
> for
> DJ> any large system will likely be an excersize in futility unless the 
> systems
> DJ> are largely just archive servers, at which point it's probably smarter 
> to
> DJ> perform backups less often, coinciding with the workflow of migrating
> DJ> archive data to it.  otherwise wouldn't the system just plain get 
> pounded?
>
> LDAP servers with several dozen millions accounts?
> Why? First you get about 2:1 compression ratio with lzjb, and you also
> get better performance.
>
>
> -- 
> Best regards,
> Robert Milkowski                            mailto:rmilkowski@xxxxxxxxxxx
>                                       http://milek.blogspot.com
>
> 

_______________________________________________
zfs-discuss mailing list
zfs-discuss@xxxxxxxxxxxxxxx
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

<Prev in Thread] Current Thread [Next in Thread>