java-user@lucene.apache.org
[Top] [All Lists]

RE: Reloading RAM Directory from updated FS Directory

Subject: RE: Reloading RAM Directory from updated FS Directory
From: "Uwe Schindler"
Date: Wed, 10 Jun 2009 12:35:41 +0200
There is currently a patch/idea from Earwin around that modifies
MMapDirectory to optionally call MappedByteBuffer.load() after mapping a
file from the directory. MappedByteBuffer.load() tells the operating system
kernel to try to swap as much as possible from the file into physical RAM.

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@xxxxxxxxxxx

> -----Original Message-----
> From: eks dev [mailto:eksdev@xxxxxxxxxxx]
> Sent: Wednesday, June 10, 2009 12:30 PM
> To: java-user@xxxxxxxxxxxxxxxxx
> Subject: Re: Reloading RAM Directory from updated FS Directory
> 
> 
> there is one case where MMAP does not beat RAM, initial warm-up after
> process restart. With MMAP it can take a while before you get up to speed.
> MMAP with reopen is the best, if you run without restart.
> 
> 
> 
> 
> 
> ----- Original Message ----
> > From: Uwe Schindler <uwe@xxxxxxxxxxx>
> > To: java-user@xxxxxxxxxxxxxxxxx
> > Sent: Wednesday, 10 June, 2009 0:24:35
> > Subject: RE: Reloading RAM Directory from updated FS Directory
> >
> > Hi Greg & Kay,
> >
> > > Lucene (IndexSearchers / IndexReaders) has the notion of cache as well
> > > so you need to check if you really want a 100% replication of
> > > RAMDirectory / FSDirectory as well concurrently in memory.  Have you
> > > tested with the FieldCache policies before moving onto the
> RAMDirectory
> > > based backup solution.
> >
> > As you have so big indexes, I think you use a 64 bit machine and 64 bit
> JVM.
> > I would recommend you in this case to use MMapDirectory instead of
> copying
> > the contents to a RAMDirectory. MMapDirectory works like FSDirectory,
> but it
> > does not read the files via normal RandomAccessFile actions, but it maps
> the
> > file into the 64 bit address space. In principle this is like a swap
> file,
> > the contents of the file are seen like memory. The operating system
> kernel
> > then uses the same strategies to map parts of the index into real RAM,
> just
> > like a swap file. For Lucene it looks like a RAMDirectory backed by a
> file.
> >
> > Changes to the underlying index can be seen immediately, and changes can
> > reloaded using IndexReader.reopen(), which is much faster than reopening
> the
> > index in complete.
> >
> > Kay Kay: Instantiating an IndexSearcher on top of a IndexReader costs
> > nothing (its just a wrapper around the IndexReader). The costly part is
> > opening the IndexReader. So using reopen() and wrapping this always with
> a
> > fresh IndexReader is faster than creating an IndexSearcher using the
> > directory path.
> >
> > Uwe
> >
> > > Diamond, Greg wrote:
> > > > Hi All -
> > > >
> > > > What is the best way to load a RAM Directory from a FS Directory,
> and
> > > periodically reload the RAM Directory to pick up new documents?
> > > >
> > > > The scenario I have is I create several large directories which I
> create
> > > to a file system, then load them into ram for faster searching.
> > > > They takes several hours to create, so i want to retain a file copy
> so
> > > in the event of a service/server crash or reboot, they can load again
> > > > in a few seconds.  Once a day I append or recreate the FS
> Directories
> > > with new entries, and then reload the RAM Directories to pick up the
> > > > new entries.
> > > >
> > > > Problem is, what I am doing seems to cause a memory leak.  The
> > > directories take ~12 GB of RAM to load, but bloats
> > > > to 24 GB (the -Xmx setting) after a few reloads and stays there.
> > > >
> > > >     // Map to hold single instance of the IndexSearcher, 1 per RAM
> > > Directory, to be reused across requests.
> > > >     private static final Mapsearchers = new
> > > HashMap();
> > > >
> > > >     // Creates the IndexSearcher and stores it in the static map.
> > > >     public static void load(IndexName index, ...) {
> > > >         // sync... etc
> > > >         RAMDirectory dir = new
> > > RAMDirectory(FSDirectory.getDirectory(directoryPath);
> > > >         IndexReader reader = IndexReader.open(dir, true);
> > > >         IndexSearcher searcher = new IndexSearcher(reader);
> > > >         searchers.put(index, searcher);
> > > >     }
> > > >
> > > >     public static IndexSearcher get(IndexName index, ...) {
> > > >         // sync... etc
> > > >         return searchers.get(index);
> > > >     }
> > > >
> > > >     // After an indexing service appends the existing FS Directory,
> this
> > > reloads it.
> > > >     public static void reload(IndexName index, ...) {
> > > >         // sync... etc
> > > >         IndexSearcher searcher = searchers.get(index);
> > > >         FSDirectory fsDirectory =
> > > FSDirectory.getDirectory(directoryPath);
> > > >         Directory ramDirectory =
> > > searcher.getIndexReader().directory();
> > > >         Directory.copy(fsDirectory, ramDirectory, false);
> > > >     }
> > > >
> > > > I've tries some theme and variations on this with the same issue.
> > > >
> > > > TIA!
> > > >
> > > > gd
> > > >
> > > >
> > > > --------------------------------------------------------------------
> -
> > > > To unsubscribe, e-mail: java-user-unsubscribe@xxxxxxxxxxxxxxxxx
> > > > For additional commands, e-mail: java-user-help@xxxxxxxxxxxxxxxxx
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@xxxxxxxxxxxxxxxxx
> > > For additional commands, e-mail: java-user-help@xxxxxxxxxxxxxxxxx
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@xxxxxxxxxxxxxxxxx
> > For additional commands, e-mail: java-user-help@xxxxxxxxxxxxxxxxx
> 
> 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@xxxxxxxxxxxxxxxxx
> For additional commands, e-mail: java-user-help@xxxxxxxxxxxxxxxxx



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@xxxxxxxxxxxxxxxxx
For additional commands, e-mail: java-user-help@xxxxxxxxxxxxxxxxx

<Prev in Thread] Current Thread [Next in Thread>