|Subject:||Re: [ZODB-Dev] ZODB Ever-Increasing Memory Usage even with cache-size-bytes|
|Date:||Mon, 10 May 2010 17:16:20 -0700|
I've incorporated everybody's advice, but I still can't get memory to obey cache-size-bytes. I'm using the new 3.10 from pypi (but the same behavior happens on the server where I was using 3.10 from the new lucid apt repos).
I'm going through a mapping where we take one long integer "docid" and map it to a collection of long integers ("wordset") and trying to invert it into a mapping for each '"wordid" in those wordsets to a set of the original docids ("docset").
I've even tried calling cacheMinimize after every single docset append, but reported memory to the OS never goes down and the process continues to allocate like crazy.
I'm wrapping ZODB in a "ZMap" class that just forwards all the dictionary methods to the ZODB root and allows easy interchangeability with my old sqlite OODB abstraction.
Here's the latest version of my code, (minorly instrumented...see below):
max_docset_size = 0
for docid, wordset in docid_to_wordset.iteritems():
for wordid in wordset:
docset = wordid_to_docset[wordid]
docset = array('L')
if len(docset) > max_docset_size:
max_docset_size = len(docset)
print 'Max docset is now %d (owned by wordid %d)' % (max_docset_size, wordid)
wordid_to_docset[wordid] = docset
n_docs_traversed += 1
if n_docs_traversed % 100 == 1:
if n_docs_traversed % 50000 == 1:
except KeyboardInterrupt, ex:
self.log_write('Caught keyboard interrupt, committing...')
I'm keeping track of the greatest docset (which would be the largest possible thing not able to be paged out) and its only 10,152 longs (at 8 bytes each according to the array module's documentation) at the point 75 seconds into the operation when the process has allocated 224 MB (on a cache_size_bytes of 64*1024*1024).
On a lark I just made an empty ZMap in the interpreter and filled it with 1M unique strings. It took up something like 190mb. I committed it and mem usage went up to 420mb. I then ran cacheMinimize (memory stayed at 420mb). Then I inserted another 1M entries (strings keyed on ints) and mem usage went up to 820mb. Then I committed and memory usage dropped to ~400mb and went back up to 833mb. Then I ran cacheMinimize again and memory usage stayed there. Does this example (totally decoupled from any other operations by me) make sense to experienced ZODB people? I have really no functional mental model of ZODB's memory usage patterns. I love using it, but I really want to find some way to get its allocations under control. I'm currently running this on a Macbook Pro, but it seems to be behaving the same way on Windows and Linux.
I really appreciate all of the help so far, and if there're any other pieces of my code that might help please let me know.
On Mon, May 10, 2010 at 3:18 PM, Jim Fulton <jim@xxxxxxxx> wrote:
Stanford Computer Science
BS '09, MS '10
_______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@xxxxxxxx https://mail.zope.org/mailman/listinfo/zodb-dev
|<Prev in Thread]||Current Thread||[Next in Thread>|
|Previous by Date:||Re: [ZODB-Dev] ZODB Ever-Increasing Memory Usage (even with cache-size-bytes), Jim Fulton|
|Next by Date:||Re: [ZODB-Dev] ZODB Ever-Increasing Memory Usage (even with cache-size-bytes), Ryan Noon|
|Previous by Thread:||Re: [ZODB-Dev] ZODB Ever-Increasing Memory Usage (even with cache-size-bytes), Jim Fulton|
|Next by Thread:||Re: [ZODB-Dev] ZODB Ever-Increasing Memory Usage (even with cache-size-bytes), Ryan Noon|
|Indexes:||[Date] [Thread] [Top] [All Lists]|