dev@directory.apache.org
[Top] [All Lists]

[jira] Commented: (DIRSERVER-1459) Adding members to a groupOfNames resu

Subject: [jira] Commented: (DIRSERVER-1459) Adding members to a groupOfNames results in polynomial increase in JDBM partition size
From: "Alex Karasulu (JIRA)"
Date: Sun, 24 Jan 2010 12:30:22 +0000 UTC
    [ 
https://issues.apache.org/jira/browse/DIRSERVER-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804233#action_12804233
 ] 

Alex Karasulu commented on DIRSERVER-1459:
------------------------------------------

I've been working on this issue for some time and have found out exactly why it 
is resulting.  First this is a general problem that occurs irregardless of 
uniqueMember attributes.  It's occurring due to the inability of JDBM to 
reclaim blocks used by secondary BTree objects for storing large numbers of 
values for duplicate keys.  The JdbmTable wrapper is designed to switch at some 
configurable threshold to begin using embedded BTrees to store values instead 
of serializing a TreeSet/ArraySet container data structure.

When this happens and a nested BTree is used, the space used cannot be 
reclaimed by JDBM when the BTree is deleted.  Plus modify operations for some 
reason are deleting and adding the values causing the data to be rewritten.  
This might be due to yet another issue all together in the design of the modify 
operation.  This will need to be investigated.  However the need to free blocks 
used by embedded secondary BTrees used for duplicate key value storage is a 
must.

I am looking at JDBM right now to see how we can make sure this does in fact 
happen properly.  This is not a small matter and is critical for proper 
operation.  I will keep you informed as this progresses.

> Adding members to a groupOfNames results in polynomial increase in JDBM 
> partition size
> --------------------------------------------------------------------------------------
>
>                 Key: DIRSERVER-1459
>                 URL: https://issues.apache.org/jira/browse/DIRSERVER-1459
>             Project: Directory ApacheDS
>          Issue Type: Bug
>    Affects Versions: 1.5.5
>         Environment: Any (tested on Linux and Mac OS X)
>            Reporter: Ben Hoyt
>            Assignee: Alex Karasulu
>            Priority: Blocker
>             Fix For: 1.5.6
>
>         Attachments: DIRSERVER-1459.tar.gz, screenshot-1.jpg
>
>
> I noticed a polynomial increase JDBM partition size and therefore disk usage 
> when adding users to groups in my ApacheDS instance.  The vast majority of 
> the usage (95+% once you hit a couple thousand users) is in 
> workingDirectory/partitionId/master.db  
> Further testing showed that simply adding a user is linear, as one would 
> expect, and as 'apacheds-tools capacity' confirms.  It is only when a user is 
> made a member of a group that the JDBM partition size shoots up.
> Example statistics:
> Add 16,000 users - JDBM partition size = ~70 megabytes
> Now add those same 16,000 users to a single group (all in the same group) - 
> JDBM partition size = ~19 GIGABYTES
> I'll work to attach a test case and some more numbers from my tests

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

<Prev in Thread] Current Thread [Next in Thread>