gmane.comp.apache.incubator.directory.devel

Re: Problem of JDBM being hard(impossible) to reconfigure once it is initialized

Alex Karasulu — 2012-05-23T19:05:00

+1

I agree with Emmanuel here. Such changes should not interrupt the
operation of the server.


I guess you mean the newly added index here above.

The real problem is that we won't be able to

OK let me see if I understand. To add an index we:

1). Note the version (V0) when the request to build the index arrives.
Start a full master table scan at version V0 to build the new index.

2). Once the index has been built for this version acquire an
exclusive write lock to prevent writes.

3a). If no changes occurred while building the index, meaning we're
still at V0. Then we enable the index and release the exclusive write
lock.

3b). If changes have occurred while building the index, then we update
the index with changes V1-V0. Then we enable the index and release the
exclusive write lock.


We just need an index state. It should be marked as deleted or offline
so when a new search comes in while waiting for the others to finish
using this index we simply do not use the index. So we need some index
state information.


+1


What other option is there?

You should not have to stop and re-start the server just to rename a
partition suffix. We simply need to mark (as we did for indices) the
partition as offline waiting until all current in progress operations
complete on that partition. New requests should not be processed for
it. Maybe we can return an unwilling to perform result response.

Then bring the partition online in the new suffix after doing the copy
and making all the needed changes. This should be a really fast
operation because the file system operation is rename. The other
suffix configuration changes are point changes. It should not take
long regardless of the size of the partition.

Re: Problem of JDBM being hard(impossible) to reconfigure once it is initialized

Göktürk Gezer — 2012-05-23T15:01:31

Quiscing the Ldap Operations is necessary at some point. Either in
DefaultOperationManager level of JdbmPartition level. If we handle that
other than DefaultOperationManager level, then we and 3th party developers
houldhave to ensure locking is ensured in consistent manner. I believe
quiscing the operations at DefaultOperationManager is our best shot, and
reconfiguration is not something occurs frequently. Clients will only get
late answers to their query when some reconfiguration is in place. Not a
big deal IMO?

What happens if some modification is done while we're executing step2 !
Again, we need locking at some level.



If we enter the deletion code while some operation is using indexes without
waiting for that operation to exit, this'll be clearly a concurrency
problem. Operation might reference to a deleted index at some point. We
need to wait for all operations to return, we shouldn't allow any new
operation using this index until we deleted the index. So we can set some
locking around index retrieval code, however this would be hard to
implement sacrifice to prevent contention. We can't run away from
contention at Jdbm, we can only divide it to sublevels anyway.


I have a question here:
Does the suffix dn should be compatible with context entry for that
partition ?


Yeah we shouldn't move partition files around directories. I believe if
partition path is changed, it must trigger partition implementation to look
at specified directory to initialize itself from scratch based on given
directory's content.

I recommend looking at the things we discussed with Alex on this thread. We
should implement some immutability concept for some component types.
According to this discussion here is what we may do for JdbmPartition.

* Any reconfiguration will try to create a new instance of JdbmPartition
with new configuration.
* If instance creation is failed due to wrong configuration,
reconfiguration is aborted
  else:
  We will switch runtime instance with created instance while quiscing
Operations, and initialize the DirectoryService and new Partition reference.

That way:
* we won't change code of JdbmPartition at all.
* We'll let operations run while we're creating new instance, removing
contention at some degree.
* Contention will only occur while we switch runtime references of
Partition and initialize new reference inside DirectoryService.

But to do that, we must be sure that side-by-side references can be created
with same configuration. For example for LdapServer whis wouldn't be
possible, because of its 'network socket' usage, it wouldn't be possible to
create new instance if it for reconfiguration operation.

[jira] [Commented] (DIRSTUDIO-801) Allow sorting of LDIF files when Exporting and/or Importing

Hendy Irawan (JIRA — 2012-05-22T23:30:41

    [ https://issues.apache.org/jira/browse/DIRSTUDIO-801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281311#comment-13281311 ] 

Hendy Irawan commented on DIRSTUDIO-801:
----------------------------------------

Thank you Emmanuel. Yes I wasn't aware of the complexities you mentioned. Thank you for explaining that.

For those lucky enough to have repositories with limited complexity, like mine, the Ruby script above works wonders.

BTW, I've just exported my ApacheDS 2.0.0-M6 content to LDIF, sorted it, then imported it into ApacheDS 2.0.0-M7 without any troubles. Yaaay! :-) Feels good :)
                

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (DIRSTUDIO-801) Allow sorting of LDIF files when Exporting and/or Importing

Emmanuel Lecharny (JIRA — 2012-05-22T23:04:41

    [ https://issues.apache.org/jira/browse/DIRSTUDIO-801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281280#comment-13281280 ] 

Emmanuel Lecharny commented on DIRSTUDIO-801:
---------------------------------------------

Regarding DN ordering :
- first of all, the RDN must be ordered, which means we must be able to order the single elements in a RDN. That mens we do have an ORDERING matching rule associated with teh AttributeType, which is not always the case. Then you have to decide how composed RDN should be ordered. Last, not least, all of the AVAs in a RDN must have a ORDERING MatchingRule
- second, if we want to order the DN itself, beside the RDN, then we have to propagate those rules all along the DN's RDNs
- third, the client must be schema aware, otherwise the client will have no idea on how to apply the MatchingRules it does not have access to

Regarding the memory consumption : if you have millions of entries to sort, then it will not only eat a hell lot of memory, but it will be time consuming : comparing DN is not a free operation. If you add that you'll probably have to store on a local disk the entries, then you add some extra complexity.

Sorting data is really far from being obvious in LDAP...

The SORT control is not yet implemented in ApachDS for the exact same reasons. As 2.0 is not out yet, and as we already have a lot in our plate to get it ready, frankly, we prefer focusing on the missing critical features atm... Sorry for that !
                

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (DIRSTUDIO-801) Allow sorting of LDIF files when Exporting and/or Importing

Hendy Irawan (JIRA — 2012-05-22T22:51:40

    [ https://issues.apache.org/jira/browse/DIRSTUDIO-801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281265#comment-13281265 ] 

Hendy Irawan edited comment on DIRSTUDIO-801 at 5/22/12 10:50 PM:
------------------------------------------------------------------

I'm not going to argue about LDAP spec, but users will appreciate when implementations make their lives easier when possible.

As for ordering, from parent to children always works in my cases. In which particular cases are the DN ordering not obvious?

I'm not aware how difficult it is to implement this feature in Apache LDAP libraries, but I'll post the Ruby code to do this. Hopefully someone else will find this helpful:

#!/usr/bin/env ruby
# Note: ActiveLdap can't read Extended LDIF format, use ldapsearch with -L
# sudo gem install activeldap
require 'active_ldap'

ldif_data = ARGF.read
parsed = ActiveLdap::Ldif.parse(ldif_data)
sorted = parsed.sort_by { |entry| entry.dn.count(',') }
sorted.each { |entry|
  puts entry
  puts
}

(credit: http://rubyforge.org/pipermail/ruby-activeldap-discuss/2008-June/000660.html )

Usage is simple:

./ldifsort.rb dc\=aksimata\,dc\=com.ldif

You can pipe or redirect the output to another file.

Granted, there are definitely cases where this simplistic tool will not work properly or use too much memory. But at least for now, for all my use cases it's working very well, luckily I haven't yet had an LDIF export bigger than the installed RAM :-D Hopefully someone finds it useful too.

I still am curious about the ApacheDS Server ordering. It seems that previous ApacheDS versions (1.5.x?) sorted the entries when the partition is indexed on objectClass, but it's not the case in 2.0.0-M6, or perhaps I'm missing something. Shall I file another bug for this issue?
                
      was (Author: ceefour):
    I'm not going to argue about LDAP spec, but users will appreciate when implementations make their lives easier when possible.

As for ordering, from parent to children always works in my cases. In which particular cases are the DN ordering not obvious?

I'm not aware how difficult it is to implement this feature in Apache LDAP libraries, but I'll post the Ruby code to do this. Hopefully someone else will find this helpful:

#!/usr/bin/env ruby
# sudo gem install activeldap
require 'active_ldap'

ldif_data = ARGF.read
parsed = ActiveLdap::Ldif.parse(ldif_data)
sorted = parsed.sort_by { |entry| entry.dn.count(',') }
sorted.each { |entry|
  puts entry
  puts
}

(credit: http://rubyforge.org/pipermail/ruby-activeldap-discuss/2008-June/000660.html )

Usage is simple:

./ldifsort.rb dc\=aksimata\,dc\=com.ldif

You can pipe or redirect the output to another file.

Granted, there are definitely cases where this simplistic tool will not work properly or use too much memory. But at least for now, for all my use cases it's working very well, luckily I haven't yet had an LDIF export bigger than the installed RAM :-D Hopefully someone finds it useful too.

I still am curious about the ApacheDS Server ordering. It seems that previous ApacheDS versions (1.5.x?) sorted the entries when the partition is indexed on objectClass, but it's not the case in 2.0.0-M6, or perhaps I'm missing something. Shall I file another bug for this issue?
                  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (DIRSTUDIO-801) Allow sorting of LDIF files when Exporting and/or Importing

Hendy Irawan (JIRA — 2012-05-22T22:43:41

    [ https://issues.apache.org/jira/browse/DIRSTUDIO-801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281265#comment-13281265 ] 

Hendy Irawan edited comment on DIRSTUDIO-801 at 5/22/12 10:43 PM:
------------------------------------------------------------------

I'm not going to argue about LDAP spec, but users will appreciate when implementations make their lives easier when possible.

As for ordering, from parent to children always works in my cases. In which particular cases are the DN ordering not obvious?

I'm not aware how difficult it is to implement this feature in Apache LDAP libraries, but I'll post the Ruby code to do this. Hopefully someone else will find this helpful:

#!/usr/bin/env ruby
# sudo gem install activeldap
require 'active_ldap'

ldif_data = ARGF.read
parsed = ActiveLdap::Ldif.parse(ldif_data)
sorted = parsed.sort_by { |entry| entry.dn.count(',') }
sorted.each { |entry|
  puts entry
  puts
}

(credit: http://rubyforge.org/pipermail/ruby-activeldap-discuss/2008-June/000660.html )

Usage is simple:

./ldifsort.rb dc\=aksimata\,dc\=com.ldif

You can pipe or redirect the output to another file.

Granted, there are definitely cases where this simplistic tool will not work properly or use too much memory. But at least for now, for all my use cases it's working very well, luckily I haven't yet had an LDIF export bigger than the installed RAM :-D Hopefully someone finds it useful too.

I still am curious about the ApacheDS Server ordering. It seems that previous ApacheDS versions (1.5.x?) sorted the entries when the partition is indexed on objectClass, but it's not the case in 2.0.0-M6, or perhaps I'm missing something. Shall I file another bug for this issue?
                
      was (Author: ceefour):
    I'm not going to argue about LDAP spec, but users will appreciate when implementations make their lives easier when possible.

As for ordering, from parent to children always works in my cases. In which particular cases are the DN ordering not obvious?

I'm not aware how difficult it is to implement this feature in Apache LDAP libraries, but I'll post the Ruby code to do this. Hopefully someone else will find this helpful:

{code}
#!/usr/bin/env ruby
# sudo gem install activeldap
require 'active_ldap'

ldif_data = ARGF.read
parsed = ActiveLdap::Ldif.parse(ldif_data)
sorted = parsed.sort_by { |entry| entry.dn.count(',') }
sorted.each { |entry|
  puts entry
  puts
}
{code}
(credit: http://rubyforge.org/pipermail/ruby-activeldap-discuss/2008-June/000660.html )

Usage is simple:

{code=bash}
./ldifsort.rb dc\=aksimata\,dc\=com.ldif
{code}

You can pipe or redirect the output to another file.

Granted, there are definitely cases where this simplistic tool will not work properly or use too much memory. But at least for now, for all my use cases it's working very well, luckily I haven't yet had an LDIF export bigger than the installed RAM :-D Hopefully someone finds it useful too.

I still am curious about the ApacheDS Server ordering. It seems that previous ApacheDS versions (1.5.x?) sorted the entries when the partition is indexed on objectClass, but it's not the case in 2.0.0-M6, or perhaps I'm missing something. Shall I file another bug for this issue?
                  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (DIRSTUDIO-801) Allow sorting of LDIF files when Exporting and/or Importing

Hendy Irawan (JIRA — 2012-05-22T22:43:40

    [ https://issues.apache.org/jira/browse/DIRSTUDIO-801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281265#comment-13281265 ] 

Hendy Irawan edited comment on DIRSTUDIO-801 at 5/22/12 10:42 PM:
------------------------------------------------------------------

I'm not going to argue about LDAP spec, but users will appreciate when implementations make their lives easier when possible.

As for ordering, from parent to children always works in my cases. In which particular cases are the DN ordering not obvious?

I'm not aware how difficult it is to implement this feature in Apache LDAP libraries, but I'll post the Ruby code to do this. Hopefully someone else will find this helpful:

{code}
#!/usr/bin/env ruby
# sudo gem install activeldap
require 'active_ldap'

ldif_data = ARGF.read
parsed = ActiveLdap::Ldif.parse(ldif_data)
sorted = parsed.sort_by { |entry| entry.dn.count(',') }
sorted.each { |entry|
  puts entry
  puts
}
{code}
(credit: http://rubyforge.org/pipermail/ruby-activeldap-discuss/2008-June/000660.html )

Usage is simple:

{code=bash}
./ldifsort.rb dc\=aksimata\,dc\=com.ldif
{code}

You can pipe or redirect the output to another file.

Granted, there are definitely cases where this simplistic tool will not work properly or use too much memory. But at least for now, for all my use cases it's working very well, luckily I haven't yet had an LDIF export bigger than the installed RAM :-D Hopefully someone finds it useful too.

I still am curious about the ApacheDS Server ordering. It seems that previous ApacheDS versions (1.5.x?) sorted the entries when the partition is indexed on objectClass, but it's not the case in 2.0.0-M6, or perhaps I'm missing something. Shall I file another bug for this issue?
                
      was (Author: ceefour):
    I'm not going to argue about LDAP spec, but users will appreciate when implementations make their lives easier when possible.

As for ordering, from parent to children always works in my cases. In which particular cases are the DN ordering not obvious?

I'm not aware how difficult it is to implement this feature in Apache LDAP libraries, but I'll post the Ruby code to do this. Hopefully someone else will find this helpful:

{code=ruby}
#!/usr/bin/env ruby
# sudo gem install activeldap
require 'active_ldap'

ldif_data = ARGF.read
parsed = ActiveLdap::Ldif.parse(ldif_data)
sorted = parsed.sort_by { |entry| entry.dn.count(',') }
sorted.each { |entry|
  puts entry
  puts
}
{code}
(credit: http://rubyforge.org/pipermail/ruby-activeldap-discuss/2008-June/000660.html )

Usage is simple:

{code=bash}
./ldifsort.rb dc\=aksimata\,dc\=com.ldif
{code}

You can pipe or redirect the output to another file.

Granted, there are definitely cases where this simplistic tool will not work properly or use too much memory. But at least for now, for all my use cases it's working very well, luckily I haven't yet had an LDIF export bigger than the installed RAM :-D Hopefully someone finds it useful too.

I still am curious about the ApacheDS Server ordering. It seems that previous ApacheDS versions (1.5.x?) sorted the entries when the partition is indexed on objectClass, but it's not the case in 2.0.0-M6, or perhaps I'm missing something. Shall I file another bug for this issue?
                  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (DIRSTUDIO-801) Allow sorting of LDIF files when Exporting and/or Importing

Hendy Irawan (JIRA — 2012-05-22T22:41:41

    [ https://issues.apache.org/jira/browse/DIRSTUDIO-801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281265#comment-13281265 ] 

Hendy Irawan commented on DIRSTUDIO-801:
----------------------------------------

I'm not going to argue about LDAP spec, but users will appreciate when implementations make their lives easier when possible.

As for ordering, from parent to children always works in my cases. In which particular cases are the DN ordering not obvious?

I'm not aware how difficult it is to implement this feature in Apache LDAP libraries, but I'll post the Ruby code to do this. Hopefully someone else will find this helpful:

{code=ruby}
#!/usr/bin/env ruby
# sudo gem install activeldap
require 'active_ldap'

ldif_data = ARGF.read
parsed = ActiveLdap::Ldif.parse(ldif_data)
sorted = parsed.sort_by { |entry| entry.dn.count(',') }
sorted.each { |entry|
  puts entry
  puts
}
{code}
(credit: http://rubyforge.org/pipermail/ruby-activeldap-discuss/2008-June/000660.html )

Usage is simple:

{code=bash}
./ldifsort.rb dc\=aksimata\,dc\=com.ldif
{code}

You can pipe or redirect the output to another file.

Granted, there are definitely cases where this simplistic tool will not work properly or use too much memory. But at least for now, for all my use cases it's working very well, luckily I haven't yet had an LDIF export bigger than the installed RAM :-D Hopefully someone finds it useful too.

I still am curious about the ApacheDS Server ordering. It seems that previous ApacheDS versions (1.5.x?) sorted the entries when the partition is indexed on objectClass, but it's not the case in 2.0.0-M6, or perhaps I'm missing something. Shall I file another bug for this issue?
                

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Problem of JDBM being hard(impossible) to reconfigure once it is initialized

Emmanuel Lécharny — 2012-05-22T22:36:24

Le 5/23/12 12:01 AM, Selcuk AYA a écrit :

I'm not comfortable with the idea of quiscing the ldap operations. The 
server is supposed to work 24x7, and if we do add some way to modify the 
backend config on the fly, then the user should not notice that.

See my comments below.
Adding an index should be done while the server is still able to process 
requests. The thing is that the newly added server should not be 
available until it has been created. The real problem is that we won't 
be able to parse the master table fast enough to have all the 
subscequent modifications into it, so this is a two steps operation :
- do a full scan search on the master table, and create the additional 
index. We should be safe here, because we will use the current version 
of the master table when we start the search.
- then for every modification done after the beginning of step 1, apply 
those changes to the index. If we have some more changes since we 
started this step, then do it again.

Now, we can use this index.

Here, it's simpler. The operation using this index will continue to use 
it, until they are done. New operations won't be allowed to use it. Once 
the number of requests using this index is 0, then the index can be 
deleted. I don't think we need to block any operation here.
changing teh suffix DN is just a matter of modifying the configuration, 
and the DnNode structure. The problem is that we rebuild all the 
entries' DN using the partition suffixDn, so we have to be careful when 
doing so. The request being processed when the rename occurs should be 
fulfilled with the initial suffixDn.

Here, this is an administrative task. Not sure we want to do that on a 
running server...

[jira] [Commented] (DIRSTUDIO-801) Allow sorting of LDIF files when Exporting and/or Importing

Emmanuel Lecharny (JIRA — 2012-05-22T22:12:41

    [ https://issues.apache.org/jira/browse/DIRSTUDIO-801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281247#comment-13281247 ] 

Emmanuel Lecharny commented on DIRSTUDIO-801:
---------------------------------------------

I think you have no idea of the implication of such a feature, client side. The fact that it's "requested" since 2009 is irrelevant. There is nothing in LDAP specification that requires the data to be sorted in any way. Add that ordering DNs is all but obvious, and sometime impossible.

Yes, it would be cool to have such a feature, but, no, it's not easy to implement, and it's not easy to use either.

There already exists some control that ask the server to order the returned entries on the server, not all servers support them.
                

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Problem of JDBM being hard(impossible) to reconfigure once it is initialized

Selcuk AYA — 2012-05-22T22:01:44

I  do not know the OSGI jargon but I believe,  at the end, changing
these should reduce to something like this:

1) quiesce the necessary ldap operations: The simplest thing to do
would be to block all operations quiesce all the outstanding
operations. In some cases, it would be enough to block only searches.
There might be some cases where you do not need any blocking at all. I
do not

2) do the configuration: In the end I am assuming the configuration is
delegated to the component itself to do the reconfig.

3) unblock the operations.


For index add:

* queisce any ldap operation that might modify the data. Then notify
the partition to add the index. Partition will scan the master table
and build the index and swap its indexed attributes. Then unlock the
operations.

For index delete:
* block all operations, and remove the index from the indexes list.
Maybe commit this change in partition config. unlock changes. Then
continue deleting  index file.


For suffixdn change:

* I believe this is a rename operation. again block all operations and
do the rename. Then unblock.

Working directory change;
* Block all opreations and let the parition change its config dir. For
jdbm, this would be a copy of it working directory.

Using the same techniques:

* We should be able to add/remove and interceptor
* add a partition. This should not require blocking of operations.
* remove a partition.

thanks
Selcuk

On Sun, May 20, 2012 at 8:22 PM, Göktürk Gezer <gokturk.gezer-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

[jira] [Commented] (DIRSTUDIO-801) Allow sorting of LDIF files when Exporting and/or Importing

Hendy Irawan (JIRA — 2012-05-22T21:46:41

    [ https://issues.apache.org/jira/browse/DIRSTUDIO-801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281235#comment-13281235 ] 

Hendy Irawan commented on DIRSTUDIO-801:
----------------------------------------

Note that as of 2.0.0-M6, even when the partition is indexed in the objectClass attribute, the returned entries order is still arbitrary.

This is contrast (a regression?) compared to the behavior described in:
http://mail-archives.apache.org/mod_mbox/directory-users/200901.mbox/%3C497925E4.1080209-udRLCcwUmhVaqNkQJhWWaQ< at >public.gmane.org%3E

                

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (DIRSTUDIO-801) Allow sorting of LDIF files when Exporting and/or Importing

Hendy Irawan (JIRA — 2012-05-22T21:42:41

    [ https://issues.apache.org/jira/browse/DIRSTUDIO-801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281232#comment-13281232 ] 

Hendy Irawan commented on DIRSTUDIO-801:
----------------------------------------

This has been requested since January 2009 :

http://mail-archives.apache.org/mod_mbox/directory-users/200901.mbox/%3C98d8c0860901230228m334cf30fjd41d3e5fec32fcfe-JsoAwUIsXosN+BqQ9rBEUg< at >public.gmane.org%3E
                

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Txn branch merge : headsup

Emmanuel Lécharny — 2012-05-22T15:14:37

Hi,

so as of today, I have merged 37 out of 54 modules. The following 
modules still have to be merged :

apacheds-core-api
apacheds-core-shared
apacheds-core
apacheds-core-avl
apacheds-xdbm-partition
apacheds-ldif-partition
apacheds-jdbm-partition
apacheds-core-annotations
apacheds-server-config
apacheds-server-jndi
apacheds-server-replication
apacheds-xdbm-tools
apacheds-core-integ
apacheds-server-integ
apacheds-kerberos-test
ldap-client-test
apacheds-service

Some of them require few modifications, some other are more complex. 
I'll keep you informed as soon as I make progress.

[jira] [Created] (DIRSERVER-1728) Replace numeric errors by textual errors

Emmanuel Lecharny (JIRA — 2012-05-22T12:35:40

Emmanuel Lecharny created DIRSERVER-1728:
--------------------------------------------

             Summary: Replace numeric errors by textual errors
                 Key: DIRSERVER-1728
                 URL: https://issues.apache.org/jira/browse/DIRSERVER-1728
             Project: Directory ApacheDS
          Issue Type: Task
    Affects Versions: 2.0.0-M7
            Reporter: Emmanuel Lecharny
             Fix For: 2.0.0-RC1


We have declarations like :

ERR_42=An entry with a ''referral'' ObjectClass must contains a ''ref'' Attribute

It would be way better to have something like for clarity sake :

ERR_42_REF_ATTRIBUTE_MISSING=An entry with a ''referral'' ObjectClass must contains a ''ref'' Attribute


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Txn/trunk merge : DefaultOperationExecutionManager issues

Emmanuel Lécharny — 2012-05-22T09:48:42

Hi guys,

I'm currently facing some interesting challenge on this class, as we 
have removed the OneLevel and SubLevel index from trunk. This has some 
direct impact on the txn branch code, as we have to update the code 
accordingly.

Methods like add/delete... are storing the changes done on those index :
             // Update the OneLevel index
             Index<?> oneLevelIdx = partition.getSystemIndex( 
ApacheSchemaConstants.APACHE_ONE_LEVEL_AT_OID );
             indexChange = new IndexChange( oneLevelIdx, parentId, id, 
IndexChange.Type.ADD, true );
             changeContainer.addChange( indexChange );

We have to remove those lines, but we also have to modify the RDN Index 
update.

I'm be working on that, it may take a bit of time.

In any case, the merge is way more complex than any other we have 
already done on the server, although I do expect to have it done by the 
end of this week.

SVN is not helping a lot here... :/

Re: < at >CreateDS annotation

Emmanuel Lécharny — 2012-05-21T21:01:22

Le 5/21/12 9:41 PM, Fletcher, Duane W (Duane) a écrit :
Probably because we never had the need for that?
The best would be to create a JIRA requesting the < at >Inherited annotation 
to be added in the base class. We most certainly can add it in the next 
release.

< at >CreateDS annotation

Fletcher, Duane W (Duane — 2012-05-21T19:41:57

Why isn't < at >CreateDS annotation defined as < at >Inherited?

I'm trying to write a base class that includes < at >CreateDS annotation, but it is not inherited.

Thanks for any explanation of this.

Duane Fletcher
Alcatel-Lucent

[jira] [Created] (DIRSTUDIO-815) Tiny translation error [DE]

Peter Schwindt (JIRA — 2012-05-21T12:29:41

Peter Schwindt created DIRSTUDIO-815:
----------------------------------------

             Summary: Tiny translation error [DE]
                 Key: DIRSTUDIO-815
                 URL: https://issues.apache.org/jira/browse/DIRSTUDIO-815
             Project: Directory Studio
          Issue Type: Bug
    Affects Versions: 1.5.3
            Reporter: Peter Schwindt
            Priority: Trivial


When connecting to an LDAP source, after entering the password, the progress popup shows "Verbindungen am Öffenen".

Despite this being a typo itself, I propose something like "Öffnen der Verbindung".

Thanks,
Peter

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [jira] [Created] (DIR-285) life issues

Emmanuel Lécharny — 2012-05-21T05:37:28

Le 5/20/12 10:41 PM, Alex Karasulu a écrit :

I have deleted this user. You don't need more love than you already have :)

Re: Problem of JDBM being hard(impossible) to reconfigure once it is initialized

Göktürk Gezer — 2012-05-21T03:22:16

Most of checkInitialized() calls are necessary for current design of
Jdbm/AbstractBTree Partition.

Moving system files of partition is only necessary when we change its
partitionId as i see?
And i believe moving partition files around directories is conflicting with
partitionId property's purpose.
Using partitionId property as a pointer to where partition system files are
is more consistent IMO. So when
somebody changes some partition's partitionId, then this partition instance
must be reinitialized to use the partition files
under specified directory. If new specified directory has incompatible
content then we switch back to previous partitionId.
( ComponentHub system supports what i described: when some active component
is reconfigured in a state which is not valid,
 then reconfiguration is reverted. further explained below.)

First 2 don't have checkInitialized() checks already. syncOnWrite() has
that check, but i guess we can remove it.
We only have to insert sync() call before setting this property to ensure
none remains in buffers. Right?

Yes, they're currently not reconfigurable and it seems hard to make them.
All three is designed to be read at beginning and no altering mechanism is
supplied.


This dynamism can be handled, but i'll cause pain of course, but not much.
.db files being incompatible with new version is
something JdbmPartition implementation must handle by itself IMO. It must
have mechanisms to migrate old .db files to its own format at
initialization time.

And i remember now that i forgot to implement hot bundle replacements(cold
replacements are supported which is simply replacing bundle when none of
its components are instantiated). But no worries, with cold replacement
implemented, it is easy to implement hot reconfiguration in minutes.

No, it would not be good. Conflicting with OSGI anyway.

Here is how i handled it, (could be changed of course):
 When configuration of some component is changed(via config partition or
webconsole plugin which i'll implement later...),
new configuration's validity is checked. Valid means all property values
are compatible with the property types defined in component's metadata.
 * for primitives no check is performed, because they are ensured by Ldap
schema which is being created for that component specifically.
 * for reference types, it's checked that referenced component's type is
compatible with property's type.
 * for collection types, same with reference types + check if all
collection items are compatible with specified collection item type.
 * some more consistency checks.
So after these checks, new configuration is considered valid and it's sent
to underlying component framework which is used to create the component in
first hand to reconfigure that component. So at that point, it is
guaranteed that underlying component framework will not complain about any
type clashes.

However, component implementation can reject some configuration options
even if it is valid schematically. At that point component implementation
can choose to set some default value for this property or simply raise an
exception. If exception is thrown, then reconfigured component is
deactivated completely and it is set as dirty, meaning any further
reconfiguration will cause it to be instantiated again.

So this is how it works now. It is some kind a mixture of 1. and 2. way you
proposed. And it does real reconfiguration on components rather than what
2. option suggest.

However using 2. option would be in my favor too, Though it is not really a
reconfiguration, it is much easier to manage and it covers much more error
scenario which my be thrown while reconfiguration. And by employing such
mechanism we don't have to worry about Jdbm begin immutable. Because we'll
create new instance anyway, if there are no some lock-file mechanism inside
partition directory, we're golden.

Also there is a facility which makes it closer to 2. option, "immutable
properties". When you reconfigure an immutable property of some component
it is deleted first from system and component listeners, and then it's
created with new configuration and introduces into system again. By
deactivating-and-activating in order rather than swapping the active
instance on the fly, we ensure that hub can detect structural
inconsistencies which may come along with that reconfiguration, for example
if SystemPartition's partitionId is changed from "system" to "dummy", hub
will abort that reconfiguration because ApacheDS can't work without system
partition. If we were handled it by swapping instances on the fly, server
would continue working but then on the next run it wouldn't start because
it's lacking partition with "system" id.

If we'll change the reconfiguration schema to 2. option, then we should put
'immutable properties' into consideration here too. Because we just can not
modify some properties which are hardwired to DirectoryService without
checking if modification is valid to defined constraints on that property.
Otherwise next run of server may not take place properly.

Yes, All frameworks have different invoking policies but in ComponentHub we
only care about configuration change. We have activated configuration in
one hand and altered configuration in other hand. We validate the altered
configuration and send it to underlying component framework.

While it destroys hot reconfiguration, yes it is much more accountable. Not
so performance wise, but reconfigurations are seldom to appear, so it won't
be a problem.

So its your call, i'm good with every option. We can go with option 1 to
make real hot reconfigurations, or we can go with option 2 to create an
illusion of hot reconfigurations while keeping existing code.
FYI Switching from 1. to 2. reconfiguration policy in ComponentHub is
trivial as i see it.

Thanks,
Gokturk