gmane.comp.search.xapian.devel

Re: GSoC xapian node binding

Liam — 2012-05-25T00:08:29


And a few more!

Use "that" instead of "pThis" as the name for the unwrapped args.This()

There are currently tests for mBusy in both _do_async and AsyncOp::AsyncOp.
We can remove the latter, and also make AsyncOp::AsyncOp take the unwrapped
object, rather than do the unwrap itself.

Use "Type aVar = value;" where possible, since we're removing try blocks
that required separate "Type aVar;".

The data argument for the _do functions should be named "data" vs "aData".

Could the DECLARE_POOL methods be defined once per class if the _process &
_convert functions are passed as args to the _do functions and then stored
in the AsyncOp object?
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Re: GSoC xapian node binding

Liam — 2012-05-23T20:35:02


Few more notes...

In op.h, tryCallCatch, GetInstance, sendToThreadPool, kBusyMsg should be
declared without static (since now used by multiple .cc files) and be
defined in util.cc or similar.

In the_do functions, since they no longer ThrowException()
  if (mBusy) throw Exception::Error(...)
similar to when _process returns error

Correcting previous code snippet to use Handle<Value>:
  Handle<Value> aResult;
  try { /// if we pull the try/catch from the _do functions
    aResult = aAsync ? GetMset_async(...) : GetMset_sync(...); /// or
GetMset_do(aAsync, ...)
  } catch (Handle<Value> ex) {
    ...
Local<Value> is for declaring a value in the current HandleScope. (Yes, my
orig code used Local in catch.)

GetMset_convert doesn't need a HandleScope, since callers have it.

Could we put the _data ptr into AsyncOp? That should eliminate the need for
OpInfo?

In your testing, make sure to trigger all the possible catch blocks. (You
might have to insert throws, e.g. to simulate a Xapian error.)
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Re: GSoC - Follow up Question about doc

Dan Colish — 2012-05-23T04:37:13

Ok, so I was really suggesting that most of the documentation that lives outside of our doxygen generated API documentation is consider to be higher level user docs. The actual user can vary widely from a new xapian user to core developers, but the intention of that documentation should be to describe in high level language the objects being documented. Describing all the methods might not be useful in that scenario. What's important is to capture the critical code path's and interfaces, and preferably show a few examples as well.
I don't think what you have do has been a waste at all; it just needs a little tweaking. All of the documentation that you've created to this point can be used, the question is where it really belongs.  As I said above, I think you need to split it into a few parts and focus on the audiences involved. For basic docs, you're usually writing for xapian end-users. For docs that cover internals at a high-level, you're writing for xapian developers. For docs that describe precise implementations in their entirety, you are mostly writing API comments for doxygen.

Your observation about breaking down the commits into smaller chunks is very good. If the changes are a little smaller we'll be able to review and correct more quickly and hopefully avoid any frustrations.
 
--Dan

Re: GSoC xapian node binding

Liam — 2012-05-23T03:12:14


Re latest code...


Commit message format:
  [add|fix|clean]: filename method (, ...) (, filename ...) - what was
added/fixed/cleaned

  add - create new functionality
  fix - repair existing functionality
  clean - no change in functionality
  If a LOT of methods are affected, the method list can be truncated to a
few key ones

  No need to change the messages you've already committed.


Should we create a single node-xapian.h header file? (Which I guess could
include the contents of op.h)


Code style:
   if/else and function opening brace on same line! (see op.h macro)
   local vars begin with letter a, to differentiate from function arguments


DECLARE_POOLS

  In the _do functions, perhaps the try/catch should move to the caller,
which would ThrowException() and delete aData on catch? (The only throw is
in AsyncOp::AsyncOp.) I'm not sure of the implications of returning
ThrowException to scope.Close() -- may be harmless; something to look into.
Even so, it's probably clearer to ThrowException from the caller. Also
there's no need to delete x when x=new X() throws, I believe. And if
_process() returns an error, you'd need to convert it to an
Exception::Error and throw that  for the caller.

  Does _process() need to return a new Xapian::Error, or can it return the
Error that was caught? I was doing new there previously to attach the error
to the GetMset_data object. Of course then you'd need to do the new in
_pool()

  The _do functions don't need a HandleScope, since the caller has one.

  Did you look at merging the two _do functions? If not mergeable, then we
prob don't need _do in the names, just _sync & _async


Enquire::GetMset:

  Remove try/catch around "new GetMset_data...", which no longer throws
since it's not derived from AsyncOp!
  ...
  bool aAsync = args.Length() == 3 && args[2]->IsFunction();
  if (args.Length() < 2 || args.Length() > +aAsync+2 ||
!args[0]->IsTypename() || ...  )
    return ThrowException(...);
  ...
  Local<Value> aResult;
  try { /// if we pull the try/catch from the _do functions
    aResult = aAsync ? GetMset_async(...) : GetMset_sync(...); /// or
GetMset_do(aAsync, ...)
  } catch (Local<Value> ex) {
    return ThrowException(ex)
  }
  return scope.Close(aResult); /// it's not necessary to Close() for
Undefined() but prob not harmful

Cheers,

Liam
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

GSoC - Follow up Question about doc

Sehaj Singh Kalra — 2012-05-22T20:36:20

Dan Colish,

I am facing major issues in the documentation. Have invested quite some
time but not able to properly figure out what really needs to be done in
the documentation.

Can you please once again, clearly mention the objective of  a "user
documentation".
Also, can you please guide as to how to give an overall view? I am not able
to figure out what really needs to be added.

Also, it seems that I have wasted much of my time and energy in explaining
the things which were not meant to be explained. Have tried to removed them
from the doc.
This could have been avoided if I had come up with smaller chunks rather
than coming up with a single large chunk.

Have also added some replies regarding the comments you made on my github
commit. Reply to them when you can.
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Re: Xapian license change

Olly Betts — 2012-05-22T05:47:01

I'm rather too busy to at the moment, but I'm happy for others to work
on such a page (assuming it doesn't create a maintenance burden which
I end up shouldering, or spread misinformation which I then have to
spend time clearing up after).


Cool.


I guess you mean this:

http://trac.xapian.org/wiki/ProjectIdeas#Replacegetoptwithpoptorsimilar

That used to incorrect say "GPL" but I fixed in back in February.


Not quite.  We've not been asking longer term contributors (who have
already given us permission to relicense their contributions) to
dual-license up front.  This isn't a carefully thought out policy, there
just didn't seem a reason to do so.

It's a fairly subtle distinction, but for example a third party can't
just take code in Xapian written after a given date and use in under
MIT/X without investigating who wrote it.  (This isn't some deliberate
masterminded policy, just a consequence of the above).

But we do have agreement from most of the longer term contributors that
we can relicense, and more recent contributors have dual licensed up
front.

So the only copyrights which should be an issue are any on external code
we've brought in which isn't liberally licensed (which there isn't much
of) and these:

* BrightStation PLC.
* Ananova Ltd.
* Orange PCS Ltd.


Buildbot keeps an updated summary (I'm not convinced the later columns
are actually useful, so I'd suggest just ignoring them):

http://lcov.xapian.org/copyright.csv

A TODO list doesn't seem worth the effort it would take to maintain it
given that "git grep BrightStation" or similar can easily give you a
completely up to date list of the files concerned.


I'd rather avoid adding more build-time optional features.  They're easier
to break if not everyone is building with them, and they're a pain for
packagers as they have to make a choice for their users.  Forcing
packagers to make a *licensing* choice for users seems particularly
unhelpful.

Perhaps as an intermediate stage it's OK, so --without-gpl might give
you a copyleft-free version, and slowly this evolve to disable less
and less until it has no effect.  Currently you wouldn't get a usable
system from this though - in particular the matcher and all backends
still include BrightStation code.


I think it's over-simplifying to say there isn't ANY difference.
Certainly the two cases aren't hugely different.


I wasn't aiming to make an explicit proposal, but rather to highlight
the issues which need to be decided, and some options we have.

I think I'm the biggest single copyright holder in Xapian at this point,
but I'd rather not let that result in us just doing what I suggest here.

As Dan notes, this shouldn't be about people who want to use Xapian in
proprietary applications making demands, but I think only listening to
those using Xapian under the existing licence would give a rather
biased sample.

I don't think the relicensing itself is controversial.  The use of GPL
is largely a historical accident - it was how BrightStation opened up
the code.  I don't think I've heard anyone disagree with the plan to
relicense, and even the FSF don't advocate using the GPL in a case like
Xapian where there are alternatives under non-copyleft licences -
http://www.gnu.org/licenses/why-not-lgpl.html says:

    Using the ordinary GPL is not advantageous for every library. There
    are reasons that can make it better to use the Lesser GPL in certain
    cases.  The most common case is when a free library's features are
    readily available for proprietary software through other alternative
    libraries.  In that case, the library cannot give free software any
    particular advantage, so it is better to use the Lesser GPL for that
    library.

Also, a big area for deploying of search currently is on web sites where
no distribution of the software occurs, which means that a lot of our
users don't actually fall under the GPL's copyleft provisions anyway.

And the choice of licence isn't wide open.  For a start, it needs to be
both Free Software and Open Source.  It should be something well known,
and definitely don't want to invent a new licence.

As Thomas Waldmann has pointed out, keeping GPLv2 compatibility is
desirable (because of existing GPLv2 users, such as moinmoin) - moving
to GPLv3+ or AGPL or the Apache licence would cause problems for some
existing users.  To avoid that, either the new licence should be clearly
compatible with GPLv2, or else we should dual license (e.g.  Apache +
GPLv2 would satisfy this requirement).

It would be good to be compatible with the PHP licence too, so the PHP
bindings can be packaged again (we can add an exception clause to cover
that if the chosen licence isn't inherently compatible).

Cheers,
    Olly

Re: Standing of the project

Parth Gupta — 2012-05-21T20:47:04

Hi Rishabh,



Sounds good. It makes sense that you and Gaurav share the same  evaluation
framework so that at the end we do not end up with two different evaluation
modules.

Of course, in our case the input to the evaluation module would be
Letor::Ranklist. So, if you overload the method like evaluate() which
returns the NDCG< at >k or MAP supplying a Letor::Ranklist (in our case) and
Xapian::MSet (in Gaurav's case). You can coordinate among yourselves.
Basically we need to use it a lot. While training, we optimise a loss
function and validate the model using the same evaluation module.



Great. Btw did you have look at the Matlab/Octave code written by Geoffrey
Hinton that is used in the Science paper. That's very informative and
understandable. Recently I played with it in some experiments and its quite
easy to follow.

Currently I am in Istanbul attending a conference so this week you might
not find me much around but most of the days I will try to answer e-mails
in the night.

Parth.

_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Re: Iterators

Michael Uvarov — 2012-05-21T06:27:19

skip_to moves a cursor forward only.
That is why, I created a wrapper around iterators.

Here is a base class for them:
<https://github.com/freeakk/xapian/blob/master/c_src/termiter_gen.h>

And then I created few realizations of this interface:
<https://github.com/freeakk/xapian/blob/master/c_src/termiter_doc_gen.h>
<https://github.com/freeakk/xapian/blob/master/c_src/termiter_spy_gen.h>

After all, I can create an exemplar of one of this objects and store a
pointer on it as a property of the TermQlcTable.
<https://github.com/freeakk/xapian/blob/master/c_src/qlc_table.h>
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Re: Iterators

Olly Betts — 2012-05-21T01:30:18

You should be able to do skip_to("") on a TermIterator with no effect.
If it's the common case that first_term_name will be empty, then it's
probably worth checking for it to avoid the overhead of the no-op call:

    if (!first_term_name.empty())
it.skip_to(first_term_name);


Not sure I follow why this makes a difference...

Cheers,
    Olly

Re: Xapian-devel Digest, Vol 85, Issue 17

Sehaj Singh Kalra — 2012-05-19T13:28:13

James,

Thanks for pointing it out, will keep that in mind for future usage.

Thanks,
Sehaj Singh Kalra

On Sat, May 19, 2012 at 4:30 PM, <xapian-devel-request< at >lists.xapian.org>wrote:

_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Re: GSoC - QueryParser Information

James Aylett — 2012-05-19T07:52:45

The problem is how some email clients figure out URI/URL boundaries; to ensure it works enclose in <>, eg <http://trac.xapian.org/wiki/GSoC2012/QueryParser/Notes>. (This is in the URI RFC I think.)

J



On 19 May 2012, at 05:37, Sehaj Singh Kalra <sehaj.sk< at >gmail.com> wrote:

_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Re: GSoC - QueryParser Information

Sehaj Singh Kalra — 2012-05-19T04:37:11

The link doesn't seems to work. I think the suffix "." caused the problem.
Anyways, sending the link again.
http://trac.xapian.org/wiki/GSoC2012/QueryParser/Notes

On Sat, May 19, 2012 at 10:00 AM, Sehaj Singh Kalra <sehaj.sk< at >gmail.com>wrote:

_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

GSoC - QueryParser Information

Sehaj Singh Kalra — 2012-05-19T04:30:53

Have added information regarding queryparser at
http://trac.xapian.org/wiki/GSoC2012/QueryParser/Notes.
It is a bit informal at present but have a look, and let me know your
views. Will be improving it and adding some more things.
The aim as I talked with dcolish, is to explain the working of QueryParser
along with appropriate examples.
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Re: GSoC xapian node binding

Marius Tibeica — 2012-05-18T08:36:20


I tend to agree. The hybrid approach is even more complicated.

template <class dataType, class className>
class PoolHelper
{
public:
  ...
  template <Handle<Value> (*convertFunction)(className *data)>;
  static void execute_done(eio_req *data)
  {
    HandleScope scope;
    OpInfo* aInfo = (OpInfo*) req->data;
    dataType *aData = (dataType*)aInfo->data;
    AsyncOp<className> *aAsOp = (AsyncOp<className>*)aInfo->op;
    Handle<Value> argv[2];
    if (aAsOp->error) {
      argv[0] =
Exception::Error(String::New(aAsOp->error->get_msg().c_str()));
    } else {
      argv[0] = Null();
      argv[1] = convertFunction(aData);
    }
    tryCallCatch(aAsOp->callback, aAsOp->object->handle_, aAsOp->error ? 1
: 2, argv);
    delete aData;
    delete aAsOp;
    delete aInfo;
  }
};


#define DECLARE_POOLS(func,classn) \
...
static int func##_done(eio_req *req) {\
  HandleScope scope;\
  PoolHelper<func##_data,classn>::execute_pool<func##_convert>(req);\
  return 0;\
}\


I will leave the macro as it is.


The question now is, what's next?
Should I start implementing other methods?



_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Re: Xapian license change

Dan Colish — 2012-05-17T16:29:25

Fair enough, maybe it was just my interpretation of the thread. Olly has be excellent at keeping track of the licensing issue since it does come up every so often.

This is a pretty good point. I had not thought about it much from the other side. I think I am inclined to use Olly's guideline that the rewritten code should be an improvement. That way we are able to gain more control our licensing and improving our code. It makes sense that just gaining more control is a worthwhile effort, but I would like to see the code improve if we're already making the effort to rewrite it. 

I should say that I'm not against relicensing. I just want it to be done for the right reasons and with a clear process. How would you feel about spearheading the organization of this effort? There are some good notes on this thread from Olly that could be rounded up into a working document.

Re: Xapian license change

Dan Colish — 2012-05-17T16:14:18

True, I think sponsored development is a good thing in most cases, but its a bit hard to do without a clear proposal. PyPy has done most of their work through academic grants and the money is always attached to very clears goals and steps. I would be hesitant to accept money without going through that same process.

Re: Xapian license change

Wichert Akkerman — 2012-05-17T15:47:39

Unfortunately not totally unexpected.


I do not think I am telling anyone how to do things, and that certainly 
is not my intention. So far I have only been asking for information 
about the status quo and which might need to be done for relicensing to 
happen, and Olly has been extremely helpful with providing that. I did 
already offer to do work myself in this area if I can help, as much as 
my time and skillset will allow. I have no expectation at all that other 
people will do anything just because I or someone else is asking them to.

I would also ask that you consider my point of view. With the current 
license unfortunately Xapian is not useful for me, so please do not 
expect me to invest energy into Xapian when that is not helping to 
scratch an itch I have. I am willing to work on Xapian, within and with 
the community as long at it helps me as well. That does include writing 
patches, updating documentation and everything else that might be needed.

Regards,
Wichert.

Re: Xapian license change

Charlie Hull — 2012-05-17T15:23:40

To rewind a little, at least one of the people asking about this is from 
a company interested in using Xapian in their products but unable to 
because of the GPL. They have indicated they may be able to fund some 
development towards this end. Surely sponsored development of Xapian is 
a good thing?

Cheers

Charlie

Re: Xapian license change

Dan Colish — 2012-05-17T15:16:26

The more I read on this thread the more I am confused. I'm starting to wonder who's asking for relicensing here because its starting to read like we, the Xapian developers, are asking how to relicense and what work we'd need to do. That is not the case. A request like this comes up once or twice a year and the requestor usually disappears a week later. 

You need to be a part of the community to have this discussion. I'm sure anyone can appreciate what it is like for an outsider to tell you how to do things. So please, join the community, get to know its members, and write patches or create documentation. Help us improve Xapian for a bit, then we can talk about why the licensing needs to change and how to go about it. Remember, our time is an extremely precious resource and I would like to use that time to make Xapian better rather than rewriting code so it can be relicensed.

Re: GSoC xapian node binding

Liam — 2012-05-17T15:02:58


OK, no need to bend over backwards. Simplicity is preferable after all.
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Re: GSoC xapian node binding

Marius Tibeica — 2012-05-17T09:38:36

I have tried multiple approaches.
The difficulty comes from the fact that each class can have multiple
sync/async functions. For example, Enquire has atleast get_mset,
get_matching_terms and get_eset.
Another problem is that from the pool methods (which should be templatized)
I have to call methods from the class (GetMset_process, GetMset_convert,
GetMatchingTerm_process, ...). This means that I also probably have to pass
a function as a template parameter.
The thirt problem is that func##_pool and func##_done need to be (int
(eio_req *req)), so I can not pass a function pointer.

At the moment I am trying a hybrid approach (bot templates and macroes).

On Wed, May 16, 2012 at 8:10 PM, Liam <xapian< at >networkimprov.net> wrote:

_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel