gmane.comp.search.xapian.devel

GSoC - Follow up Question about doc

Sehaj Singh Kalra — 2012-05-22T20:36:20

Dan Colish,

I am facing major issues in the documentation. Have invested quite some
time but not able to properly figure out what really needs to be done in
the documentation.

Can you please once again, clearly mention the objective of  a "user
documentation".
Also, can you please guide as to how to give an overall view? I am not able
to figure out what really needs to be added.

Also, it seems that I have wasted much of my time and energy in explaining
the things which were not meant to be explained. Have tried to removed them
from the doc.
This could have been avoided if I had come up with smaller chunks rather
than coming up with a single large chunk.

Have also added some replies regarding the comments you made on my github
commit. Reply to them when you can.
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

GSoC - QueryParser Information

Sehaj Singh Kalra — 2012-05-19T04:30:53

Have added information regarding queryparser at
http://trac.xapian.org/wiki/GSoC2012/QueryParser/Notes.
It is a bit informal at present but have a look, and let me know your
views. Will be improving it and adding some more things.
The aim as I talked with dcolish, is to explain the working of QueryParser
along with appropriate examples.
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Xapian license change

Wichert Akkerman — 2012-05-16T15:37:52

Like Amish Shah who posted earlier this month to this list I find myself 
interested in Xapian, but currently not being able to use it due to its 
GPL license. I am aware of the CommercialLicence wiki page, but have not 
been able to find any current status or anything willing to work on it, 
so I am wondering if perhaps I could help. As far as I can see several 
things would need to happen:

  * a new license (or set of licenses) will need to be chosen
  * current copyright holders will need to sign off on the relicensing
  * dependencies on GPLed libraries (think getopt) will need to replaced


Looking at older posts and the xapian website it seems that LGPL has 
been suggested as an alternative license, but I can not find a clear 
decision.

Judging by the copyright statements in the sources (for xapian-core) 
there are 18 copyright holders, which is a manageable list:

  * Action Without Borders
  * Adam Sjøgren
  * Ananova Ltd
  * Brandon Schaefer
  * BrightStation PLC
  * Dan Colish
  * Dr Martin Port

Iterators

Michael Uvarov — 2012-05-16T10:44:18

Hello,

Can we use an iterator after deletion of a parent object?

For example, like this;

xapian gpl replacement

Amish Shah — 2012-05-15T17:12:07

Hi,
Xapian seems to work for us, but the GPL license is going to prevent us
from using it. Is there an update on the efforts to remove the GPL code
from xapian? Also, my company might be willing to fund such efforts if the
scope of the work is known.

thanks
Amish
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Patch for compiling xapian with gcc4.7

Gaurav Arora — 2012-05-08T10:49:26

Hi,

I updated my system and shifted from gcc4.6 --> gcc4.7 .While building
xapian code with gcc4.7 there is error returned due to some changes in
GCC.Pastbin of error is http://pastebin.com/7ee5WGiu .I searched and found
it to some other packages also seems to face such issue like
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=667403 .

Attached herewith is patch.

GSoC 2012 admin

Olly Betts — 2012-05-08T02:43:22

If you're one of our six selected GSoC students, please read through this
email and attend to anything which needs doing.  If anything is unclear,
please ask.

I've created a page on the wiki for each of the selected projects, which
are all linked from this overview page:

http://trac.xapian.org/wiki/GSoC2012

There's a table of useful information on each summary page.  Please check
that this is correct for you, and fill in the missing information.  For
most of you this is:

 * "work hours" - we realise you probably won't be working exactly the
   same hours every day, but it's useful for us to have some idea when we
   can expect you to be around.  We also encourage you to be on IRC in
   #xapian at least while you are working, as that will make interaction
   easier.

 * repository location - I've filled this in where I already know it, but
   for the rest of you please clone/fork our repository, push it to
   a public location, and put a link to that on your project's wiki page.
   There are already xapia

Erlang examples

Michael Uvarov — 2012-05-07T17:19:46

Hello,

Here are few examples of Erlang binding usage:

https://github.com/freeakk/xapian-examples

Index Size comparison

Jaguar Xiong — 2012-04-23T14:16:51

Hi,
I did a comparison based on similar steps as in the blog
(zooie.wordpress.com/2009/07/06/a-comparison-of-open-source-search-engines-and-indexing-twitter),
against lucene-3.4 and xapian-1.3.0. The overall index sizes are:
lucene 89M, xapian 189M (chert backend and compacted).
Since I'm more interested in index size, I dig a little further to dump
the full term list. There are about 360000 terms from lucene index, and
about 285000 terms from xapian index. But surprisingly, the termlist.DB
of xapian index is already 122M.
Is there some idea/plan on reducing the index size? I'll glad if I could
help.

Thanks!
Jaguar

Handling Negative value due to logarithm ofprobabilities.

Gaurav Arora — 2012-04-27T04:58:30

Hi,

In continuation of the discussion of melange comments,about negative value
returned in matcher due to logarithm of probabilities.

*I**f we make K suitably large, we could clamp each log(K.Pi) to be >= 0,
and this change will only affect really low probability terms (those with
Pi < 1/K, so you can adjust K to suit):*

*W' = sum(i=1,...,n, max(log(K.Pi), 0))*

Did you mean for low probability the the value returned by log(K.Pi) would
be negative. So replace lower probability, which still gives negative value
by 0?

Assigning 0 will be equivalent to rejecting term from the query
completely,which hurts the retrieval performance in Language Model as the
term missing from the document are smoothened with collection frequency.

I think we must try the smoothing from collection statistics if the
document term probability doesn't work(is generating negative value).

*sum(*

*i=1,...,n, if( max(log(K.Pi), 0) == 0)*

*max(max(log(K.Pcollec.i),0)*

*else*

***log(K.Pi)*

*)*

In case both doesnt work return 0 wou

Letor re-factored code

Parth Gupta — 2012-04-24T09:11:34

Hello Rishabh,

The attached diff file should help you to refactor the existing letor code
and plug in your code easily. I have defined the header files with the
necessary flow. I have also specified the ranker class, which should be
implemented by a new LTR models, ListNet and ListMLE in your case. The
evaluation file should be used for validating the performance of the
algorithm in the training so which you plan to implement. You should take
the corresponding definitions from the letor_internal.cc and create the
corresponding .cc files. Feel free to do necessary changes and in case of
any doubt, feel free to ask.

Regards,
Parth.
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

filter by facets and objective c version

Amish Shah — 2012-04-23T19:21:44

1. How do I perform filter by facets using the C++ api?
2. Has anyone ported xapian to objective C


thanks
Amish
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

MeCab as a Japanese tokenizer

Harrynson Hidalgo — 2012-04-20T07:56:51

The first version of a Japanese Tokenizer implementation using MeCab.
It hasn't been tested yet.

In order to compile property both MeCab and libiconv should be installed.

In ubuntu the following packages are needed: libmecab-dev mecab-ipadict-utf8
It's also recommended to install libiconv from source.
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Japanese tokenization

Harrynson Hidalgo — 2012-04-20T05:04:27

This is the first version I had, it's full of bugs... I had already
debugged it but forget to run gitt add... before doing git commit.
I'm re-debugging it right now.
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Japanese tokenization

Harrynson Hidalgo — 2012-04-20T04:46:13

I send the first prototype of the patch.
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Implementing the tf-idf weighting scheme

Jiuding Duan — 2012-04-20T03:47:22

And the patch file is in the attachment.
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Implementing the tf-idf weighting scheme

Jiuding Duan — 2012-04-20T03:33:36

Hi, all:

This is the basic implementation of tf-idf scheme (basic scheme used in
SMART) that can be used in the Xapian. It might still  need some futher
revision, but I believe it works anyway.:)

I modified the weight.h to define a subclass Tf_idfWeight and add a new
file tf_idf.cc in ../weight in the repo, to implement Tf_idfWeight.

Here is the git diff patch:

https://gist.github.com/2422049

I think the next thing to do is register this scheme to Xapian and write
some test to see whether or not it works?

I'm grepped the current BM25Weight, TradWeight and BoolWeight, and find
clues about Enquire::set_weighting_scheme( ). But something more should be
done to understand it.


Best,

Jiuding
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Posting list encoding improvements - pfd encoding & var len encoding comparison program

Weixian Zhou — 2012-04-20T02:51:45

Hi all,
I wrote a program that implement the variable length encoding and fixed
length encoding, and compares their index size and speed of search doc
length.
You can see the comparison result from the attachment snapshot.

1. The posting list is in all memory;
2. The search strategy of fixed length encoding is skipping with
exponential step (1, 2, 4, 8, ...). Once exceeds the desired doc id, back
to previous step and skip with step 1.
3. The implemented fixed length encoding uses 4 bytes as fixed length. This
is not optimal and can be further optimized in PFD.
4. The program generates uniform random doc id gap and doc len to make
posting list.

*You can access the code via my github:
https://github.com/zwxxx/pfd_simple_test*

Xapian::Database->close() for perl missing

Websuche :: Felix Ostmann — 2012-04-19T13:45:36

I have a xapian-daemon, which can be queried via http. A background-process
generated every hour one new index and then remove and create a new symlink
to the current database.

/path/to/index/20120419010000
/path/to/index/20120419020000
/path/to/index/20120419030000
/path/to/index/default => /path/to/index/20120419030000

So the daemon only check the mtime of /path/to/index/default/iamchert befor
every request and if it is a new one, he close/reopen the database.

The problem is: There is no ->close in perl for a database! So currently i
override the object. After some days, the prozess have many open
filepointers to allready removed databases.

I also tried ->reopen, but xapian dont reopen every file, so after reopen,
i have some files opened from the old database and the record.DB from the
new one. Also the filepointer to the old record.DB still exists (i tested
with lsof -p $PID).

So what is the right way to make a clean shutdown of a opened xapian
database?

I tested with xapian-core 1.2.9 / Search::Xa

Patch: New features for Learning to Rank

Rishabh Mehrotra — 2012-04-19T03:18:41

Hello,
Please find attached the patch which contains new features for the LETOR
framework which increases the total features from 19 to 37. In my proposal
I had mentioned of having a final count of 44 IR specific features apart
from those learnt by unsupervised feature learning part via Deep Learning.
So I am left with 7 out of those 44,of which 6 I will be implementing
within the next 2 days(before 22nd).
This patch also contains a 1 line change to
xapian-core/examples/questletor.cc correcting the usage of the questletor
file. I will send in another patch as soon as I am done with those 6
features.

Best regards,
Rishabh.
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Patch for Initial Prototype implementation of Unigram Langauage Modelling in xapian-core.

Gaurav Arora — 2012-04-15T01:09:33

Hi,

  I have implemented initial prototype of the  Xapian::Weight subclass for
Unigram Language Modelling to support UnigramLM weighing in xapian.Other
changes include adding collection_frequency to TermFreqs struct to store
collection frequency of terms and some changes to support it xapian
Framework,Changing simplesearch.cc to search using UnigramLMWeight class.

Following issues have not being addressed in this patch(I am working on
following issues):

1. Log trick for handling multiplication for LM need to made more robust
than just adding some random number to avoid rejecting document due to
negative value returned by log.

     Since each term contribution is probability(b/w 0 and 1). Hence
finding log will result in negative value and eventually rejection of
document.Hence a random linear weight has been added.It need to be
addressed by using log diffrent bases and some other techniques.

Discussion about log trick needed to be used are here for reference:
http://comments.gmane.org/gmane.comp.search.