gmane.comp.search.xapian.devel

Better parsing of BM25 parameters in Omega

aarsh shah — 2013-05-15T13:27:08

Hello guys, as discussed on IRC, I have written some code for better
parsing of BM25 parameters in Omega. If no parameters are specified ,it
defaults all of them. However, if there some are specified and some are not
or if the invalid values are given for any of them,it throws an error.

https://github.com/aarshkshah1992/xapian/commit/ac0a11f5d8ff975fad1e96e63764eab9b04dfcfb

-Regards
-Aarsh
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Re: Added support for TfIdf to Omega

aarsh shah — 2013-04-15T06:40:20

Thanks a lot :)


On Mon, Apr 15, 2013 at 12:00 PM, Olly Betts <olly< at >survex.com> wrote:

_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Re: Added support for TfIdf to Omega

Olly Betts — 2013-04-15T06:30:54

Great - now merged.

Cheers,
    Olly

Added support for TfIdf to Omega

aarsh shah — 2013-04-11T10:19:14

Hello guys,I have added code for tfidf to the weight.cc file in omega/ .

Here is the patch : -

https://github.com/aarshkshah1992/xapian/commit/5ff41a15f574e6780cc61e67e7f3da3d97ff4ec8

It compiles well and I think it'll work well.

Here's the link to the documentation file omegascript.rst where I've added
tfidf.

https://github.com/aarshkshah1992/xapian/commit/9434ad15ad8b69691ad45f2d340450b3070f524e

Please let me know what you'll think. Also,I somehow seem to have messed my
branch by trying to merge the tfidf branch with the master branch and so am
not sending a pull request.I'll sort it out soon.

-Regards
-Aarsh
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Re: GSOC - Posting list encoding improvements

Marius Tibeica — 2013-04-07T07:56:19

Yes, that's what i was planning to do. And probably finding the size
of the stack (sizeof(int) comparings) is a lot less costly that all
the reallocations which will happen otherwise.


On Sun, Apr 7, 2013 at 12:04 AM, Olly Betts <olly< at >survex.com> wrote:

Re: GSOC - Posting list encoding improvements

Olly Betts — 2013-04-06T21:04:26

Cool, I'll take a look.


If you use a vector for the stack storage, then you can use reserve()
if you know how big it will grow.  That would also probably be a bit
more compact - std::stack uses std::deque by default, which doesn't
guarantee to use contiguous storage, but std::vector does.

Cheers,
    Olly

Question about Project: Posting list encodingimprovements

张旭东 — 2013-04-05T13:42:28

Dear all:

My name is Xudong Zhang, from Peking University. I'm very interested in the
Project: Posting list encoding improvements, and I have had some experience
on encoding, and so I want to be involved in the project.

About the project，I have a question to consult:

What's the main goal of the improvement of the speed for the encoding and
decoding of posting lists? Is it OK to increase the encoding/decoding speed
greatly with the slight loss of compression?

Below is about my personal information. I'm a master student from Peking
University, Beijing, China. My research area is search engine and web
mining, and I've been focusing on compression algorithms of posting lists
in search engines since last February. I have proposed a novel decoding
algorithm called SIMD-PFORDelta, which exploit SIMD instructions in
PFORDelta. It is fast and can achieve good compression ratio. Recently, I
also did some experiments to test the state-of-the-art and proposed
encoding algorithms of posting lists. We used our own s

Re: (no subject)

Abhishek Tyagi — 2013-04-05T17:30:33

Hi Olly,
On Fri, Mar 22, 2013 at 7:07 AM, Olly Betts <olly< at >survex.com> wrote:

I have checked these links and also those related to java bindings.
I have also checked the source code of these two projects. Now I think I
have better grip over these projects.
Can you suggest something else that I should do or should i start working
on my proposal?

Thanks,
Abhishek
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Re: GSOC - Posting list encoding improvements

Marius Tibeica — 2013-04-06T10:22:27

Implemented it on https://github.com/mtibeica/xapian

I could also find the size to which the stack will get (the order of
the highest bit in k - j) and initialize it to avoit reallocations?
I tested by using it in BrassPositionList::read_data. All the unit tests passed.

I will focus on finding a way to reduce the size of DIState.


On Tue, Apr 2, 2013 at 4:37 PM, Marius Tibeica <mtibeica< at >gmail.com> wrote:

Re: Merging of the TfIdf patch

aarsh shah — 2013-04-03T20:39:12

Hello .Is any additional normalization/code needed before the code for
TfIdf can be merged ? Please do let me know.I want to get the code merged
as soon as we can.

-Regards
-Aarsh


On Wed, Apr 3, 2013 at 1:04 AM, aarsh shah <aarshkshah1992< at >gmail.com> wrote:

_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Re: Merging of the TfIdf patch

aarsh shah — 2013-04-02T19:34:05

No Olly,there is no specific reason for that .I just decided to go with
uppercase strings as that makes them  easily visible and conspicuous when
reading the documentation and also while reading the code.However, the case
wont make any difference to the code.

-Regards
-Aarsh


On Tue, Apr 2, 2013 at 8:39 AM, Olly Betts <olly< at >survex.com> wrote:

_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

GSOC - Posting list encoding improvements

Marius Tibeica — 2013-04-02T13:37:57

Done! Tried to get as much information in here as possible.


Guess I understood the request wrong.
Using a binary search tree in order traversal modified version we get
this solution, which is pretty close to what you said:

extra items in the BitReader (or a derived class)
stack<DIState> di_stack; //will get to a maximum of O(logn) space size indeed
DIState di_current;

struct DIState {
  DIState() { set(0, 0, 0, 0); }
  DIState(int p_j, int p_k, Xapian::termpos pos_j, Xapian::termpos
pos_k) { set(j, k, pos_j, pos_k); }
  void set(int j, int k) { this.j = j; this.k = k;, this.pos_j =
pos_j; this.pos_k = pos_k; }
  bool is_nextterm() { return j + 1 < k; };
  bool is_uninitialized() { return j == 0 && k == 0 && pos_j == 0 &&
pos_k == 0; }
  Xapian::termpos pos_j, pos_k;
  int j, k;
};

// after calling this method, decode_interpolative_next will
sequentially return the termpos from j + 1 to k - 1
// on all the decoding methods, including decode_interpolative, we
must also check if a decode_interpolative has

FastMail's search using Xapian

Greg Banks — 2013-04-02T09:01:27

G'day,

I thought some folks on this list would be interested to know that the new full text email search feature rolled out today by FastMail

http://blog.fastmail.fm/2013/04/02/fast-full-message-searching-across-all-folders/

is powered by Xapian. All the code for indexing and searching emails is Open Source and in Fastmail's version of the Cyrus IMAP server.

https://github.com/gnb/cyrus-imapd/tree/fastmail

That isn't the most recent code but its pretty close.
We expect that code will be merged into a future release of Cyrus (but not the next release which is imminent).

We evaluated Solr, Sphinx and Xapian. Sphinx was the fastest but had a number of deployment issues due to our unique requirements.
I must say that the Xapian learning curve put us off for a while, but after gaining experience with other search technologies that became less of a problem.
In the end the ability to run Xapian as a library linked with our C application, and thus to control startup times and memory usage more easily than with

Re: Merging of the TfIdf patch

Olly Betts — 2013-04-02T03:09:58

One question - the 3 character codes that TfIdfWeight takes are (I
assume) modelled on those used by SMART:

http://people.csail.mit.edu/jrennie/ecoc-svm/smart.html

But TfIdfWeight requires them to be in upper-case, while SMART documents
them as being lowercase.  Is there a reason for that?

Cheers,
    Olly

Re: Doubt about GSOC proposal

Olly Betts — 2013-04-02T01:33:10

It should definitely cover the 14 weeks of the work period (2013-06-17
to 2013-09-23).  It's useful to also sketch out any preparation work
you're planning to do during the community bonding period:

http://www.google-melange.com/gsoc/events/google/gsoc2013

The GSoC timeline doesn't fit with university holidays perfectly in some
countries, so if you have exams, courses, or other commitments which
will eat into the work period much, then it's good if you're able to get
started earlier during the community bonding period, and if you're
planning to do that, you should include the details in your timeline
(both the commitments you have during the work period, and any work
you're planning to do during the community bonding period).


A better approach is to split the project into a series of smaller
projects, each of which in turn can be implemented (along with
documentation and testcases), debugged, reviewed and merged.

There are lots of advantages to this, both for the student and for
the mentors, for instan

Doubt about GSOC proposal

aarsh shah — 2013-04-01T16:30:04

Hello guys.I have begun work on writing my proposal as discussed on IRC and
will submit a draft in a couple of days so that I can make it detailed and
refine it after getting feedback.

I wanted to know about the number of weeks a proposal should cover and
also,is it okay if I set aside a buffer week somewhere in the middle of the
summer  for something like cleaning the code,working on the feedback  and
getting it merged(merging whatever code Ive completed till then.) ?

-Regards
-Aarsh
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Re: Need help as Pl2 tests not performing as expected

aarsh shah — 2013-03-28T01:59:19

Ill have a look at database statistics.

But interestingly, I just removed the collection frequency statistic(which
will be constant for one term) from all the formula and substituted it with
1. All the tests passed and max possible was now greater than max attained.
So,I'm sure that the problem is with the collection frequency statistic.As
Olly had correctly predicted,adding a new map to weight::internal didn't
work.

I talked to Gaurav Arora about when he will be merging his code and he said
he'll be doing it pretty soon.So yeah,I think the tests will work well
after I use his code to obtain the collection frequency.

-Regards
-Aarsh

On Thu, Mar 28, 2013 at 4:09 AM, Olly Betts <olly< at >survex.com> wrote:

_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Re: Need help as Pl2 tests not performing asexpected

Olly Betts — 2013-03-27T22:39:37

That makes sense.


"db" is just the variable name, not the path on disk.

The databases used for the tests are stored in subdirectories of the
tests/ directory.  The names start with a dot, so are hidden by
default in "ls", but "ls -a" will show them.

So look in .brass and you'll see all the cached databases in brass
format.

I'd guess you want something like .brass/db=apitest_dfr - one easy way
to be sure which you want is to wipe them all and then run just the
testcase you want the database for - then there should only be one
database there - for example:

rm -rf .brass
./apitest -b brass dfr_pl2weight4
ls -l .brass


I think that would probably happen if you're returning a value above the
upper bound you gave.

Cheers,
    Olly

Need help as Pl2 tests not performing as expected

aarsh shah — 2013-03-27T14:14:49

Hello guys. I just ran the updated tests for PL2 and they are not giving
the mset order I expect.Now,the thing is, dfr's behavior is a bit hard to
predict and so even if I expect a particular order ,it may give another
order and still be correct.So,the only way to write correct tests for PL2
is to manually calculate the weight of the documents to decide the expected
order.For that,I need to have a look at the statistics  stored in the
database for my tests.I ran the tests and after it failed ,I tried "delve
db" at the terminal,but it says that it can't open the database.Please can
someone help me this.

Also,the *max possible  *statistic of the Mset *is less than max
attained*and so ,Ill have to have a look at the code again.This may
take some
time,as PL2 has a very complex formula and it's a bit hard to understand
what's happening where.

-Regards
-Aarsh

On Wed, Mar 27, 2013 at 6:23 PM, aarsh shah <aarshkshah1992< at >gmail.com>wrote:

_______________________________________________
Xapian-devel mailing list
Xa

Major Mistake in pL2 tests in the pull request

aarsh shah — 2013-03-27T12:53:01

Hello guys.I just  realized that Ive not set the weighting scheme to PL2 in
the tests for PL2 and so a default weighting scheme of BM25 is used. I am
extremely sorry for this and am updating the tests by setting the weighting
scheme to PL2.

-Regards
-Aarsh
_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel

Re: GSOC - 2013 - Introduction (Learning to Rank)

Parth Gupta — 2013-03-27T01:05:02

That sounds good.

Parth.

On Mon, Mar 25, 2013 at 8:17 PM, Mudit Gupta <mudit.raaj.gupta< at >gmail.com>wrote:

_______________________________________________
Xapian-devel mailing list
Xapian-devel< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-devel