gmane.comp.search.xapian.general

Any update on a python 3.x compatible bindings forXapian

Steve Zatz — 2010-02-09T12:53:13

I see this trac ticket:  http://trac.xapian.org/ticket/346 which looks like
it was last updated 6 months ago.  Is there any information available on the
time frame?  Thanks,

Steve

trac error adding comment to #394

Henry C. — 2010-02-05T09:35:06

Greets,

Just tried to add a comment to trac ticket 394.  Resulted in error:

Trac detected an internal error:
AttributeError: 'NoneType' object has no attribute 'split'

----

My comment, for posterity, was:

You may want to seriously consider pushing this patch into trunk since the
performance benefit is, well, significant.

My phrase test:  from 15s+ down to 1s

----

regards
Henry

Optimal usage of xapian-compact for merging

Henry C. — 2010-02-02T12:49:46

Greets,

I've been wondering, what's the sane/optimal use of xapian-compact when
merging many indexes with a view to maximum merging performance?

The obvious:
- only use -F on the final db.
- use -m since I'm merging more than 3 dbs.

Best strategy?
a)  loop:  merge batches (of say 50, where the individual db's are small)
into a temp index, then merge the (larger) temp into the final product...
end-loop

b)  loop:  merge batches (of say 50, where the individual db's are small)
into many temp indexes... end-loop
Then merge those (larger) temps into the final product.

Finally, presumably it's best to use the same blocksize (-b) as the
underlying filesystem?  I see the default is 8K, but the default blocksize
on (eg) ext3 is 4k...  or am I way off here?

Thanks
Henry

How to use a custom stemmer from Python bindings?

Eugene! — 2010-02-02T07:22:11

Hi,

I'm using Xapian bindings for Python in my project. How could I use a
custom stemmer instead of the included one (Snowball)? The one I'm
looking at right now is Hunspell (http://hunspell.sourceforge.net/)
which has Python bindings (http://code.google.com/p/pyhunspell/).

Thanks in advance,

Eugene

Xapian core dumping on Solaris

Tom — 2010-02-01T09:41:24

Hi everyone,

I'm having a problem with xapian (the matchspy branch, with Python
bindings) on Solaris 10 / SPARC. Users can run queries for a few hours
with no problems, then it returns "inflate failed (invalid code
lengths set)"  to Python and dumps core. gdb reports:

#0  0xfe6dc910 in inflate_table () from /usr/lib/libz.so.1
#1  0xfe6d9d6c in inflate () from /usr/lib/libz.so.1
#2  0xfe2ebe20 in FlintTable::read_tag (this=0x75ed70, C_=0x75ede0,
    tag=0xfdcf8568, keep_compressed=false)
    at backends/flint/flint_table.cc:1254
#3  0xfe2eff70 in FlintTable::get_exact_entry (this=0x75ed70, key=< at >0xfdcf8578,
    tag=< at >0xfdcf8568) at backends/flint/flint_table.cc:1190
#4  0xfe2ddc04 in FlintSpellingTable::open_termlist (this=0x75ed70,
    word=< at >0xfdcf8998) at backends/flint/flint_spelling.h:46
#5  0xfe20dcb0 in Xapian::Database::get_spelling_suggestion (this=0x75b5a8,
    word=< at >0xfdcf8998, max_edit_distance=2) at include/xapian/base.h:476
#6  0xfe39ba90 in Xapian::QueryParser::Internal::parse_query (this=0x75b5

Solaris core dump

Tom — 2010-02-01T09:48:56

Hi everyone,

I'm having a problem with xapian (the matchspy branch, with Python
bindings) on Solaris 10 / SPARC. Users can run queries for a few hours
with no problems, then it returns "inflate failed (invalid code
lengths set)"  to Python and dumps core. gdb reports:

#0  0xfe6dc910 in inflate_table () from /usr/lib/libz.so.1
#1  0xfe6d9d6c in inflate () from /usr/lib/libz.so.1
#2  0xfe2ebe20 in FlintTable::read_tag (this=0x75ed70, C_=0x75ede0,
   tag=0xfdcf8568, keep_compressed=false)
   at backends/flint/flint_table.cc:1254
#3  0xfe2eff70 in FlintTable::get_exact_entry (this=0x75ed70, key=< at >0xfdcf8578,
   tag=< at >0xfdcf8568) at backends/flint/flint_table.cc:1190
#4  0xfe2ddc04 in FlintSpellingTable::open_termlist (this=0x75ed70,
   word=< at >0xfdcf8998) at backends/flint/flint_spelling.h:46
#5  0xfe20dcb0 in Xapian::Database::get_spelling_suggestion (this=0x75b5a8,
   word=< at >0xfdcf8998, max_edit_distance=2) at include/xapian/base.h:476
#6  0xfe39ba90 in Xapian::QueryParser::Internal::parse_query (this=0x75b590,

Failure trying to update document.

Jesper Krogh — 2010-01-30T13:30:35

Hi list.

I have a specific document that does not handle updates sitting in the
index. What can I do about that?

2010-01-30T13:58:07     Eval failure: Exception: No termlist for
document 287376 at /usr/lib/perl5/Search/Xapian/Enquire.pm line 56.
2010-01-30T13:58:07     job failed.  considering retry.  is max_retries
of 1000 >= failures of 1?
2010-01-30T13:58:07     job failed: Exception: No termlist for document
287376 at /usr/lib/perl5/Search/Xapian/Enquire.pm line 56.


Xapian version 1.0.14

Problem getting Xapian working with Burmese

2010-01-28T10:50:12

 On Fri, Aug 21, 2009 at 02:44:44PM +0200, emmanuel at engelhart.org wrote:

Hi Olly,

thank vor your answer (and sorry not having answered before).

Your answer helped me and I think I now understand why "it does not work".

For test purpose I index one document with one string  with index_text_without_positions() (C++ API) the string "ဝီကီပိသုံးစွဲသူများက"
See this log: http://tmp.kiwix.org/tmp/kiwix-index.log (utf8 encoded)

But if I run "delve -r 1 /path/to/db" on the index I get following answer:
Term List for record #1: test က စ ပ မ ဝ သ  (utf8 encoded)
See the log : http://tmp.kiwix.org/tmp/delve.log

So, it seems to be clear for me why "it does not work" : my word is splitted in single lletters and a lot of letters are removed.

Do I'm right? Do we can avoid that and index "ဝီကီပိသုံးစွဲသူများက" as only one word?

Regards
Emmanuel




________________________________________

Xapian under 360 MB VPS

Marlon Baculio — 2010-01-21T16:31:26

Hello,

I wish to get some feedback on the use of Xapian in a virtual machine hosting plan with 360MB. The processes to share the 360MB will be the following:

0. nginx web server as front (estimated 5MB)

1. custom C++ FastCGI for dynamic requests (estimated 10MB)

2. Xapian writer (1 process and 1 thread)

3. Xapian readers (1 process with n threads for n readers)

4. PostgreSQL (estimated 50MB or lower)


That leaves about 300MB for Xapian and the rest of the Linux OS. The main UI will be a Google style search box.

Questions:

0. How would you configure Xapian for such low memory systems (e.g. how many readers, flush threshold for writer)?

1. Will file handle limitation be a problem for multithreaded Xapian reader?

2. What are advantages of multiprocess readers (compared to multithreaded) aside from crash isolation


Thanks so much!
Marlon

Error when creating trac bug ticket

Henry — 2010-01-20T09:18:12

Greets

Just tried to create a bug ticket on trac.xapian.org and it croaked with
the error:

-----------
Trac detected an internal error:
IntegrityError: columns ticket, name are not unique

The action that triggered the error was:
POST: /newticket
-----------

Clicking on the Create button to report the error results in an invalid URL.

What's the best way to proceed to report my bug?

Thanks
Henry

QueryParser: aliases and OP_AND

Menard, Daniel — 2010-01-19T15:35:59

Hello,

I'm wondering about how the QueryParser parses a query containing an "alias" when the default operator is OP_AND
(by "alias", I mean a search field mapped to multiple term prefixes).

With the following php code :
<?php
$parser=new XapianQueryParser();
$parser->set_default_op(XapianQuery::OP_AND);
$parser->add_prefix('alias', 'AUT1:');
$parser->add_prefix('alias', 'AUT2:');
echo $parser->parse_query('alias:(john smith)')->get_description();
?>

I get:

Xapian::Query(((AUT1:john:(pos=1) OR AUT2:john:(pos=1)) AND (AUT1:smith:(pos=2) OR AUT2:smith:(pos=2))))
i.e. (AUT1:john OR AUT2:john) AND (AUT1:smith OR AUT2:smith)

I was expecting something like:

Xapian::Query(((AUT1:john:(pos=1) AND AUT1:smith:(pos=2)) OR (AUT2:john:(pos=1) AND AUT2:smith:(pos=2))))
i.e. (AUT1:john AND AUT1:smith) OR (AUT2:john AND AUT2:smith)

I think that I understand why I get the current result: "alias:(john smith)" is parsed as "alias:john AND alias:smith" and the alias is then expanded.

But for my application, it produces s

Index indexed words

double — 2010-01-18T21:04:19

Hello,

We would like to create Google or Firefox like "search hints".
If someone types "abc", the search system should name
some possible hints.

I think, Firefox does it by indexing 3-characters of the domain
name. If you enter parts, you get some hints.

Thank you very much
Marcus

postlist: Tag containing meta information iscorrupt.

Henry — 2010-01-18T08:07:24

Greetings,

Using latest svn.

I've noticed the following error when performing index merging:

postlist:
baseB blocksize=8K items=33962 lastblock=534 revision=1 levels=2 root=459
B-tree checked okay
Tag containing meta information is corrupt.
postlist table errors found: 1

I can still search on this index (I've only checked very small indexes),
but merging is now a problem since I check the return code when merging
batches (this obviously returns code 1).

Is this something that I may have caused?

Thanks
Henry

PHP XapianTermIterator/XapianPositionIterator usage

Graham Kann — 2010-01-16T10:04:38

Hello again,

/thanks to Peter for previous response.

I've been digging around trying to find sample usage of
XapianTermIterator/XapianPositionIterator in PHP.  The idea is to code up a
test case in PHP to perform snippet extraction (with a possible view to
coding a pecl extension in C).   I found a C++ sample, but that wasn't much
help.

I must be dense this morning though, since I can't get my head around how to
use XapianTermIterator to step through the terms, and to iterate through all
the positions of each term with XapianPositionIterator.

Can someone provide a basic PHP example of how to:

//pseudo code
$position_iterator = new XapianPositionIterator();
$term_iterator     = new XapianTermIterator();

foreach $term ($position_iterator)
    foreach $position ($term_iterator($term))
...


Any assistance would be appreciated.

Thanks

Searching log files?

Moazam Raja — 2010-01-16T06:20:33

Hi all, has anyone used Xapian/Omega to index and search large amounts
of (Unix) server logs?

I'm looking to create a search application which will allow me to
index and search logs from roughly 20-100 servers but I'm not sure
which engine to use that will provide near real time indexing and good
search performance. Is this something that can be accomplished with
Xapian/Omega?

Thanks!

Text snippets

Graham Kann — 2010-01-15T10:10:23

Hello!

I'm new to the list, so please bare with me.

This is a follow-on to Shripad Bodas's question about snippets/highlighting.

I too have tried the Perl Search::Tools mode, and it works well.  Only
problem is that it's slow - eg, when displaying results which exceeds (say)
50, the time taken to render the page (snipping and highlighting with
Search::Tools) can actually exceed the time to perform the search itself...

Seeing that our search frontend is coded in PHP, it makes sense to use PHP
exclusively (calling a Perl routine works, but you pay a double penalty -
the one mentioned above, plus the usual costs associated with calling
Perl).  Is anyone aware of PHP code I can use to create excerpts/snippets
and keyword highlighting (with usage of stemming of course)?

Any pointers would be appreciated.

Thanks!
Graham

Text snippets

Shripad Bodas — 2010-01-14T21:40:19

Hi,

The problem I am facing using Search::Tools is xapian treats query
string in terms eg. if the query is "universities" xapian may have a
corresponding term for this query as "universit" and while searching
it uses this term rather than "universities". Now if one of the page
has "university" in its text I want to highlight that as well. But
since the actual query is "universities" Search::Tools doesnt
highlight this word though xapian::search shows this page in the
result.
Now in my overall architecture I can supply this term list (i.e.
universit) to the perl script but is there any way that Search::Tool
will match half-word but highlight full-word?

Thanks,

Shripad Bodas

Latest revision and backwards compatibility

2010-01-14T09:10:45

Greetings,

I've been wondering about the index format and backwards compatibility.

We're using the dev version (for chert) and each svn up means that any
indexes created prior to this revision cannot be read.  Is this purely a
cautious move to prevent errors, and, barring any obvious index format
changes, can I safely force the current revision to read existing indexes?
 eg, by commenting out the throw in
xapian-core/backends/chert/chert_version.cc.

Or is there a better way?

Thanks
Henry

Revision 13824 compilation failure

2010-01-13T12:42:40

g' day

Configured with --disable-maintainer-mode.

Compilation of revision 13824 is failing with:

common/serialise-double.cc: In function double unserialise_double(const
char**, const char*):
common/serialise-double.cc:141: error: SerialisationError is not a
member of Xapian
common/serialise-double.cc:157: error: SerialisationError is not a
member of Xapian
common/serialise-double.cc:169: error: SerialisationError is not a
member of Xapian

Regards
Henry

London Open Source Search meetup - Tue 12 January/ Meetup.com

René Kriegler — 2010-01-05T10:35:53

Hi all,

We are organising another open source search social evening (OSSSE?) in
London on Tuesday the 12th of January. 

The plan is to get together and chat about search technology, from
Lucene to Solr, Hadoop, Mahout, Xapian, Ferret and the like - bringing
together people from across the field to discuss ideas and ask questions
over a quiet drink.

We've also created a group on Meetup.com where you can sign up to get
notifications by email about upcoming search socials.

For directions to this meetup and for the Meetup.com group see:

http://www.meetup.com/london-search-social/ 

Please come along if you can!


René & Rich (Marr)



_______________________________________________
Xapian-discuss mailing list
Xapian-discuss< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-discuss

Multiple spelling suggestions

Garrett Smith — 2010-01-01T16:50:31

Is there a way to get a list of spelling suggestions from Xapian?
Database::get_spelling_suggestion (obviously) returns just a single
suggestion.

Garrett