gmane.comp.search.xapian.general

How to omindex some sub-directories?

Charles — 2013-05-15T06:35:01

Given a directory tree like ...

/foo
|
+-- A
|
+-- B
|
+-- C

... what is the best way to index A and C into a single Xapian database?

AFAIK the alternatives are:

omindex --db /my_db --no-delete /foo /foo/A
omindex --db /my_db --no-delete /foo /foo/B

or

omindex --db /my_A_db /foo /foo/A
omindex --db /my_B_db /foo /foo/B
xapian-compact /my_A_db /my_B_db /my_db

The first alternative does not delete files deleted from the file system
from the database.  Is there any way around this except by emptying the
database and starting over?

The second alternative increases storage and processing requirements. 
OK if needs must but would prefer to avoid.

Are there any better alternatives?

Best, Charles

Match positions of a queryresult

Cséri Tamás — 2013-05-15T02:11:07

Hello,

I've just started learning Xapian and I'm facing the following problem.

I've indexed many text files (using a TermGenerator from std::string), each
document in my database is a single file on the disk.
The search works pretty well and finds the files that match the query
string, but I can't figure out how I can determine the location of the
actual matched terms. I want to show the user the row and column number of
the match (to somehow highlight the match).

So far I haven't found a solution. The closest I've got is the
Enquire::get_matching_terms_* functions but this does not really work for
phases and I'm still far from character positions.

I hope someone can give me some hints where to look to begin with.

Thanks,
- Tamás Cséri -
_______________________________________________
Xapian-discuss mailing list
Xapian-discuss< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-discuss

Xapian 1.3.1 development snapshot released

Olly Betts — 2013-05-04T04:40:38

After rather longer than expected, Xapian 1.3.1 is now available.

Please note that 1.3.x releases are development releases - they are made
to encourage earlier and wider use and testing of new and changed code
so that 1.4.0 can both happen sooner and be better than otherwise.

Our record with 1.1.x was very good - all the bugs I am aware of were
either in new features, or were also present in the corresponding 1.0.x
release.  But if you main concern is minimising risk of accidental
breakage, sticking with 1.2.x would be prudent, at least for deployment.

The 1.3.x development series will lead to a stable 1.4.x release series.
We don't have a date set, but towards the end of this year seems plausible.

If you make packages of this release, please make sure that they are very
clearly labelled as not being a stable version, and ensure that they can be
installed in parallel with the stable version (the default paths and program
suffix are set to help make this easier).  If they are binary packages you
should al

Tutorial on OS X

Tae — 2013-05-01T18:42:27

Hi, guys.

Is there any example Cocoa project which uses Xapian?

Best,
Tae

What is the significance of 22 in the Debianlibxapian22 packages

Charles — 2013-04-30T03:54:06

What is the significance of 22 in the Debian libxapian22 and
libxapian22-dbg packages?  According to
http://packages.debian.org/search?keywords=libxapian22 they are built
from upstream versions 1.2.3 to 1.2.12 (of xapian-core?)

The reason for asking is that we need xapian-omega (and hence
xapian-core) at 1.2.8 or later.  Previously we have built and installed
directly from source but a package would be more convenient.  So I am
studying the existing Debian packages for guidance while creating 1.2.15
packages..

Charles

remote backend

Michael Lewis — 2013-04-26T14:42:33

So, given what I've read in the documentation I would create a text file named document_database.txt that might have the following:



remote 192.168.1.10:30000

chert /var/lib/xapian_database/segment1

remote 192.168.1.10:30000 chert /var/lib/xapian_database/segment2

remote 192.168.1.10:30000 chert /var/lib/xapian_database/segment3

etc.



I would then in my PHP program open document_database.txt as the database and then perform normal Xapian calls. The same with Omega, I would, in my template, change the database name to document_database.txt.



Given the above, it appears that I do not have to refer to each segment (segment1, segment2, etc.) for my calls, as it appears they are aggregated and treated as a single database and Xapian takes care of which actual database segment to search. IE, searching all the segments for my search terms.



Michael



-----Original Message-----

From: simon.roe< at >gmail.com<mailto:simon.roe< at >gmail.com> [mailto:simon.roe< at >gmail.com] On Behalf Of Sym Roe

Sent: Friday, April 2

Compiling Xapian within a Cocoa project

Tae Won Ha — 2013-04-26T06:45:49

Hi, folks.

I don't know much about C++, so please excuse my newbie question: I just tried to include Xapian in a dummy Cocoa project, ie created a new project and added the libxapian.a file installed via MacPorts. When I include xapian.h in an Objective-C++ file (either mm or h), the compilation fails with the following message:

=================
In file included from /opt/local/include/xapian.h:34:
In file included from /opt/local/include/xapian/database.h:32:
In file included from /opt/local/include/xapian/document.h:33:
/opt/local/include/xapian/valueiterator.h:143:10: error: expected member name or ';' after declaration specifiers
     bool check(Xapian::docid docid);
     ~~~~ ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk/usr/include/AssertMacros.h:1291:28: note: expanded from macro 'check'
         #define check(assertion)  __Check(assertion)
                                   ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform

Converting MySQL database to Xapian

Michael Lewis — 2013-04-25T21:56:26

I am looking for some guidance on converting a large MySQL database to Xapian. The current structure is that the database is broken up into 160 "sub-databases". There are 50,000 or so records in each stub database. Each record has content that I am full-text indexing. The average size of the text is about 59k characters. The database is broken up into sub-databases because the MySQL full-text handling is so abysmal. I use MySQL-Proxy and some Lua modules to make the searching and inserting transparent to the client programs. This data is written once and never (or once in a blue moon a record is deleted) modified. What I am looking to do is:

1. Create a xapian database on a remote server where each record's indexed text content would be a document (storing the document ID in the MySQL data record). I will then load the DB from the old MySQL server into the xapian server (eventually retiring the old server). It must remain in operation during the change-over.

2. Be able to access the xapian database via PHP

Xapian 1.2.15 released

Olly Betts — 2013-04-17T02:10:53

I've uploaded Xapian 1.2.15 (including Search::Xapian 1.2.15.0), which
you can download from:

http://xapian.org/download

You can see a summary of the most notable changes on the wiki:

http://trac.xapian.org/wiki/ReleaseOverview/1.2.15

As always, if you encounter problems, please report them here, or to
the bug-tracker: http://xapian.org/bugs

Here are the SHA1 checksums of the released files:

497071a70b93f04d71596f6212078d3e9d38b70a  Search-Xapian-1.2.15.0.tar.gz
88424067be668f3566b5921099d82032a7a88289  xapian-bindings-1.2.15.tar.gz
3d2ea66e9930451dcac4b96f321284f3dee98d51  xapian-core-1.2.15.tar.gz
a6aef085b1e63262c2f8ff7cd9419f28c8a53623  xapian-omega-1.2.15.tar.gz

And SHA256 checksums (more secure, but sha256sum is less widely installed):

179223bcee0c2c8f00f690ede478a1feef34e8f16da61c77277813fde36dee68  Search-Xapian-1.2.15.0.tar.gz
89d30cbd38dcfb9d9e0fccc8e4ba7d6452ffad45afb3f1e9614a3be0a3a33e77  xapian-bindings-1.2.15.tar.gz
cde4f5d1b111b66643fa41c11b9e5962bff7ce7244ca34cbbcbd2d2caa0c4df0  xapia

confusion about term prefixes

Chris Purves — 2013-04-16T17:57:37

I am confused about using term prefixes for omega searches.  There are a number of term prefixes that are reserved and used by omindex.  In order to use those for searching with omega, do I need to use the $setmap{} function in the omega template or are the reserved ones built in?

problems with indexing xlsx files

Chris Purves — 2013-04-05T18:47:11

Hello,

I have a number of Excel .xlsx files that aren't indexed properly.  To illustrate, I have a file called "this is a test.xlsx".  It consists of four cells:

| this |
| is   |
| a    |
| test |

It gets indexed but I am unable to search for it.

I was able to determine the index number and use delve to see the term list:

#delve users -r 16496
Term List for record #16496: D20130405 Hvesuvius M201304 P/ Tapplication/vnd.openxmlformats-officedocument.spreadsheetml.sheet Ufile://vesuvius/cpurves/this is a test.xlsx Y2013 Zthisisatest thisisatest

You can see that the words are all concatenated together as if they are a single word.  If I search for "thisisatest" it comes up, but not otherwise.

I'm using version 1.2.3 on Debian.

Newbie questions about omega

Julien Pfefferkorn — 2013-04-03T13:16:50

Dear all,

 

I am a newbie to Xapian-Omega.

I installed Xapian-Omega (1.2.14) a few days ago on my freebsd NAS server,
and intend to use it to search my CIFS/SMB share.

So far it works fine (I am able to search my share using the omega web
interface), nevertheless I have few newbie questions.

Hoping that somebody can answer my (hopefully not to silly) newbie
questions.

 

Indexing file names:

 

I noticed that Omega indexes file names. The file name seems to indexed as
several words if the name contains space characters.

In my share I often separate words in the file name using "-" or "_" or even
using a capital letter at the beginning of each word (I guess this is also
the case for many other users):

Examples:

this-is-a-file.txt

this_is_a_file.txt

thisIsAFile.txt

In those cases, a noticed that omega does not index the individual words,
but only the full basename as one single word.

It would be helpful, if omega would index each respective word, to ease the
search.

Is it planned to add that fea

Xapian wiki: typo in docid to sub-db translation?

2013-03-26T08:04:03

On the Xapian wiki page:
   http://trac.xapian.org/wiki/FAQ/MultiDatabaseDocumentID

It seems to me that:
  subdatabase_number = docid_combined % number_of_databases; 

Should read:
  subdatabase_number = (docid_combined - 1) % number_of_databases; 

Otherwise I'm seriously confused ...

Cheers,

jf

Incremental indexing

陈振 — 2013-03-18T05:52:01

Hi all,

I am trying to implement an Incremental indexing scheme. The problem
is that usually the modified documents are large but the modifications
are limited. Ideally, I would like to reindex only the modified parts
of these documents. If I am not mistaken, xapian cannot do that. Are
there any other approaches?

It would be nice if xapian supported something like the SQL "group
by". If it did, then it would be possible to break large documents
into several pieces which could be indexed independently. When
querying, these pieces would be then combined again using some
aggregate function similar to the SQL function sum.

Thanks

Xapian 1.2.14 released

Olly Betts — 2013-03-15T05:31:23

I've uploaded Xapian 1.2.14 (including Search::Xapian 1.2.14.0), which
you can download from:

http://xapian.org/download

You can see a summary of the most notable changes on the wiki:

http://trac.xapian.org/wiki/ReleaseOverview/1.2.14

As always, if you encounter problems, please report them here, or to
the bug-tracker: http://xapian.org/bugs

Here are the SHA1 checksums of the released files:

f06fc937a1124b876937b9836a32321919384b0f  Search-Xapian-1.2.14.0.tar.gz
e2b3cec358124bb32d28090ba39db5799ee0eceb  xapian-bindings-1.2.14.tar.gz
0199483021b7e8704ddbc9c27c3d1531566a847a  xapian-core-1.2.14.tar.gz
d82f978ee5fb3a7c53145f27e97d6b19e8bc3b64  xapian-omega-1.2.14.tar.gz

And SHA256 checksums (more secure, but sha256sum is less widely installed):

6b726d1ab9f5243be4996e03c012f6c1541fbff3b63efa811e7be494f28e6b72  Search-Xapian-1.2.14.0.tar.gz
a55815ddff0bd5f79814b5964cc94299ecf8f7ebafea11523aa0787f6082e3d7  xapian-bindings-1.2.14.tar.gz
daec6292b595b57e3ab48f44a1d1643866545bf494da8b9d3b9955e059b236a9  xapia

GSoC 2013

Olly Betts — 2013-03-10T11:13:37

Google are running their "Summer of Code" again this year - for
background info see:

http://code.google.com/soc/

We took part in 2011 and 2012, and we're going to apply again this
year.  Dan Colish and I are happy to act as admins again.  I'll submit
the application (which needs to be done between 18th and 29th March).

I've updated the list of project ideas for students on the wiki from
last year, removing those done last year, and updating those where work
has been done outside GSoC:

http://trac.xapian.org/wiki/GSoCProjectIdeas

If you're interested in acting as a mentor for one of the ideas there,
or have an idea for a project with a scope suitable for a student to
complete in about 12 weeks, please update that page.  Ideas without a
potential mentor aren't very useful though, so being willing to mentor
your new idea is helpful.

Ideas don't have to be for work on Xapian itself - projects related to
Xapian in other software are within scope.  A wider range of project
ideas will give us a broader appeal

Xapian web fronends

Omer Segev — 2013-03-06T14:29:02

Hi,

Is there any php or other web (not cli) front end (same as omega) for
Xapian index?

I would like to use Xapian for an intranet search portal, and modify the
style of it.

Many thanks,

Omer

Sorting by value - direction

Andrew Betts — 2013-03-05T10:58:15

Hi,

I'm trying to sort a list of results alphabetically, but it's coming out in the wrong direction.  Indexing is done like this (PHP bindings):

// Tag name as a value (for sorting)
$doc->add_value(2, strtolower($obj->tag));

And searching like this:

$enquire->set_sort_by_value(2);

But for some reason, the results are coming out Z-A, not A-Z.  I've tried setting true as the second argument to set_sort_by_value, even though we shouldn't need to reverse the direction because a standard sort should produce the right order already.  But even setting the second argument to true, I still get Z-A.

Using delve I can see the value (slot 2) is set correctly:

# delve -r 25 -V tags
Values for record #25: 1:? 2:finance
Term List for record #25: [snip]

Any ideas?

Andrew

Parsing fields with phrases.

Jim Lynch — 2013-02-21T02:15:45

I have the following code:

     my $par = 
$qp->parse_query($query,Search::Xapian::FLAG_SPELLING_CORRECTION);
     print LOG "Query $query, par $par\n";
     my $enq = $xDatabase->enquire( $par );

The output from the LOG file is:
Query title:"new dolphin", par Xapian::Query(0 * Snew dolphin)

No results are returned.  If I change the search to title:dolphin it 
finds a number of hits including one who's title is
Introducing the New Dolphin 
<http://localhost/mail/cur/mail.fdcga.com/Inbox/1905.html>

What do I do to make this work the way I want it to?

Thanks,
Jim.

Sticky results

Andrew Betts — 2013-02-20T10:10:32

Hi there,

I have a xapian index whose results are being sorted by a value, with (PHP bindings):

$enquire->set_sort_by_value($sort_data_value);

This is because I want the results returned in chronological order of publication date.  However, I now have a need to have certain results be 'sticky' at the top of the resultset, regardless of their publication date.  Obviously there are hacks such as setting a publication date in the distant future, but I'm wondering if there is a best practice way to do this within the Xapian world.

I considered that perhaps the answer was to remove the sort by value and replace it with a custom posting source that combined that value with the post's stickiness to give the post a weight.  Would that be the right approach?  If I do that how do I turn off all the default relevance weight logic?

Cheers,

Andrew

Go (golang) bindings for Xapian?

Marinos Yannikos — 2013-02-14T17:59:51

Hi,

is anyone working on Xapian bindings for Go? SWIG supports Go since
version 2.0 (http://www.swig.org/Doc2.0/Go.html), but there's some
Go-specific code that needs to be written.

Unfortunately, I have 0 experience both with SWIG and hacking on the
Xapian bindings, so I probably cannot do this as a weekend project. It
would come in very handy though.

Regards,
 Marinos