gmane.comp.search.xapian.general

[q] phrase replacement in thousands of text files

V S P — 2012-05-21T02:59:11

Hello, first post.
I searched through docs and examples but did not see this particular
problem answered.

I have thousands of text files total size about 17GB.
Within a file I need for find a phrase (typically up to 3 words together
separated by spaces, commas, and non period punctuation mark).

I have a dictionary of about 3 million phrases and their replacement.

So I need to replace all of the matching phrases from the dictionary
with their replacements

The most brute force approach I though was
a) build an index on all of the 17GB of documents
b) for every one of the 3 million search phrases do search
c) expect to return from ( b ) xapian match where I would get the start
and end byte location in a file for every search
remember that location, and the found phrase in a 'future replacement
list'

d) when done , use the 'future replacement list' -- to perform the
replacement operation


Obviously couple of problems 3 million times search 17GB worth of text
Second -- I do not understand how (if at all pos

Building php5-xapian on Ubuntu 12.04 Server x64

2012-05-20T11:27:53

As Xapian's been dropped from the Ubuntu repositories, I'm attempting to
build my own .deb from the instructions here:

 

http://article.gmane.org/gmane.comp.search.xapian.general/8855

 

http://beeznest.wordpress.com/2011/07/06/howto-build-your-own-binaries-of-ph
p-xapian-bindings-for-debian/

 

I can only get things to progress beyond the first few seconds by leaving
out 'rm debian/control', but if I do, it looks as if the Python and Ruby
bindings are building and passing their versions of smoketest correctly.

 

However, the PHP part of the build is failing with this error:

 

    /home/charlie/xapian-bindings-1.2.8/php/smoketest.php:38:
include(xapian.php): failed to open stream: No such file or directory

    FAIL: smoketest.php

 

There's a xapian.php file in /home/charlie/xapian-bindings-1.2.8/php/php5/
but if I copy it to /home/charlie/xapian-bindings-1.2.8/php/ or change the
path to it in smoketest.php, the build fails right near the start with:

 

    dpkg-source: error: aborting due to unex

php bindings won't build with MAMP

Mark Grandi — 2012-05-17T23:29:09

Hello, I'm trying to build the xapian bindings to use with php, i've
installed using MAMP to make it easier to set up the apache/php/myssql
stuff (http://www.mamp.info/en/mamp/index.html), but when I try to
configure the bindings, i get this:

configure: error: '/usr/bin/php-config --includes' reports
'-I/Applications/MAMP/bin/php/php5.3.6/include/php
-I/Applications/MAMP/bin/php/php5.3.6/include/php/main
-I/Applications/MAMP/bin/php/php5.3.6/include/php/TSRM
-I/Applications/MAMP/bin/php/php5.3.6/include/php/Zend
-I/Applications/MAMP/bin/php/php5.3.6/include/php/ext
-I/Applications/MAMP/bin/php/php5.3.6/include/php/ext/date/lib' but zend.h
can't be included from there


I made it so its using the php-configure from MAMP, so its reporting the
correct paths, those paths did not exist, but I downloaded the header files
for the correct php version and stuck them in there, but its still claiming
it can't find them. I've tried a lot of things and nothing work =(

I read this blog post:
http://blog.geertvd.be/2011/

Xapian 1.2.10 released

Olly Betts — 2012-05-10T01:31:34

I've uploaded Xapian 1.2.10 (including Search::Xapian 1.2.10.0).

Unsurprisingly, you can download it from:

http://xapian.org/download

You can read an overview of the release here:

http://trac.xapian.org/wiki/ReleaseOverview/1.2.10

The full lists of user-visible changes are linked to from there, and also
from the "[news]" links on the download page.

As always, if you encounter problems, please report them here, or to
the bug-tracker: http://xapian.org/bugs

Here are the SHA1 checksums of the released files:

121e0ab529a3e5a70931f61463a56e944805ae10  Search-Xapian-1.2.10.0.tar.gz
631c7650ac2ca47f8a72434a06d463da5b7596f4  xapian-bindings-1.2.10.tar.gz
1be1896ab11a3a66c6c0ade962c700d96678116e  xapian-core-1.2.10.tar.gz
59fca635728d5ce2ca18e5f39b8d3ef081e46f6c  xapian-omega-1.2.10.tar.gz

And SHA256 checksums (more secure, but sha256sum is less widely installed):

9695b45b978ef41b367fc5ea2bcf5ecbc2b4c5aff6d138d7dbc31946fdb86bc7  Search-Xapian-1.2.10.0.tar.gz
dd191ad2e1134b7008b3b71c7cab41d2d80df2fe82ecdb9ad

Incorrect line nums displayed in full text search

Babu Neelam — 2012-05-03T02:34:04

Hi,

Could you please clarify a query regarding full text search in Xapain?

I have installed LXRng (cross-referencer), which internally uses Xapain for
text search. I am using CentOS 6 & have installed the following Xapian
packages:

$ yum list installed | grep -i xapian
perl-Search-Xapian.x86_64
                         1.2.9.0-1.el6  < at >rpm-eprints-org-xapian
perl-Search-Xapian-debuginfo.i686
                         1.2.7.0-1.el6  < at >rpm-eprints-org-xapian
perl-Search-Xapian-debuginfo.x86_64
                         1.2.7.0-1.el6  < at >rpm-eprints-org-xapian
rpm-eprints-org-xapian.noarch
xapian-bindings.x86_64   1.2.9-1        < at >rpm-eprints-org-xapian
xapian-bindings-perl.x86_64
                         1.2.9-1        < at >rpm-eprints-org-xapian
xapian-bindings-php.x86_64
                         1.2.9-1        < at >rpm-eprints-org-xapian
xapian-core.x86_64       1.2.9-1        < at >rpm-eprints-org-xapian
xapian-core-debuginfo.x86_64
                         1.2.7-1        < at >rpm-eprints-org-xapian
xapian-core-devel.x86_64 1.2

GSoC 2012 Update

Olly Betts — 2012-04-27T06:06:04

The accepted students were announced earlier this week.  I wrote a blog
post yesterday with the details, which I'll link to rather than repeat
here:

http://survex.com/~olly/blog/xapian/xapian-gsoc-2012-projects.html

We're intending to use the xapian-devel mailing list for development
discussion of the projects - if you want to follow along, please
subscribe, or follow it on gmane, or similar.  We'll also be using the
#xapian IRC channel on Freenode.

Please help answer questions where you feel able, and if you're
interested in the projects, do try them out and offer constructive
feedback.

Cheers,
    Olly

Rebuilding corrupt databases from .DB files.

Graham Jones — 2012-04-16T00:15:08

We've had some catastrophic filesystem failures that have left us with corrupted databases with empty files and no backup for about 15TB of our data. Recreating the 15TB from source data backups is possible but will take a very very long time. 

I'm hoping that, given all of the .DB files are still intact, there my be some way to extract their contents and rebuild the other tables. 

This is one of our 800 databases that has been corrupted and you can see which files are empty:
drwxr-xr-x 1152 search search       40960 Apr 15 23:06 ..
drwxr-xr-x    2 search search        4096 Jan 31 17:39 .
-rw-r--r--    1 search search           0 Jan 31 17:39 iamchert
-rw-r--r--    1 search search           0 Jan 31 17:39 position.baseB
-rw-r--r--    1 search search  2191785984 Jan 31 17:39 position.DB
-rw-r--r--    1 search search           0 Jan 31 17:37 position.baseA
-rw-r--r--    1 search search           0 Jan 31 17:37 termlist.baseB
-rw-r--r--    1 search search  3953393664 Jan 31 17:37 termlist.DB
-rw-r--r--    1 s

Omega: favicon.ico?

Charles — 2012-04-14T03:14:51

Hello :-)

Does omega support favicon.ico files?  If so, how can they be set up?
If not, please consider enhancing Omega to do so and providing a default
favicon.ico file.

There are two reasons for wanting favicon.ico support:

1. Apache generates 404 errors for /favicon.ico when serving an
Omega-generated page.

2. It would provide a more complete UI appearance.

Best

Charles

Incremental Index update

Jaguar Xiong — 2012-04-04T02:26:19

Hi there,
I'm looking for a full-text-search engine/library and find xapian after
googling. The application I'm working on get some specific characteristic:
1.The current data-set are already quite huge: multi-tera bytes, even
after compression.
2.Existing data are mostly read-only.
3.New data is coming every minute. The daily total could be several
gigabytes (before compressing).
4.Query rate are not huge, yet. But I do expect a real-time search,
that's, new data is expected to be available for searching after a few
minutes.

Could you share some thought about xapian with regard to above aspects?
Especially on incremental index update.

Best Regards!
Jaguar

php fatal error no matching for overloadedfunction Enquire_set_sort_by_key

Xiao-Wei (Shawn) Xu — 2012-04-01T03:51:10

I'm using PHP. Xapian is giving me this:

Fatal error: No matching function for overloaded 'Enquire_set_sort_by_key'
in PATH/xapian.php line 1068

Confirmed that the xapian.so extension is loaded successfully.
var_dump(function_exists('Enquire_set_sort_by_key') outputs true.

in xapian.php line 1068 :

case 1: Enquire_set_sort_by_key($this->_cPtr,$sorter); break;

As the source code of the function suggests, it has to be passed a third
parameter so I did:

case 1: Enquire_set_sort_by_key($this->_cPtr,$sorter, FALSE); break;

but after this, I got 'segmentation fault from the apache error log and no
response is processed and sent back to browser.

Is it a but in xapian.so or what, how to solve this?

Thanks.

Compiling Omega 1.2.9 on Windows 7 64-Bit

Noel Huebers — 2012-03-28T01:54:00

Hi All,

I 'm trying to build Omega 1.2.9, configure ran through, but when I do make I get error below.

Any thoughts what I am missing?

Thanks

Noel

make  all-recursive
make[1]: Entering directory `/xapian/xapian-omega-1.2.9'
Making all in .
make[2]: Entering directory `/xapian/xapian-omega-1.2.9'
/bin/sh ./libtool   --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I./commo
n -I/usr/local/include  -Wall -W -Wredundant-decls -Wpointer-arith -Wcast-qual -
Wcast-align -Wno-long-long -Wformat-security -fno-gnu-keywords -Wundef -Wstrict-
null-sentinel -Wshadow -Wstrict-overflow=1 -Winit-self -Wlogical-op -Wmissing-de
clarations -fvisibility=hidden -I/xapian/xapian-core-1.2.9/include -g -O2 -MT li
btransform_la-transform.lo -MD -MP -MF .deps/libtransform_la-transform.Tpo -c -o
libtransform_la-transform.lo `test -f 'transform.cc' || echo './'`transform.cc
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I./common -I/usr/local/include -Wall
-W -Wredundant-decls -Wpointer-arith -Wcast-qual -Wcast-align -Wno-long-lon

Xapian 1.2.9 uses UNIVERSAL->import,deprecated in perl-5.14.2

Johan Sydseter — 2012-03-22T09:34:33

Hello.

Xapian 1.2.9 uses UNIVERSAL->import, deprecated in perl-5.14.2

We noticed that we could not get our perl tests to pass since 
UNIVERSAL->import is deprecated in perl-5.14.2.

The message given was:

UNIVERSAL->import is deprecated and will be removed in a future perl at 
/usr/local/share/perl/5.14.2/Search/Xapian.pm line 3158

Johan Sydseter

ABCStartsiden

Writing a MatchDecider and exposing it to PHP

Paul Dixon — 2012-03-20T11:16:42

I would like to write a MatchDecider in C++ and expose it to PHP.

I'm able to compile a simple extension and use SWIG to expose a simple
function to PHP, but I can't seem to hit upon the right incantation to
expose my MatchDecider class.


I have a header file declaring my decider, MyDecider.h

#ifndef _INC_MYDECIDER_H
#define _INC_MYDECIDER_H

#include <xapian/enquire.h>
#include <xapian/types.h>

class MyDecider : public Xapian::MatchDecider {

    Xapian::valueno valuenum;
    bool inclusive;

  public:
     MyDecider  (Xapian::valueno slot, bool inclusive_)
: valuenum(slot), inclusive(inclusive_) { }

    bool operator()(const Xapian::Document& doc) const;
};


#endif



I have a very simple SWIG interface file, MyDecider.i


%module mydecider


%{
extern int demo_function(int x);
#include <xapian.h>
#include "mydecider.h"
%}


extern int demo_function(int x);
#include <xapian.h>
#include "mydecider.h"


This successfully exports my 'demo_function' to PHP, but not the
MyDecider

Incremental indexing

Marios Titas — 2012-03-20T06:01:07

Hi all,

I am trying to implement an Incremental indexing scheme. The problem
is that usually the modified documents are large but the modifications
are limited. Ideally, I would like to reindex only the modified parts
of these documents. If I am not mistaken, xapian cannot do that. Are
there any other approaches?

It would be nice if xapian supported something like the SQL "group
by". If it did, then it would be possible to break large documents
into several pieces which could be indexed independently. When
querying, these pieces would be then combined again using some
aggregate function similar to the SQL function sum.

Thanks

MSetIterator::get_percent Shows Increased ValuesWhen Using Query::OP_FILTER

Michael Henson — 2012-03-16T20:04:58

I'm seeing some rather odd behavior with respect to the match percent that
is returned when performing a simple query. It appears that the addition of
a filter to a simple OR query will not affect the weight of the match, but
will increase the percent of the resulting match. According to the
documentation for MSetIterator::get_percent (
http://xapian.org/docs/apidoc/html/classXapian_1_1MSetIterator.html#99514c11a9291175d20fafa73a0839fd)
a match should only get 100% if it matches all of the weight contributing
terms in the query.

I've attached a test case that shows the odd behavior that I am seeing. I
had a database with 2 documents, each with one term 'cold' and 'kid'. Each
document also has a boolean term for the owner of the document 'Ome'.

Below are the results for a search for 'cold kids' on that set with and
without the owner filter. The weight 0.405465108108 stays the same for both
queries, but the percentage increases from 50 to 100 with the addition of
the filter.

Query: Xapian::Query((cold OR ki

Xapian 1.3.0 development snapshot released

Olly Betts — 2012-03-14T06:14:07

Xapian 1.3.0 is now available.

Please note that 1.3.x releases are development releases - they are made
to encourage earlier and wider use and testing of new and changed code
so that 1.4.0 can both happen sooner and be better than otherwise.

Our record with 1.1.x was very good - all the bugs I am aware of were
either in new features, or were also present in the corresponding 1.0.x
release.  But if you main concern is minimising risk of accidental
breakage, sticking with 1.2.x would be prudent, at least for deployment.

The 1.3.x development series will lead to a stable 1.4.x release series.
We don't have a date set, but I'd expect it to be later this year.

If you make binary packages of this release, please make sure that
they are very clearly labelled as not being a stable version, and if
possible ensure that they can be installed in parallel with the stable
version (the default paths and program suffix are set to help make this
easier).  I would probably recommend not rushing to package this right
away.

128 bit Document IDs (Please don't hurt me)

Shane Spencer — 2012-03-09T23:39:48

I apologize for what may be a sore subject.  4 billion documents is a
heck of a lot.  64 bit vs 32 bit would be an incredibly large database
with an average document and term size.  Why 128 bit?  Simply for
address space.

Mapping a UUID (128 bit) or MongoDB ObjectID (96 bit) directly into
the Xapian document space removes the need for referencing one or the
other from one or both.  I see a common tendency to write a document
to the Xapian, return the document ID, and then write to the database
backing the document in some way.

This is nothing new.. but I really would like to remove that extra
write and optionally throw a way the Xapian response by specifying the
document ID as the UUID associated to the document.  This is starting
to become much more important as people are walking away from
auto-increment fields and aiming more toward universal identification
which, from a sparseness standpoint, is amazingly wasteful but
incredibly useful.

Thanks for your consideration.  I have no idea how complicated it

Xapian 1.2.9 released

Olly Betts — 2012-03-09T12:50:29

I've uploaded Xapian 1.2.9 (including Search::Xapian 1.2.9.0).

As usual you can download from:

http://xapian.org/download

You can read an overview of the release here:

http://trac.xapian.org/wiki/ReleaseOverview/1.2.9

The full lists of user-visible changes are linked to from there, and also
from the "[news]" links on the download page.

As always, if you encounter problems, please report them here, or to
the bug-tracker: http://xapian.org/bugs

Here are the SHA1 checksums of the released files:

911f3b0aede0ad323d8c0570691ffa1ec13a9502  Search-Xapian-1.2.9.0.tar.gz
3cd457058ae89ff62588c56308fdba0c311afbc0  xapian-bindings-1.2.9.tar.gz
ce66725d74790639e0703db4b117b7ab5b326216  xapian-core-1.2.9.tar.gz
6340a83621822e6b2adcdebe282cce84c2ae075b  xapian-omega-1.2.9.tar.gz

And SHA256 checksums (more secure, but sha256sum is less widely installed):

9d608cd4afabe24323cffa1a96526fd0adcd99a5ed7c07a6845abcdb854f9485  Search-Xapian-1.2.9.0.tar.gz
49a4555313cd03afc8a969b6939d275fe651604af0825f3fcc52a2cc8a34f13b  x

how to enable debugging

Matteo Zandi — 2012-03-05T23:11:25

Ciao,

sorry for the silly question, but how do you enable debugging in
xapian? I'm using xapian-tpcsrc and would like it to print debugging
info (connections, queries, etc).

I'm configuring xapian 1.2.8 on OSX with "./configure
--enable-debug-verbose", then

$ export XAPIAN_DEBUG_FLAGS=-1
$ export XAPIAN_DEBUG_LOG=test.log
$ sudo xapian-tcpsrv /path/to/db --port 972
Starting server on port 972
Listening...
Connection from 127.0.0.1, port 48102
Connection from 127.0.0.1, port 48358
Got exception NetworkError: Received EOF (context: /path/to/db)
Closing connection.
Connection from 127.0.0.1, port 48614
Got exception NetworkError: Received EOF (context: /path/to/db)
Closing connection.

The client can connect and make queries but I can't find the expected test.log.

I read NEWS and HACKING but they seem to have conflicting information

Thanks,
Matteo

PS: what are those NetworkError exceptions?

Dezi

Peter Karman — 2012-02-24T03:41:31

The recent thread about http://restpose.org/ (which looks like a cool project)
reminded me that I had not yet announced the existence of Dezi[0] to this group.

Dezi is a search server similar to RestPost and Apache Solr that supports
OpenSearch[1] XML and JSON response types. You can index any format of document
supported by SWISH::Filter[3] via a REST API. Clients are available already in
Perl and PHP.

The Xapian backend[2] is available via Search::OpenSearch::Engine::Xapian, which
relies on the Search::Xapian Perl bindings.

Contributors, users, critics all welcome.

cheers,
pek



[0] http://dezi.org/
[1] http://www.opensearch.org/
[2] http://dezi.org/node/4
[3] https://metacpan.org/module/SWISH::Filter

DatabaseModifiedError on get_data - best practice?

Andrew Betts — 2012-02-17T12:54:36

Hi,

I have previously had a problem with getting this error on a get_mset
call, and solved it by subclassing XapianEnquire with a
backoff-and-retry algorithm (as suggested by this list, many thanks!).
 However, I now get it intermittently when calling get_data on a
XapianDocument.  The same solution doesn't seem to be quite as easy in
this case, because:

1. The document is not instantiated by my code, it's returned from the
Iterator, so I can't easily subclass it without editing the bindings.

2. The document doesn't have a reference to the database, so I can't
reopen it from that scope.

So, first is it necessary to reopen the database in these situations,
or could I simply call get_data on the same document object after a
brief delay?  And second, how/where would you suggest I insert the
retry procedure?  Currently I can only see a few options, none of
which seem very good, and the first two don't solve the reopen
problem):

A) Subclass XapianDocument, and in order to make the bindings use the
subclass,