gmane.comp.search.xapian.general

Re: [q] phrase replacement in thousandsoftext?files

Olly Betts — 2012-05-22T05:48:58

No, Xapian only stores word offsets.


No.

Cheers,
    Olly

Re: [q] phrase replacement in thousands of textfiles

V S P — 2012-05-22T00:40:00

Thanks Olly
can Xapian, at least return the line number where the matches where
found (the text files are a few, but big)
also is there a way to 'batch up' a search to send multiple queries in
one shot ?

I did not quite understand the the 'pass each chunk of text when
indexing' -- but I could probably figure this out as long the above
functionality is present in some form

thank you again
vlad

On Mon, May 21, 2012, at 06:46 AM, Olly Betts wrote:

Re: [q] phrase replacement in thousands oftextfiles

Olly Betts — 2012-05-21T05:46:18

I'm not sure I see why this a problem unless the run time of this
is highly sensitive.


Xapian doesn't store the byte offsets (only word offsets), so this isn't
possible.  It can narrow down the number of files you need to go and
look at for each replacement though, which could make quite a difference
if many of the replacements are rarely done.


When indexing, pass each chunk of text between periods to
TermGenerator::index_text(), calling increase_termpos() after each
index_text() call.  Then phrases can't span a period.

Cheers,
    Olly

Re: Building php5-xapian on Ubuntu 12.04 Serverx64

Olly Betts — 2012-05-21T03:27:00

Ah, looks like this probably got broken by a backported patch, but works
in a VPATH build from SVN (as the generated sources are then in
builddir) so I didn't notice.


Yes, that should work with suitably recent dpkg.

A slightly simpler option instead of "dpkg-source --commit" is to do:

echo auto-commit >> debian/source/options

That doesn't require the user to pick a patch name, and with older
dpkg where "dpkg-source --commit" isn't supported, the options file
will be ignored, and the patch should just get included anyway.
dpkg should include the change with a generic name automatically.

I'll update the FAQ with that.

It's probably worth noting here that smoketest.php currently fails
with a segmentation fault under PHP 5.4.  If you don't try to subclass
Xapian classes in PHP, everything seems fine, but there's an issue
with the director code SWIG generates.  I've investigated a bit, but
it isn't clear to me whether it's PHP or SWIG at fault.

Cheers,
    Olly

[q] phrase replacement in thousands of text files

V S P — 2012-05-21T02:59:11

Hello, first post.
I searched through docs and examples but did not see this particular
problem answered.

I have thousands of text files total size about 17GB.
Within a file I need for find a phrase (typically up to 3 words together
separated by spaces, commas, and non period punctuation mark).

I have a dictionary of about 3 million phrases and their replacement.

So I need to replace all of the matching phrases from the dictionary
with their replacements

The most brute force approach I though was
a) build an index on all of the 17GB of documents
b) for every one of the 3 million search phrases do search
c) expect to return from ( b ) xapian match where I would get the start
and end byte location in a file for every search
remember that location, and the found phrase in a 'future replacement
list'

d) when done , use the 'future replacement list' -- to perform the
replacement operation


Obviously couple of problems 3 million times search 17GB worth of text
Second -- I do not understand how (if at all possible) to get the
start/end offset of the found phrase within the source file
Third  how do I insure that the phrase words are together (and the one
with period between them is not concidered a find).


thank you in advance for any suggestions,
vsp

Re: Building php5-xapian on Ubuntu 12.04 Serverx64

James Aylett — 2012-05-20T13:12:00



This doesn't seem to work at the moment (checked against Ubuntu 12.04 here), because the PHP smoketest won't run in a VPATH build (checked using the 1.2.10 bindings without the debian patches or build system). I think in php/Makefile.am the TESTS_ENVIRONMENT needs `$(srcdir)` in the `-d include_path=php5` stanza.

As a short-term fix, if you do the following before the `debuild` line, it seems to build okay:

sed -i 's/include_path=php5$/include_path=$(srcdir)\/php5/' php/Makefile.in
dpkg-source --commit

(give it a patch name of something like fix-php-buildtests, then just save the file it prompts you to edit)

J

Re: Building php5-xapian on Ubuntu 12.04 Serverx64

Olly Betts — 2012-05-20T11:56:09

I would recommend using the instructions in the FAQ, which are based on
those in the first URL, but have a couple of fixes:

http://trac.xapian.org/wiki/FAQ/PHP%20Bindings%20Package

Cheers,
    Olly

Building php5-xapian on Ubuntu 12.04 Server x64

2012-05-20T11:27:53

As Xapian's been dropped from the Ubuntu repositories, I'm attempting to
build my own .deb from the instructions here:

 

http://article.gmane.org/gmane.comp.search.xapian.general/8855

 

http://beeznest.wordpress.com/2011/07/06/howto-build-your-own-binaries-of-ph
p-xapian-bindings-for-debian/

 

I can only get things to progress beyond the first few seconds by leaving
out 'rm debian/control', but if I do, it looks as if the Python and Ruby
bindings are building and passing their versions of smoketest correctly.

 

However, the PHP part of the build is failing with this error:

 

    /home/charlie/xapian-bindings-1.2.8/php/smoketest.php:38:
include(xapian.php): failed to open stream: No such file or directory

    FAIL: smoketest.php

 

There's a xapian.php file in /home/charlie/xapian-bindings-1.2.8/php/php5/
but if I copy it to /home/charlie/xapian-bindings-1.2.8/php/ or change the
path to it in smoketest.php, the build fails right near the start with:

 

    dpkg-source: error: aborting due to unexpected upstream changes

 

Unfortunately I'm out of my comfort zone building from source. Anyone got
any ideas?

Re: php bindings won't build with MAMP

Olly Betts — 2012-05-18T14:23:44

You probably have a system install of xapian-core 1.2.8, which is
getting picked up from /usr/lib in preference to the 1.2.10 you
installed (which would be in /usr/local/lib if you didn't specify a
different prefix).

I'd guess you're running Ubuntu 12.04, as that has xapian-core 1.2.8
and it gets installed by default on Ubuntu.

Cheers,
    Olly

Re: php bindings won't build with MAMP

Mark Grandi — 2012-05-18T13:07:58

I'll mess with this later, For now I just compiled it on my linux box and
it all went smoothly. However, in phpinfo, it says that the xapian compiled
version is 1.2.10, but the linked version is 1.2.8...I used both the latest
versions of the bindings/core from teh website (1.2.10), so why the
inconsistency in phpinfo?

On Thu, May 17, 2012 at 5:20 PM, Olly Betts <olly< at >survex.com> wrote:

Re: php bindings won't build with MAMP

Olly Betts — 2012-05-18T00:20:49

I know nothing about MAMP, but I find it hard to believe you're expected
to download the headers and install them by hand.  A quick web search
suggests they are actually in /Applications/MAMP/Library/include - if
that's true, then MAMP's packaging is buggy - php-config --includes
should point there.

I suspect that when MAMP make their packages, they build and install
PHP, then move some of the resulting files around without fixing up
references to their locations.  You should file a bug with MAMP about
this.

My guess as to why installing the headers by hand didn't work is that
you've not copied the headers over correctly.  If you look in config.log
you can see what happened when configure tried to compile a test file
including zend.h.


That doesn't look like a reliable guide.  It says "make sure you
download and use the latest stable Xapian version", yet the examples use
a *development* version which was over a year old when the post was
written, and had been superseded by a stable release nearly a year
before.

You really should NOT copy files out of .libs either - tell configure
where you want to install them (use PHP_EXTENSION_DIR as INSTALL
explains) and then use "make install".  The extension should by default
be installed to somewhere which works, so I suspect this is another
instance of MAMP's php-config reporting bad information.

Cheers,
    Olly

Re: php bindings won't build with MAMP

Peter Van Dijk — 2012-05-18T00:15:14


Let me start by saying that this is not a solution: i had (the misfortune
of trying) to configure apache/php on a mac the other day and i found it
considerably easier to use the apache/php installation that comes with OSX
than use MAMP.
Any time i did anything remotely complicated with MAMP i'd get random
segfaults and weird stuff.

Additionally, the built-in php is (i think) 5.3.9 or newer, which is a fair
bit newer than the MAMP version.

Perhaps a 'Mac Person' can provide a better answer?

php bindings won't build with MAMP

Mark Grandi — 2012-05-17T23:29:09

Hello, I'm trying to build the xapian bindings to use with php, i've
installed using MAMP to make it easier to set up the apache/php/myssql
stuff (http://www.mamp.info/en/mamp/index.html), but when I try to
configure the bindings, i get this:

configure: error: '/usr/bin/php-config --includes' reports
'-I/Applications/MAMP/bin/php/php5.3.6/include/php
-I/Applications/MAMP/bin/php/php5.3.6/include/php/main
-I/Applications/MAMP/bin/php/php5.3.6/include/php/TSRM
-I/Applications/MAMP/bin/php/php5.3.6/include/php/Zend
-I/Applications/MAMP/bin/php/php5.3.6/include/php/ext
-I/Applications/MAMP/bin/php/php5.3.6/include/php/ext/date/lib' but zend.h
can't be included from there


I made it so its using the php-configure from MAMP, so its reporting the
correct paths, those paths did not exist, but I downloaded the header files
for the correct php version and stuck them in there, but its still claiming
it can't find them. I've tried a lot of things and nothing work =(

I read this blog post:
http://blog.geertvd.be/2011/04/15/using-xapian-on-os-x-with-mamp/ but he
doesn't list a solution, other then just use php 5.2 (which i really don't
want to do since i'm using some features that are only in 5.3)

Any ideas? ~Mark

Re: Xapian 1.2.10 released

Jim Lynch — 2012-05-17T20:22:59

That error message is most likely an Apache error message telling you 
that it didn't like the format of the data coming back from some web 
application, probably omega if I can read between the lines.  Look in 
the Apache error log for a specific error.  The log may be in 
/var/log/httpd or var/log/apache2 depending on your distro.

Also I think omega has to have the execute bit set for it to run as CGI, 
so check to see that omega is chmod 755 or 555.

Jim.

Re: Xapian 1.2.10 released

Marc Fromm — 2012-05-17T16:26:18

I updated xapian as shown in the attached text file. I then updated the omega database. When I do a search I get this message:

Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.

I have no idea what has happened. Did I miss a step in the update procedure?

Thanks


-----Original Message-----
From: Olly Betts [mailto:olly< at >survex.com] 
Sent: Tuesday, May 15, 2012 6:20 PM
To: Marc Fromm
Cc: xapian-discuss< at >lists.xapian.org
Subject: Re: [Xapian-discuss] Xapian 1.2.10 released

On Tue, May 15, 2012 at 11:08:34PM +0000, Marc Fromm wrote:

The most recent packages for RHEL 5 seem to be of Xapian 1.2.5.

For RHEL 6, 1.2.9 looks to be available at rpm.eprints.org, I guess
1.2.10 should appear in the near future, but it depends how busy Tim is.
You can probably rebuild from the source RPM if you want a newer 1.2.x for RHEL 5.


I suspect you're using the old repo at xapian.org, which has 1.0.4.

You want the repo at rpm.eprints.org instead - follow the instructions
at:

http://xapian.org/download#RPM

Cheers,
    Olly

Step1:
[root< at >finaid45 tmp]$ rpm -ivh http://rpm.eprints.org/rpm-eprints-org-key-1-1.noarch.rpm
Retrieving http://rpm.eprints.org/rpm-eprints-org-key-1-1.noarch.rpm
warning: /var/tmp/rpm-xfer.6J08Hb: Header V3 RSA/SHA1 signature: NOKEY, key ID c093f9be
Preparing...                ########################################### [100%]
   1:rpm-eprints-org-key    ########################################### [100%]

Step2:
[root< at >finaid45 tmp]$ rpm -ivh http://rpm.eprints.org/xapian/5/noarch/rpm-eprints-org-xapian-5-1.noarch.rpm
Retrieving http://rpm.eprints.org/xapian/5/noarch/rpm-eprints-org-xapian-5-1.noarch.rpm
warning: /var/tmp/rpm-xfer.tgy9ko: Header V3 RSA/SHA1 signature: NOKEY, key ID c093f9be
Preparing...                ########################################### [100%]
   1:rpm-eprints-org-xapian ########################################### [100%]

Step3:
[root< at >finaid45 tmp]$ yum update
Loaded plugins: rhnplugin, security
rpm-eprints-org-xapian                                                                                                                |  951 B     00:00     
rpm-eprints-org-xapian/primary                                                                                                        |  14 kB     00:00     
rpm-eprints-org-xapian                                                                                                                                 88/88
Skipping security plugin, no data
Setting up Update Process
Resolving Dependencies
Skipping security plugin, no data
--> Running transaction check
---> Package xapian-bindings.i386 0:1.2.5-1 set to be updated
---> Package xapian-bindings-php.i386 0:1.2.5-1 set to be updated
---> Package xapian-core-libs.i386 0:1.2.5-1 set to be updated
---> Package xapian-omega.i386 0:1.2.5-1 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================
 Package                                    Arch                        Version                          Repository                                     Size
=============================================================================================================================================================
Updating:
 xapian-bindings                            i386                        1.2.5-1                          rpm-eprints-org-xapian                        111 k
 xapian-bindings-php                        i386                        1.2.5-1                          rpm-eprints-org-xapian                        136 k
 xapian-core-libs                           i386                        1.2.5-1                          rpm-eprints-org-xapian                        929 k
 xapian-omega                               i386                        1.2.5-1                          rpm-eprints-org-xapian                        333 k

Transaction Summary
=============================================================================================================================================================
Install       0 Package(s)
Upgrade       4 Package(s)

Total download size: 1.5 M
Is this ok [y/N]: y
Downloading Packages:
(1/4): xapian-bindings-1.2.5-1.i386.rpm                                                                                               | 111 kB     00:00     
(2/4): xapian-bindings-php-1.2.5-1.i386.rpm                                                                                           | 136 kB     00:00     
(3/4): xapian-omega-1.2.5-1.i386.rpm                                                                                                  | 333 kB     00:00     
(4/4): xapian-core-libs-1.2.5-1.i386.rpm                                                                                              | 929 kB     00:00     
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                        479 kB/s | 1.5 MB     00:03     
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA1 signature: NOKEY, key ID c093f9be
rpm-eprints-org-xapian/gpgkey                                                                                                         |  979 B     00:00     
Importing GPG key 0xC093F9BE "Tim Brody (rpm.eprints.org) <rpm< at >eprints.org>" from /etc/pki/rpm-gpg/RPM-GPG-KEY-rpm-eprints-org
Is this ok [y/N]: y
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating       : xapian-core-libs                                                                                                                      1/8 
  Updating       : xapian-omega     
  Updating       : xapian-bindings                                                                                                                       3/8 
  Updating       : xapian-bindings-php                                                                                                                   4/8 
  Cleanup        : xapian-bindings                                                                                                                       5/8 
  Cleanup        : xapian-bindings-php                                                                                                                   6/8 
  Cleanup        : xapian-core-libs                                                                                                                      7/8 
  Cleanup        : xapian-omega                                                                                                                          8/8 

Updated:
  xapian-bindings.i386 0:1.2.5-1        xapian-bindings-php.i386 0:1.2.5-1        xapian-core-libs.i386 0:1.2.5-1        xapian-omega.i386 0:1.2.5-1       

Complete!

Step4:
omindex --db /var/lib/omega/data/default --url /lan/files /var/www/html/lan/files/ _______________________________________________
Xapian-discuss mailing list
Xapian-discuss< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-discuss

Re: Xapian 1.2.10 released

Tim Brody — 2012-05-16T11:26:12

rpm.eprints.org has been updated to 1.2.10.

/Tim.
_______________________________________________
Xapian-discuss mailing list
Xapian-discuss< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-discuss

Re: Xapian 1.2.10 released

Olly Betts — 2012-05-16T01:20:13

The most recent packages for RHEL 5 seem to be of Xapian 1.2.5.

For RHEL 6, 1.2.9 looks to be available at rpm.eprints.org, I guess
1.2.10 should appear in the near future, but it depends how busy Tim is.
You can probably rebuild from the source RPM if you want a newer 1.2.x
for RHEL 5.


I suspect you're using the old repo at xapian.org, which has 1.0.4.

You want the repo at rpm.eprints.org instead - follow the instructions
at:

http://xapian.org/download#RPM

Cheers,
    Olly

Re: Xapian 1.2.10 released

Marc Fromm — 2012-05-15T23:08:34

I am running the version below, on a RHEL 5 system.
xapian-omega-1.0.4-1
xapian-core-libs-1.0.4-1
xapian-bindings-php-1.0.4-1
xapian-bindings-1.0.4-1

How can I update to version 1.2.10? 

If I run yum update xapian, nothing is found, even though I have xapian.repo in my yum.repos.d directory.
No Match for argument: xapian
No package xapian available.
No Packages marked for Update

Thanks

-----Original Message-----
From: xapian-discuss-bounces< at >lists.xapian.org [mailto:xapian-discuss-bounces< at >lists.xapian.org] On Behalf Of Olly Betts
Sent: Wednesday, May 09, 2012 6:32 PM
To: xapian-discuss< at >lists.xapian.org
Subject: [Xapian-discuss] Xapian 1.2.10 released

I've uploaded Xapian 1.2.10 (including Search::Xapian 1.2.10.0).

Unsurprisingly, you can download it from:

http://xapian.org/download

You can read an overview of the release here:

http://trac.xapian.org/wiki/ReleaseOverview/1.2.10

The full lists of user-visible changes are linked to from there, and also from the "[news]" links on the download page.

As always, if you encounter problems, please report them here, or to the bug-tracker: http://xapian.org/bugs

Here are the SHA1 checksums of the released files:

121e0ab529a3e5a70931f61463a56e944805ae10  Search-Xapian-1.2.10.0.tar.gz
631c7650ac2ca47f8a72434a06d463da5b7596f4  xapian-bindings-1.2.10.tar.gz 1be1896ab11a3a66c6c0ade962c700d96678116e  xapian-core-1.2.10.tar.gz 59fca635728d5ce2ca18e5f39b8d3ef081e46f6c  xapian-omega-1.2.10.tar.gz

And SHA256 checksums (more secure, but sha256sum is less widely installed):

9695b45b978ef41b367fc5ea2bcf5ecbc2b4c5aff6d138d7dbc31946fdb86bc7  Search-Xapian-1.2.10.0.tar.gz
dd191ad2e1134b7008b3b71c7cab41d2d80df2fe82ecdb9ad68c11209ba19382  xapian-bindings-1.2.10.tar.gz 05cf0c374cdda2b0a655f4d8897b37729a0d0d6887fd12bf86c778b486ef874f  xapian-core-1.2.10.tar.gz 464a681647c1348fe7ab7ab5d8e32c27dc0d48a863ce5d78881e18334dfaad4f  xapian-omega-1.2.10.tar.gz

Cheers,
    Olly

Xapian 1.2.10 released

Olly Betts — 2012-05-10T01:31:34

I've uploaded Xapian 1.2.10 (including Search::Xapian 1.2.10.0).

Unsurprisingly, you can download it from:

http://xapian.org/download

You can read an overview of the release here:

http://trac.xapian.org/wiki/ReleaseOverview/1.2.10

The full lists of user-visible changes are linked to from there, and also
from the "[news]" links on the download page.

As always, if you encounter problems, please report them here, or to
the bug-tracker: http://xapian.org/bugs

Here are the SHA1 checksums of the released files:

121e0ab529a3e5a70931f61463a56e944805ae10  Search-Xapian-1.2.10.0.tar.gz
631c7650ac2ca47f8a72434a06d463da5b7596f4  xapian-bindings-1.2.10.tar.gz
1be1896ab11a3a66c6c0ade962c700d96678116e  xapian-core-1.2.10.tar.gz
59fca635728d5ce2ca18e5f39b8d3ef081e46f6c  xapian-omega-1.2.10.tar.gz

And SHA256 checksums (more secure, but sha256sum is less widely installed):

9695b45b978ef41b367fc5ea2bcf5ecbc2b4c5aff6d138d7dbc31946fdb86bc7  Search-Xapian-1.2.10.0.tar.gz
dd191ad2e1134b7008b3b71c7cab41d2d80df2fe82ecdb9ad68c11209ba19382  xapian-bindings-1.2.10.tar.gz
05cf0c374cdda2b0a655f4d8897b37729a0d0d6887fd12bf86c778b486ef874f  xapian-core-1.2.10.tar.gz
464a681647c1348fe7ab7ab5d8e32c27dc0d48a863ce5d78881e18334dfaad4f  xapian-omega-1.2.10.tar.gz

Cheers,
    Olly
_______________________________________________
Xapian-discuss mailing list
Xapian-discuss< at >lists.xapian.org
http://lists.xapian.org/mailman/listinfo/xapian-discuss

Re: Incorrect line nums displayed in fulltextsearch

Olly Betts — 2012-05-03T03:24:11

[...]

Xapian itself has no concept of "line numbers", so I think you'll need
to ask the LXRng developers.

Cheers,
    Olly

Incorrect line nums displayed in full text search

Babu Neelam — 2012-05-03T02:34:04

Hi,

Could you please clarify a query regarding full text search in Xapain?

I have installed LXRng (cross-referencer), which internally uses Xapain for
text search. I am using CentOS 6 & have installed the following Xapian
packages:

$ yum list installed | grep -i xapian
perl-Search-Xapian.x86_64
                         1.2.9.0-1.el6  < at >rpm-eprints-org-xapian
perl-Search-Xapian-debuginfo.i686
                         1.2.7.0-1.el6  < at >rpm-eprints-org-xapian
perl-Search-Xapian-debuginfo.x86_64
                         1.2.7.0-1.el6  < at >rpm-eprints-org-xapian
rpm-eprints-org-xapian.noarch
xapian-bindings.x86_64   1.2.9-1        < at >rpm-eprints-org-xapian
xapian-bindings-perl.x86_64
                         1.2.9-1        < at >rpm-eprints-org-xapian
xapian-bindings-php.x86_64
                         1.2.9-1        < at >rpm-eprints-org-xapian
xapian-core.x86_64       1.2.9-1        < at >rpm-eprints-org-xapian
xapian-core-debuginfo.x86_64
                         1.2.7-1        < at >rpm-eprints-org-xapian
xapian-core-devel.x86_64 1.2.9-1        < at >rpm-eprints-org-xapian
xapian-core-libs.x86_64  1.2.9-1        < at >rpm-eprints-org-xapian
xapian-omega.x86_64      1.2.9-1        < at >rpm-eprints-org-xapian
xapian-omega-debuginfo.x86_64
                         1.2.7-1        < at >rpm-eprints-org-xapian
$


However, when I search for more than one word with double quotes, displayed
line numbers are incorrect (but, displayed file names are correct).

Please help me how I go about solving this issue.


Thanks,
Babu