gmane.comp.jakarta.lucene.user

RE: IndexSearcher using Collector

Uwe Schindler — 2013-06-19T09:30:21

Hi,

The search function does block. IndexSearcher.search(Query, Collector) returns when all collecting is done. You can do the after-collect work after it returns.

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe< at >thetaphi.de

IndexSearcher using Collector

amit nanda — 2013-06-19T08:59:28

Hi,

I have multiple index that i want to search against, thus i am
using MultiReader for that. Along with this I also want all the matches to
the query so i am using Collector class for this.

The issue i am facing is that I am not able to know when all the matches
are done, i.e. for each matching doc the collect function on the Collector
class will be called but, when all the matches are done how can i come to
know about that. The search function doesn't block.

Is there any way to get this done?

Thanks
Amit

Re: segments and sorting

Adrien Grand — 2013-06-19T08:10:17

Hi,

On Wed, Jun 19, 2013 at 12:16 AM, Sriram Sankar <sankar< at >gmail.com> wrote:

Given that online sorting is rather new to Lucene, we tried to keep it
simple. Merging segments in parallel by maintaining a priority queue
is totally doable and is probably one of the next steps for online
sorting but it would require some non-trivial work to reimplement
merging for all formats (postings lists especially) and to be able to
plug a custom SegmentMerger into the IndexWriter.

For now, we just make sure that sorting a SlowCompositeReaderWrapper
which wraps several sorted segments is faster than sorting a random
AtomicReader by using TimSort to compute the mapping between the old
and the new doc IDs and to sort all individual postings lists.

--
Adrien

Re: how to reterieve all results from lucene searcher.search() method

neha yadav — 2013-06-19T07:51:21

You can try like :
long docSizeQ1= docs.getNumFound() ;
will give you the total number of results.
and this value can be inserted in
like
TopDocs hits = searcher.search(qu,docSizeQ1);

Hope this helps.



On Wed, Jun 19, 2013 at 1:09 PM, Swapnil Patil <ping.swapnil< at >gmail.com>wrote:

Re: how to reterieve all results from lucene searcher.search() method

Swapnil Patil — 2013-06-19T07:39:46

you can get all results for given query  "q". By giving documents to fetch
 a very big number like Integer.MAX_VALUE.

But if u want to loop over all documents. Use  index reader and get
document by its doc id from 0 to numDocs (Index reader has method for num
docs).


On Wed, Jun 19, 2013 at 3:11 PM, neeraj shah <neerajshah84< at >gmail.com> wrote:

how to reterieve all results from lucene searcher.search() method

neeraj shah — 2013-06-19T07:11:19

hello,

Is there any way to get all the search result.
In lucene we get top documents by giving the limit like top 100,1000... etc.
but if i want to get all results.

How can I achieve that??

 Query qu  = new QueryParser(Version.LUCENE_36,"field",
analyzer).parse(query);

 TopDocs hits = searcher.search(qu,1000);

Re: New Lucene User

Ashwin Tandel — 2013-06-18T22:28:47

Raghav,


I would like to second Jack, Solr would take care of indexing your document
without writing any code and it has scalability features like replication
and sharding if required that would handle large volume of data.

http://lucene.apache.org/solr/

Regards,
Ashwin


On Tue, Jun 18, 2013 at 8:38 AM, <raghavendra.k.rao< at >barclays.com> wrote:

Re: segments and sorting

Sriram Sankar — 2013-06-18T22:16:23

Is it possible to do this more efficiently using a merge sort?  Assuming
the individual segments are already sorted, is there a wrapper that I can
use where I can pass the same sorting function?  I'm guessing the
SlowCompositeReaderWrapper does not assume that the individual segments are
already sorted and therefore would repeat the work?

Thanks,

Sriram.



On Sat, Jun 15, 2013 at 1:52 AM, Adrien Grand <jpountz< at >gmail.com> wrote:

Re: TestGrouping.Java seems to combine multiple tests into one huge test

Tom Burton-West — 2013-06-18T18:08:39

Thanks Mike and Robert,

on
:)

TestRandomGrouping, and add some brand new very simple-easy-to-understand
tests to a new file(s).

I opened LUCENE-5065.  I'll try your "easier step. Since the reason I'm
looking at the test is that I'm trying to understand
BlockGroupingCollector, I'll start by trying to just pull out as simple a
test as I can figure out for BlockGroupingCollector and add that.   It may
take me a while, since I need to walk through the code a lot more before I
understand it enough to feel confident that I can pull out an appropriate
test.

Tom



On Tue, Jun 18, 2013 at 1:14 PM, Michael McCandless <
lucene< at >mikemccandless.com> wrote:

Re: TestGrouping.Java seems to combine multiple tests into one huge test

Michael McCandless — 2013-06-18T17:14:04

+1 to somehow refactor this scary test to make it more understandable!

Mike McCandless

http://blog.mikemccandless.com


On Tue, Jun 18, 2013 at 12:48 PM, Tom Burton-West <tburtonw< at >umich.edu> wrote:

Re: TestGrouping.Java seems to combine multiple tests into one huge test

Robert Muir — 2013-06-18T17:13:42


+1, I think it would be great to simplify this. This test seems to be
testing the entire kitchen sink: grouping by function, simulating
distributed grouping, grouping by docvalues, etc.

Refactoring this test would be fantastic, but I wouldn't want to take it on
:)

Maybe an easier step would be to rename this test something like
TestRandomGrouping, and add some brand new very simple-easy-to-understand
tests to a new file(s).

Something that looks more like
http://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/core/src/test/org/apache/lucene/search/TestSort.java:)

TestGrouping.Java seems to combine multiple tests into one huge test

Tom Burton-West — 2013-06-18T16:48:03

Hello,

I'm trying to understand BlockGroupingCollector.   I thought I would start
by running the tests in the debugger.  However the only test I can find is
lucene/grouping/src/test/org/apache/lucene/search/grouping/TestGrouping.java


In TestGrouping.java,  in the second test, "testRandom" it seems like a
fairly large number of things are tested, including a test of
BlockGroupingCollector.

I'm having a hard time understanding the tests and wonder if perhaps
TestGrouping.java might be refactored to make it easier to understand.

Is there some reason that separate tests are not broken out?
I also could not find setup and teardown methods.  Is there some reason
they are not being used?
Would it make sense to open a JIRA issue to refactor  TestGrouping.java to
make a clearer separation between different things being tested?
In particular, perhaps BlockGroupingCollector could have a separate test?

Tom

Looking for Search Engineers

Jagdish Nomula — 2013-06-18T16:17:45

Hello,

SimplyHired.com, a job search engine with the biggest job index in the
world is looking for engineers to help us with our core search and auction
systems.

Some of the problems you will be working on are,
a) Scaling to millions of requests
b) Working with millions of jobs
c) Maximizing the revenue and relevance for search queries(aka
multi-objective maximization problem)
d) Helping people find jobs

If you are interested in helping us out, please send your resume across.

Thanks,

***Jagdish Nomula*
Sr. Manager Search
Simply Hired, Inc.
370 San Aleso Ave., Ste 200
Sunnyvale, CA 94085

office - 408.400.4700
cell - 408.431.2916
email - jagdish< at >simplyhired.com <youremail< at >simplyhired.com>

www.simplyhired.com

[ANNOUNCE] Apache Lucene 4.3.1 released

Shalin Shekhar Mangar — 2013-06-18T15:28:08

June 2013, Apache Lucene™ 4.3.1 available

The Lucene PMC is pleased to announce the release of Apache Lucene 4.3.1

Apache Lucene is a high-performance, full-featured text search engine
library written entirely in Java. It is a technology suitable for
nearly any application that requires full-text search, especially
cross-platform.

The release is available for immediate download at:
http://lucene.apache.org/core/mirrors-core-latest-redir.html

Lucene 4.3.1 includes 12 bug fixes and 1 optimizations, including
fixes for a serious bug that can cause deadlock.

See the CHANGES.txt file included with the release for a full list of
changes and further details.

Please report any feedback to the mailing lists
(http://lucene.apache.org/core/discussion.html)

Note: The Apache Software Foundation uses an extensive mirroring
network for distributing releases. It is possible that the mirror you
are using may not have replicated the release yet. If that is the
case, please try another mirror. This also goes for Maven

RE: New Lucene User

2013-06-18T13:38:32

Heikki,

Thank you very much. I tried it out and the initial results look good.

Although I get "java.lang.OutOfMemoryError: Java heap space" when I search for a single TextField over 70 million records. Probably my code needs tuning.

I'll research more to figure it out. But this is a great start, thanks to everyone who provided suggestions.

Regards,
Raghu


-----Original Message-----
From: heikki [mailto:tropicano< at >gmail.com] 
Sent: Monday, June 17, 2013 5:35 PM
To: java-user< at >lucene.apache.org
Subject: Re: New Lucene User

hi,

I think Lucene is an excellent option for you.

You don't need to export the data to a flat file first. You can just access your database (in whatever way you normally like, e.g. using JDBC or Hibernate). You can do this for example once a day, retrieving only modified records. For each record you retrieve, you create a so-called Lucene Document. You add fields to these documents as you see fit -- for example, you want to search in 20 of your 30 columns, so you

Re: Upgrading from 3.6.1 to 4.3.0 and Custom collector

Peyman Faratin — 2013-06-18T13:04:35

Hi Adrien

thank you very much. It worked. 

have a good day

On Jun 18, 2013, at 5:35 AM, Adrien Grand <jpountz< at >gmail.com> wrote:

Programmatically create proximity or slop queries using Lucene's flexible parser

Kasper van den Berg — 2013-06-18T09:44:47

Hi all,

(Cross posted to stackoverflow yesterday (http://stackoverflow.com/q/17154510/814206), no answers at SO yet, perhaps java-users< at >lucene is a better place for this question; I hope not to annoy any of you with these duplicate messages)

In Lucene (currently using version 4.1) using Lucene's flexible query parser , is it possible to build(…) proximity queries from ProximityQueryNode?

The code uses a flexible parser in order to parse (partial) queries read from a file into QueryNode; process these QueryNodes; and finally search using the built queries.

The three phases of the flexible parser:
* Parsing, does what I want it to do.
  The StandardSyntaxParser parses even complex queries with proximity operators (for example "(\"letter comes\"~3 \"agitated employees\"~4 \"worried about salaries\"~1) AND (\"letter comes agitated employees worried about salaries\"~8)" parses correctly (to a group containing SlopQueryNode; having slop queries is adequate).

* Processing, removes proximity and slop nodes fr

Re: Upgrading from 3.6.1 to 4.3.0 and Custom collector

Adrien Grand — 2013-06-18T09:35:17

Hi,

You didn't say specifically what your problem is so I assume it is
with the following method:

On Tue, Jun 18, 2013 at 4:37 AM, Peyman Faratin <peymanfaratin< at >gmail.com> wrote:

setNextReader now takes an AtomicReaderContext as an argument and
FieldCache.getStrings is now FieldCache.getTerms, so this would give
something like

private BinaryDocValues store;

public void setNextReader(AtomicReaderContext ctx) throws IOException{
    this.docBase = ctx.docBase;
    this.store = FieldCache.DEFAULT.getTerms(ctx.reader(), "title");
}

public void collect(int doc) throws IOException {
    BytesRef page = new BytesRef();
    store.get(doc, page);
    if (page.bytes != BinaryDocValues.MISSING) {
        outLinks.add(page.utf8ToString());
    }
}

--
Adrien

Re: segments and sorting

Adrien Grand — 2013-06-18T09:25:59

Doc values don't have this limit anymore. However, there is a limit of
~32kb per term, but this shouldn't be a problem with reasonable
use-cases for doc values.

These slides are talking about the pre-4.0 API, and the doc values API
has been completely refactored in 4.2. Although the concepts are the
same, it may be non-trivial to translate the code examples to the new
API.

--
Adrien

Upgrading from 3.6.1 to 4.3.0 and Custom collector

Peyman Faratin — 2013-06-18T02:37:58

Hi 

I am migrating from Lucene 3.6.1 to 4.3.0. I am however not sure how to migrate my custom collector below to 4.3.0 (this page http://lucene.apache.org/core/4_3_0/MIGRATE.html gives some hints but is the instructions are incomplete and looking at the source examples of custom collectors make me dizzy!!!) 

Any advise would be very much appreciated 

thank you


public class AllInLinks extends Collector {
  private Scorer scorer;
  private int docBase;
  private String[] store;
  private HashSet<String> outLinks = new HashSet<String>();

  public boolean acceptsDocsOutOfOrder() {
    return true;
  }
  public void setScorer(Scorer scorer) {
    this.scorer = scorer;
  }
  public void setNextReader(IndexReader reader, int docBase) throws IOException{
this.docBase = docBase;
store = FieldCache.DEFAULT.getStrings(reader,"title");
  }
  public void collect(int doc) throws IOException {
  String page = store[doc];    
  outLinks.add(page);
  }
  public void reset() {

Re: segments and sorting

Sriram Sankar — 2013-06-17T23:05:08

I'm sorry - I meant "DocValue" not "FieldValue".  Slide 20 in the following
deck talks about the 2Gb limit.

Sriram.

http://www.slideshare.net/lucenerevolution/willnauer-simon-doc-values-column-stride-fields-in-lucene


On Sat, Jun 15, 2013 at 1:52 AM, Adrien Grand <jpountz< at >gmail.com> wrote: