gmane.org.user-groups.marklogic.xqzone.general

[MarkLogic Dev General] Join two different set of docs using cts:search

2013-05-24T06:33:33

Hi,

I have two sets of documents in ML as:

fn:doc()/a. (a is root element)
Sample structure:
<a>
<b>"sample1"</b>
</a>

And fn:doc()/c. (c is root element)
Sample structure:
<c>
<b>"sample2"</b>
</c>

Now I have millions of both these types of docs.
I have to join these two sets of docs on the element b.
Return the matching values.

Currently I am using nested for loop as:

for $doc1 in fn:doc()/a
for $doc2 in fn:doc()/c
where ($doc1/b = $doc2/b)
return 
(
<output>
{
<b>{doc1/b}</b>
}
</output>
)

This takes very long even for a few thousands. I am trying to join them using cts:search. Any help will be appreciated.

Thanks..

Re: [MarkLogic Dev General] Updates creating additional documents

Danny Sokolsky — 2013-05-23T23:24:21

Is there a CPF pipeline attached to the database (or some other triggers)?  If so, that might be what is causing the updates.

In general, batches are more efficient than single updates, maybe batches of 100 or 1000 (or more or less...), depending.  As you said, tho, a failure meant that the whole batch fails.  But that might not be a very big deal, as long as you have logic to deal with that.   Another thing that might work, even with single updates, is to run many threads from different clients, especially if you have enough resources.  If you have a single client, it is likely serializing in a single thread, so ewach transaction might have to wait for the previous one to commit.  Another option is to have a single client spawn a bunch of tasks to do updates.

Again, without knowing what the code is doing it is hard to speculate; 1 per second might be slow or might not, depending on what it is doing.

-Danny

From: general-bounces-ld4jwAGwUXTgXEvjvSGRgMKenhbt+owO< at >public.gmane.org [mailto:general-bounces< at >de

Re: [MarkLogic Dev General] Sorting on aggregation in ML4.2-7

Danny Sokolsky — 2013-05-23T23:14:21

Is there always one <value> per metadata?  If so, you might be able to fragment on metadata, put a range index on value, and then do something like this:

cts:sum-aggregate(cts:element-reference(xs:QName("value"), (),
  cts:element-word-query(xs:QName("country"), (), "usa") )

This should give you the sum of all values in fragments matching that cts:query.

Do you want to sort them by value or by country?  Either way, having a range index on whichever you want to sort on, you should be able to do that using the range index optimization that I mentioned earlier (the flowr with a positional predicate and an order by on a key that has a range index).

-Danny

From: general-bounces-ld4jwAGwUXTgXEvjvSGRgMKenhbt+owO< at >public.gmane.org [mailto:general-bounces< at >developer.marklogic.com] On Behalf Of Abhishek53 S
Sent: Thursday, May 23, 2013 12:36 PM
To: MarkLogic Developer Discussion
Subject: Re: [MarkLogic Dev General] Sorting on aggregation in ML4.2-7
Importance: High

Hi Danny,

You are right, I need sorting based on

Re: [MarkLogic Dev General] Querying attributes

Dak Tapaal — 2013-05-23T22:55:59

Thanks for this..

-----Original Message-----
From: general-bounces-ld4jwAGwUXTgXEvjvSGRgMKenhbt+owO< at >public.gmane.org
[mailto:general-bounces-ld4jwAGwUXTgXEvjvSGRgMKenhbt+owO< at >public.gmane.org] On Behalf Of Erik Hennum
Sent: 23 May 2013 13:28
To: MarkLogic Developer Discussion
Subject: Re: [MarkLogic Dev General] Querying attributes

Hi, Dak:

Your XPath expression is copying the id attribute itself (not its value), so
XQuery is adding the attribute to the div element.

If you want to copy the value of the id attribute as the text of the div
element, you can use an expression like:

       <div>{$college/< at >id/string(.)}</div>

If $college is not empty and $college contains an element with an id
attribute, the XPath expression will get its value.

Hoping that helps,

Erik Hennum

________________________________________
From: general-bounces-ld4jwAGwUXTgXEvjvSGRgMKenhbt+owO< at >public.gmane.org
[general-bounces-ld4jwAGwUXTgXEvjvSGRgMKenhbt+owO< at >public.gmane.org] on behalf of Mailing Lists Mail
[daktapaal-Re5JQEeQ

[MarkLogic Dev General] Updates creating additional documents

Bob O — 2013-05-23T19:57:12

Hello again!

So, I'm doing some investigation to my new ML project and I'm finding out
some weird things happening:

-When updates are done, it creates additional documents (creating rather
than updating).  We would get new documents when we try to update a
particular field that is part of the Data Access Descriptor (DAD) object,
such as the URI that points to the document's product. What would cause
this? I'm thinking some logic on their code.

-The time to ingest documents takes about one per second which seems really
slow to me (average size of document is approximately 15Kb). On my last
project, we would batch 1,000 documents in one file and that seems to work
better for us then. The only drawback is that if one document rejects
during the ingest, the entire batch of 1,000 doesn't get sent. It sends up
to the point when the corrupted document comes up. For example, the 999th
document fails, only 998 get sent through. Is batch processing something we
should consider now?

Any thoughts?

Any suggestions i

Re: [MarkLogic Dev General] Sorting on aggregation in ML4.2-7

Abhishek53 S — 2013-05-23T19:36:07

Hi Danny,
 
You are right, I need sorting based on the "summation of all value items". The perrformance is not acceptable with for loop or cts:search since 1.5 millions of documents required to get sorted. I was planning to keep the summation of value in some seperate field but the issue more complex where I need to dynamically summation the value, which means that sum of all <value/> where <country> is USA.
 
Can any type of document model change help here? 


Thanks
Abhishek Srivastav
Tata Consultancy Services
Cell:- +91-9883389968
Mailto: abhishek53.s-/cgKlWMm3F8< at >public.gmane.org
Website: http://www.tcs.com
____________________________________________
Experience certainty. IT Services
Business Solutions
Consulting
____________________________________________



-----general-bounces-ld4jwAGwUXTgXEvjvSGRgMKenhbt+owO< at >public.gmane.org wrote: -----


To: MarkLogic Developer Discussion <general-ld4jwAGwUXTgXEvjvSGRgMKenhbt+owO< at >public.gmane.org>
From: Danny Sokolsky <Danny.Sokolsky-efBvD/aTHCF8UrSeD/g0lQ< at >public.

Re: [MarkLogic Dev General] Sorting on aggregation in ML4.2-7

Danny Sokolsky — 2013-05-23T18:07:09

Hi Abeshek,

First off, I am not totally sure what you mean by "summation value of one field", but I am guessing you mean to sum up all of the value elements under your content element.  To start, the naive way to do this is to just do a flowr with an order by, something like this:

xquery version "1.0-ml";
let $node := <content>
<id>C1</id>
<metadata>
<country>USA</country>
<value>200</value>
</metadata>
<metadata>
<country>CAN</country>
<value>400</value>
</metadata>
<metadata>
<country>AUS</country>
<value>300</value>
</metadata>
</content>
let $node2 := <content>
<id>C1</id>
<metadata>
<country>USA</country>
<value>200</value>
</metadata>
<metadata>
<country>CAN</country>
<value>400</value>
</metadata>
<metadata>
<country>AUS</country>
<value>500</value>
</metadata>
</content>

for $x in ($node, $node2)
order by fn:sum(fn:data($x//value)) descending
return $x

Now that might give an expanded tree cache error, not sure, depends on how much memory you have.

Now if that is not fast enough (or fills up the

[MarkLogic Dev General] Sorting on aggregation in ML4.2-7

Abhishek53 S — 2013-05-23T17:53:53


Hi All,
 
I need to do some sorting over summation value of one field in MarkLogic 4.2-7. 
 
Below is the way our content is organized and the sorting is required over sum of all value tag over 1.5 million document
 
Please advise!!!
 
<content>
<id>C1</id>
<metadata>
<country>USA</country>
<value>200</value>
</metadata>
<metadata>
<country>CAN</country>
<value>400</value>
</metadata>
<metadata>
<country>AUS</country>
<value>300</value> 
</metadata>
</content>


Abhishek Srivastav
Tata Consultancy Services
Cell:- +91-9883389968
Mailto: abhishek53.s-/cgKlWMm3F8< at >public.gmane.org
Website: http://www.tcs.com
____________________________________________
Experience certainty. IT Services
Business Solutions
Consulting
____________________________________________
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain 
confidential or privileged information. If you are 
not the intended recipient, any dissemination, use, 
review, distribution, printin

Re: [MarkLogic Dev General] Querying attributes

Erik Hennum — 2013-05-23T12:28:21

Hi, Dak:

Your XPath expression is copying the id attribute itself (not its value), so XQuery is adding the attribute to the div element.

If you want to copy the value of the id attribute as the text of the div element, you can use an expression like:

       <div>{$college/< at >id/string(.)}</div>

If $college is not empty and $college contains an element with an id attribute, the XPath expression will get its value.


Hoping that helps,


Erik Hennum

________________________________________
From: general-bounces-ld4jwAGwUXTgXEvjvSGRgMKenhbt+owO< at >public.gmane.org [general-bounces-ld4jwAGwUXTgXEvjvSGRgMKenhbt+owO< at >public.gmane.org] on behalf of Mailing Lists Mail [daktapaal-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org]
Sent: Thursday, May 23, 2013 1:31 AM
To: MarkLogic Developer Discussion
Subject: [MarkLogic Dev General] Querying attributes

Dear All,
I was trying to query the atribute value using Xquery in MarkLogic and
no matter what, I dont seem to be printing the value...
So this is what I do...

 for $college a

Re: [MarkLogic Dev General] Updating ML license(s) progamatically

Bob O — 2013-05-23T11:48:33

ThanKS AGAIN, KEN!

I'LL DOWNLOAD IT.

BOB O.


On Thu, May 23, 2013 at 4:17 AM, Ken Tune <Ken.Tune-efBvD/aTHCF8UrSeD/g0lQ< at >public.gmane.org> wrote:

Re: [MarkLogic Dev General] License Key "screen" locks up

Bob O — 2013-05-23T11:46:34

Thanks, Ken! I missed that on my initial post.

~~Bob O.


On Thu, May 23, 2013 at 3:49 AM, Ken Tune <Ken.Tune-efBvD/aTHCF8UrSeD/g0lQ< at >public.gmane.org> wrote:

Re: [MarkLogic Dev General] Updating ML license(s) progamatically

Ken Tune — 2013-05-23T09:17:12

Bob
If you download this project from git
https://github.com/mustard57/MarkLogicEC2ClusterSetup
git clonehttps://github.com/mustard57/MarkLogicEC2ClusterSetup.git myDir

you will find MarkLogicSetup.py in the for_remote directory.

RE Do I understand that correctly---that this functionality to programatically update MarkLogic licenses --will not be available until version 7?

There is no published API for this. You may be able to make use of /license-go.xqy however.

Ken


From: general-bounces-ld4jwAGwUXTgXEvjvSGRgMKenhbt+owO< at >public.gmane.org [mailto:general-bounces< at >developer.marklogic.com] On Behalf Of Bob O
Sent: 22 May 2013 17:25
To: MarkLogic Developer Discussion
Subject: Re: [MarkLogic Dev General] Updating ML license(s) progamatically

Ken,

Where would I find the MarkLogicSetup.py? Is it in the binaries?

Do I understand that correctly---that this functionality to programatically update MarkLogic licenses --will not be available until version 7?

~~Bob O.
On Tue, May 21, 2013 at 4:28 AM, Ken Tune <Ke

Re: [MarkLogic Dev General] License Key "screen" locks up

Ken Tune — 2013-05-23T08:49:07

Bob

You need this ...

/agree-go.xqy?accepted-agreement=<LICENSE-TYPE>&ok.x=1

which was mentioned a little further down in the response to "Updating ML license(s) progamatically"

The full sequence is

/license-go.xqy?license-key=$LICENSE_KEY&licensee=$LICENSEE
/agree-go.xqy?accepted-agreement=$ACCEPTED_AGREEMENT&ok.x=1
/initialize-go.xqy?auto=true&user=$ADMIN_USER_NAME,&password=$ADMIN_PASSWORD&password2=$ADMIN_PASSWORD&realm=public

RESTART
/security-install-go.xqy

Ken

From: general-bounces-ld4jwAGwUXTgXEvjvSGRgMKenhbt+owO< at >public.gmane.org [mailto:general-bounces< at >developer.marklogic.com] On Behalf Of Bob O
Sent: 22 May 2013 19:53
To: General-ld4jwAGwUXTgXEvjvSGRgMKenhbt+owO< at >public.gmane.org
Subject: [MarkLogic Dev General] License Key "screen" locks up

Hello,

Is there a way to read the EULA License Key Agreement page without having to stop the ML server and awaiting for a restart?

I plugged this in the URL: http://HOST:8001/license-go.xqy?licensekey=<KEY>&licensee=<LICENSEE<http://host:8001/license-

[MarkLogic Dev General] Querying attributes

Mailing Lists Mail — 2013-05-23T08:31:49

Dear All,
I was trying to query the atribute value using Xquery in MarkLogic and
no matter what, I dont seem to be printing the value...
So this is what I do...

 for $college at $position in  $doc//college
   return
        <div>
            {$college/name}
        </div>
       <div>
            {$college/< at >id}
        </div>

This is interesting as the first div is prints the name of the
college... the second is empty..  if I change the xpath to:
 <div>
            {$college/< at >*[1]/name()}
        </div>
 this does print the text "id"  , which is correct.

so I tried

 <div>
            {$college/< at >*[1] }
        </div>

I thought that this should print the value in the attribute id, which
is also the first attribute...

There is probably something very basic, I am missing... So please help!

Thanks
--Dak

Re: [MarkLogic Dev General] CLass Diagram for xQuery files

Stu Salsbury — 2013-05-22T22:25:01

Modules are probably best mapped to components in uml-speak. There isnt
much in the way of a design class paradigm in uml and youre free to think
of your functions as mapping to whatever analysis classes you want since
analysis classes by definition dont lap directly to code.  If you want to
depict an xquery program in uml you'd probably get more bang for your buck
by focusing primarily on the behavioral diagrams (sequence, collaboration,
activity, statechart).

Re: [MarkLogic Dev General] CLass Diagram for xQuery files

David Lee — 2013-05-22T21:42:32

There is some question as to whether diagramming functional programs is useful, or whether a functional program is (ideally) by its nature the simple expression of its own meaning already.
<

Well all MY code is self-documenting :)

Seriously though not sure what an XQuery diagram would look like.
I could imagine every module file being treated as a "class" where every function is a "method"
but without inheritance this would not be all that useful.  Probably about as useful as
   grep function *.xquery

Now for something more like JavaDocs might be more useful

https://github.com/xquery/xquerydoc

This might be of use to you





-----------------------------------------------------------------------------
David Lee
Lead Engineer
MarkLogic Corporation
dlee-efBvD/aTHCF8UrSeD/g0lQ< at >public.gmane.org
Phone: +1 812-482-5224
Cell:  +1 812-630-7622
www.marklogic.com<http://www.marklogic.com/>

Re: [MarkLogic Dev General] CLass Diagram for xQuery files

Stu Salsbury — 2013-05-22T21:11:25

Of course XQuery isn't LISP and it does have non-functional features
(especially in ML), but you may want to search for "functional programming
diagrams"... there is plenty of discussion out there.

There is some question as to whether diagramming functional programs is
useful, or whether a functional program is (ideally) by its nature the
simple expression of its own meaning already.

Re: [MarkLogic Dev General] Reindexing estimation

Danny Sinang — 2013-05-22T20:20:05

Hi Michael,

Does this behavior (of queries using the old indexes while reindexing is
going on) apply :

1. to manually invoked reindexing - not just to reindexing caused by an ML
upgrade ?
2. whether or not "index detection" is set to "automatic" or "none" ?

Regards,
Danny


---------- Forwarded message ----------
From: Michael Blakeley <mike-dWgsRPgt/GtWk0Htik3J/w< at >public.gmane.org>
Date: Tue, May 14, 2013 at 4:19 PM
Subject: Re: [MarkLogic Dev General] Reindexing estimation
To: MarkLogic Developer Discussion <general-ld4jwAGwUXTgXEvjvSGRgMKenhbt+owO< at >public.gmane.org>


The database is designed so that indexes are versioned and
forward-compatible. Thus reindexing is a fully online operation. After a
major upgrade the old indexes are still there. The forests want to build
new ones, hence the reindex. But until that happens, queries use whatever
is available.

This passage from http://docs.marklogic.com/admin-help/database might help:

Server applies to the indexes for this database during request evaluation

Re: [MarkLogic Dev General] CLass Diagram for xQuery files

Eric Bloch — 2013-05-22T20:18:51

Kashif,

It sounds like you're trying to take your experience with other programming languages (Java, C++, others) and map that onto XQuery.

XQuery doesn't have a notion of Class.  XQuery has Library Modules and Main Modules.  I have not yet seen a tool that generated you a diagram of your modules.  But a module is not really like a Class at all.

There is xqdoc, which might help you if the XQuery code you're trying to understand is documented accordingly.

E

Eric Bloch
Director, Community
MarkLogic Corporation

desk +1 650 655 2390 | mobile +1 650 339 0376
email  eric.bloch-efBvD/aTHCF8UrSeD/g0lQ< at >public.gmane.org<mailto:eric.bloch-efBvD/aTHCF8UrSeD/g0lQ< at >public.gmane.org>
web    developer.marklogic.com<http://developer.marklogic.com>
twitter < at >eedeebee

On May 22, 2013, at 8:20 AM, "Khan, Kashif" <Kashif.Khan-Azcf/Un3YmUAvxtiuMwx3w< at >public.gmane.org<mailto:Kashif.Khan-Azcf/Un3YmUAvxtiuMwx3w< at >public.gmane.org>>
 wrote:

Hello Everyone, Has anyone used a tool to create class diagrams for the xQuery files begin

[MarkLogic Dev General] License Key "screen" locks up

Bob O — 2013-05-22T18:52:59

Hello,

Is there a way to read the EULA License Key Agreement page without having
to stop the ML server and awaiting for a restart?

I plugged this in the URL:
http://HOST:8001/license-go.xqy?licensekey=<KEY>&licensee=<LICENSEE<http://host:8001/license-go.xqy?licensekey=<KEY>&licensee=<LICENSEE>
and it hung up MarkLogic to the point that no one can access it at all. I
had to "accept" the license query (accept or decline) in order for it to
come back up. I DID NOT have to go to Configure>Status>Restart button. Is
this normal? Does it uses another license key everytime this happens?

Thanks in advance!

~~Bob O

Re: [MarkLogic Dev General] Updating ML license(s) progamatically

Bob O — 2013-05-22T16:24:31

Ken,

Where would I find the MarkLogicSetup.py? Is it in the binaries?

Do I understand that correctly---that this functionality to programatically
update MarkLogic licenses --will not be available until version 7?

~~Bob O.

On Tue, May 21, 2013 at 4:28 AM, Ken Tune <Ken.Tune-efBvD/aTHCF8UrSeD/g0lQ< at >public.gmane.org> wrote: