gmane.comp.freedesktop.tracker

Re: Semantic Desktop for Gnome

Martyn Russell — 2013-05-10T12:39:30

Hello Tom.


This is really a question which divides the team. Email is the classic 
example, do we/could we store emails in the database?

Sure we could. But some on the team feel Tracker is more about storing 
the metadata and location of the information than the information 
itself. This saves the database from being so bloated too.

Some data types are exempt as Ivan says. Contacts and SMS/MMS content on 
the Nokia devices were saved in the database (for example).


This divides the community too. There are 4 approaches here:

1. Many of the application developers prefer to NOT have an indexer 
running in the background but to call an API which indexes their files 
on demand. This depends on your content of course. Maybe you don't want 
the file holding the data indexed too like with your tasks?

2. The miner-fs runs for all other cases, e.g. indexing PDFs, music, 
etc. left in your common XDG directories.

3. You can pick up your data with a specific miner as you say. This 
really means another process

Re: Semantic Desktop for Gnome

Ivan Frade — 2013-05-08T00:51:42

Hi Anatoly,


On Tue, May 7, 2013 at 11:56 AM, אנטולי קרסנר <tombackton-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:


 There is no specific ontology for tasks, although "To do" items are part
of the calendar ontology (scal).

https://developer.gnome.org/ontology/unstable/scal-ontology.html#scal-Todo



 Best option is to take a look to the scal shipped with Tracker and
contribute some improvements to it. We can review and help.

 If the changes are big enough, you could also start a new ontology for
Tasks. Read carefully NIE and NAO (at least) because much of what you need
is there. You just need to create the right subclasses (and some extra
properties).

 The ontologies are in the turtle format, and all the information for
applications comes and goes in SparQL. Both are easy to learn (although
Sparql queries can grow quickly). There is no XML involved.



Sure. If scal is not good, you could translate your XSD into an ontology.
Take a look to any of our ontologies for inspiration:

 https://

Re: Semantic Desktop for Gnome

Philip Van Hoof — 2013-05-07T20:24:01



[cut excellent reasons]

Add this reason: Because Tracker has, with tracker-writeback, a service
to write such metadata to files.

Kind regards,

Philip

Re: Semantic Desktop for Gnome

אנטולי קרסנר — 2013-05-07T18:56:55

Hi Ivan,

Thanks for the answers. I'll write directly to the Tracker database,
since tasks are mostly semantically-useful data, and contain simple
plain text (even header/footer/comment are marked in WYSIWYM style).

I have a new question :)

In the process of examining existing task-management apps, I discovered
that - not surprising at all - they have lots of things in common. The
arrangement of the data may vary, but the details - start date, due
date, tags, priority, assigned people, related mail/notes/source
code/references/bibliography, it's common to many apps.

So what I'm asking is: Is there an ontology for tasks? I didn't fully
dive into RDF and Tracker ontologies yet, but I saw ontologies for music
tags, for mail, for files, for contacts, for documents, etc.

I didn't see a special ontology for tasks on the list. Does it mean I
need to write my own XML/RDF format for semantic storage of tasks?

I already store tasks in XML with clear rules (XSD schema is
work-in-progress), so it shouldn't be diffi

Re: Semantic Desktop for Gnome

Ivan Frade — 2013-05-07T17:48:06

Hi Anatoly,

 Good questions!


On Tue, May 7, 2013 at 1:08 AM, אנטולי קרסנר <tombackton-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:


Information that is only metadata (e.g. Contacts, a simple note) can be
stored directly in Tracker. No need to use a file. A file representation
could be useful to share the information (like a vcard) but you can even
generate it on demand.



Every resource in Tracker is an InformationElement and has a
nie:plainTextContent (IIRC) that you can use to store those small texts.

Those texts are very useful for plain text search. Then you can find your
objects in tracker-needle and you can use it as clause in your structured
queries: "All notes with its title sorted by modification date with "todo"
in the content".




We prefer the applications writing directly into tracker. Some reasons:
  * It saves the indexing roundtrip: Using files as intermediate format
means that the app writes the file, miner-fs discover it, send it to a
extractor which reads the file, par

Re: Semantic Desktop for Gnome

אנטולי קרסנר — 2013-05-07T08:08:34

Great news! And thanks for the feedback :)

I went through the various Tracker APIs visible in Devhelp, and I have a
few design questions, which you may be able to help me with.


1. Data storage:
Some file formats, such as binary data of all kinds (music, executables,
video, etc.), are stored as regular files, and only the metadata is
stored in Tracker's database. But many file formats are just text. For
example, my task management app represents the data as a graph of tasks,
and 100% of the data has semantic value. Same thing with notes and
probably many other file formats which describe data in simple XML.

So the question is: Is all the data supposed to be stored in the Tracker
database and fecthed from it every time, or I still should store the XML
data as local files, and "report" semantic data to the Tracker database?

Clearly, my data *does* have some plain text, such as the task
descriptions. I guess it's not valuable to Semantic Desktop, but still,
I'm wondering how to store the data. Another idea

Re: Semantic Desktop for Gnome

Ivan Frade — 2013-05-06T22:51:13

Hi Anatoly,

 Glad to hear about your plans. Tracker was build with use cases like those
in mind.

On Mon, May 6, 2013 at 2:22 PM, אנטולי קרסנר <tombackton-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:



Yes, it does. It is Tracker :)



You won't need to augment it much. Tracker as it is solves most of your
storage problems. It is a shared storage accessible from any application
(in the user session) and the information is represented as a graph. You
can link resources coming from different sources and so on.

Maybe you need to tune the ontologies, but that shouldn't be dramatic.



Tracker was created as backend for applications like you were describing,
but for multiple reasons we never wrote the actual "semantic apps". It is
great if you can use it and show the potential of linking information.

We are glad to help and receive feedback. Do not hesitate on report bugs
and discuss in this mailing list or in the IRC channel.

Best regards,

Ivan

Semantic Desktop for Gnome

אנטולי קרסנר — 2013-05-06T21:22:25

Hello Tracker team,

I'd like to discuss an issue with you, related to Semantic Desktop.

I'm working on a desktop app, basically a task/project manager, with
semantic desktop capabilities. The tasks can be marked with tags, and
tags are created in a tag-graph and have semantic meaning. For example,
a tag holding the name of a software package allows click-to-install,
company names may allow go-to-website by clicking.

And there are many other awesome examples, like automating the creation
of shopping lists using recipes you choose as tasks.

The issue is, this technology does its best when all apps integrate.
Clearly there should be a desktop-wide system for semantic data storage,
but Gnome doesn't have one yet. So I decided to start planning, using
the requirements and use cases I have.

Instead of re-inventing the wheel, I looked for existing similar
technology. And I found you.

Do you think I can use a possibly augmented tracker-based data store
as a general purpose desktop-wide data storage interface,

Re: tracker store timeout

Carlos Garnacho — 2013-04-19T10:37:06

Hi,

On vie, 2013-04-19 at 12:14 +0300, MrJojohn1987 . wrote:



This still fits the symptoms that Ivan suspects about, usually readonly
queries are run on a direct connection to the database (tracker-store is
not involved), and updates/deletes are run through the dbus connection,
and handled centrally by tracker-store.

One thing worth checking is whether the tracker-store PID stays the same
while tracker-miner-fs is doing its business. If not, you could perhaps
attach to the tracker-store process with gdb and get a backtrace of it
crashing

  $ gdb /usr/libexec/tracker-store `pidof tracker-store`

Re: tracker store timeout

MrJojohn1987 . — 2013-04-19T09:14:58

Hello ,


The timeout seem to be when doing tracker_sparql_update_async.

Found this on google search
https://mail.gnome.org/archives/commits-list/2010-September/msg12239.html




On Sun, Apr 14, 2013 at 7:37 AM, MrJojohn1987 . <putineiionut-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org>wrote:

Re: make error on latest

Martyn Russell — 2013-04-18T06:18:27

It's highly suspicious that you're installing GLib in this way (i.e. 
hacking about removing some files, etc. It also looks like with 
configure you're installing to /usr/local as a prefix and so there would 
be concurrent GLib versions installed. I had to use jhbuild to get this 
working properly. I tried multiple and newer GLib versions but it just 
caused problems in one way or another due to the changes that have 
occurred in recent months.

The errors you report below are symptomatic of what I was seeing.

make error on latest

Björn Johansson — 2013-04-18T05:12:53

Hi, I tried to install the latest version from git like this:

git clone git://git.gnome.org/tracker

./autogen.sh --enable-tracker-needle
--with-nautilus-extensions-dir=/usr/lib/nautilus/extensions-3.0/

It complained about an old Glib, so
I got a later Glib 2.36.1
http://freecode.com/projects/glib

installed like this

./configure                             # run the `configure' script
make                                    # build GLIB

sudo rm -rf /install-prefix/include/glib.h /install-prefix/include/gmodule.h
sudo make install                            # install GLIB

which seemed to have worked as there were no error msg.

I returned to ./autogen.sh --enable-tracker-needle
--with-nautilus-extensions-dir=/usr/lib/nautilus/extensions-3.0/

and got the error below. IS there som dependency still missing?
thank you for your help,
/Bjorn


make[3]: Entering directory
`/home/bjorn/Desktop/tracker/src/libtracker-sparql-backend'
  VALAC  libtracker_sparql_0.18_la_vala.stamp
tracker-backend.vala:212.22-212.2

Re: tracker store timeout

Ivan Frade — 2013-04-16T22:07:38

Hi,

 With those symptoms, it looks like the tracker-store daemon is crashing
and then it is restarted automatically (dbus autoactivation).

 Maybe the logs can give a clue on what is going on.

 Regards,

Ivan






On Sat, Apr 13, 2013 at 9:37 PM, MrJojohn1987 . <putineiionut-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org>wrote:

tracker store timeout

MrJojohn1987 . — 2013-04-14T04:37:25

Hello,

I run tracker with fd passing enabled and with sparql backend bus.
Recently i observed in logs that some of the queries/updates are not
executed  because they receive a message like tracker-store is no started.
The situation i kind of odd because now query is not executed and in 2-3
sec is executed.


I am thinking that is because of thread pool.......seen that there are 2
threads for query pool and 1 thread for update pool.
Increasing the number of thread could help?

Or if you have another ideea please share it.



Thanks,
Ionut

Re: Tracker Internal Documentaion

Vishesh Handa — 2013-04-11T12:42:53


If it's possible could you please elaborate as to how this is implemented
in the database?



Oh. So, suppose a music file existed with a performer "Coldpay" and
modified the id3 tags to now say "Coldplay". The file would get reindexed
and it would now have both performers? The old and the new one?

The performer is generally stored with a 'nmm:performer' property and does
not have a max cardinality of 1.

Re: Tracker Internal Documentaion

Vishesh Handa — 2013-04-11T10:23:58

Hey Philip

Thanks for the detailed explaintaion

On Thu, Apr 11, 2013 at 12:20 PM, Philip Van Hoof <philip-P+RAIPACLmdwemrocBI00Q< at >public.gmane.org>wrote:


I'm curious as to how you handle -

1. Type Inference - Say something like this 'select ?r where { ?r a
nco:Contact . }. Lets say one has some 10 nco:Contacts and some 15
nco:PersonContacts. In this case one would have to iterate over both the
tables. Does tracker do that?

2. Property Inference - Do you handle cases such as 'select ?r ?l where {
?r nao:prefLabel ?l . }', where the 'nao:prefLabel' has not been explicitly
defined. Lets assume that the 'nie:title' has been set.

The nie:title is a rdfs:subPropertyOf nao:prefLabel

Does tracker handle cases like this? Cause this is a rather common usecase
in Nepomuk where we want to fetch a good label for the resource and we do
not want to query specific properties.

3. I read that you have some support for graphs - How is that implemented?
From what I understand from your db schema each property has their

Re: Tracker Internal Documentaion

Philip Van Hoof — 2013-04-11T13:42:46




It's implemented in the SQLite tables as an extra column.




No, multivalue properties are first clearned and then rewritten (by
miner-fs). When another client adds a value to a multivalue property
then the graph of all is set to the clients', so miner-fs wont overwrite
it anymore.

An example:

1) file.mp3 a nmm:MusicPiece has nmm:performer 'somebody' as extracted
by tracker-extract/miner-fs and written in tracker-store:

GRAPH <miner-fs> {
<urn> a nmm:MusicPiece, nfo:FileDataObject;
   nie:url 'file://.../file.mp3';
   nie:title 'test';
   nmm:performer 'somebody' .
}

2) client updates nmm:performer:

GRAPH <client> {
<urn> nmm:performer 'somebody else' .
}

3) Result:

GRAPH <miner-fs> {
<urn> a nmm:MusicPiece, nfo:FileDataObject;
   nie:title 'test';
   nie:url 'file://.../file.mp3' ;
   nfo:lastModifiedSomething '...' .
}

GRAPH <client> {
<urn> nmm:performer 'somebody else' .
}

4) miner-fs sees an update on file.mp3, update is on performer and
title:

GRAPH <miner-fs> {
  <urn> nie:title 'new ti

Re: Tracker Internal Documentaion

Philip Van Hoof — 2013-04-11T12:34:56


Let me rephrase that: there is a solution for this in Tracker, but it's
somewhat specific to miner-fs.

The miner-fs file indexer will only overwrite triples that have as
origin the miner-fs's graph. When you write a triple you'll always
overwrite its graph value (it's recommended to always provide it and to
never use the one of miner0fs).

Tracker's graph support is like supporting a default graph value, but
only a default one (not more than one).

So basically, miner-fs indexes a file and stores its metadata in sparql
using its own graph.

Another app stores or updates new metadata and uses its own graph.

When miner-fs updates the metadata of the file, it will only overwrite
data in its own graph.

Kind regards,

Philip

Re: Tracker Internal Documentaion

Philip Van Hoof — 2013-04-11T12:29:21


This gets translated to something like

SELECT Uri FROM "nco:Contact", to translate from nco:Contact to the SQL
table "nco:Contact" is something Tracker does internally. I think there
is a environment variable that you can turn on to print the SQLite
statements the first time they are prepared (there is a LRU cache of
SQLite statements). These statements include the query.

Type interference itself (select ?p ?o { nco:Contact ?p ?o }) works
because Tracker stores its own ontology in itself, and an ontology are
just a bunch of rdf statements.



I don't remember.


Afaik yes.


The graph support is limited. More explanation on its limitations here:
https://live.gnome.org/Tracker/Documentation/SparqlFeatures#Named_Graphs


Only for storing the origin of a statement (which was the only required
use-case for the N9, which we were mainly targeting while developing
Tracker's SPARQL endpoint and Nepomuk ontology support).


This isn't supported by Tracker.


It doesn't. Full graph support was not a design conside

Re: Tracker Internal Documentaion

Philip Van Hoof — 2013-04-11T06:50:11

Hi Vishesh,

It's always a good idea to pose these questions on the Tracker public
mailing list, so I replied with the mailing list in CC:

So you have a denormalized schema in SQL where each multi value field in
Nepomuk is represented by a table, and each RDF class is represented by
a table with the exception of some of the xsd primitive ones (which are
implied because SQLite knows how to handle these things, for example
xsd:int, xsd:string, etc).

Then the libtrackersparql makes a WAL SQLite connection, parses your
SPARQL and generates on the fly SQL for that. This happens often using
subqueries, and without building an AST first - making the
parse-translate phase relatively fast and resource friendly, which is a
design-choice as Tracker is indented to run on devices with few
resources.

This code does that:
https://git.gnome.org/browse/tracker/tree/src/libtracker-data/tracker-sparql-query.vala

That design-choice of course has a draw back in that the queries have to
manually optimized very often and/or to

Re: Tracker 0.16 from jhbuild shell

Martyn Russell — 2013-03-28T14:24:04

It by default goes to .xsession-errors or the systemd journal unless you 
define the env var TRACKER_USE_LOG_FILES, you can man tracker-miner-fs 
for details.

I did hear from others recently that this wasn't working for them, let 
me know if this is the case for you. It may need fixing. It worked when 
I tested it before though.