gmane.comp.db.metakit

Re: Extract a single column

Thadeus Burgess — 2011-09-07T16:12:02

newvw = vw.project(vw.column_name)

for row in newvw:
  print row.column_name

--
Thadeus




On Wed, Mar 30, 2011 at 10:21 AM, Thadeus Burgess <thadeus.burgess< at >gmail.com

Re: Compiling for python-windows

Thadeus Burgess — 2011-09-07T16:10:55

Here is the solution...

Windows
~~~~~~~

Install MinGW for your version of windows (32 bit or 64 bit version)

Download ``metakit <http://equi4.com/pub/mk/metakit-2.4.9.7.tar.gz>`` and
extract it using 7-zip.

.. code-block:: bash

    $: set PATH=PATH;C:\mingw\msys\1.0\bin\;C:\mingw\bin\
    $: cd metakit-2.4.9.7\builds
    $: sh ..\unix\configure
    $: make
    $: cd ..\python
    $: C:\python27\python setup.py clean build --compiler=mingw32 # or
mingw64

Find the files in ``..\builds\lib.windows.....\Mk4py.pyd``.

You will also need the following files alongside the ``Mk4py.pyd``.

* ``C:\mingw\bin\libgcc_s_dw2-1.dll``
* ``C:\mingw\bin\libstdc++-6.dll``

--
Thadeus




On Thu, Mar 31, 2011 at 11:41 AM, Thadeus Burgess <thadeus.burgess< at >gmail.com

Compiling for python-windows

Thadeus Burgess — 2011-03-31T16:41:08

Is there anyone who could help get a version of metakit with python on
windows?

I attempted to use the pre-compiled version from the site which does not
work. (Spits out a "module Mk4py not found").

Could either a pre-compiled python version working or a step-by-step guide
to get a version compiled.

Thanks for your help and time


--
Thadeus

Extract a single column

Thadeus Burgess — 2011-03-30T15:21:59

Is it possible to quickly extract a list of a single column without having
to do list comprehension?

--
Thadeus

Re: Over 1.5GB file limit?

Pat Thoyts — 2011-03-29T17:57:15

We store spectroscopic data into metakit files. We've been doing so
for about 8 years now and last time I counted up the local datafiles
we have nearly 0.75TB stored in these things. We store each individial
spectrum as a row of some values and a blob of binary data. We found
that over some threshold (10 000 rows or so) the default configuration
doesnt scale too well. If you intend to have lots of rows, ensure you
use blocked views.

Speed wise - you will have to test your setup. I'm now moving away
from our metakit based files so that we can memory map the data
directly as both the 2GB limit and the time taken to extract an
individual dataset are now limiting. (The time issue is not metakit's
fault - we also serialized COM objects into the data stream - in
retrospect this was a mistake). If your data is constant sized, then
fastest and least resource intensive is to simply append blocks onto
the end of the file when writing and use memory mapped I/O with a
suitable window size when reading them later.

Any

Re: Over 1.5GB file limit?

Thadeus Burgess — 2011-03-29T17:21:11

3rd tries a charm to post this message to the group !

----------

It is glad to know metakit is being actively used.

We are using it to log data that comes in every second from many nodes, all
time stamped and need to be able to store potentially millions of rows. We
do plan on running metakit on some cloud servers, however this also has to
run on some Atom class processor typically found in netbooks (since this is
where the data is coming from).

Currently we are using PostgreSQL in production, yet unfortunately it is
VERY slow and is unable to keep up with the data demand. Yes Postgres is
configured, has proper indexing etc etc....

In preliminary tests...

We can stuff about 11million records into 2GB. There shouldn't be more than
2million records in a given day. Using a time-series based file partitioning
(year -> month -> day -> data.mk) the 2GB limit should not occur.

Performing a select on a range in the 11M records takes about .12 seconds
(SQL -> 15 minutes)
Performing the above select and sorting

Re: Over 1.5GB file limit?

Steve Landers — 2011-03-29T08:10:20

On 29/03/2011, at 3:51 PM, schmir< at >gmail.com wrote:


Or perhaps just not broken?   Pat Thoyts earlier today referred to bug fixes in the mk::channel support. Did you miss that?


There are a number of devices where 2Gb is more than appropriate.  There are a number of devices where slot-based access to memory mapped data structures is very appropriate and significantly more efficient (and predictable) than other approaches.

Of course, it may not be appropriate for your needs.  But for you to say Metakit is dead is unfortunate and quite misleading.


Never let the facts get in the way of personal opinion.

Now, back to the original topic. It may well be that Thadeus would be better served by considering another database library, depending on his needs. But neither you nor I know that based on the information he provided.

Steve

Re: Over 1.5GB file limit?

2011-03-29T07:51:35


The last release is from 2007, it's probably just finished?

5 out of 8 related projects links on http://equi4.com/metakit/links.html
do not work. The "failure feedback forum" link, which you can use to
report bugs on that same page, points to http://equi4.com/fff.html,
which just says:

,----
| The Failure Feedback Forum area is no longer being tracked or
| maintained. 
`----

A 2 GB file limit just doesn't seem appropriate any more, and there is
no plan to lift this limit.  

That's why I call metakit a dead project.

Cheers,
- Ralf

Re: Over 1.5GB file limit?

Steve Landers — 2011-03-29T07:00:52

On 29/03/2011, at 2:54 PM, schmir< at >gmail.com wrote:


That is a ridiculous assertion, Metakit is stable and maintained.


SQLite is an excellent choice if you want SQL.    But there is a class of problems that Metakit is particularly well suited to. I use and recommend both, depending on the circumstances.

Steve

Re: Over 1.5GB file limit?

2011-03-29T06:54:49


It's still 2GB. The file format itself doesn't allow bigger databases. 

My advise is to stay away from metakit, it's dead. Choose something
else. sqlite comes to mind.

Cheers,
- Ralf

metakit vs HDF5

Thadeus Burgess — 2011-03-28T21:20:53

Are there any good detailed comparisons of metakit vs HDF5 (mk4py vs
pytables).

--
Thadeus

Re: Over 1.5GB file limit?

Thadeus Burgess — 2011-03-28T21:20:08

I am planning on running metakit on some amazon ec2 servers, so I can choose
32bit or 64bit linux.

So for a 32bit linux server, plan on a max filesize of around 2GB, and on a
64bit linux server would be ____GB ?

--
Thadeus




On Mon, Mar 28, 2011 at 3:30 PM, Pat Thoyts <patthoyts< at >gmail.com> wrote:

Re: Over 1.5GB file limit?

Pat Thoyts — 2011-03-28T20:30:13

Metakit memory maps the whole database file in one single view. So on
32 bit systems the total addressable memory tends to be 2GB (this is
certainly the case for Windows) so taking into account your process
plus any dll's loaded you are likely to only be able to map in a file
of about 1.5 GB. Metakit itself permits larger files. There are more
than 32 bits available to hold the size of the data file (its not a
64bit number - its a 32bit + some bits from the next slot - I forget
how many).
On a 64 bit system with a 64 bit process you should be able to address
more space.
Its not clear to me what limit you are discussing here - but its
likely just the Windows 32bit process addressing limit.

Over 1.5GB file limit?

Thadeus Burgess — 2011-03-28T20:13:12

What happens when the file goes over 1.5GB ?

In my testing, generating random data to provide a metakit file of about 2GB
(about 11milliion records). Metakit seems to be able to open and parse the
file as normal.

Is there any specific reason the GB file limit has been placed?

--
Thadeus

Get the latest rows (by date property) from a view?

Jeremy Cowgar — 2010-09-14T00:10:46

  I have a view that contains ~20,000 rows. Each row has a startedAt
property, when the action began. I would like to get the 20 most recent
actions. Using the Tcl interface I tried all sorts of things, for example:

foreach row [mk::select logdb.log -rsort startedAt -count 20] {
     puts $row
}

That simply returns 19 18 17 16 15 14 ... 0 ... Those are the oldest
ones (not in order). I also tried:

mk::select logdb.log -rsort startedAt -start [expr {[mk::view size
logdb.log] - 20}]

but that didn't work either as the entries are not placed in the db in
sequence according to date. That returned the last 20 rows in what
seemed to be in random order.

I typically program with SQL and what I want to accomplish is this:

SELECT * FROM log ORDER BY startedAt DESC LIMIT 20

How can I accomplish this using the Tcl interface to MetaKit?

Thanks,

Jeremy Cowgar

Filling a hash view

Marcin Krol — 2010-03-04T19:37:48

Hello everyone,

I'm "testdriving" metakit 4 py now, and I have a little trouble with 
hashed view: when the number of keys in view starts exceeding 2000, 
filling up a hash view becomes extremely slow.

Is there some way to fill a view that is hashed to begin with, so the db 
knows its way around db and adding rows becomes faster?

The code is below.

Also, could someone please explain to me what's the meaning of 'del 
map[:]' in

def hash_ini(n=1000):
   t0 = time();
   for i in xrange(n):
     del map[:]
     vwh = vw.hash(map)
   print ' hash_ini %d times, size %d: %g sec' % (n, len(map), time() - t0)


in examples/find.py in metakit distribution? And why vw.hash(map) has to 
be repeated n times? Sadly, the docs do not explain what's this all about.




#!/usr/bin/python

##import wingdbstub


import Mk4py, sys
mk = Mk4py

print sys.argv[0], '- Mk4py', mk.version, '-', sys.platform

db = mk.storage("demo2.db",1)
vw = db.getas('words[idxs1:S,idxs2:S,idxi:I,val:S]')
map = db.getas('map[_H:I,_R:I]')
vwh = N

Re: Filling a hash view

Jean-Claude Wippler — 2010-03-05T00:12:36


This is the normal way to open a hash view:

db = mk.storage("demo2.db",1)
vw = db.getas('words[idxs1:S,idxs2:S,idxi:I,val:S]')
map = db.getas('map[_H:I,_R:I]')
vwh = vw.hash(map)

After this point you must make all changes to vwh to keep everything in sync. Changes to the (non-persistent / virtual) vwh view will internally get translated to the proper changes to the persistent vw and map view.

To fill it more quickly:
- make changes directly to the underlying "vw", ignoring map and vwh
- empty the map (that's what "del map[:]" does)
- reconstruct a new vwh with "vwh = vw.hash(map)"

This means you reconstruct the hash after filling the underlying view, instead up updating it with each change. It's similar to reconstructing vs. maintaining an index in a relational database.


 t0 = time();
 for i in xrange(n):
   del map[:]
   vwh = vw.hash(map)
 print ' hash_ini %d times, size %d: %g sec' % (n, len(map), time() - t0)

That's a benchmark, it reports the number of seconds needed to perform n hash const

Re: Filling a hash view

Pat Thoyts — 2010-03-05T00:05:32

When a view has a lot of rows you can fix the performance issues using
a blocked view. This is another wrapper around the view but it keeps
the performance in check. I've no idea about python but in C++
   c4_View v = rFiles(view[nInode]);
   m_vFiles = v.Blocked().Ordered();

When defining the layout you add a _B[] around the view description.

The performance issue is that metakit is column oriented. So when you
add a new row it moves whole columns around. By breaking this into
blocks it restricts the size of the columns. I use this kind of view
to manage 200 000 rows per view quite handily.

Pat.

Re: Metakit and disk access.

Dmitrii — 2009-11-25T20:02:11

Effectively, Metakit does the "storing a package of data" in memory
for you until you commit. So, perhaps, you should keep the database
open and call commit from time to time to actually persist the data on
disk (the frequency of commits depends on your tolerance to data loss
if something bad happens and the data collecting process dies).

On Nov 2, 10:09 am, Jose Brasil <automacao....< at >gmail.com> wrote:

Reducing file size be eliminating free space

Bill — 2009-11-11T19:41:07

Hello:

Thanks again for Metakit... Big fan...

When the size of the data in my application
is reduced, I would like to reflect this
in the size of the associated data file.

The Metakit file format documentation states:

    "For those cases where slack is unacceptable,
     a simple full re-save can be used to produce a fresh,
     optimally-packed datafile."

Given a c4_View created with a backing c4_Storage (which
is associated with a file), how do I do a "full re-save" ???

Here, again, my goal here is to reduce the size of the data
file by removing all of the "free space".

Thanks.

Bill

Metakit and disk access.

Jose Brasil — 2009-11-02T14:09:15

Dear Members List,

I'm using Metakit to store data sent by a lab analyser each 1 second.

I have a doubt if I should open, commit and close the database every time to
store data or should store data in a variable and store a package of data in
database to decrease the disk access.

Thanks for any sugestion,
Jose