gmane.comp.db.firebird.python

Re: RFC: Metadata extensions for FDB

Gerdus van Zyl — 2013-05-15T09:41:16

I would prefer option 2: Database class that takes Connection object as
constructor parameter possibly with convenience function on connection
object e.g con.databasemeta()
don't think publishing it as separate package is needed.

Having database metadata object would be very useful to me.




On Tue, May 14, 2013 at 5:54 PM, Pavel Cisar <pcisar-MMa1HB0MvMoheNAkBk4TRQ< at >public.gmane.org> wrote:

RFC: Metadata extensions for FDB

Pavel Cisar — 2013-05-14T15:54:13

Hi all,

I have "metadata" extension for FDB in development for some time now, 
and I'm getting close to release it (intended for 1.2 release). The 
basic idea is to provide database metadata (and other selected useful 
information like common queries/operations on top of monitoring tables) 
as collections of Python objects for easy access and manipulation. 
However, this is huge addition to "basic" functionality mandated by 
Python DB API 2.0, and it's not necessary for work with databases. So I 
tend to keep it separated from core functionality. On the other hand, 
this all naturally fits to Connection class, so I would appreciate some 
feedback from end FDB users here.

Right now these extensions are separated into independent submodule 
fdb.utils. The main "access" point is Database class, but this module 
also contains other metadata classes (tables, domains, indices etc.). 
The submodule was introduced to keep indirectly used classes/code from 
main FDB namespace, and import only Database class into it

FDB release 1.1.1

Pavel Cisar — 2013-05-14T11:41:08

FDB release 1.1.1 is out:
http://pypi.python.org/pypi/fdb

Bugs Fixed:

- http://tracker.firebirdsql.org/browse/PYFB-34
- http://tracker.firebirdsql.org/browse/PYFB-35

best regards
Pavel Cisar
IBPhoenix


------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscribe-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Re: FDB Blocking Cursor

Pavel Cisar — 2013-05-07T11:07:29

Hi,

You're right, fetch should report correct error for statements that 
doesn't return any result. However, your proposed fix is not correct.

Bug logged (PYFB-35), fixed and fix & test committed.

best regards
Pavel Cisar
IBPhoenix

Dne 6.5.2013 16:55, josef_gschwendtner napsal(a):


------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscribe-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Re: Huge Firebird memory usage using FDB

Pavel Cisar — 2013-05-06T16:38:46

Philippe,

This one was really interesting. CORE-1249 is about operations after 
database shutdown, so we shut down the database, try if subsequent sql 
statement fails and then bring it back online so it could be properly 
finalized.

Actual error here was in sql statement release via 
isc_dsql_free_statement (DSQL_drop) that raises error that db is 
shutdown, although it wasn't at the time this function is called. To 
make it more interesting, it's only Linux problem since 2.5.1, while 
2.5.0 and Windows doesn't throw this error.

However, after consultation with Dmitry it's clear that actually this 
error is as designed (and Windows somehow doesn't comply :) , and 
intended behavior (since 2.5 even more aggresive as even idle 
connections are affected) is that all connections after they hit the 
"shutdown wall" are unusable, and this state is cached in client library 
so all subsequent calls to any API function on this database ends with 
this error. Connectivity library developers must detect 335544528 e

FDB Blocking Cursor

josef_gschwendtner — 2013-05-06T14:55:58

We depend on a third party framework which uses one Cursor Instance for
the execution of many queries. There are some select queries and also
ddl statements which get executed with one Cursor instance.

After each execution of a statement the fetch method gets called. Also
for the ddl statements which produces no result.  But in fdb this leads
to an exception "WindowsError: exception: access violation reading
0x0000000000000000". After this exception the cursor is no longer
usable. The next call of cur.execute() with a select statement blocks
and never returns.

This situation can be simulated with the following script. Instead of a
ddl statement it is also possible to use an update statement.



import fdb

import traceback





def execute(cur, stmt, parameters = None):

     print "-----------------------------------"

     print "Execute ", stmt

     cur.execute(stmt, parameters)

    try:

         for row in cur.fetchall():

             print row

     except Exception, e:

         print traceback.

Re: Huge Firebird memory usage using FDB

Philippe Makowski — 2013-05-04T13:57:48

Pavel Cisar  [2013-05-03 11:42] :
seems better but still problems

see:
http://ci.ibphoenix.fr/job/firebird25_Linux_64/Architecture=SuperServer,System=System_Icu/65/console

AttributeError: "'Connection' object has no attribute '_Connection__ic'"
in <bound method Connection.__del__ of <fdb.fbcore.Connection object at
0x3698550>> ignored

during fbtest

and an error during fdb test:
ERROR: test_restore (testfdb.TestServices2)



------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscr

Re: Huge Firebird memory usage using FDB

Pavel Cisar — 2013-05-03T09:42:52

Hi,

Initial fix for PYFB-34 was incomplete, which should be now corrected 
(at least fdb nosetests and fbtest now run fine). Please verify it too 
in your environment.

best regards
Pavel Cisar
IBPhoenix



------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscribe-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Re: Huge Firebird memory usage using FDB

Philippe Makowski — 2013-05-02T11:26:47

Pavel,
FDB tests have problem on 2.5 on the automated build system
and this, only after your last commit about PYFB-34

+ nosetests --with-xunit --verbose --exclude=test_limbo_transactions
test_basic (testfdb.TestArrays) ... ok
test_read_full (testfdb.TestArrays) ... ok
test_write_full (testfdb.TestArrays) ... ok
test_write_wrong (testfdb.TestArrays) ... ok
test_pyfb_17 (testfdb.TestBugs) ... ok
test_pyfb_22 (testfdb.TestBugs) ... ok
test_pyfb_25 (testfdb.TestBugs) ... ok
test_pyfb_30 (testfdb.TestBugs) ... ok
testBlob (testfdb.TestCharsetConversion) ... ok
testCharVarchar (testfdb.TestCharsetConversion) ... ok
test_octets (testfdb.TestCharsetConversion) ... ok
test_utf82win1250 (testfdb.TestCharsetConversion) ... ok
test_connect (testfdb.TestConnection) ... ok
test_connect_role (testfdb.TestConnection) ... ok
test_database_info (testfdb.TestConnection) ... ok
test_db_info (testfdb.TestConnection) ... ok
test_execute_immediate (testfdb.TestConnection) ... ok
test_transaction (testfdb.TestConnection) ... ok
t

Re: Huge Firebird memory usage using FDB

Pavel Cisar — 2013-05-02T09:59:11

Philippe,

I never tested FDB with 3.0, so it may fail. Strange it fails with 
fbtest on 2.5, as it passed all FDB tests on 2.5 (I never commit changes 
if fdb tests fail). I'll investigate.

Pavel


------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscribe-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Re: Huge Firebird memory usage using FDB

Philippe Makowski — 2013-05-02T07:28:51

Le 01/05/13 10:09, Pavel Cisar a écrit :
seems that there is a problem with Firebird Trunk
see (at the end of 
http://ci.ibphoenix.fr/job/Firebird_trunk_Arch_Linux/Architecture=Default/23/console) 
:

test_rowcount (testfdb.TestCursor) ... FAIL
pure virtual method called
terminate called without an active exception
/home/vagrant/buildfirebirdtrunk.sh: line 22: 16965 Aborted 
     (core dumped) nosetests2 --with-xunit --verbose 
--exclude=test_limbo_transactions



And whith Firebird 2.5

we get a lot of :
Exception fdb.fbcore.DatabaseError: DatabaseError('Error while closing 
SQL statement:\n- SQLCODE: -901\n- invalid statement handle', -901, 
335544485) in <bound method Cursor.__del__ of <fdb.fbcore.Cursor object 
at 0x2ad2f50>> ignored


during tests

see : 
http://ci.ibphoenix.fr/job/firebird25_Linux_64/Architecture=Classic,System=Firebird_Icu/64/console






------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-p

Re: Huge Firebird memory usage using FDB

Pavel Cisar — 2013-05-01T08:09:22

Hi,

Good catch, thanks. Logged as PYFB-34 and fix committed. Please checkout 
the the latest version from our Subversion at:

http://svn.code.sf.net/p/firebird/code/python/fdb/trunk

and let me know whether it solved your problem.

best regards
Pavel Cisar
IBPhoenix

Dne 30.4.2013 12:20, josef_gschwendtner napsal(a):


------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscribe-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yaho

Huge Firebird memory usage using FDB

josef_gschwendtner — 2013-04-30T10:20:14

Hi,

We got a problem when we use the Firebird driver for Python (fdb) inside
a connection pool. If a single connection is used for the execution of

many queries and each query uses a separate cursor instance we observed
an increasing memory usage on the server side (the fbserver process).

Only when the connection gets closed the memory usage of the fbserver
process gets back to a "normal" value.

 An equal situation can be simulated with this short python script.

 

import fdb





con = fdb.connect(dsn='localhost/3050:C:\\Development\\DB\\mydb.fdb',
user='sysdba',
                  password='masterkey')



for i in range(10000):

    cur = con.cursor()

    cur.execute( "select * from RDB$Relations")

    for row in cur.fetchall():

        print i, row[0]

    con.commit()

t = raw_input('Press Key')

con.close()



print "Finished"



When the loop has finished and before closing the connection the
fbserver process uses almost one GB of memory. This is independe

WG: FDB Bug Report

Josef Gschwendtner — 2013-04-30T09:28:04

Von: Markus Holzner 
Gesendet: Dienstag, 30. April 2013 10:30
An: Josef Gschwendtner
Betreff: FDB Bug Report



We got a problem when we use the Firebird driver for Python (fdb) inside a connection pool. If a single connection is used for the execution of 

many queries and each query uses a separate cursor instance we observed an increasing memory usage on the server side (the fbserver process). 

Only when the connection gets closed the memory usage of the fbserver process gets back to a "normal" value. 

 

An equal situation can be simulated with this short python script.

 

import fdb

 

 

con = fdb.connect(dsn='localhost/3050:C:\\Development\\DB\\mydb.fdb', user='sysdba', 
                  password='masterkey')

 

for i in range(10000):

    cur = con.cursor()

    cur.execute( "select * from RDB$Relations")

    for row in cur.fetchall():

        print i, row[0]

    con.commit()

         

 

t = raw_input('Press Key')

con.close()

 

print "Finished"

 

When the loop has finished and before

FDB release 1.1

Pavel Cisar — 2013-04-09T10:55:58

FDB release 1.1 is out:
http://pypi.python.org/pypi/fdb

New Features:

- Context Manager for transactions.

Bugs Fixed:

- http://tracker.firebirdsql.org/browse/PYFB-30

best regards
Pavel Cisar
IBPhoenix


------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscribe-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Re: reading blob from table

Pavel Cisar — 2013-04-08T10:12:50

Hi,

Bug confirmed and fixed. Unfortunately our tracker is down at the 
moment, so I can't make an entry, commit changes to Subversion, close it 
and make a bugfix 1.0.1 release. So meanwhile you can use this hotfix 
release:

http://ibphoenix.cz/download/fdb-1.0.1.dev.tar.gz

It was tested (unit test added to test suite), so it shouldn't break 
anything else.

best regards
Pavel Cisar
IBPhoenix


------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscribe-hHKSG33TihhbjbujkaE4pw< at >publ

Re: reading blob from table

Hamish Moffatt — 2013-04-08T13:48:52

Thanks Pavel, I confirm that my problem is fixed with the above release.


regards,
Hamish

Re: reading blob from table

Hamish Moffatt — 2013-04-08T04:16:33

Hi,

Thanks Pavel. I tried with charset NONE set when opening the connection 
but the behaviour did not change.


regards
Hamish

On 8/04/2013 2:08 AM, Pavel Cisar wrote:

Re: reading blob from table

Hamish Moffatt — 2013-04-08T07:05:33

I'm not usually, but was experimenting with Pavel's suggestion that the 
blob truncation I saw might be a character set issue. I am now back to 
my original definition of the column as simply "BLOB", the connection 
character set as UTF8 and I'm able to fetch the blobs successfully in 
materialized mode using kinterbasdb.

thanks
Hamish

Re: reading blob from table

Helen Borrie — 2013-04-08T07:02:45



Why are you applying a character set attribute at all, to binary blobs?



Helen Borrie, Support Consultant, IBPhoenix (Pacific)
Author of "The Firebird Book" and "The Firebird Book Second Edition"
http://www.firebird-books.net
__________________________________________________________________ 



------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscribe-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Re: reading blob from table

Hamish Moffatt — 2013-04-08T05:28:03

I found that it works with kinterbasdb for UTF8 or NONE connection type.

I tried changing the column definition to "blob binary character set 
none" and some other things but none of them helped with fdb (1.0). For 
some combinations of the column type and the connection character set I 
got Python utf8 decode errors, others just truncated the data. Also 
tried with a Firebird 2.5 server (I was using 2.1) without any success.

Hamish

On 8/04/2013 2:16 PM, Hamish Moffatt wrote: