gmane.comp.db.firebird.python

Re: RFC: Metadata extensions for FDB

Gerdus van Zyl — 2013-05-15T09:41:16

I would prefer option 2: Database class that takes Connection object as
constructor parameter possibly with convenience function on connection
object e.g con.databasemeta()
don't think publishing it as separate package is needed.

Having database metadata object would be very useful to me.




On Tue, May 14, 2013 at 5:54 PM, Pavel Cisar <pcisar-MMa1HB0MvMoheNAkBk4TRQ< at >public.gmane.org> wrote:

RFC: Metadata extensions for FDB

Pavel Cisar — 2013-05-14T15:54:13

Hi all,

I have "metadata" extension for FDB in development for some time now, 
and I'm getting close to release it (intended for 1.2 release). The 
basic idea is to provide database metadata (and other selected useful 
information like common queries/operations on top of monitoring tables) 
as collections of Python objects for easy access and manipulation. 
However, this is huge addition to "basic" functionality mandated by 
Python DB API 2.0, and it's not necessary for work with databases. So I 
tend to keep it separated from core functionality. On the other hand, 
this all naturally fits to Connection class, so I would appreciate some 
feedback from end FDB users here.

Right now these extensions are separated into independent submodule 
fdb.utils. The main "access" point is Database class, but this module 
also contains other metadata classes (tables, domains, indices etc.). 
The submodule was introduced to keep indirectly used classes/code from 
main FDB namespace, and import only Database class into it (or not).

There are several arrangements I currently consider:

1. Database is subclass of Connection returned by 
connect/create_database instead Connection on request. These functions 
will have optional connection_class named parameter (defaults to 
Connection) to return instance of any (either Database from fdb.utils or 
user created) Connection subclass instead Connection. Other option is 
addition of connect/create_database also in fdb.utils, but these 
functions would return Database instead Connection, but parametrized 
ones are imho better.

The obvious advantage of this scenario is that you have both 
functionalities in single object. Disadvatage is that you would get big 
ball of code/attributes which you may not use very often - if at all.

2. Database is independent class working with Connection instance 
provided on creation. This arrangement clearly separates both 
functionalities. Database instance provides extended attributes and 
functions only, for basic access you'll use the Connection instance, so 
you may need to keep around both. On the other hand it's easier and more 
effective arrangement if you don't want to use metadata functionality 
extensively, as you can easily ignore it or drop the Database object any 
time without disruption to core (Connection) functionality of your 
application.

3. Hybrid case between 1. a 2. Database is independent class (like in 
2.) but works as transparent proxy to Connection provided, i.e. has 
getattr that passes calls to directly unhandled names to Connection.

Extreme cases:

4. Don't bother and extend Connection class directly.

5. Create separate (namespace?) package with this extended functionality 
to keep driver package pure. Who needs such extension can easily install 
it from PyPI. The connection to FDB Connection could be either 1. or 2. 
method.

My personal preference is slightly toward option 1 (maybe as 5.), but 
I'm open to any suggestions.

What is your preference?

best regards
Pavel Cisar
IBPhoenix



------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscribe-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

FDB release 1.1.1

Pavel Cisar — 2013-05-14T11:41:08

FDB release 1.1.1 is out:
http://pypi.python.org/pypi/fdb

Bugs Fixed:

- http://tracker.firebirdsql.org/browse/PYFB-34
- http://tracker.firebirdsql.org/browse/PYFB-35

best regards
Pavel Cisar
IBPhoenix


------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscribe-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Re: FDB Blocking Cursor

Pavel Cisar — 2013-05-07T11:07:29

Hi,

You're right, fetch should report correct error for statements that 
doesn't return any result. However, your proposed fix is not correct.

Bug logged (PYFB-35), fixed and fix & test committed.

best regards
Pavel Cisar
IBPhoenix

Dne 6.5.2013 16:55, josef_gschwendtner napsal(a):


------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscribe-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Re: Huge Firebird memory usage using FDB

Pavel Cisar — 2013-05-06T16:38:46

Philippe,

This one was really interesting. CORE-1249 is about operations after 
database shutdown, so we shut down the database, try if subsequent sql 
statement fails and then bring it back online so it could be properly 
finalized.

Actual error here was in sql statement release via 
isc_dsql_free_statement (DSQL_drop) that raises error that db is 
shutdown, although it wasn't at the time this function is called. To 
make it more interesting, it's only Linux problem since 2.5.1, while 
2.5.0 and Windows doesn't throw this error.

However, after consultation with Dmitry it's clear that actually this 
error is as designed (and Windows somehow doesn't comply :) , and 
intended behavior (since 2.5 even more aggresive as even idle 
connections are affected) is that all connections after they hit the 
"shutdown wall" are unusable, and this state is cached in client library 
so all subsequent calls to any API function on this database ends with 
this error. Connectivity library developers must detect 335544528 error 
code and ignore these errors on object cleanups.

Fix committed.

best regards
Pavel Cisar
IBPhoenix

Dne 4.5.2013 15:57, Philippe Makowski napsal(a):


------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscribe-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

FDB Blocking Cursor

josef_gschwendtner — 2013-05-06T14:55:58

We depend on a third party framework which uses one Cursor Instance for
the execution of many queries. There are some select queries and also
ddl statements which get executed with one Cursor instance.

After each execution of a statement the fetch method gets called. Also
for the ddl statements which produces no result.  But in fdb this leads
to an exception "WindowsError: exception: access violation reading
0x0000000000000000". After this exception the cursor is no longer
usable. The next call of cur.execute() with a select statement blocks
and never returns.

This situation can be simulated with the following script. Instead of a
ddl statement it is also possible to use an update statement.



import fdb

import traceback





def execute(cur, stmt, parameters = None):

     print "-----------------------------------"

     print "Execute ", stmt

     cur.execute(stmt, parameters)

    try:

         for row in cur.fetchall():

             print row

     except Exception, e:

         print traceback.format_exc()





def main():

     con = fdb.connect(dsn='localhost/3050:employee', user='sysdba',
password='masterkey')

     cur = con.cursor()





     execute(cur, "select * from COUNTRY")

     #execute(cur, "UPDATE COUNTRY SET CURRENCY = 'Euro' where COUNTRY =
                    'Germany'")

     execute(cur, "CREATE SEQUENCE TEST_SEQ_1")

     execute(cur,  "select * from COUNTRY")



     con.commit()

     con.close()

     print "Finished"



main()







We got the following output:



-----------------------------------

Execute  select * from COUNTRY

('USA', 'Dollar')

('England', 'Pound')

('Canada', 'CdnDlr')

('Switzerland', 'SFranc')

('Japan', 'Yen')

('Italy', 'Lira')

('France', 'FFranc')

('Germany', 'Euro')

('Australia', 'ADollar')

('Hong Kong', 'HKDollar')

('Netherlands', 'Guilder')

('Belgium', 'BFranc')

('Austria', 'Schilling')

('Fiji', 'FDollar')

-----------------------------------

Execute  CREATE SEQUENCE TEST_SEQ_1

Traceback (most recent call last):

   File "C:\Development\Workspaces\PySnippets\fdb_fetch_problem.py", line
10, in execute

     for row in cur.fetchall():

   File
"C:\Development\VirtualEnvs\web\lib\site-packages\fdb-1.1-py2.7.egg\fdb\\
fbcore.py", line 3188, in fetchall

     return [row for row in self]

   File
"C:\Development\VirtualEnvs\web\lib\site-packages\fdb-1.1-py2.7.egg\fdb\\
fbcore.py", line 2951, in next

     row = self.fetchone()

   File
"C:\Development\VirtualEnvs\web\lib\site-packages\fdb-1.1-py2.7.egg\fdb\\
fbcore.py", line 3148, in fetchone

     return self._ps._fetchone()

   File
"C:\Development\VirtualEnvs\web\lib\site-packages\fdb-1.1-py2.7.egg\fdb\\
fbcore.py", line 2820, in _fetchone

     ctypes.cast(ctypes.pointer(self.out_sqlda), XSQLDA_PTR))

WindowsError: exception: access violation reading 0x0000000000000000



-----------------------------------

Execute  select * from COUNTRY



The execution of the second select statement never returns. The
application is still running.



We think that a call to fetch after a sql statement without a result
should not lead to an access violation.

The same test with kinterbasdb leads to an exception like this:



Error  (0L, 'Attempt to fetch row of results after statement that does
not produce result set.  That statement was:  CREATE SEQUENCE
TEST_SEQ_1')



After this exception the cursor from kinterbasdb is still usable.



This behavior seems to be conform to PEP 249 -- Python Database API
Specification v2.0



From: http://www.python.org/dev/peps/pep-0249/#id19
<http://www.python.org/dev/peps/pep-0249/#id19>



.fetchone <http://www.python.org/dev/peps/pep-0249/#fetchone> ()

Fetch the next row of a query result set, returning a single sequence,
or None when no more data is available. [6]
<http://www.python.org/dev/peps/pep-0249/#id45>

An Error <http://www.python.org/dev/peps/pep-0249/#error>  (or subclass)
exception is raised if the previous call to .execute*()
<http://www.python.org/dev/peps/pep-0249/#id14>  did not produce any
result set or no call was issued yet.



One solution  could be, that inside the fetchone operation of the Cursor
is a check like this:

  if not self._ps.statement_type in [isc_info_sql_stmt_select,
isc_info_sql_stmt_select_for_upd,

                                          
isc_info_sql_stmt_exec_procedure]:

             raise Database.Error("Attempt to fetch row of results  after
statement that does not
                                   produce result set.")


We found a similar check inside the C source of kinterbasdb (see
function *_Cursor_fetchtuple from
http://python-kinterbasdb.sourcearchive.com/documentation/3.2/__kicore__\
cursor_8c-source.html
<http://python-kinterbasdb.sourcearchive.com/documentation/3.2/__kicore_\
_cursor_8c-source.html> ).
    Is it possible that a patch like this could be integrated to fdb?

Thank you,

Josef

Re: Huge Firebird memory usage using FDB

Philippe Makowski — 2013-05-04T13:57:48

Pavel Cisar  [2013-05-03 11:42] :
seems better but still problems

see:
http://ci.ibphoenix.fr/job/firebird25_Linux_64/Architecture=SuperServer,System=System_Icu/65/console

AttributeError: "'Connection' object has no attribute '_Connection__ic'"
in <bound method Connection.__del__ of <fdb.fbcore.Connection object at
0x3698550>> ignored

during fbtest

and an error during fdb test:
ERROR: test_restore (testfdb.TestServices2)



------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscribe-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Re: Huge Firebird memory usage using FDB

Pavel Cisar — 2013-05-03T09:42:52

Hi,

Initial fix for PYFB-34 was incomplete, which should be now corrected 
(at least fdb nosetests and fbtest now run fine). Please verify it too 
in your environment.

best regards
Pavel Cisar
IBPhoenix



------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscribe-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Re: Huge Firebird memory usage using FDB

Philippe Makowski — 2013-05-02T11:26:47

Pavel,
FDB tests have problem on 2.5 on the automated build system
and this, only after your last commit about PYFB-34

+ nosetests --with-xunit --verbose --exclude=test_limbo_transactions
test_basic (testfdb.TestArrays) ... ok
test_read_full (testfdb.TestArrays) ... ok
test_write_full (testfdb.TestArrays) ... ok
test_write_wrong (testfdb.TestArrays) ... ok
test_pyfb_17 (testfdb.TestBugs) ... ok
test_pyfb_22 (testfdb.TestBugs) ... ok
test_pyfb_25 (testfdb.TestBugs) ... ok
test_pyfb_30 (testfdb.TestBugs) ... ok
testBlob (testfdb.TestCharsetConversion) ... ok
testCharVarchar (testfdb.TestCharsetConversion) ... ok
test_octets (testfdb.TestCharsetConversion) ... ok
test_utf82win1250 (testfdb.TestCharsetConversion) ... ok
test_connect (testfdb.TestConnection) ... ok
test_connect_role (testfdb.TestConnection) ... ok
test_database_info (testfdb.TestConnection) ... ok
test_db_info (testfdb.TestConnection) ... ok
test_execute_immediate (testfdb.TestConnection) ... ok
test_transaction (testfdb.TestConnection) ... ok
test_create_drop (testfdb.TestCreateDrop) ... ok
test_description (testfdb.TestCursor) ... ok
test_exec_after_close (testfdb.TestCursor) ... ok
test_fetchall (testfdb.TestCursor) ... ok
test_fetchallmap (testfdb.TestCursor) ... ok
test_fetchmany (testfdb.TestCursor) ... ok
test_fetchmanymap (testfdb.TestCursor) ... ok
test_fetchone (testfdb.TestCursor) ... ok
test_fetchonemap (testfdb.TestCursor) ... ok
test_iteration (testfdb.TestCursor) ... ok
test_name (testfdb.TestCursor) ... ok
test_pscache (testfdb.TestCursor) ... ok
test_rowcount (testfdb.TestCursor) ... ok
test_use_after_close (testfdb.TestCursor) ... ok
test_context_manager (testfdb.TestDistributedTransaction) ... ok
test_simple_dt (testfdb.TestDistributedTransaction) ... ok
test_20_events (testfdb.TestEvents) ... ok
test_flush_events (testfdb.TestEvents) ... ok
test_multiple_events (testfdb.TestEvents) ... ok
test_one_event (testfdb.TestEvents) ... ok
test_insert_blob (testfdb.TestInsertData) ... ok
test_insert_char_varchar (testfdb.TestInsertData) ... ok
test_insert_datetime (testfdb.TestInsertData) ... ok
test_insert_float_double (testfdb.TestInsertData) ... ok
test_insert_integers (testfdb.TestInsertData) ... ok
test_insert_numeric_decimal (testfdb.TestInsertData) ... ok
test_insert_returning (testfdb.TestInsertData) ... ok
test_basic (testfdb.TestPreparedStatement) ... ok
test_execution (testfdb.TestPreparedStatement) ... ok
test_get_plan (testfdb.TestPreparedStatement) ... ok
test_wrong_cursor (testfdb.TestPreparedStatement) ... ok
test_attach (testfdb.TestServices) ... ok
test_query (testfdb.TestServices) ... FAIL
test_running (testfdb.TestServices) ... ok
test_wait (testfdb.TestServices) ... ok
test_activateShadowFile (testfdb.TestServices2) ... ok
test_backup (testfdb.TestServices2) ... ok
test_getLimboTransactionIDs (testfdb.TestServices2) ... ok
test_getStatistics (testfdb.TestServices2) ... ok
test_getUsers (testfdb.TestServices2) ... ok
test_log (testfdb.TestServices2) ... ok
test_manage_user (testfdb.TestServices2) ... ok
test_nbackup (testfdb.TestServices2) ... ok
test_nrestore (testfdb.TestServices2) ... ok
test_repair (testfdb.TestServices2) ... ok
test_restore (testfdb.TestServices2) ... ERROR
test_setAccessMode (testfdb.TestServices2) ... ok
test_setDefaultPageBuffers (testfdb.TestServices2) ... ok
test_setSQLDialect (testfdb.TestServices2) ... ok
test_setShouldReservePageSpace (testfdb.TestServices2) ... ok
test_setSweepInterval (testfdb.TestServices2) ... ok
test_setWriteMode (testfdb.TestServices2) ... ok
test_shutdown_bringOnline (testfdb.TestServices2) ... ok
test_sweep (testfdb.TestServices2) ... ok
test_trace (testfdb.TestServices2) ... ok
test_callproc (testfdb.TestStoredProc) ... ok
testBlobBasic (testfdb.TestStreamBLOBs) ... ok
testBlobExtended (testfdb.TestStreamBLOBs) ... ok
test_context_manager (testfdb.TestTransaction) ... ok
test_cursor (testfdb.TestTransaction) ... ok
test_fetch_after_commit (testfdb.TestTransaction) ... ok
test_fetch_after_rollback (testfdb.TestTransaction) ... ok
test_savepoint (testfdb.TestTransaction) ... ok
test_tpb (testfdb.TestTransaction) ... ok
test_transaction_info (testfdb.TestTransaction) ... ok

======================================================================
ERROR: test_restore (testfdb.TestServices2)
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/home/vagrant/fdb/test/testfdb.py", line 1214, in test_restore
     report = self.svc.readlines()
   File "/home/vagrant/fdb/fdb/services.py", line 546, in readlines
     return [line for line in self]
   File "/home/vagrant/fdb/fdb/services.py", line 266, in next
     line = self.readline()
   File "/home/vagrant/fdb/fdb/services.py", line 537, in readline
     return self.__fetchline()
   File "/home/vagrant/fdb/fdb/services.py", line 307, in __fetchline
     self.__read_buffer()
   File "/home/vagrant/fdb/fdb/services.py", line 292, in __read_buffer
     "Services/isc_service_query:")
DatabaseError: ('Services/isc_service_query:\n- SQLCODE: -901\n- could 
not drop database /home/vagrant/fdb/test/test_employee.fdb (database 
might be in use)', -901, 336330985)

======================================================================
FAIL: test_query (testfdb.TestServices)
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/home/vagrant/fdb/test/testfdb.py", line 1074, in test_query
     assert len(x) == 2
AssertionError

----------------------------------------------------------------------
XML: nosetests.xml
----------------------------------------------------------------------
Ran 83 tests in 20.632s

FAILED (errors=1, failures=1)
Exception fdb.fbcore.DatabaseError: DatabaseError('Error while closing 
SQL statement:\n- SQLCODE: -901\n- invalid statement handle', -901, 
335544485) in <bound method Cursor.__del__ of <fdb.fbcore.Cursor object 
at 0x2ad2f50>> ignored
Exception fdb.fbcore.DatabaseError: DatabaseError('Error while closing 
SQL statement:\n- SQLCODE: -901\n- invalid statement handle', -901, 
335544485) in <bound method Cursor.__del__ of <fdb.fbcore.Cursor object 
at 0x2ad2c10>> ignored
Exception fdb.fbcore.DatabaseError: DatabaseError('Error while closing 
SQL statement:\n- SQLCODE: -901\n- invalid statement handle', -901, 
335544485) in <bound method Cursor.__del__ of <fdb.fbcore.Cursor object 
at 0x2ad28d0>> ignored
Exception fdb.fbcore.DatabaseError: DatabaseError('Error while closing 
SQL statement:\n- SQLCODE: -901\n- invalid statement handle', -901, 
335544485) in <bound method Cursor.__del__ of <fdb.fbcore.Cursor object 
at 0x2ad2810>> ignored
Exception fdb.fbcore.DatabaseError: DatabaseError('Error while closing 
SQL statement:\n- SQLCODE: -901\n- invalid statement handle', -901, 
335544485) in <bound method Cursor.__del__ of <fdb.fbcore.Cursor object 
at 0x2cde1d0>> ignored




------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscribe-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Re: Huge Firebird memory usage using FDB

Pavel Cisar — 2013-05-02T09:59:11

Philippe,

I never tested FDB with 3.0, so it may fail. Strange it fails with 
fbtest on 2.5, as it passed all FDB tests on 2.5 (I never commit changes 
if fdb tests fail). I'll investigate.

Pavel


------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscribe-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Re: Huge Firebird memory usage using FDB

Philippe Makowski — 2013-05-02T07:28:51

Le 01/05/13 10:09, Pavel Cisar a écrit :
seems that there is a problem with Firebird Trunk
see (at the end of 
http://ci.ibphoenix.fr/job/Firebird_trunk_Arch_Linux/Architecture=Default/23/console) 
:

test_rowcount (testfdb.TestCursor) ... FAIL
pure virtual method called
terminate called without an active exception
/home/vagrant/buildfirebirdtrunk.sh: line 22: 16965 Aborted 
     (core dumped) nosetests2 --with-xunit --verbose 
--exclude=test_limbo_transactions



And whith Firebird 2.5

we get a lot of :
Exception fdb.fbcore.DatabaseError: DatabaseError('Error while closing 
SQL statement:\n- SQLCODE: -901\n- invalid statement handle', -901, 
335544485) in <bound method Cursor.__del__ of <fdb.fbcore.Cursor object 
at 0x2ad2f50>> ignored


during tests

see : 
http://ci.ibphoenix.fr/job/firebird25_Linux_64/Architecture=Classic,System=Firebird_Icu/64/console






------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscribe-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Re: Huge Firebird memory usage using FDB

Pavel Cisar — 2013-05-01T08:09:22

Hi,

Good catch, thanks. Logged as PYFB-34 and fix committed. Please checkout 
the the latest version from our Subversion at:

http://svn.code.sf.net/p/firebird/code/python/fdb/trunk

and let me know whether it solved your problem.

best regards
Pavel Cisar
IBPhoenix

Dne 30.4.2013 12:20, josef_gschwendtner napsal(a):


------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscribe-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Huge Firebird memory usage using FDB

josef_gschwendtner — 2013-04-30T10:20:14

Hi,

We got a problem when we use the Firebird driver for Python (fdb) inside
a connection pool. If a single connection is used for the execution of

many queries and each query uses a separate cursor instance we observed
an increasing memory usage on the server side (the fbserver process).

Only when the connection gets closed the memory usage of the fbserver
process gets back to a "normal" value.

An equal situation can be simulated with this short python script.

import fdb

con = fdb.connect(dsn='localhost/3050:C:\\Development\\DB\\mydb.fdb',
user='sysdba',
password='masterkey')

for i in range(10000):

cur = con.cursor()

cur.execute( "select * from RDB$Relations")

for row in cur.fetchall():

print i, row[0]

con.commit()

t = raw_input('Press Key')

con.close()

print "Finished"

When the loop has finished and before closing the connection the
fbserver process uses almost one GB of memory. This is independent of
the table being used.

RDB$Relations is only used for demonstration. The same happens also with
every other table. After closing the connection everything is fine on
the server

(fbserver memory usage is back to 7 MB).

We found out that releasing the Cursor instance does not permanently
free the prepared statement on the server side. During the destructor of
the Cursor instance

the internal managed PreparedStatement instances will also be released.
This will call the method _close from PreparedStatement. Inside the
_close Method the

release of the server side statement resource will only be triggered if
there is an open connection. But to get this connection the
PreparedStatement instance uses the

cursor Attribute which is a weak reference to the Cursor instance. But
this Cursor instance is no longer available because the program
execution is inside the

destructor (__del__) of this cursor. The weak reference is dead.

It seems that Cursor and PreparedStatement instances are released on the
client side but the following statement from the _close method of
PreparedStatement doesn't get executed.

api.isc_dsql_free_statement(self._isc_status, stmt_handle,

ibase.DSQL_drop)

But this statement is required for releasing the server side resources
and also the memory used for these resources.

Thanks for your help,

Josef

WG: FDB Bug Report

Josef Gschwendtner — 2013-04-30T09:28:04

Von: Markus Holzner
Gesendet: Dienstag, 30. April 2013 10:30
An: Josef Gschwendtner
Betreff: FDB Bug Report

We got a problem when we use the Firebird driver for Python (fdb) inside a connection pool. If a single connection is used for the execution of

many queries and each query uses a separate cursor instance we observed an increasing memory usage on the server side (the fbserver process).

Only when the connection gets closed the memory usage of the fbserver process gets back to a "normal" value.

An equal situation can be simulated with this short python script.

import fdb

con = fdb.connect(dsn='localhost/3050:C:\\Development\\DB\\mydb.fdb', user='sysdba',
password='masterkey')

for i in range(10000):

cur = con.cursor()

cur.execute( "select * from RDB$Relations")

for row in cur.fetchall():

print i, row[0]

con.commit()

t = raw_input('Press Key')

con.close()

print "Finished"

When the loop has finished and before closing the connection the fbserver process uses almost one GB of memory. This is independent of the table being used.

RDB$Relations is only used for demonstration. The same happens also with every other table. After closing the connection everything is fine on the server

(fbserver memory usage is back to 7 MB).

We found out that releasing the Cursor instance does not permanently free the prepared statement on the server side. During the destructor of the Cursor instance

the internal managed PreparedStatement instances will also be released. This will call the method _close from PreparedStatement. Inside the _close Method the

release of the server side statement resource will only be triggered if there is an open connection. But to get this connection the PreparedStatement instance uses the

cursor Attribute which is a weak reference to the Cursor instance. But this Cursor instance is no longer available because the program execution is inside the

destructor (__del__) of this cursor. The weak reference is dead.

It seems that Cursor and PreparedStatement instances are released on the client side but the following statement from the _close method of PreparedStatement doesn't get executed.

api.isc_dsql_free_statement(self._isc_status, stmt_handle,

ibase.DSQL_drop)

But this statement is required for releasing the server side resources and also the memory used for these resources.

-------------------------------------------------------------

Quattro-Soft GmbH

Traunsteiner Straße 29, 83334 Inzell

Tel. 08665/1708, Fax 08665/1710

Markus.Holzner-a/aOGbxzuYeapf4nk6Pq9w< at >public.gmane.org <mailto:Markus.Holzner-a/aOGbxzuYeapf4nk6Pq9w< at >public.gmane.org>

www.quattro-soft.de <http://www.quattro-soft.de>

Amtsgericht Traunstein HR B 12426

Geschäftsführer: Richard Hütter

-------------------------------------------------------------

FDB release 1.1

Pavel Cisar — 2013-04-09T10:55:58

FDB release 1.1 is out:
http://pypi.python.org/pypi/fdb

New Features:

- Context Manager for transactions.

Bugs Fixed:

- http://tracker.firebirdsql.org/browse/PYFB-30

best regards
Pavel Cisar
IBPhoenix


------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscribe-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Re: reading blob from table

Pavel Cisar — 2013-04-08T10:12:50

Hi,

Bug confirmed and fixed. Unfortunately our tracker is down at the 
moment, so I can't make an entry, commit changes to Subversion, close it 
and make a bugfix 1.0.1 release. So meanwhile you can use this hotfix 
release:

http://ibphoenix.cz/download/fdb-1.0.1.dev.tar.gz

It was tested (unit test added to test suite), so it shouldn't break 
anything else.

best regards
Pavel Cisar
IBPhoenix


------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscribe-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Re: reading blob from table

Hamish Moffatt — 2013-04-08T13:48:52

Thanks Pavel, I confirm that my problem is fixed with the above release.


regards,
Hamish

Re: reading blob from table

Hamish Moffatt — 2013-04-08T04:16:33

Hi,

Thanks Pavel. I tried with charset NONE set when opening the connection 
but the behaviour did not change.


regards
Hamish

On 8/04/2013 2:08 AM, Pavel Cisar wrote:

Re: reading blob from table

Hamish Moffatt — 2013-04-08T07:05:33

I'm not usually, but was experimenting with Pavel's suggestion that the 
blob truncation I saw might be a character set issue. I am now back to 
my original definition of the column as simply "BLOB", the connection 
character set as UTF8 and I'm able to fetch the blobs successfully in 
materialized mode using kinterbasdb.

thanks
Hamish

Re: reading blob from table

Helen Borrie — 2013-04-08T07:02:45



Why are you applying a character set attribute at all, to binary blobs?



Helen Borrie, Support Consultant, IBPhoenix (Pacific)
Author of "The Firebird Book" and "The Firebird Book Second Edition"
http://www.firebird-books.net
__________________________________________________________________ 



------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/firebird-python/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/firebird-python/join
    (Yahoo! ID required)

<*> To change settings via email:
    firebird-python-digest-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org 
    firebird-python-fullfeatured-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> To unsubscribe from this group, send an email to:
    firebird-python-unsubscribe-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Re: reading blob from table

Hamish Moffatt — 2013-04-08T05:28:03

I found that it works with kinterbasdb for UTF8 or NONE connection type.

I tried changing the column definition to "blob binary character set 
none" and some other things but none of them helped with fdb (1.0). For 
some combinations of the column type and the connection character set I 
got Python utf8 decode errors, others just truncated the data. Also 
tried with a Firebird 2.5 server (I was using 2.1) without any success.

Hamish

On 8/04/2013 2:16 PM, Hamish Moffatt wrote: