gmane.comp.web.zope.zodb

Re: RelStorage breaks history tab - solution?

Chris Withers — 2012-05-23T18:05:16

Hi Shane,

On 23/05/2012 16:27, Chris Withers wrote:

Upon investigation, this turns out to be the history method itself! :-)

Switching to the following patch also solves the problem:

diff --git a/relstorage/storage.py b/relstorage/storage.py
index 2d592d3..3fa792b 100644
--- a/relstorage/storage.py
+++ b/relstorage/storage.py
< at >< at > -1053,6 +1053,7 < at >< at > class RelStorage(
                          break
              return res
          finally:
+            self._load_conn.rollback()
              self._lock_release()

      def undo(self, transaction_id, transaction):

However, that raised more questions for me:

- How does RelStorage usually rollback the _load_conn in a non-write 
transaction?

- Why doesn't this kick in when viewing the history method?

My guess as to why 
ZODB.tests.HistoryStorage:HistoryStorage.checkSimpleHistory passes is 
that everything is done with the same thread/connection, and so a 
consistent view of the data is seen.

More confusing: having dropped the zserver-threads down to one and from 
looking at the results of:

select * from pg_catalog.pg_stat_activity where datname='myrelstorage';

...it appears that there's:

- one connection for reads

- one connection for writes, which is opened when first used.

Okay, as expected.

- one connection for viewing history.

I can't see where/why this final connection is opened...

Shane, little help?

Chris

Re: RelStorage breaks history tab

Shane Hathaway — 2012-05-23T17:37:45

That is what RelStorage is designed to do when you set poll-interval. 
Does the bug go away when you set poll-interval to 0?

Shane
_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev< at >zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: RelStorage breaks history tab

Chris Withers — 2012-05-23T15:27:48

Hi Shane,

On 22/05/2012 06:37, Chris Withers wrote:

Okay, the issue appears to be that, in some circumstances, RelStorage is 
leaving the read connection with an open transaction that isn't rolled back.

I couldn't find the source of this, but in the meantime, I less 
heavyweight hack is:

diff --git a/relstorage/storage.py b/relstorage/storage.py
index 2d592d3..459469b 100644
--- a/relstorage/storage.py
+++ b/relstorage/storage.py
< at >< at > -1025,6 +1025,7 < at >< at > class RelStorage(
          self._lock_acquire()
          try:
              self._before_load()
+            self._load_conn.rollback()
              cursor = self._load_cursor
              oid_int = u64(oid)
              try:

However, that open transaction is likely to cause other problems (I'm 
fairly certain I've seen other effects of this) so do you have any idea 
where to look for it?

cheers,

Chris

RelStorage breaks history tab

Chris Withers — 2012-05-22T05:37:20

Hi All,

I've been chatting with Shane about this off-list but wanted to see if 
anyone else has experience of this and has found a solution.

So, the problem is that, with a Zope 2 app server, any revisions made to 
an object (say, a Page Template) don't show in the history tab of that 
object *until* the app server is restarted.

I've verified this on both MySQL and Postgres with trunk of RelStorage 
as well as 1.5.2 and 1.4.2. It appears to be some kind of cursor re-use 
/ relational database transaction isolation bug given that the following 
hack fixes the problem:

--- a/relstorage/storage.py
+++ b/relstorage/storage.py
< at >< at > -1025,7 +1025,7 < at >< at > class RelStorage(
          self._lock_acquire()
          try:
              self._before_load()
-            cursor = self._load_cursor
+            conn, cursor = self._adapter.connmanager.open()
              oid_int = u64(oid)
              try:
                  rows = self._adapter.dbiter.iter_object_history(
< at >< at > -1053,6 +1053,7 < at >< at > class RelStorage(
                          break
              return res
          finally:
+            self._adapter.connmanager.close(conn, cursor)
              self._lock_release()

It seems a little heavyweight to have to open a whole new connection 
each time someone views a history tab, though :-/

A couple of surprising things:

- ZODB.tests.HistoryStorage:HistoryStorage.checkSimpleHistory passes. 
I'd expect this to fail, so I wonder what it's doing differently?

- The read connection is obviously getting the transaction's data, 
otherwise the wrong version of the object would be rendered, right?

Any ideas very gratefully received...

cheers,

Chris

Re: A little help in configuring a better solution forZEO deployment

Alan Runyan — 2012-05-18T15:07:37

Any reason for 7 threads?  Try 1 thread?  Is it a classical web application? Or
you creating threads in your application?


You can change the tmp location in Python by setting TMPDIR environment variable
for your process.

I would advise against persistent disk cache and increase the ZODB
cache-size. If you
are using latest ZODB you can use the cache-size-bytes which mostly
works.  You can
set it so zodb cache for each process will not exceed, say, 512MB.



If your ZEO server process is not bottlenecking -  it is your application.


dont know your application but it is doubtful that ZODB is your bottleneck.

A little help in configuring a better solution for ZEOdeployment

Войнаровський Тарас — 2012-05-18T13:53:00

Good day comunity.

My project is a ZOPE3/BlueBream project, that runs mainly of ZODB using
ZEO. I use wsgi to get multiple instances(4) of my project and connect to
ZEO Server from them.
The database is around 3 GB in unpacked state and around 800 MB in packed.

I use 4 processes with 7 threads each for serving data

WSGIDaemonProcess doba.ua user=www group=www processes=4 threads=7
maximum-requests=2000000

I have little space on /tmp partition, so the cache of ZEO is only 40 MB

    pool-size 30
  <zeoclient>
    server localhost:8200
    storage 1
    # ZEO client cache, in bytes
    cache-size 40MB
    # Uncomment to have a persistent disk cache
    #client zeo1
  </zeoclient>

The server has those specs:
8GB ram. 8 CPU 3.2 Ghz.  Raid-2 in a mirror mode.

The load of instances:
1 GB per instance - 8 GB total. Other apps consum up to 1.5 GB. So there is
2.5 GB in inactive mode all the time.

The problem, that I have:
The server has a Raid-2 in a mirror mode, so I only have 1 hard drive,
which contain all my data. After a while of serving data the load gets
highter and this hard drive just starts having 100% of disk usage. Then
everything starts to fall. I have problems accessing the server and even
restarting the project.
I can't find any ther explanation, that ZEO does not handle the load well
enough.

Does someone see any way to ease the load?
_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev< at >zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: Relstorage and MSQL

Dylan Jay — 2012-05-16T13:51:04


as would we. But it looks like there isn't someone with the time and  
licenses at the moment to make this happen so we'll go ahead without  
sql server support.
but count this as a vote of support if anyone wants to give it a go.


_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev< at >zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: Relstorage and MSQL

Alan Runyan — 2012-05-15T17:24:09


I would agree.


This really depends on usage patterns and capacity.  I would recommend keeping
the BLOBs on the filesystem.


If you can depend on mxODBC then I do not believe this is a huge problem.
I would be skeptical of pyodbc stability/performance - its worked ok for us but
we dont have anything 24x7 running with it.

As someone said earlier it is not difficult just very time consuming to test.
I would be more than happy to help test.  Some more thoughts:

  - If you run mxODBC you will have much less adoption/testing by the community.
  due to license and pain to install mxODBC.

  - pyodbc i would skeptical of and test.  you would get the most
usage using this
  driver.

  - pywin32 is another candidate.  it should work just fine but you
may need to watch
  for scaling issues (you may have to add some smarts the mssql storage)

I would love to see this support added.

git clone of relstorage

Chris Withers — 2012-05-15T12:30:42

Hi All,

In case anyone else is interested, I've git svn clone'd 
svn://svn.zope.org/repos/main/relstorage and pushed the result up here:

https://github.com/cjw296/relstorage

It's only a master branch, sine git svn clone does svn branches and 
separate remotes by default and I didn't want to do my manual hack-fix 
steps since Shane likely wants to stay with svn:

cp -Rf .git/refs/remotes/tags/* .git/refs/tags/
rm -rf .git/refs/remotes/tags
cp -Rf .git/refs/remotes/* .git/refs/heads/
rm -rf .git/refs/remotes

cheers,

Chris

Re: Relstorage and MSQL

Shane Hathaway — 2012-05-09T02:47:41

I don't have enough cycles now to do it myself, but I will happily 
assist anyone who wants to take on this project.  The code already has 3 
mature implementations to serve as a reference.

I would plan to spend at least as much time on testing as on 
development.  Like any database, ZODB storage bugs are often expensive 
to debug, so you really need a reliable storage before you can safely 
build layers on top of it.

Shane
_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev< at >zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: Relstorage and MSQL

Dylan Jay — 2012-05-09T01:37:48


sql server 2008. not sure about 64/32bit.


Windows 2008 server standard. Python 2.6/2.7


I'm not sure. I think living without clustering and memcached for the  
moment is fine.


I don't know enough to answer this but for our purposes support SQL  
Server in a basic capacity so Plone runs ok is enough to start with.


Our need is to 'state' support for SQL server to our client. They have  
SQL server licenses and they'd prefer to use them. The job is for 3  
plone sites, public, extranet and intranet but with no estimate yet of  
the data size. Availability concerns are an issue.
In the case where the work goes ahead and if performance is an issue  
then we;d be in a better position to either implement greater support  
for sqlserver or recommend they switch to a more mature relstorage  
option. Hope that helps put things in context.



_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev< at >zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: Relstorage and MSQL

Hanno Schlichting — 2012-05-09T01:26:41

It might be helpful if you could add more constraints.

Which version of SQL server do you want to support? Just 2008 or 2012 or some Express Edition? Only 64bit servers and clients or 32bit?

Are all clients also on Windows, which OS version, which versions of Python?

What clustering options are you interested in if any? Do you want memcached support?

And what about blob storage? Are blobs inside the DB in little chunks enough, do you want them on the filesystem via shared network drive or new efficient filestream support in SQL server 2008+

The fewer variables you have, the easier an estimate should be. And just stating something like the expected data size, number of clients and availability concerns might help.

Hanno
_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev< at >zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Relstorage and MSQL

Dylan Jay — 2012-05-09T00:34:22

Hi,

Previous threads on this subject didn't seem to go anywhere.

I know it's not all about money but if we were to sponsor development  
of microsoft sql server support for relstorage, is there someone who  
knows how and has an estimated cost and available time?


---
Dylan Jay
Technical Solutions Manager
PretaWeb: Multisite Performance Support
P: +612 80819071 | M: +61421477460 | twitter.com/djay75 | linkedin.com/ 
in/djay75

_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev< at >zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: all webserver threads blocking on db.open()

Claudiu Saftoiu — 2012-05-07T17:14:18


Thanks, I'll check out the Repoze.BFG IRC channel as well.

From what I can tell you are dealing with hung requests. I'd look at

The problem from my previous email was indeed hung requests. However, the
stack trace for those looked different:

Thread 140605868680960:
  File "/usr/lib/python2.6/threading.py", line 504, in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
line 878, in worker_thread_callback
    runnable()
  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
line 1052, in <lambda>
    lambda: self.process_request_in_thread(request, client_address))
  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
line 1068, in process_request_in_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.6/SocketServer.py", line 322, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.6/SocketServer.py", line 617, in __init__
    self.handle()
  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
line 442, in handle
    BaseHTTPRequestHandler.handle(self)
  File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
    self.handle_one_request()
  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
line 437, in handle_one_request
    self.wsgi_execute()
  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
line 287, in wsgi_execute
    self.wsgi_start_response)
  File
"/home/tsa/env/lib/python2.6/site-packages/repoze/zodbconn/connector.py",
line 21, in __call__
    result = self.next_app(environ, start_response)
  File
"/home/tsa/env/lib/python2.6/site-packages/repoze/zodbconn/cachecleanup.py",
line 25, in __call__
    return self.next_app(environ, start_response)
  File
"/home/tsa/env/lib/python2.6/site-packages/repoze/retry/__init__.py", line
65, in __call__
    chunk = original_wsgi_input.read(rest)
  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
line 474, in read
    data = self.file.read(length)
  File "/usr/lib/python2.6/socket.py", line 377, in read
    data = self._sock.recv(left)

Note the line it blocks on is "self._sock.recv(left)", well after the
response started.
In the trace I just provided, the block was on opening the DB connection
*at the start of the request*:

  ...
*  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
line 287, in wsgi_execute*
*    self.wsgi_start_response)*
  File
"/home/tsa/env/lib/python2.6/site-packages/repoze/zodbconn/connector.py",
line 18, in __call__
    conn = self.db.open()
*  File "/home/tsa/env/lib/python2.6/site-packages/ZODB/DB.py", line 729,
in open*
*    self._a()*
  File "/usr/lib/python2.6/threading.py", line 123, in acquire
    rc = self.__block.acquire(blocking)

Why would the database start blocking on opening a new database connection?
The
issue does indeed seem to be with ZODB.

Thanks,
- Claudiu
_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev< at >zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: all webserver threads blocking on db.open()

Hanno Schlichting — 2012-05-07T16:13:42

I think you might get better help on one of the Pyramid support channels.

Your problems all seem to be related to configuring a web server in
production mode, rather than database issues.

From what I can tell you are dealing with hung requests. I'd look at
either the paster configuration options for anything related to
timeouts, thread pools, handling of incomplete requests and so on. Or
use a more production quality web server like Apache (mod_wsgi), Nginx
(gevent/gunicon) which likely has better default configuration values
for these things.

Hanno

On Mon, May 7, 2012 at 5:38 PM, Claudiu Saftoiu <csaftoiu< at >gmail.com> wrote:
_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev< at >zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

all webserver threads blocking on db.open()

Claudiu Saftoiu — 2012-05-07T15:38:21

Hello all,

I'm using Repoze.BFG, with paster to launch the webserver. This is a
similar issue to the one I emailed about before titled
"server stops handling requests - nowhere near 100% CPU or Memory
used"

The situation is the same. I used z3c.deadlockdebugger , and what
I notice is that, when the server is blocked, there are about 100
threads running (as opposed to 15 or so when the server has
just started), and all their stack traces look like this:

Thread 140269004887808:
  File "/usr/lib/python2.6/threading.py", line 504, in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
line 878, in worker_thread_callback
    runnable()
  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
line 1052, in <lambda>
    lambda: self.process_request_in_thread(request, client_address))
  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
line 1068, in process_request_in_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.6/SocketServer.py", line 322, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.6/SocketServer.py", line 617, in __init__
    self.handle()
  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
line 442, in handle
    BaseHTTPRequestHandler.handle(self)
  File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
    self.handle_one_request()
  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
line 437, in handle_one_request
    self.wsgi_execute()
  File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
line 287, in wsgi_execute
    self.wsgi_start_response)
  File
"/home/tsa/env/lib/python2.6/site-packages/repoze/zodbconn/connector.py",
line 18, in __call__
    conn = self.db.open()
  File "/home/tsa/env/lib/python2.6/site-packages/ZODB/DB.py", line 729, in
open
    self._a()
  File "/usr/lib/python2.6/threading.py", line 123, in acquire
    rc = self.__block.acquire(blocking)

The server gets to a blocked state every 24 hours or so. Simply restarting
the webserver works fine; however, i'd like to know what the problem is so
restarting won't be necessary, and to prevent it from getting worse. Any
ideas/
suggestions?

Thanks,
- Claudiu
_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev< at >zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: ZODB-Dev Digest, Vol 110, Issue 1

Claudiu Saftoiu — 2012-05-03T18:27:48

That makes sense. I wasn't sure where to get a db object from so here's what
I ended up doing. Let me know if there's a better way.

I use paster to run the webserver, so in run.py I now have:

    GLOBAL_CONFIG = [None]
    def app(global_config, **settings):
        GLOBAL_CONFIG[0] = global_config
        #...

Now in the thread I start to do the processing I have:

    def proc_f(self, i):
        from repoze.zodbconn.uri import db_from_uri
        import run
        global_config = run.GLOBAL_CONFIG[0]
        uri = global_config['zodb_uri']
        db = db_from_uri(uri)

        conn = db.open()
        app_root = conn.root()['app_root']

        while True:
            #wait for request...

            t = conn.transaction_manager.begin()
            #do a bunch of read-only processing, queue results
            conn.abort(t)


I figure since I never write anything to the database in my worker thread I
can
always abort the transaction.

This all seems to work. Anything horribly wrong I'm doing?

Thanks muchly,
- Claudiu
_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev< at >zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: how to get root obj from a new transaction?

Marius Gedminas — 2012-05-02T21:54:52

Right.


Also correct: you should not share instances of persistent objects between
threads.


You need to pass the database object to the thread, and the thread needs
to open a connection (connection = db.open()).  Then connection.root()
will give you the root object (or you could pass OIDs to the thread and
use connection.get(oid) to find the objects you need to work with).

Don't forget to commit or abort the transaction, and also don't forget
that you may need to implement some kind of retry logic if commit()
raises a ConflictError due to conflicting updates.

Marius Gedminas

how to get root obj from a new transaction?

Claudiu Saftoiu — 2012-05-02T16:49:32

Hey all,

I'm using a thread to do some server-side work. The thread will be asked by
different
requests to do the same thing at the same time, so I want the thread to do
all the work
once and return the data to the requests. The problem is that the requests
each have
their own transaction and the thread essentially has none.

I can communicate between the two only using identifiers - not persistent
objects - that
way the thread can process data in a different transaction than the
requests yet still
return a meaningful reply. However, the thread has to start a new
transaction each
time it processes something - which I know how to do:

    while True:
        #wait until asked to do something
        import transaction
        transaction.begin()

However, the thread needs access to the root object in order to turn the
identifiers
gotten from the requests into persistent objects... how would I go about
accessing
the root object in such a circumstance?

Thanks,
- Claudiu
_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev< at >zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: ValueError: Non-zero version length. Versions aren't supported.

Andreas Jung — 2012-04-25T07:52:14

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Mikko Ohtamaa wrote:


I would say that the sauna.reload 0.4.1+2 versions were the primary
reason for the ZODB corruption. It did not happen with 0.3.x versions
and Asko's fixes in 0.4.3 seem to work now.

Andreas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQGUBAEBAgAGBQJPl60uAAoJEADcfz7u4AZjyZ4Lv1GyS6GX8DThVwM3Ffvg+WLK
YeAGpH+ttqgu9r6j4QFlwDV2YKi64kCT6iItTaiuNxhi6SRMc6F+ONuzvaFoCquO
yK+I9eYlC/x0FzIbcZrNQ9lvkIUkOLffvVf0+QxLyg7QQTi/vM3n51nhdJd+sBc3
ECWLwauAGAONvlnCr+qxjZLGe55R8PmKaFGiK3zWtsNjU81HTpWXXwABws2yWIDx
dWVKjypTPe8zmv62tHvZjAlGRsJk9TOKAsIU4qxMH1cq5xarOPML9I1MqY58s3sm
4FPRTbEG+L4PNCQxp6UGNfpdGokCmSHK/HBOwtHGsHvlH6BfG9dLjcqlCw+rAPTT
uaBSIeDXOkfZzc0IfDkwsuJEnMbn9F3/bAHcCUTAr8SHay/j2eI3pebjtM9xTeFl
prHYLF+GUw95uuCsNEvrjdLqS3v05Vzrux+5I/Y3x96pna4VbVyBLb78YOTvEcd2
IuffFL4MLT5+Tw5F1OX7qQKrMXJmF34=
=KQFh
-----END PGP SIGNATURE-----
_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev< at >zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: ValueError: Non-zero version length. Versions aren'tsupported.

Mikko Ohtamaa — 2012-04-25T06:58:14

Hi,

I can confirm this is often caused by corrupted Data.fs. Other potential
causes are bad rsync transfers.

http://collective-docs.readthedocs.org/en/latest/troubleshooting/exceptions.html#valueerror-non-zero-version-length-versions-aren-t-supported

sauna.reload increases your changes of corruption Data.fs as it forcefully
restarts Zope instance very often and makes other bad things happen. I
suggest you run sauna.reload in new ZEO mode (client + db) in which case
the database process stays intact between restarts.

-Mikko

On Wed, Apr 25, 2012 at 07:56, Andreas Jung <lists< at >zopyx.com> wrote: