http://permalink.gmane.org/gmane.comp.python.durus hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://permalink.gmane.org/gmane.comp.python.durus/998 <pre> As always, thanks for Durus. It's a great (and under-recognized) database. We may end up patching Durus 3.7 to avoid having to upgrade everything for the moment. If anybody is interested in that patch, let me know. Dave On Aug 17, 2011, at 7:38 AM, David Binger wrote: </pre> David Hess 2011-08-17T13:38:16 http://permalink.gmane.org/gmane.comp.python.durus/997 <pre>Good news. The issue discussed in this thread has now been diagnosed and solved by David Hess. The problem turned out to be this: when a persistent instance becomes temporarily unreachable during the period that an incremental pack is in progress, the object's record might be left out of the newly packed file. All subsequent attempts to access that object, including subsequent packs, would get an exception. This rare trap has existed for years: for all versions that supported incremental packing. The patch will be included in the next release (expected soon). Thanks, David, for your good work on this. </pre> David Binger 2011-08-17T12:38:19 http://permalink.gmane.org/gmane.comp.python.durus/996 <pre> On Aug 15, 2011, at 9:55 AM, David Binger wrote: Agree. This particular BTree is an index of all instances of a particular Persistent subclass. My repair involved using a monkey patched get_crawler that could handle an exception in __setstate__ and keep going. I then walked through the entire database finding all instances of my particular class that didn't blow up when loaded and then loading them into a brand new btree. I then deleted the old btree and packed the resulting database. Based on the debugging I added to the monkey patched get_crawler, the magnitude of the problem reported there matched the amount of change that went on during the packing. Didn't confirm yet that it was exactly the changes that occurred during the packing. We are definitely one thread and one process for this FileStorage. We do use Twisted so there's a lot of asynchronous things going on and interleaved with each crank forward of the packing generator. We commit every 5 seconds rather than on a transactional basis. We n</pre> David Hess 2011-08-15T15:13:08 http://permalink.gmane.org/gmane.comp.python.durus/995 <pre> On Aug 15, 2011, at 10:50 AM, David Hess wrote: OK. I haven't studied the issue, but I would guess that it is safe. If we have that, then we can solve it for sure. _______________________________________________ Durus-users mailing list Durus-users&lt; at &gt;mems-exchange.org http://mail.mems-exchange.org/mailman/listinfo/durus-users </pre> David Binger 2011-08-15T14:59:32 http://permalink.gmane.org/gmane.comp.python.durus/994 <pre> Yes, the oid is now called "durus_id". I just meant that the BNode class doesn't have any special state management code that could be causing the problem. I think BNode is involved only because that is where your actions happen to be. I would be interested in the details of the repair. I don't remember the details of how you are using FileStorage. Do you have just one thread/process using the FileStorage? I hope so. That does seem safe. Is there any code in the cache manager, though, that calls the ghosting function and then takes some action based on the assumption that the ghosting worked? The new code adds self.file.flush(). Without it, certain file systems lose track of the correct end of the file, so a subsequent seek to the end of the file ends up in an incorrect place. In a pack, an offset is written into the header of the file and then there is a seek to the end of the file. It is important for this to work correctly. It could possibly be the reason for the trouble you have seen. </pre> David Binger 2011-08-15T14:55:39 http://permalink.gmane.org/gmane.comp.python.durus/993 <pre> On Aug 15, 2011, at 9:12 AM, David Hess wrote: I should note that this mis-mapping was not just in process memory. It was a permanent change to the database and occurs each time you open it and access an affected object. Thinking about this some more: if that object was worked with during the pack and yet it was ghosted and was trying to reload its state - that implies that not only commits occurred during a pack but also a cache shrink. Is it safe for a cache shrink to occur during a pack? We're going to work on building a test app that can recreate this problem. Dave </pre> David Hess 2011-08-15T14:50:07 http://permalink.gmane.org/gmane.comp.python.durus/992 <pre> On Aug 15, 2011, at 8:50 AM, David Binger wrote: Yes, if the packing implementation was an issue, I would have thought we would just have missing oids (or is that a durus_id now?) - but I also have evidence of an object whose class is BNode trying to load state of a pickle that was clearly associated with an application object. Based on what I did to repair the database, it appears this corruption only affected the items that were touched when the modification to the BTree was made during the pack (interior BNodes and the application objects). It's almost as if there is a race problem during packing having to do with either durus_id allocation or file writing. That's on one particular and isolated class that was not involved in this situation. All it does is ignore the ghosting request from the cache manager (which is safe because we never have conflicts). Here's the implementation we are running from on Durus 3.7: def write(self, s): self.obtain_lock() self.file.write(s) I noti</pre> David Hess 2011-08-15T14:12:18 http://permalink.gmane.org/gmane.comp.python.durus/991 <pre>This sounds like the offset mapping in memory is somehow out of sync with the file. We've never seen this condition, which is obviously bad. I'm concerned about this and trying to figure out how it could occur. It seems unlikely to have anything in particular to do with BTree or BNodes: those structures are implemented in the same way as other persistent objects. I think, from your other message, that you are looking closely at the packing code, and that seems like a good idea. There was a time when you talked about overriding a '_p_' method. Does the code you are running involve any such low-level modifications? In the current code tree, File.write() ends with self.file.flush() under all conditions. Is that the case in the code you are running? On Aug 14, 2011, at 12:44 PM, David Hess wrote: _______________________________________________ Durus-users mailing list Durus-users&lt; at &gt;mems-exchange.org http://mail.mems-exchange.org/mailman/listinfo/durus-users </pre> David Binger 2011-08-15T13:50:25 http://permalink.gmane.org/gmane.comp.python.durus/990 <pre> Yep, that seems like a good fix. Dave On Aug 15, 2011, at 8:18 AM, David Binger wrote: _______________________________________________ Durus-users mailing list Durus-users&lt; at &gt;mems-exchange.org http://mail.mems-exchange.org/mailman/listinfo/durus-users </pre> David Hess 2011-08-15T13:30:41 http://permalink.gmane.org/gmane.comp.python.durus/989 <pre>I think I'd prefer to fix this by having FileStorage.load() raise a DurusKeyError instead of KeyError. Does that seem reasonable to you? On Aug 14, 2011, at 11:37 PM, David Hess wrote: _______________________________________________ Durus-users mailing list Durus-users&lt; at &gt;mems-exchange.org http://mail.mems-exchange.org/mailman/listinfo/durus-users </pre> David Binger 2011-08-15T13:18:13 http://permalink.gmane.org/gmane.comp.python.durus/988 <pre> I think there might be a bug in load_state in connection.py - shouldn't it also catch KeyError to handle the case when working from a local FileStorage instead of a ClientStorage (as the traceback below shows)? Regardless, that's a case that shouldn't have happened in our app. In looking at the Shelf packer in Durus 3.8, it seems a bit more sophisticated than the one in 3.7. I'm guessing that an upgrade to 3.8 is likely to prevent this corruption? Thanks. Dave On Aug 14, 2011, at 11:44 AM, David Hess wrote: ------ David K. Hess 877.343.4947 x114 dhess&lt; at &gt;fishtechnology.com _______________________________________________ Durus-users mailing list Durus-users&lt; at &gt;mems-exchange.org http://mail.mems-exchange.org/mailman/listinfo/durus-users </pre> David Hess 2011-08-15T03:37:17 http://permalink.gmane.org/gmane.comp.python.durus/987 <pre> After some more investigation - this occurred right after live packing the database and changes to this BTree were made and committed during the pack. Afterwards, the oids seem to be jumbled up in this BTree (at least - maybe elsewhere in the database too). Unpickled Persistent objects are not what they should be - interior BNodes are sometimes application classes and stored values are sometimes BNodes rather than application classes. Dave On Aug 14, 2011, at 10:45 AM, David Hess wrote: </pre> David Hess 2011-08-14T16:44:35 http://permalink.gmane.org/gmane.comp.python.durus/986 <pre> Thanks! FYI, after some more diagnosis, it turns out that this particular kind of corruption seems to be correlated with running out of disk space. Dave On Aug 12, 2011, at 2:44 PM, David Binger wrote: ------ David K. Hess 877.343.4947 x114 dhess&lt; at &gt;fishtechnology.com _______________________________________________ Durus-users mailing list Durus-users&lt; at &gt;mems-exchange.org http://mail.mems-exchange.org/mailman/listinfo/durus-users </pre> David Hess 2011-08-12T20:11:27 http://permalink.gmane.org/gmane.comp.python.durus/985 <pre>This change looks good to me. Nice work. On Aug 11, 2011, at 10:37 AM, David Hess wrote: </pre> David Binger 2011-08-12T19:44:33 http://permalink.gmane.org/gmane.comp.python.durus/984 <pre>We have servers that deal with power outages (and dirty power in general) and have ended up with some corrupted durus databases. We've used the normal "repair" feature to handle a lot of the cases but we have another that's occurring on occasion that is not handled by repair. It is failing as this exception: Traceback (most recent call last): File "/usr/local/bin/durus", line 22, in &lt;module&gt; client_main() File "/usr/local/lib/python2.6/dist-packages/durus/client.py", line 108, in client_main options.startup) File "/usr/local/lib/python2.6/dist-packages/durus/client.py", line 35, in interactive_client storage = FileStorage(file, readonly=readonly, repair=repair) File "/usr/local/lib/python2.6/dist-packages/durus/file_storage.py", line 73, in __init__ self.shelf = Shelf(filename, readonly=readonly, repair=repair) File "/usr/local/lib/python2.6/dist-packages/durus/shelf.py", line 91, in __init__ self.file, repair=repair) File "/usr/local/lib/python2.6/dist-packages/durus/shelf.p</pre> David Hess 2011-08-11T14:37:31 http://permalink.gmane.org/gmane.comp.python.durus/983 <pre> NB: I don't have to worry about data invalidation due to aborts and conflicts - there's only one writer to this database so by design those cannot occur. So, my question is: if I have a long lasting reference to a PersistentObject instance, is there a safe way for that object to veto being ghosted by the cache manager? It looks like reimplementing _p_set_status_ghost as a "pass" might work but there may be some not so obvious side-effects. Dave _______________________________________________ Durus-users mailing list Durus-users&lt; at &gt;mems-exchange.org http://mail.mems-exchange.org/mailman/listinfo/durus-users </pre> David Hess 2011-06-03T18:17:38 http://permalink.gmane.org/gmane.comp.python.durus/982 <pre>-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 16/01/11 22:44, Neil Schemenauer wrote: You have the weak reference callback, for instance. It is not different to current cache object count. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea&lt; at &gt;jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea&lt; at &gt;jabber.org _/_/ _/_/ _/_/_/_/_/ .. _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTX+3Y5lgi5GaxT1NAQKZnAP+LJM0JeSnjLiAa0pxx+zRaziPh2d2S0ss n5daZmogs9IzjWs2qp0p8rZnymACniJWDb2GEDLyYfdDFvc73OP/R1hh2ngmfHVI 8c5+etJwWwgsoOdvxJgx1K2dM+wskBN31Q+GLnFcIw8Ve5rPU83HEoyM5ASAYaO5 KDtOAzdYw</pre> Jesus Cea 2011-03-15T19:00:51 http://permalink.gmane.org/gmane.comp.python.durus/981 <pre> I played with this idea a little. It seems like the bookkeeping will get pretty complicated. Iterating over all objects in the cache to find the current size does not seem feasible, for performance reasons. Therefore, the cache would have to keep track of the total size as objects are added and removed. The fact that objects are weakly referenced by the cache futher complicates things. Maybe I'm missing a simple implementation but it doesn't appear to be worth the effort, IMHO. Regards, Neil </pre> Neil Schemenauer 2011-01-16T21:44:14 http://permalink.gmane.org/gmane.comp.python.durus/980 <pre>-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Durus objects, when in RAM, could keep the object size in a (volatile, not stored in disk) attribute. This attribute can be generated when loading the pickle from disk (you directly have the size), or when storing the object in the disk (you have to create the pickle, so you have the size too). In fact, this size bookkeeping could be managed in a separate internal dictionary, don't have to be inside the object. I have objects of very dissimilar sizes in my storage, so current cache control (that is, object count) is not representative of actual memory usage. I have objects 50 bytes long, and 60Kbytes long :-(. I would suggest to change the cache code to control cache size, instead of object count. Comments?. PS: I could consider patching this myself, if durus developers are interested but not spare time. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea&lt; at &gt;jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber </pre> Jesus Cea 2010-09-24T17:57:04 http://permalink.gmane.org/gmane.comp.python.durus/979 <pre>-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 This is a draft I wrote three years ago. Maybe can be useful. I would like durus to be more "community driven". """ This is a preview version. As ever, I'm ready to help to implement this, if you ask. * "Factorize" the durus server socket management (in particular, socket creation for incomming connections and socket "select") to be able to reuse the server code in other communication media, like shared memory, intraprocess queues or mmaped files. * "connection" objects should provide a method to query how much accumulated idle time was spent waiting for the storage. * A precompiled DURUS distribution for Windows users. Please!. Durus programmers can't do it. Any other gentle soul able to provide this service?. Durus deserves it!. * When getting a "late conflict", the client should get the *real* OID list of the conflicting objects. Since an storage implementation could choose to only report conflicts in a "late" way, this check should be done even for </pre> Jesus Cea 2010-09-23T17:02:20 http://permalink.gmane.org/gmane.comp.python.durus/978 <pre>While waiting for a fairly big Durus DB to pack today, I remembered an optimization idea I had but never had time to play with. The layout of objects in the disk file makes a difference for performance. For best packing performance, the reads should be sequential since that would be most efficient for IO scheduling. Also, it would be nice if packing moved objects around a little as possible in order to speed up rsyncing of DBs. I haven't had time to dig into it, but I suspect the relevant logic is in file_storage.py:gen_oid_record(). It looks like references are found using a depth first search. Perhaps it would be better to use breath-first (e.g. using collection.deque for efficiency). One way to approach this would be to have file_storage.py log file offsets and examine how their distribution is affects by different packing algorithms. It's possible that optimizing the layout for packing would slow down normal access. Anyhow, I thought it was an interesting idea. Neil </pre> Neil Schemenauer 2010-07-15T01:47:48 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.comp.python.durus