gmane.comp.python.cython.devel

Re: [Cython] Temp allocation flow

Dag Sverre Seljebotn — 2008-12-01T23:49:00

The best way of getting the context would be searching for disposal_code in the source until you find such a case. My situation is that I don't know how that code got there or who wrote it, I just have to adapt to it when moving the temps to code generation, preferably while making as few other changes at the same time as possible. I can guess though .. take BoolBinopNode as an example. def f(): return [] def g(n): if n == 0: return 5 else: return f() or g(n-1) print g(100000000) Since the "or" can decref the result of f() inside the if-branch evaluating g(..), memory usage for the code in Cython is O(1) rather than O(n). (Not saying that it is not O(1) in Pyrex, I wouldn't know). I'm not 100 percent sure about this but it is what I figure. Dag Sverre Seljebotn -----Original Message----- From: Greg Ewing public.gmane.org> Date: Tuesday, Dec 2, 2008 12:03 am Subject: Re: [Cython] Temp allocation flow To: cython-dev-F/1GfIIGwJsClooVUwDQFiifJy3iqPdu< at >public.gman

Re: [Cython] Recursive structure, typedef

Greg Ewing — 2008-12-01T23:27:52

Not sure about Cython, but the following works in Pyrex: cdef struct foo ctypedef foo foo_t cdef struct foo: foo_t *f i.e. although you can't forward-declare and typedef in one go, you can forward declare the struct and then typedef it.

Re: [Cython] Temp allocation flow

Greg Ewing — 2008-12-01T23:06:14

Maybe you could provide a bit more context still? I'm just wondering why you can't generate the disposal code after the whole if-statement, rather than duplicating it in each branch.

Re: [Cython] Refcount safe-mode?

Dag Sverre Seljebotn — 2008-12-01T18:30:26

Right. It's something to enable while running runtests.py, in order to get more proper regression tests on refcounting than a segfault. (Also leaks are currently not regression tested as far as I know.) It wouldn't help for leaks in end-user code, it would help ensure that Cython itself doesn't introduce leaks. The former would construct a new object which is stored in a global stack (basically a dict of pointers-casted-to-integer to a "refcount that the curren function holds"), and which __Pyx_???REF would record information in. (Hmm. Perhaps it would be better to conditinally define a local variable with a fixed name. But these are details unecesarry to discuss at this point.) The latter would count up everything and make sure that there was exactly zero references (that the function knows that it holds) to everything but the parameter, __pyx_r, which should have exactly one reference (which the function holds). Schemes using the Python ob_refcnt could work as well, but in general I thoug

Re: [Cython] Refcount safe-mode?

Robert Bradshaw — 2008-12-01T18:06:37

This is to ease development, not for end-users, right? I think this could be a good idea, though it's unclear exactly what "__Pyx_PushRefCountFrame" and "__Pyx_PopRefcountFrame" would do. And would __Pyx_DECREF be more useful than running under gdb? - Robert

Re: [Cython] Disabled FlattenInListTransform

Robert Bradshaw — 2008-12-01T18:01:41

Sure. Your new temps framework, once it all gets shaken out, makes more sense for something like this anyways. - Robert

Re: [Cython] Python aborts when running Cython on petsc4py sources

Stefan Behnel — 2008-12-01T13:32:45

I would also consider that revision safe. But I agree that for releasing software, the latest official Cython release is the safest bet. Stefan

Re: [Cython] Python aborts when running Cython on petsc4py sources

Dag Sverre Seljebotn — 2008-12-01T13:30:52

Opps, I was looking at the description for "Child". So this was wrong, but my point still stands :-)

Re: [Cython] Python aborts when running Cython on petsc4py sources

Dag Sverre Seljebotn — 2008-12-01T13:29:22

Yes, and that is the very commit that broke a lot of things :-) It just doesn't show up because the testsuite is not complete, and it only strikes in special circumstances. Anything including the new dict for-loop optimizations is NOT SAFE. Please, use the latest stable. It's great that you /develop/ using -devel so that bugs are found, but when it comes to doing a release of your software, I strongly advice to always go back to the latest release.

Re: [Cython] Python aborts when running Cython on petsc4py sources

Lisandro Dalcin — 2008-12-01T13:04:22

On Mon, Dec 1, 2008 at 11:03 AM, Dag Sverre Seljebotn public.gmane.org> wrote: OK No, no. This is happening with cython-devel. BTW, I switched to the revision above (a few commits away for latest release), and all seems to work. Revision: 1393 Branch: default Author: Stefan Behnel public.gmane.org> 2008-11-26 18:00:35 Committer: Stefan Behnel public.gmane.org> 2008-11-26 18:00:35 Parent: 1392:a53116f21261 (fixed typo) Child: 1394:71d8a06957eb (make TupleNode and ListNode a NewTempExprNode) let types inherit from object to work around hashing problems

Re: [Cython] Python aborts when running Cython on petsc4py sources

Dag Sverre Seljebotn — 2008-12-01T13:03:25

If you're in the process of releasing software, use the latest stable. -devel has had bugs for at least a week because of temps stuff, but the latest release should be safe. Or is this happening in the latest release?

Re: [Cython] Recursive structure, typedef

David Cournapeau — 2008-12-01T12:40:02

Yes, I use 0.10.2. I upgraded because indeed, before, the following did not work: cdef struct foo: foo *foo ctypdef foo foo_t It works on 0.10.2. But my problem is slightly different: cdef struct foo: foo_t *foo ctypdef foo foo_t Notice how the recursive is referenced through a typedef. In C, this is solved by forward declaring the struct through a typedef before the struct declaration. But AFAIK, you can't do that in cython. My question may be stupid: I know nothing about parsing languages, and maybe this is just a problem in my own code generator which is "hand-built" from the AST given by gccxml. cheers, David

Re: [Cython] Recursive structure, typedef

Lisandro Dalcin — 2008-12-01T12:31:40

Hi, David. Did you tried latest release, http://cython.org/Cython-0.10.2.tar.gz ? It contains fixes for recursive typedef struct. Please, give a try a let us know of any problem. On Mon, Dec 1, 2008 at 4:27 AM, David Cournapeau public.gmane.org> wrote:

[Cython] Python aborts when running Cython on petsc4py sources

Lisandro Dalcin — 2008-12-01T12:20:35

I've got this when running Cython to generate petsc4py sources. cython --cleanup 9 -I. -Iinclude -w src petsc4py.PETSc.pyx -o petsc4py_PETSc.c python: Modules/gcmodule.c:276: visit_decref: Assertion `gc->gc.gc_refs != 0' failed. I've tried to manually remove the Cython extension modules $ rm `find ~/lib/python/Cython -name '*.so'` but then (I reported this some time ago), things fails (because of pure-python mode) cython --cleanup 9 -I. -Iinclude -w src petsc4py.PETSc.pyx -o petsc4py_PETSc.c Traceback (most recent call last): File "/u/dalcinl/bin/cython", line 7, in from Cython.Compiler.Main import main File "/u/dalcinl/lib/python/Cython/Compiler/Main.py", line 17, in import Code File "/u/dalcinl/lib/python/Cython/Compiler/Code.py", line 11, in from Scanning import SourceDescriptor File "/u/dalcinl/lib/python/Cython/Compiler/Scanning.py", line 14, in import cython ImportError: No module named cython I have no time to dive onto this. I'm in t

Re: [Cython] Temp allocation flow

Dag Sverre Seljebotn — 2008-12-01T11:49:00

I don't know Pyrex that well, and weren't around at the time of the fork. I assume it is Cython-only then. We could probably change these cases. Then again they were probably added in the first place as features/optimizations. Anyway, now the necessary changes to support it is up in -devel. Dag Sverre Seljebotn -----Original Message----- From: Greg Ewing public.gmane.org> Date: Monday, Dec 1, 2008 11:16 am Subject: Re: [Cython] Temp allocation flow To: cython-dev-F/1GfIIGwJsClooVUwDQFiifJy3iqPdu< at >public.gmane.org: cython-dev-F/1GfIIGwJtbRRN4PJnoQQ< at >public.gmane.org Dag Sverre Seljebotn wrote:

Re: [Cython] Temp allocation flow

Greg Ewing — 2008-12-01T10:19:05

Are there any examples of this in Pyrex, or is this from Cython?

[Cython] Recursive structure, typedef

David Cournapeau — 2008-12-01T06:27:42

Hi, I am using cython to wrap some fairly large C API, with a big numbers of enumerations. Those API are big enough so that I got tired of manually declaring the definitions (in particular enum), and start working on a code generator based on gccxml. It almost works for some non trivial headers (CoreAudio framework, Alsa headers), but I have a problem with the following kind of C declaration: typedef struct foo foo_t; struct foo { foo_t *foo; } AFAIK, cython does support typedefing a struct directly, and you should use something like that instead: cdef struct foo: foo_t *foo ctypdef foo foo_t Which does not work as the recursive reference uses the typedef'd type name. The following works, though: cdef struct foo: foo *foo ctypdef foo foo_t For now, I can get away by modifying on the fly the gccxml parsed tree when I detect a recursive reference, but this feels hackish. Is there a better way ? David

Re: [Cython] Temp allocation flow

Dag Sverre Seljebotn — 2008-12-01T01:12:02

Actually, this is not true. Pushing if-tests like the above to tree nodes would amount simply to moving the generate_disposal_code to the end of the C if-test.

Re: [Cython] Temp allocation flow

Dag Sverre Seljebotn — 2008-12-01T01:09:03

I didn't supply enough context. It was meant for situations like this: self.arg.generate_evaluate_code(code) ... code.putln("if (...) {") ... self.arg.generate_disposal_code(code) ... code.putln("} else {") ... self.arg.generate_disposal_code(code) ... code.putln("}") Here, one can't free the temp in the first generate_disposal_code (because then the else-block could reuse and overwrite the temp etc.). As mentioned this is now solved by considering temp freeing as a seperate but parallell step to disposal. To make the example up-to-date, simply add at the end, but NOT inside the C if-blocks, self.arg.free_temps(code) I even found one example of the opposite construction, where evaluation_code was called multiple times but disposal once! (In TupleNode.generate_assignment_code). Luckily I was able to shuffle it a bit to avoid the problem. There might be more of this stuff in Cython than in Pyrex? This all points to a fundamental problem with the current scheme: Temp allocation doesn't branch whe

Re: [Cython] Temp allocation flow

Greg Ewing — 2008-11-30T23:08:26

Rather than a free_temp parameter, I'd be more inclined to have an owns_temp flag on the node, set according to whether the node allocated the temp itself or it was supplied by the parent (currently done via an optional parameter to allocate_temps(), which would become a parameter to generate_evaluation_code()).

Re: [Cython] Temp allocation flow

Dag Sverre Seljebotn — 2008-11-30T14:57:40

I suddenly realized that this is completely wrong and won't work. I'll change things back and implement Stefan's proposal of a seperate "free_temps" call instead. I'm sure there's a lesson to be learned from this.