gmane.lisp.picolisp.general

Re: Are PicoLisp's coroutine stacks legal?

Rand Dow — 2013-05-21T17:21:12

Hi Alex,

Yes, by all means, keep the coroutines!

Perhaps add some of this mailing list chain as a caveat about the stack in
the
coroutines documentation: http://picolisp.com/5000/!wiki?coroutines
In particular, the picture of the stack and the warning to allocate enough
stack if
there is any chance of using a lot of stack.

Rand


On Tue, May 21, 2013 at 6:55 PM, Alexander Burger <abu-rbQqYLFhARZybvfhVmRDVg< at >public.gmane.org>wrote:

Re: Are PicoLisp's coroutine stacks legal?

Alexander Burger — 2013-05-21T16:55:33

Hi Rand,


Yes


Yes, I'm aware of that. I wrote something similar also in my mail
earlier today:

   This doesn't happen usually. PicoLisp does runtime stack checks, so the
   above would result in an stack overflow error (I say "usually", because
   this is not 100% safe. You may call some 'native' code which does
   unpredictable things to the stack, for example).


Yep. That's the trade-off. As soon as you use a coroutine in PicoLisp,
you have to be concious about your stack. For a program that has no
coroutine active when a lot of stack space is needed, these limitations
don't apply. As soon as the last coroutine finishes or is stopped, the
stack becomes unlimited again (at least if you called "ulimit -s
unlimited").



Then you could call (stack 3) before starting the first coroutine. This
allocates 12 MB for the main task, and 3 MB for each coroutine.



True.



Great! That's good news.

So let's keep the coroutines in PicoLisp, unless somebody objects. :)

♪♫ Alex

Re: Are PicoLisp's coroutine stacks legal?

Rand Dow — 2013-05-21T16:30:37

Hi Alex,

On Tue, May 21, 2013 at 7:46 AM, Alexander Burger <abu-rbQqYLFhARZybvfhVmRDVg< at >public.gmane.org>wrote:
you *could* have the first scenario I mentioned in a previous mail
(unlikely, but, think of a native C code, that allocated huge
structures and recursed -- it could overwrite many megabytes of
stack). This would be my concern: a C library function that uses
(in the above picture) more than 4 MB of memory. That is easy to
imagine with modern reentrant graphics libraries. The programmer
would have to have a very clear picture of the above to know to use
the 'stack' function to allocate adequate stack.

Anyway, enough is *not ever* enough. I would only feel comfortable
seeing at least 10MB of *non-mapped* memory between the main program's
stack and each of the the co-routines' stacks.

Again, generally, I would feel uncomfortable having anything "below"
my stack pointer than a large region of writable memory followed
by a very large region of memory that would give me a "sigsegv" if
I tried to access

Re: Small URL problem at Cloud9

Alexander Burger — 2013-05-21T16:00:06

Hi Jon,


Hmm, difficult to tell remotely. I suspect that this can be only the
'Path' local variable, i.e. the return value of (file) in

     (let (Path (car (file)) ...

Can you insert a 'trace'

   (trace 'file)
   (html NIL "-index-" "< at >lib.css" ...
   ...

in the beginning of the "default" file, to see what it returns?

♪♫ Alex

Re: Are PicoLisp's coroutine stacks legal?

Alexander Burger — 2013-05-21T13:41:11

setjmp()/longjmp() is what PicoLisp basically does, though it is done
directly in assembly, not by using a library. In any case, the
underlying mechanisms (i.e. the logic of above/below the stack pointer)
are the same.

So this would mean we are on the safe side.



Yes, in assembly it is almost trivial. But it doesn't help if the OS has
a different idea of what you're doing ;-)


True.

♪♫ Alex

Re: Are PicoLisp's coroutine stacks legal?

Jorge Acereda Maciá — 2013-05-21T11:46:13

Hi,

On May 21, 2013, at 7:46 AM, Alexander Burger <abu-rbQqYLFhARZybvfhVmRDVg< at >public.gmane.org> wrote:


Some OSs do provide the required functionality (e.g, the fibers API on Windows). On POSIX, it's usually a matter of using some dirty hack (setjmp()/longjmp() being the most portable, swapcontext() is also quite popular). When working in embedded/mobile I have seen a couple of platforms with broken setjmp()/longjmp() implementations, so that can also be troublesome.

The problem is so complex that perhaps your approach is the simplest. Just take a look at how nasty the thing can get:

http://code.google.com/p/libaw/source/browse/coroutine/source/Coro.c

Regards,
  Jorge

Small URL problem at Cloud9

Jon Kleiser — 2013-05-21T08:52:58

Hi,

I have a tiny PicoLisp (64-bit) web server running at c9.io (not full 
time, only now and then). My code looks like this:

(load "lib/http.l" "lib/xhtml.l" "lib/form.l")

(allowed ("doc/") "lib.css" "< at >lib/form.js" "favicon.ico" "!foo" "README")

(de foo ()
     (html 0 "-PicoLisp-" "lib.css" '(style . "margin: 40px; font-size: 
large;")        # body style
         (<h1> '(style . "color: #933") "Hello PicoLisp World!")
         (<p> NIL (ht:Prin "It's now " (tim$ (time) T) " (around here)"))
         (<p> NIL (<href> "Some PicoLisp docs right here" "doc/app.html"))
         (<p> NIL (<href> "More info about the PicoLisp system" 
"http://picolisp.com" "_blank")) ) )

(de go (Port)
     (unless Port (setq Port (or (format (sys "PORT")) 8080)))
     (out NIL (prinl "Starting server using port " Port))
     (server Port "!foo") )


I start it like this:
/pil web.l -go +


To be able to produce a listing ("index") of all the files in the doc/ 
folder, I have put this "default" script into the doc/ folder:

Re: Are PicoLisp's coroutine stacks legal?

Jorge Acereda Maciá — 2013-05-21T07:31:41

Hi,

On May 21, 2013, at 6:56 AM, Alexander Burger <abu-rbQqYLFhARZybvfhVmRDVg< at >public.gmane.org> wrote:

Each task has a different stack using a single user-mode stack pointer. In Amiga there is no memory protection, a context switch just involves saving the registers in the stack, saving the stack pointer itself in the stack structure, restoring the stack pointer of the next task and returning to it in a way that restores the registers.


I thought you were talking about micro stacks. If your stacks are 1 Mb I guess there won't be any problem.

If at some point you face problems on a system that happens to unmap stack pages (I doubt it), you can probably workaround it by marking the stack pages with mlock() or similar.

Re: Are PicoLisp's coroutine stacks legal?

Alexander Burger — 2013-05-21T05:46:02

Hi Rand,


OK, that would be no problem, as there should always be sufficient space
reserved on each coroutine's stack frame.



Me too, that's why I started this discussion. :)


To make things more clear, let's consider the following situation. It is
after the program started two coroutines, and then execution continued
in the main program. The two coroutines are suspended, and the hardware
stack points into the main program's stack frame:

                  +---------------+
Stack frame #1    |               |
Main program      |               |
                  |    Stack      |
                  |    Area       | <- SP (Stack pointer register points here)
                  |               |
                        ...
                  |  e.g. 4 MB    |
                  |               |
                  |               |
                  |               |
                  +---------------+
Stack frame #2    |     tag       |
Coroutine #1      +---------------+
                  |   Saved SP    | +

Re: Are PicoLisp's coroutine stacks legal?

2013-05-21T05:23:34

Hi Alex,

On May 21, 2013, at 7:02 AM, Alexander Burger <abu-rbQqYLFhARZybvfhVmRDVg< at >public.gmane.org> wrote:


Good!
..


The hardware (CPU) will write at least two words (instruction pointer and CPU status register, and maybe more) onto the user stack before switching to the system stack.


I'm not sure about this in the general case. I would feel uncomfortable about it.


I'm stilled amazed at PicoLisp! I wish I could use it again in my work!

Rand

Re: Are PicoLisp's coroutine stacks legal?

Alexander Burger — 2013-05-21T05:02:21

Hi Rand,

On Tue, May 21, 2013 at 05:56:52AM +0200, Rand Dow wrote:

OK. That's exactly how I implemented it.



The stack pointer is decremented by the stack frame size (default 1 MB),
then the memory is accessed with normal operations.



This doesn't happen usually. PicoLisp does runtime stack checks, so the
above would result in an stack overflow error (I say "usually", because
this is not 100% safe. You may call some 'native' code which does
unpredictable things to the stack, for example).



As I said, these would use the system stack. And even if they would use
the user stack, there should be sufficient space. As I said, we have 1
MB per stack frame, and this can be easily extended with the 'stack'
function.


Again, my problem is not to have a stack frame for each task, but having
stack frames BELOW the active stack pointer.

♪♫ Alex

Re: Are PicoLisp's coroutine stacks legal?

Alexander Burger — 2013-05-21T04:56:46

Hi Jorge,


No. From what I read there, I believe that they do it the same way I do.

They talk about "task stacks" and "preserved on this stack". I assume
that these "stacks" all reside on the single hardware stack, right? So
there is a stack *frame* for each task, and this frame might also be
used to save exception context. That's all right.

It seems not forbidden, however, to allocate several frames below each
other, where the active stack pointer points into some upper frame,
right? That's the problem I'm talking about.

♪♫ Alex

Re: Are PicoLisp's coroutine stacks legal?

Rand Dow — 2013-05-21T03:56:52

I won't try to "point" to documentation....
Each co-routine should have it's own separate stack. Best practices with
stack management today have a sufficiently large stack that grows and then
terminates in unmapped memory. If it is attempted to grow the stack too
much, then the program gets a memory exception (SIGSEGV on systems that use
it). Think about a recursive routine with relatively large local data
structures (stack-based). Such a routine could overwrite any amount of
stack, finish, and then return with no errors. The current PicoLisp
implementation sounds like, then, when a co-routine switch occurs, the
switched to routine could find it's stack corrupted.
And certainly operating system events could overwrite things deeper on the
stack during a context switch.

Rand


On Mon, May 20, 2013 at 6:43 PM, Alexander Burger <abu-rbQqYLFhARZybvfhVmRDVg< at >public.gmane.org>wrote:

Re: Are PicoLisp's coroutine stacks legal?

Jakob Eriksson — 2013-05-20T22:28:44

Wow, I just have to comment this. Amiga + picolisp in the same post = awesome.
/me crawls back under rock


On May 20, 2013 at 11:13 PM "Jorge Acereda Maciá" <jacereda-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: Are PicoLisp's coroutine stacks legal?

Jorge Acereda Maciá — 2013-05-20T21:13:45

Hi,

I can only say that under AmigaOS it would certainly be a no-no:

http://wiki.amigaos.net/index.php/Exec_Tasks#Task_Exceptions

"When an exception occurs, Exec stops executing the tasks normal code and jumps immediately into the exception routine, no matter what the task was doing. The exception routine operates in the same context the task's normal code; it operates in the CPU's user mode and uses the task's stack."

http://wiki.amigaos.net/index.php/Exec_Tasks#Task_Stack

"Task stacks are normally used to store local variables, subroutine return addresses, and saved register values. Additionally, when a task loses the processor, all of its current registers are preserved on this stack (with the exception of the stack pointer itself, which must be saved in the task structure)."


On May 20, 2013, at 6:43 PM, Alexander Burger <abu-rbQqYLFhARZybvfhVmRDVg< at >public.gmane.org> wrote:

Are PicoLisp's coroutine stacks legal?

Alexander Burger — 2013-05-20T16:43:50

Hi all,

since nearly three years PicoLisp supports coroutines (64-bit version).
Now suddenly it occurred to me that the way I implemented them might be
illegal.

The problem is how individual stacks for the coroutines are allocated. I
do this by reserving space on the stack (by decrementing the stack
pointer) upon starting a new coroutine. Execution of that coroutine may
be suspended (with 'yield', so that execution continues in the main
program or some other coroutine), to be resumed later.

Switching between individual (co)routines is done by moving the stack
pointer up and down between the corresponding stack frames. Everything
seems to work fine - though I haven't used coroutines in serious
projects yet.


Now I remember to have read that (in POSIX?) no valid data should be
stored *below* the current position of the stack pointer. Is that really
the case? I can't find any conclusive information about that.

This switching between stack frames, as described above, of course
leaves temporarily suspended s

Re: How to set IP address for socket?

Jon Kleiser — 2013-05-19T18:07:26

Hi Alex,

No, it's not in Norway. Could be in Canada ... It's in the "cloud"
somewhere, Cloud9. ;-)

/Jon

Re: How to set IP address for socket?

Alexander Burger — 2013-05-19T17:29:55

Taking a closer look, it seems in Canada, right?


BTW, if I'd want to make a form with a time-field, which updates its
value whenever you press a button:

   (allowed ()
      "!home" setTime "< at >lib.css" )

   (load "< at >lib/http.l" "< at >lib/xhtml.l" "< at >lib/form.l")

   (de setTime (H M S)
      (set> (: time) (time H M S)) )

   (de home ()
      (app)
      (action
         (html 0 "Local Time" "< at >lib.css" NIL
            (form NIL
               (gui 'time '(+Lock +TimeField) 10 "Local Time")
               (----)
               (gui '(+OnClick +Button)
                  "var t = new Date();
                     return lisp(this.form, 'setTime',
                        t.getHours(),
                        t.getMinutes(),
                        t.getSeconds())"
                  "Now!" ) ) ) ) )

   (de go ()
      (server 8080 "!home") )

Started as

   pil <file>.l -go +      # Debug
   pil <file>.l -go -wait  # Production

♪♫ Alex

Re: How to set IP address for socket?

Alexander Burger — 2013-05-19T16:45:33

However, the time is strange. It says

   It's now 12:44:27 (around here)

Is the server not in Norway? I'd expect 19:44:27 then.

♪♫ Alex

Re: How to set IP address for socket?

Alexander Burger — 2013-05-19T16:27:21

Hi Jon,


Work perfectly! Congratulations! :)

♪♫ Alex

Re: How to set IP address for socket?

Jon Kleiser — 2013-05-19T16:12:24

Hi Rowan & Alex,

Yes, that ipv6-mapped address made the difference! My tiny PicoLips server
is now running at <http://demo-project.jkleiser.c9.io/>. It probably won't
run for a very long time, but I'm very satisfied. ;-)
(I obviously have to fix something re. the "default" file, in doc/, but
that's something else.)

/Jon