gmane.comp.python.ipython.user

Re: [0:execute]: IOError: [Errno 4] Interruptedsystem call

Brian Granger — 2008-12-01T23:01:51

Just checked in Twisted and IPython and there are no os.kill's anywhere in the core that would show up here. That leaves: 1. User code 2. Some other process. Is there a chance that the engines are being started using some batch system (like PBS) that ends up sending the process signals? Brian On Mon, Dec 1, 2008 at 2:21 PM, Andrew Straw astraw.com> wrote:

Re: [0:execute]: IOError: [Errno 4] Interrupted system call

Andrew Straw — 2008-12-01T22:21:22

Sorry, I wasn't clear here. EINTR is the errno value that's returned here. The signal could be anything. I've never experienced the kernel sending unrequested signals other than the usual things like SIGINT and SIGKILL... But I suppose it could be configured to do so somehow. My best guess is that this is more likely to be in user code than in Twisted or the other sources you list. I suspect that if signals were used as part of Python or Twisted, folks attempting to use non-EINTR-safe code (there's a lot of it) would've come screaming by now. My relatively naive understanding of this stuff is that signals are the blunt tool of inter-process communication and I see the Twisted crew as more the fine surgeon types... But I haven't grepped for os.kill() in the Twisted sources, either, so I could be wrong. Likewise I don't know about IPython, but I'd be surprised if it was doing IPC via signals. If it is, I'd suggest it be removed, as it will come back to haunt anyone without non-EINTR-safe code. It's debugging these types of things that turns my hair gray... -Andrew

Re: [IPython-dev] gathering statics/infomation fromrunning TaskController

Brian Granger — 2008-12-01T22:16:21

What is the usage case for this. My only hesitation to add this is that there is no promise that the engine that is currently working on a task will be the engine that actually completes the task. Brian

Re: OS X Terminal iPython history display bug

Brian Granger — 2008-12-01T22:14:46

Yes, could you file a bug report here: https://bugs.launchpad.net/ipython Thanks! Brian On Wed, Oct 22, 2008 at 11:38 AM, Lawrence Johnston gmail.com> wrote:

Re: cd doesn't work on Windows

Brian Granger — 2008-12-01T22:12:03

Can you file a bug report on our launchpad site: https://bugs.launchpad.net/ipython Thanks, Brian On Tue, Nov 25, 2008 at 10:25 PM, Elias Bröms eliasit.se> wrote:

Re: [0:execute]: IOError: [Errno 4] Interruptedsystem call

Brian Granger — 2008-12-01T22:09:11

Yep, I think that is what is going on. Is there a chance that the signal is anything other than EINTR? I ask as that will help us track this down. There are a couple of possibilities: 1. Something deep in the internals of Python itself. 2. Something deep in Twisted 3. It wouldn't be in IPython as we (as far as I know) are not sending any signals. 4. Deep somewhere in user code that they are not aware of. My best guesses are Twisted or in user code. I will look at Twisted to see if it sends signals anywhere. Is it also possible that the kernel itself sends the signal? Thanks Brian

Re: Trivial parallelisation

Brian Granger — 2008-12-01T19:00:25

Yes, IPython will handle this beautifully. Here is an example, with everything done interactively: pcp025387pcs:~ bgranger$ ipython Python 2.5.1 (r251:54863, Apr 15 2008, 22:57:26) Type "copyright", "credits" or "license" for more information. IPython 0.9.1 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object'. ?object also works, ?? prints more. In [1]: files = ['file%i.txt' % i for i in range(10)] In [2]: files Out[2]: ['file0.txt', 'file1.txt', 'file2.txt', 'file3.txt', 'file4.txt', 'file5.txt', 'file6.txt', 'file7.txt', 'file8.txt', 'file9.txt'] In [3]: from IPython.kernel import client # this won't load balance, but has low overhead In [4]: mec = client.MultiEngineClient() In [5]: def doit(file): ...: pass ...: In [6]: mec.map(doit,files) Out[6]: [None, None, None, None, None, None, None, None, None, None] # this does load balance, but has higher overhead In [7]: tc = client.TaskClient() In [8]: tc.map(doit,files) Out[8]: [None, None, None, None, None, None, None, None, None, None] Our documentation has more details on how to use all of this: http://ipython.scipy.org/doc/rel-0.9.1/html/parallel/index.html But, please don't hesitate to ask us questions. Cheers, Brian On Mon, Dec 1, 2008 at 9:50 AM, Jose Gómez-Dans gmail.com> wrote:

Trivial parallelisation

Jose Gómez-Dans — 2008-12-01T17:50:29

Hi, I do have some trivial parallel code (I need to apply some analysis algorithm to a bunch of data files), and access to either an SMP linux machine with loads of CPUs or to a linux cluster. In the past, I have launched my python scripts with a different command line to get them to run concurrently, but I was thinking whether there's some pythonic way of dealing with this that is simple. I seem to remember that ipython would be a nice tool for this, but I'm not sure of it, or indeed, any other alternatives. My code is simple: for file in list_of_files: #I want spawn this next line ProcessFile ( file ) Thanks for your help, Jose

Re: OS X Terminal iPython history display bug

2008-12-01T05:26:36

_______________________________________________ IPython-user mailing list IPython-user< at >scipy.org http://lists.ipython.scipy.org/mailman/listinfo/ipython-user

Re: Qt gui now usable (sort of) :-)

Ville M. Vainio — 2008-11-30T20:49:58

On Sun, Nov 30, 2008 at 10:13 PM, Richard Riley googlemail.com> wrote: Glad that someone asked! - One reason is integration with other tools (embedding). The console takes "too much" control, and doesn't always play well with gui event loops. The "threading" versions of ipython shells (to be used with qt, gtk et al) are a constant source of woes. My particular use case for this is "ILeo": http://webpages.charter.net/edreamleo/IPythonBridge.html - Leo is being ported to Qt, and I want IPython to be first class citizen there. Sometimes the console ipython can be comfortably integrated with gui event loop using PyOS_InputHook, but esp. with Qt & windows, it was too slow. Tk is a well-behaved exception it this respect. - We can offer better feature set with gui shells. For example, the qt shell allows seamless multiline editing using the qscintilla feature set, while multiline editing with console ipython is not that stellar (unless you switch to an external editor using %edit, losing tab completion). Other features are basically limited by people's imagination (think object browsers, namespace searchers, clickable %pycat, ...) - readline is a recurrent pain, and cause of bugs that just can't be fixed - whereas with gui shells, we can make things behave pretty much the way we want. Obviously the "classic" terminal ipython isn't going anywhere, since it can be used be where the gui shells can't (and for simple jobs, plain terminal is probably the most elegant interface).

Qt gui now usable (sort of) :-)

Ville M. Vainio — 2008-11-30T19:28:19

You can now actually use the qt ui. It's at "technology demo" right now, but basic stuff like tab completion now works. The colors suck (typical ipython + white background issue), KeyboardInterrupt doesn't work (apart from doing it in the console window) and it probably has lots of other problems, but nothing that hasn't been solved in other gui's already, so it probably won't take forever to polish it. Input happens in separate scintilla widget, so it requires a bit of a mindset switch, but nothing too drastic. The big win is simplicity of multiline editing, and general "statelessness" (autoindent etc). You can test it out by doing: bzr branch lp:~villemvainio/ipython/ipython-qt-ui and running qtipy.py. Screenshot: http://img84.imageshack.us/img84/5875/ipyqteo3.png

Re: How to create a new file

Ville M. Vainio — 2008-11-30T16:48:02

I figured as much :-) Note that %quickref should be your first stops in problems like this.

Re: How to create a new file

Mr.SpOOn — 2008-11-30T16:29:05

2008/11/30 Ville M. Vainio gmail.com>: The fact that I didn't know about that may be a good reason. Anyway, thanks.

Re: How to create a new file

Ville M. Vainio — 2008-11-30T16:22:11

Any reason you can't do: !vim newfile.py ?

How to create a new file

Mr.SpOOn — 2008-11-30T15:22:51

Hi, is there any way for creating a new blank file from the iPython shell? I'm on a linux machine and I'm wondering if is it possible to do something like: vim newfile.py Or at least using touch. Thanks.

Re: [0:execute]: IOError: [Errno 4] Interrupted system call

Andrew Straw — 2008-11-30T07:27:11

Hi Mark, Sorry I don't have any specific information concerning your situation, but as it's been a few days with no response, I figured I'd chime in. My understanding is that an interrupted system call (EINTR) happens when a system call (e.g. select(), fread(), fwrite(), and so on) is interrupted by a signal that the kernel decides your thread is going to handle. The correct behavior is to deal appropriately with the signal (possibly ignoring it) and then repeat your system call. I've found lots of code in the wild that is not robust to being interrupted this way (including in core Python), but luckily very little code that sends signals and thus interrupts code that way. So, I think the best solution will be to find what is sending the signal and eliminate it. Hopefully this will ring some bells with someone more knowledgeable in the ipython internals than I. Also, are you running any third party code that could be sending signals? -Andrew mark starnes wrote:

Re: cd doesn't work on Windows

Elias Bröms — 2008-11-26T06:25:03

Now I've uninstalled ipython, python and panda3d. Then I reinstalled python and ipython in their default locations. I still have this problem! Elias 2008/11/17 Elias Bröms eliasit.se>: _______________________________________________ IPython-user mailing list IPython-user< at >scipy.org http://lists.ipython.scipy.org/mailman/listinfo/ipython-user

Re: [0:execute]: IOError: [Errno 4] Interrupted system call

mark starnes — 2008-11-25T10:19:32

Hi again, I cannot replicate this behaviour from the IPython console. I attempted it, using: from IPython.kernel import client mec = client.MultiEngineClient() # Create file. import cPickle a = file('/tmp/temp.file', 'wb') cPickle.dump('0' * int(1E5), a) a.close() # Execute repeated loads on engines. mec.execute('import cPickle') for j in xrange(100): for i in xrange(3): command = "a = file('/tmp/temp.file')" mec.execute(command, targets = [i]) command = "b = cPickle.load(a)" mec.execute(command, targets = [i]) command = "a.close()" mec.execute(command, targets = [i]) performing 100 reads, with no errors. BR, Mark. mark starnes wrote:

[0:execute]: IOError: [Errno 4] Interrupted systemcall

mark starnes — 2008-11-25T09:51:12

Hi everyone, I'm performing blocking file read commands on remote engines, one at a time, but regularly get the error: ******************************************************************************************************** CompositeError Traceback (most recent call last) in () /fe2.pyc in setup(self, append) 365 diskpush({'a':self}, picklemode = 0) # use mode 0 (others fail) 366 else: --> 367 diskpush({'a':self}) 368 369 /reference.pyc in diskpush(a, targets, block, isanobject, picklemode) 3359 time.sleep(5.0); mprint('Done. ') 3360 #mec.execute('tmpdata = dpfile.read()', targets = [i], block = True) -> 3361 mec.execute('dpvar = cPickle.load(dpfile)', targets = [i], block = True) 3362 mec.execute('dpfile.close()', targets = [i], block = True) 3363 #mec.barrier(a) /usr/local/lib64/python2.5/site-packages/IPython/kernel/multiengineclient.pyc in execute(self, lines, targets, block) 520 targets, block = self._findTargetsAndBlock(targets, block) 521 result = blockingCallFromThread(self.smultiengine.execute, lines, --> 522 targets=targets, block=block) 523 if block: 524 result = ResultList(result) /usr/local/lib64/python2.5/site-packages/IPython/kernel/twistedutil.pyc in blockingCallFromThread(f, *a, **kw) 67 < at >raise: any error raised during the callback chain. 68 """ ---> 69 return twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw) 70 71 else: /usr/local/lib64/python2.5/site-packages/Twisted-8.1.0-py2.5-linux-x86_64.egg/twisted/internet/threads.pyc in blockingCallFromThread(reactor, f, *a, **kw) 81 result = queue.get() 82 if isinstance(result, failure.Failure): ---> 83 result.raiseException() 84 return result 85 /usr/local/lib64/python2.5/site-packages/Twisted-8.1.0-py2.5-linux-x86_64.egg/twisted/python/failure.pyc in raiseException(self) 317 information if available. 318 """ --> 319 raise self.type, self.value, self.tb 320 321 CompositeError: one or more exceptions from call to method: execute [0:execute]: IOError: [Errno 4] Interrupted system call ******************************************************************************************************************* It doesn't always happen; sometimes it will occur after a few hours of execution (ruining overnight runs). The active parts of the routine falling over are: ============================================= for i in targets: # Sequential engine reads. mec.execute('dpfile = bz2.BZ2File(tmpfn, "rb")', targets = [i], block = True) mec.execute('dpvar = cPickle.load(dpfile)', targets = [i], block = True) mec.execute('dpfile.close()', targets = [i], block = True) ============================================= with targets = mec.get_ids() Following the error, subsequent requests to the engines get results like: / in () /fe2.pyc in setup(self, append) 365 diskpush({'a':self}, picklemode = 0) # use mode 0 (others fail) 366 else: --> 367 diskpush({'a':self}) 368 369 /reference.py in diskpush(a, targets, block, isanobject, picklemode) 3359 #time.sleep(5.0); mprint('Done. ') 3360 #mec.execute('tmpdata = dpfile.read()', targets = [i], block = True) -> 3361 mec.execute('dpvar = cPickle.load(dpfile)', targets = [i], block = True) 3362 mec.execute('dpfile.close()', targets = [i], block = True) 3363 #mec.barrier(a) /usr/local/lib64/python2.5/site-packages/IPython/kernel/multiengineclient.pyc in execute(self, lines, targets, block) 520 targets, block = self._findTargetsAndBlock(targets, block) 521 result = blockingCallFromThread(self.smultiengine.execute, lines, --> 522 targets=targets, block=block) 523 if block: 524 result = ResultList(result) /usr/local/lib64/python2.5/site-packages/IPython/kernel/twistedutil.pyc in blockingCallFromThread(f, *a, **kw) 67 < at >raise: any error raised during the callback chain. 68 """ ---> 69 return twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw) 70 71 else: /usr/local/lib64/python2.5/site-packages/Twisted-8.1.0-py2.5-linux-x86_64.egg/twisted/internet/threads.pyc in blockingCallFromThread(reactor, f, *a, **kw) 81 result = queue.get() 82 if isinstance(result, failure.Failure): ---> 83 result.raiseException() 84 return result 85 /usr/local/lib64/python2.5/site-packages/Twisted-8.1.0-py2.5-linux-x86_64.egg/twisted/python/failure.pyc in raiseException(self) 317 information if available. 318 """ --> 319 raise self.type, self.value, self.tb 320 321 CompositeError: one or more exceptions from call to method: execute [0:execute]: AttributeError: 'module' object has no attribute 'cbook' and then, when I quit IPython, I get, Closing threads... Done. Exception exceptions.TypeError: "'NoneType' object is not callable" in > ignored The engines need a restart after this error. Any ideas on how to fix this, would be appreciated. It's hobbling my parallel processing attempts! Thanks in advance, Mark. matplotlib version 0.98.3 verbose.level helpful interactive is False units is False platform is linux2 Could not load matplotlib icon: 'module' object has no attribute 'window_set_default_icon_from_file' backend QtAgg version 0.9.1 Activating auto-logging. Current session state plus future input saved. Filename : ipython_log.py Mode : rotate Output logging : False Raw input log : False Timestamping : False State : active Python 2.5.1 (r251:54863, Jan 10 2008, 18:00:49) Type "copyright", "credits" or "license" for more information. IPython 0.9.rc1 -- An enhanced Interactive Python.

Re: pass command flags to script

Ville M. Vainio — 2008-11-25T07:26:48

In [3]: cat t.py a= "hello" print a,a [~]|16> ipython -i -c '%run t.py hello' hello hello In [2]: print a hello Set sys.argv before that.

pass command flags to script

John Hunter — 2008-11-24T17:16:24

I would like to pass command args to a script and after the script runs, drop into an ipython shell. Something like > ipython 'agg_oo.py --verbose-helpful' Which does not work. Is such a thing possible? If not, what is the proper incantation to add at the end of the script to open an interactive ipython shell with the namespace of the script? I tried:: from IPython.Shell import IPShellEmbed ipshell = IPShellEmbed() ipshell() but it did not like the command args either :-( Thanks, JDH IPython 0.9.0.bzr.r1016