gmane.comp.python.ipython.user

Re: Notebook rename vs "save as"

Brian Granger — 2012-05-26T05:16:13

But notebooks can be saved in non-file system stores (such as a
mongodb database) that don't track items by filenames.  Because of
this, we have to preserve the notebook name inside the notebook.
There is no way around that.  It is very likely that the
NotebookManager class will become configurable and will support
multiple notebook stores such as this.

Re: memory problem with passing numpy arrays in IPython parallel

Brian Granger — 2012-05-26T05:01:41

Charles,

On Fri, May 25, 2012 at 8:32 AM, Charles Cadieu <c.cadieu< at >gmail.com> wrote:

My first response is that if you are sending 1000x1000 arrays back and
forth 100k times you need to restructure your computation to minimize
the data movement.  The communication overhead is going to kill any
parallel speedup you would get from using multiple cores or CPUs.  How
long does each time step take in parallel?

Cheers,

Brian

Re: iPython Notebook profile under Windows

MinRK — 2012-05-25T20:38:17


Is this a double-click launcher from EPD? If so, it sounds like a bug to
report to them.




NotebookManager.notebook_dir


_______________________________________________
IPython-User mailing list
IPython-User< at >scipy.org
http://mail.scipy.org/mailman/listinfo/ipython-user

iPython Notebook profile under Windows

federico vaggi — 2012-05-25T20:21:42

Hi,

I usually use iPython and the notebook under Linux, but I ran into some
rather trivial problems under Windows.  After installing EPD (thank you
very much to Enthought) I noticed that the default save directory for
iPython notebooks is windows/system - which of course, the default user
does not have write access to.  This means that I can open existing iPython
notebooks, but I can't create new ones, because I am not able to change the
starting directory, and pressing new crashes Python with an error about
'lacking write access''

For reference, here is the log:

[NotebookApp] Using existing profile dir:
u'C:\\Users\\Fede\\.ipython\\profile_d
efault'
[NotebookApp] The IPython Notebook is running at: http://127.0.0.1:8888
[NotebookApp] Use Control-C to stop this server and shut down all kernels.
ERROR:root:Uncaught exception GET /new (127.0.0.1)
HTTPRequest(protocol='http', host='127.0.0.1:8888', method='GET',
uri='/new', ve
rsion='HTTP/1.1', remote_ip='127.0.0.1', body='', headers={'Connection':
'keep-a
l

Re: parallel data capture and write

MinRK — 2012-05-25T20:20:26


There's also something funny in your ventilator, where you do `d =
data.copy() * task_nbr`

That means your last message is 1500 * 1000 * 2000 * 2  = 6 GB all on its
own.  Also note that the default behavior in send_array is copy=True,
unless you changed that, so this single message will occupy 12 GB of memory
in your main process, before it even gets to zeromq.

I also might recommend using the HWM socket option on your ventilator.
 This allows you to limit the number of messages you have passed along to
zeromq still in the pipe, waiting to send.  For instance, if your reading
from the source (in this toy test, simple memcopy, and thus super fast)
faster than zeromq can send, your zeromq buffer will simply grow at a rate
equal to the difference between these speeds.

Here is a version of your script with some edits and a dummy worker, that
seems to work okay: https://gist.github.com/2790332

-MinRK


_______________________________________________
IPython-User mailing list
IPython-User< at >scipy.org
http://ma

Re: 0.12.1 install problem

Christoph Gohlke — 2012-05-25T19:46:45

The ipython_win_post_install script, which creates the shortcuts, 
requires setuptools or distribute to be installed on the target Python:

https://github.com/ipython/ipython/blob/master/scripts/ipython_win_post_install.py#L37

Christoph


On 5/25/2012 12:36 PM, Paul Hobson wrote:

Re: 0.12.1 install problem

Ryan Krauss — 2012-05-25T19:46:12

EPD Free saves me some hassle going forward, so this little hitch lead
to something good for me.  Thanks again for the tip.

Ryan

On Fri, May 25, 2012 at 2:11 PM, Fernando Perez <fperez.net< at >gmail.com> wrote:

Re: 0.12.1 install problem

Paul Hobson — 2012-05-25T19:36:50

That's my understanding, yes. But it appears that on Windows this
/might/ not be the case.

Unfortunately, this is hard to test since I only use Windows at work
and don't have time to set up a new VM.

All I'm proposing is to add a couple of sentences to the Windows
README saying something along the lines of "If the installer runs to
completion, but didn't work, make sure you have distribute and run the
installer again. If that didn't work, trying using easy_install to get
pyzmq and pygments before you re-run the installer. If *that* doesn't
work, ping the email list"
-paul

Re: 0.12.1 install problem

MinRK — 2012-05-25T19:28:15


But IPython installation should absolutely not behave any differently if
those packages are present or not, right?


_______________________________________________
IPython-User mailing list
IPython-User< at >scipy.org
http://mail.scipy.org/mailman/listinfo/ipython-user

Re: 0.12.1 install problem

Paul Hobson — 2012-05-25T19:26:03

Fernando,

I'm struggling to remember this fully. But I think I ran into this
issue with my new work machine trying a fresh python setup with
ipython 12.0. I think pyzmq and pygments fixed it. I recently
installed 12.1 without a hitch.

I guess what I'm trying to say is that it might be worth adding some
language to the documentation encouraging people to grab those
libraries before hand. I can edit the README file and send a pull
request if you'd like.

-paul

Re: Notebook rename vs "save as"

Thomas Kluyver — 2012-05-25T19:17:57

What about something like this:

- The server stores a filename for each notebook
- This *isn't* saved inside the notebook
- When you first save a notebook with a title, the filename is
calculated based on the title (probably with changes like replacing
spaces and slashes).
- When you open a notebook, the filename is set to the file you opened
it from, so that when you save, it won't suddenly change location.

Thomas

Re: 0.12.1 install problem

Fernando Perez — 2012-05-25T19:11:21

OK, good to know.  I wish we knew how to make this more robust...
Thanks for your patience!

Cheers,

f

Re: 0.12.1 install problem

Ryan Krauss — 2012-05-25T19:09:35

I logged into my office computer remotely and couldn't reproduce the
problem.  It already had 0.11 installed.  I uninstalled it and ran the
0.12.1 installer.  There were no issues.

On Fri, May 25, 2012 at 1:58 PM, Ryan Krauss <ryanlists< at >gmail.com> wrote:

Re: Notebook rename vs "save as"

Brian Granger — 2012-05-25T19:00:30

Hmm, this should not be the case as the server doesn't cache anything.


One important point in all of this: while the notebook server is
running you should not move notebook files around using the filesystem
or rename them in their metadata.  There is simply no way we can have
the notebook server do something sensible in these situations with our
current design.  We have ideas about how to improve this situation in
the future but it will involve major development and refactoring of
much of the notebook.

Re: 0.12.1 install problem

Ryan Krauss — 2012-05-25T18:58:50

I will make a note to try this on my office computer next week to help
investigate this.  I installed EPD Free on my laptop and don't want to
mess with uninstalling it and then reinstalling everything.

On Fri, May 25, 2012 at 1:44 PM, Paul Hobson <pmhobson< at >gmail.com> wrote:

Re: Notebook rename vs "save as"

Brian Granger — 2012-05-25T18:57:57

The fact that we use the file system to store notebook is an
implementation detail.  At some point in the future we may offer other
storage methods.  Because of this we have to store the title in the
notebook document.  The only other option would be to use a filename
that is some UUID, but that would be ugly as hell.

The main constraint that exists right now is that you have to keep the
two names in sync or unexpected things will happen.

Re: 0.12.1 install problem

Paul Hobson — 2012-05-25T18:44:55


On Fri, May 25, 2012 at 11:34 AM, MinRK <benjaminrk< at >gmail.com> wrote:

Understood. Just trying to figure out if the absence of those
libraries on the OP's machine lead to his initial installation
problem.
-paul

Re: 0.12.1 install problem

MinRK — 2012-05-25T18:34:07


I'm pretty sure EPD Free has pyzmq now.

-paul
_______________________________________________
IPython-User mailing list
IPython-User< at >scipy.org
http://mail.scipy.org/mailman/listinfo/ipython-user

Re: parallel data capture and write

MinRK — 2012-05-25T18:32:58


Can you post the *complete* code, including worker?  Even a dummy worker
that just dumps to devnull is fine.

-MinRK


_______________________________________________
IPython-User mailing list
IPython-User< at >scipy.org
http://mail.scipy.org/mailman/listinfo/ipython-user

Re: 0.12.1 install problem

Paul Hobson — 2012-05-25T18:32:03

I think this makes sense if you have students. For your own machine,
can you go ahead and install pygments and pyzmq (via pip or
easy_install) and see if that fixes your problem? It's been a while,
but I think that did the trick for me.
-paul

Re: 0.12.1 install problem

Ryan Krauss — 2012-05-25T17:58:12

EPD Free seems like the winner.

On Fri, May 25, 2012 at 12:28 PM, Ryan Krauss <ryanlists< at >gmail.com> wrote: