gmane.comp.lang.r.hpc

Error in socketConnection ... all connections are in use

Rob Steele — 2012-05-26T15:33:26

This is occurring with the R parallel package.  How do I increase the 
maximum number of connections?

Thanks!


Error in socketConnection("localhost", port = port, server = TRUE, 
blocking = TRUE,  :
   all connections are in use
Calls: makeCluster ... makePSOCKcluster -> newPSOCKnode -> socketConnection
Execution halted


platform       x86_64-redhat-linux-gnu
arch           x86_64
os             linux-gnu
system         x86_64, linux-gnu
status
major          2
minor          15.0
year           2012
month          03
day            30
svn rev        58871
language       R
version.string R version 2.15.0 (2012-03-30)
nickname

fork() warning from mpi

Jim Maas — 2012-05-19T14:11:13

I get the following message when running a job using R 2.15.0, Rmpi, 
doMPI, and a cluster system that uses a LSF operating system and 
infiniband connections.  I'm just a user and know little about the 
workings of such networks but the warning unsettles me a little.  I've 
asked people who write and maintain open-mpi and they are suggesting 
that something has changed in the latest version of R that causes this.

I'm wondering if there has been a change or if we have done something 
incorrectly when installing Rmpi on the system, not sure here but this 
might have to be built/compiled against open-mpi libraries.  The 
open-mpi people have suggested that R must be doing some king of fork 
after the call MPI-Init.

Could anyone tell me if this is a new problem, or have we done something 
incorrectly?  I have a toy model file that creates it and am happy to 
supply this.

Thanks a bunch,

J

===============================
An MPI process has executed an operation involving a call to the
"fork()" system call t

New version of cloudRmpi

Barnet Wagman — 2012-05-14T16:07:18

cloudRmpi v 1.2 is now available on CRAN.

cloudRmpi is a means for doing parallel processing in R, using MPI on a  
cloud-based network.  It currently
supports the use of Amazon's EC2 cloud computer service.

Changes in v 1.2:

Support for RStudio. RStudio Server is available on new AMIs.  cloudRmpi 
now has a function for securely connecting to an RStudio session running 
on the master node of an EC2-MPI network, via ssh port forwarding.  
(RStudio Server is a browser based, IDE-like interface to R).

The network specification dialog shows more information about AMIs.

The network manager has a command to repeat Open MPI network network 
configuration.


Regards,

Barnet Wagman

Condor, grid-appliance or pool-of-virtual machine

Stefan Janse van Rensburg — 2012-05-14T11:08:14

Dear List,

I am hoping somebody would be kind enough to share their recent 
experiences with R and Condor or R and BOINC. I am aware of the November 
2005 RNews article, but I would think that a lot has happened in this 
area since then. E.g. The gridR package.

I am writing a preliminary report on the viability of implementing some 
form of a grid computing setup for cycle-scavanging at Rhodes 
University, South Africa. Most of our lab machines are Windows XP and 7 
desktops. My preliminary readings suggest that running a VM with Linux 
guest OS would be the best option. Is see that there are several 
pre-prepared bundles for this sort of thing, such as Grid Appliance and 
Pool-of-Virtual Machines. Has any one used any of these with R?

Any help, whatsoever, would be much appreciated.

Kind regards,

Stefan Janse van Rensburg 
<http://www.ru.ac.za/statistics/staff/mrstefanjansevanrensburg/>

<http://www.ru.ac.za/statistics/staff/mrstefanjansevanrensburg/>

Lecturer, Dept. of Statistics, Rhodes University

Quickest way to make a large "empty" file on disk?

Jonathan Greenberg — 2012-05-02T22:23:35

R-helpers:

What would be the absolute fastest way to make a large "empty" file (e.g.
filled with all zeroes) on disk, given a byte size and a given number
number of empty values.  I know I can use writeBin, but the "object" in
this case may be far too large to store in main memory.  I'm asking because
I'm going to use this file in conjunction with mmap to do parallel writes
to this file.  Say, I want to create a blank file of 10,000 floating point
numbers.

Thanks!

--j

Analytics & Data Mining Conference, San Diego,California

Lisa Solomon — 2012-05-02T19:08:35

Subject: Analytics & Data Mining Conference, San Diego, California

Don't miss it, it's this month!

The 2012 Salford Analytics & Data Mining Conference aims to bring together
researchers, practitioners, and data enthusiasts to exchange ideas and
experiences.

Attendees will have the chance to have one-on-one meetings with the creators of the CART and RandomForests algorithms
(Dr. Adele Cutler, Dr. Jerome Friedman and Dr. Richard Olshen).

In addition, the following companies will give presentations:
Johnson & Johnson, Genentech, Visa, Fidelity, Qualcomm, Union Bank and more!

Conference website: http://www.salforddatamining.com/
Agenda: http://www.salforddatamining.com/agenda.html
Registration: http://www.salforddatamining.com/register.html

For additional information, contact info-JdWS0y8hGXeRvUXW8jGz4vpXobYPEAuW< at >public.gmane.org<mailto:info-JdWS0y8hGXeRvUXW8jGz4vpXobYPEAuW< at >public.gmane.org> or phone 619-543-8880.

Best regards,
Lisa Solomon
lisas-8ugdV5V/5NuhEniVeURVKkEOCMrvLtNR< at >public.gmane.org<mailto:li

'Rmpi' issue

Libo Sun — 2012-04-30T04:07:51

Hi,

I am trying to test 'Rmpi' on Blacklight,
http://www.psc.edu/machines/sgi/uv/blacklight.php, in PSC.

The R code I am trying to run is very simple:

mpi.remote.exec(paste("I am",mpi.comm.rank(),"of",mpi.comm.size()))
mpi.remote.exec(paste("I am",Sys.info()1,"of",mpi.comm.size()))

The batch file:

#!/bin/csh
#PBS -l walltime=5:00
#PBS -l ncpus=16
#PBS -o test.out
#PBS -j oe
#PBS -q debug

# define module command
source /usr/share/modules/init/csh

set echo
date

# load R module
module load R/2.11.1
module load Rmpi
module swap mpt/2.04 mpt/2.01

# move to scratch space to run job
cd $SCRATCH

# copy input file to scratch space
cp $HOME/test.R .
cp $HOME/.Rprofile .

mpirun -np 16 R --no-save -q < test.R > test.Rout

However, here are some error messages I got:

 *** caught segfault ***
address (nil), cause 'memory not mapped'

Traceback:
 1: dyn.load(file, DLLpath = DLLpath, ...)
 2: library.dynam("Rmpi", pkg, lib)
 3: f(libname, pkgname)
 4: firstlib(which.lib.loc, package)
 5: doTryCatch(return(expr),

parallel and openblas

Martin Renner — 2012-04-23T19:53:11

Parallel and openblas don't seem to mix well on my machine. If I link openblas, a job executed through parallel (using either the multicore or snow (local socket cluster) setup), each of my 8 cores only operates at 1/8 of 100% (taking a little longer than serial execution). Linking to the reference blas or to single-threaded atlas does not cause this handicap when running snow or multicore. 

Is this a known problem (My google attempts were fruitless)? If yes, is there a fix for it? Do MKL or multi-threaded atlas have the same issues? 

Thank you for your time. 

Martin



Martin Renner
Post-doctoral Fellowphone: 907-226 4672
University of Washington   or: 907-235 0728
School of Aquatic and Fishery SciencesSeattle, USA





debian squeeze on 8-core Xeon
R version 2.15.0 (2012-03-30)
Platform: x86_64-unknown-linux-gnu (64-bit)

locale:
[1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
[3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
[5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UT

What to Experiment With?

ivo welch — 2012-04-20T22:16:47

Dear R HPC experts:

I have about $5,000 to spend on building fast computer hardware to run
our problems.  if it works well, I may be able to scrounge up another
$10k/year to scale it up.  I do not have the resources to program very
complex algorithms, administer a full cluster, etc.  (the effective
programmer's rate here is about $50/hour and up, and I have severe
restrictions against hiring outsiders.)  the programs basically have
to work with minimum special tweaking.

There are no real-time needs.  Typically, I operate on historical CRSP
and Compustat data, which are about 1-5GB (depending on subset).  most
of what I am doing involves linear regressions.  I often need to
calculate Newey-West/Hansen-Hodrick/White adjusted standard errors,
and I often do need to sort and rank, calculate means and covariances.
 these are not highly sophisticated stats, but it entails lots of it.
most of what I do is embarrassingly parallel.

Now, I think in the $5k price range, I have a couple of options.
Roughly, the lands

What happened to the vignette of 'parallel'?

Marius Hofert — 2012-04-15T00:03:42

Hi,

I am looking for the latest version of the vignette of 'parallel'. I can't find
it (not on CRAN, not via vignette()). How can I obtain it?

Cheers,

Marius

compiling R 2.15.0 to link to open MPI

Jim Maas — 2012-04-11T11:28:54

I use R in parallel on a cluster.  Our Sys Op has compiled the versions 
for me, up to R 2.13.1 against openMPI libraries so that I can run 
programmes across 404 cores.  I've just asked him to compile 2.15 for me 
but he can't find how to connect it to openMPI.  Has that capability 
disappeared or has the method of connecting changed?

Thanks

J

libRblas.so => not found problem

Jonathan Greenberg — 2012-04-05T16:32:15

r-sig-hpc'ers:

I'm banging my head against a wall with this.  I recently updated R to 2.15 via:

./configure --prefix=/pathtoinstall/ --enable-R-shlib --enable-BLAS-shlib
make
make install

I noticed that I wasn't getting a multithreaded response on, e.g.:
a = matrix(rnorm(5000*5000), 5000, 5000)
b = matrix(rnorm(5000*5000), 5000, 5000)
c = a%*%b

[This used to light up all 12 cores on our system, now it only lights
up one and takes MUCH longer].

I have an update-alternatives set up for /pathto/R/lib/libRblas.so (I
backed up the installed one to libRblas.so_bak) where I have a few
flavors of BLAS I've been playing with (including one that USED to
work, an alpha of OpenBLAS).  I also have the most recent OpenBLAS and
Intel MKL.  The update-alternatives IS working.

When I ldd R I see I no longer have a link to libRblas.so:
 ldd /pathto/R/bin/exec/R
        linux-vdso.so.1 =>  (0x00007fffd4a9e000)
        libR.so => not found
        libRblas.so => not found
        libgomp.so.1 => /usr/lib64/libgomp.so.1 (0

parallel and snow

Jonathan Greenberg — 2012-04-03T20:21:15

r-sig-hpc'ers:

Now that parallel is coming along with R, has anyone written a summary
of how it compares?  Should I be using snow or parallel for (e.g.)
clusterMap() calls?

--j

Rmpi spawning across nodes.

Ben Weinstein — 2012-03-29T18:26:49

Hello all,

I've seen multiple posts on this subject, but haven't clearly been able to
understand the issue. There must be something small.

I am trying to get Rmpi, foreach and snow working on a beowulf cluster,
using debian.

I have succeeded in installing Rmpi, changing the paths, etc.

However, i remain the with my original problem, when

cl<-makeCluster(4,type="MPI")

spawns slaves, they are always on the same node!

There are two processor per node on my cluster. What i need is a way to
spawn slaves across all nodes i've requested in my qsub script.

export PATH=$PATH:/usr/local/pkg/openmpi-1.4.4/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/pkg/openmpi-1.4.4/lib
#!/bin/bash
#PBS -o 'qsub.out'
#PBS -e 'qsub.err'
#PBS -l nodes=4:ppn=1
#PBS -m bea
$PBS_NODEFILE
$hostname

cd $PBS_O_WORKDIR

# Run an R script
R --vanilla < /nfs/user08/bw4sz/Files/Seawulf.R > output.Rout

I have tried using mpirun -np 4 in front of the R - call, but this just
fails without message.

In my R script, i call (alot of

spawning MPI workers through RDCOM on Windows 7

Y R — 2012-03-29T05:31:10

Hello,

I am having performance issues using Statconnector, working on Windows 7,
Intel(R) Core(TM) i7. CPU < at > 2.69GHz. operating on a 64-bit. After reading
archives and docs about Parallel R, I have installed the following Rmpi and
Snow packages:

   * > sessionInfo()
    R version 2.13.1 Patched (2011-07-22 r56481)
    Platform: i386-pc-mingw32/i386 (32-bit)

    locale:
    [1] LC_COLLATE=English_United Kingdom.1252
    [2] LC_CTYPE=English_United Kingdom.1252
    [3] LC_MONETARY=English_United Kingdom.1252
    [4] LC_NUMERIC=C
    [5] LC_TIME=English_United Kingdom.1252

    attached base packages:
    [1] stats     graphics  grDevices datasets  utils     methods
base

    other attached packages:
    [1] Rmpi_0.5-9     rcom_2.2-3.1   rscproxy_1.3-1

    loaded via a namespace (and not attached):
    [1] tools_2.13.1*

Spawning for workers, I have the following:

*    mpi.spawn.Rslaves(nslaves=2, needlog=TRUE)
    2 slaves are spawned successfully. 0 failed.
    master (rank 0, comm 1) of size 3 is runnin

New AMD Bulldozer for Parallel Computing with R

Andreas Recktenwald — 2012-03-23T11:14:40

Dear R enthusiasts,

i`m planning to set up a new PC and consider to buy an AMD FX-8120. For 
my estimations (with floating-point data) I use the genoud optimizer in 
the package rgenoud which (at least I think) would be able to handle the 
8 cores. Is anybody using the AMD proc for computing with R and could 
tell about the performance? What scares me a bit to just buy it are the 
huge amount of benchmarks on the net which tell that AMD has really bad 
performance ( especially for single-threaded apps). Would the Intel I7 
2600 be the better alternative? All the Intel 6-core processors are out 
of budget for me.

Best regards,

Andreas Recktenwald

indexing/mmap database package

Daniel Cegiełka — 2012-03-20T11:33:31

Hi,

I try to understand the indexing package operation. I want to store data
from a C-apps on disk to be able to natively read it into R with
indexing/mmap (and vice versa from R-indexing to C-apps). If I understand
correctly, we have:

 # each column in the envir contains a list of entries:
    #  d: original data mapping
    #  s: sorted data
    #  o: ordered data (location of sorted in data)

We use binsearch() to find index for "value" in sorted data:

binsearch(value, sorted[], n) -> return sorted_idx;

then we use the sorted_idx as a index in ordered data to locate index in
original data:

ordered [sorted_idx] -> return idx;

original [idx]  -> return value;


Is this how it works?

Best regards,
Daniel

[[alternative HTML version deleted]]

generate a DLL on 64-bit windows

Michael Spiegel — 2012-03-16T17:24:40

Hi everyone,

Our team is trying to compile a numerical optimization library that we use with our R package (OpenMx). We've been able to compile it on all platforms + architectures except for 64-bit windows. The library is written in fortran (I think it was written in the Fortran 77 dialect and we're using gfortran to compile it). I've tried compiling with the Rtools 2.14 suite and I get the following segfault captured below in gdb when I try to use the library. I'm using R CMD SHLIB to build the dll. I've also tried using the Rtools 2.15 suite and the resulting .dll crashes on both 32-bit and 64-bit platforms (I haven't run gdb on those crashes, that is on the TO DO list). Has anyone experienced the following error?

Program received signal SIGTRAP, Trace/breakpoint trap.
0x00000000772440c0 in ntdll!RtlUnicodeToCustomCPN ()
  from C:\Windows\system32\ntdll.dll


[[alternative HTML version deleted]]

Question about output the objects on slave by SNOW

Libo Sun — 2012-03-16T16:23:54

Hi all,

I am using SNOW package on my multi-core Windows machine. I am wondering if
there is a way to output the objects on slave, as far as I know 'cat' or
'print' doesn't work. One simple example:

+ y=sin(x)
+ cat('x=', x,'\n')
+ return(y)
+ }
[1] 0.841471 0.841471

I would like to get the information about 'x', like this,

x= 1
x= 1
[1] 0.841471 0.841471

Thanks,
Libo

[[alternative HTML version deleted]]

a CUDA question solved

Hodgess, Erin — 2012-03-15T22:29:44

Hello again:

I found all of my answers (at least for a little while) on the following website:

http://possiblybrainful.blogspot.com/2011/08/using-gpu-in-r-scripts.html

Thanks,
Erin


Erin M. Hodgess, PhD
Associate Professor
Department of Computer and Mathematical Sciences
University of Houston - Downtown
mailto: hodgesse-aB/zTajLQNc< at >public.gmane.org


[[alternative HTML version deleted]]

a CUDA question

Hodgess, Erin — 2012-03-15T21:30:11

Dear R SIG HPC:

I have a CUDA question, please:  I have a small subroutine (stolen from the Extensions manual) and fixed for CUDA

#include <R.h>
#include <Rinternals.h>

__global__ void conv1(double *a, int *na, double *b, int *nb, double *ab)
{
R_len_t i, j, nab = *na + *nb - 1;
for(i = 0; i < nab; i++)
ab[i] = 0.0;
for(i = 0; i < *na; i++)
for(j = 0; j < *nb; j++)
ab[i + j] += a[i] * b[j];
}

and its name is conv1.cu

I then have a Makefile:
EXT := cu 

OBJS := conv1.o

#compiler/preprocessor options
INCS := -I/usr/local/cuda/include
PARAMS := -Xcompiler "-I/usr/share/R/include -fpic"

#linker options
LD_PARAMS := -Xlinker "-L/usr/lib/R/lib -lR -Wl,-rpath,/usr/local/cuda/lib64"
LIBS :=  -L/usr/local/cuda/lib64 -lcublas -L/usr/lib/nvidia-current/-lcuda

TARGETS := conv1.so

CC := /usr/local/cuda/bin/nvcc

all: $(TARGETS) 

$(TARGETS): $(OBJS)
$(CC) -shared $(LD_PARAMS) $(LIBS) $(OBJS) -o $< at >

$(OBJS): %.o: %.$(EXT)
$(CC) -c $(INCS) $(PARAMS) $^ -o $< at >

clean:
rm -rf *o

.PHONY: all clean

And when I run