gmane.comp.distributed.boinc.devel

Re: Can we do shared memory with no disk usage?

Nicolás Alvarez — 2013-06-19T19:39:52

2013/6/18 Heinz-Bernd Eggenstein <Heinz-Bernd.Eggenstein< at >aei.mpg.de>:

Or drop iostat and use iotop instead, which gives realtime I/O stats
per *thread*.

Re: Can we do shared memory with no disk usage?

Steffen Möller — 2013-06-19T13:26:18

Re: William

William — 2013-06-19T12:57:33

Hello!
 http://hotelinter.org/hfvsfa/ogent/qdedm/dwdtz.html?ze=bobtb


 William




 vtevy

Re: Can we do shared memory with no disk usage?

Michael Goetz — 2013-06-18T23:38:51

My guess on the 1 GB laptop is that it is thrashing because it doesn't have
enough memory.  That would cause a lot of disk i/o, and could account for
the behavior you're describing.  On cpu-bound tasks, such as what you're
running on boinc, if you exceed physical memory and the OS starts swapping
to disk, it's going to be writing to the swap file **continuously**.

Furthermore, I know Linux has a "swapiness" setting whereby it will write
stuff out to the swap file *before* it needs to, in order to make it easier
for it to potentially make room for something else later.  Therefore, even
if you haven't actually run out of memory, conceivably it could be still be
continuously writing stuff to the swap file, but never reading anything
back in.  I don't know if Windows is as aggressive with regards to
preemptively write stuff out to the swap file.

I would try running just one instance at a time by setting the "use xx% of
the cpus" to whatever value makes boinc run only one task at a time, i.e.,
25% on a quad cor

Re: Can we do shared memory with no disk usage?

David Anderson — 2013-06-18T20:07:27

The state file is written mostly when jobs start and end.
(A long time ago it was written on each checkpoint,
but we changed this).

Re: Can we do shared memory with no disk usage?

David Anderson — 2013-06-18T20:05:36

How much I/O do these machines do when BOINC is not running?

4 GB/hr is about 1 MB/sec.
I can't think of any BOINC activity that would write this much.
We need to look at the system call trace to get more info.

Re: Can we do shared memory with no disk usage?

Heinz-Bernd Eggenstein — 2013-06-18T08:29:07

Hi all,

Interesting. I guess it would be useful to run BOINC on a dedicated 
partition (e.g. ext hard disk/ USB stick)  to isolate BOINC's contribution 
to the stats.

How often is the client_state.xml file updated? It can grow to enormous 
sizes if you have a huge number of  waiting tasks  or unreported finished 
tasks (stderr outputs !!). 

Cheers
HB


-----------------------------------------------------------------
Heinz-Bernd Eggenstein
Max Planck Institute for Gravitational Physics
Callinstrasse 38
D-30167 Hannover,  Germany
Tel.: +49-511-762-19466 (Room 037)



From:   "Steffen Möller" <steffen_moeller-Mmb7MZpHnFY< at >public.gmane.org>
To:     "boinc_dev-C9EgComYM8RUAgJt6FLh2g< at >public.gmane.org" <boinc_dev-C9EgComYM8RUAgJt6FLh2g< at >public.gmane.org>, 
Date:   06/18/2013 09:03 AM
Subject:        Re: [boinc_dev] Can we do shared memory with no disk 
usage?
Sent by:        "boinc_dev" <boinc_dev-bounces-C9EgComYM8RUAgJt6FLh2g< at >public.gmane.org>






My little centrino laptop had SETI (official client) run over

Re: Can we do shared memory with no disk usage?

Steffen Möller — 2013-06-18T07:03:49



My little centrino laptop had SETI (official client) run over night.

$ iostat -h -m
Linux 3.8-1-rt-amd64 (Toshiba)  06/18/2013      _x86_64_        (2 CPU)
...
Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda
                 33.80         0.17         0.39       3040       6859
$ date
Tue Jun 18 00:30:06 CEST 2013
$ iostat -h -m
...
Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda
                 65.32         0.07         0.98       3387      45737
$ date
Tue Jun 18 08:36:50 CEST 2013

Looking only at the last column, in those 8 hours this were

[1] 37.9668

GB and consequently

[1] 4.74585

GB/h


another machine, running about 10 clients in parallel (all Rosetta, one WCG),
had a bit less of IO ... here iostat was run twice with 1h interim sleep

twin1a:~ $ date ; iostat -m -h
Mon Jun 17 23:34:12 CEST 2013
Linux 3.2.0-3-amd64 (twin1a)    06/17/2013      _x86_64_        (24 CPU)

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn

Re: Can we do shared memory with no disk usage?

David Anderson — 2013-06-17T19:29:51

In situations were BOINC is causing unexpectedly large
(> 1 GB/hour) disk I/O,
we need to figure out the source of the I/O.

Re: Can we do shared memory with no disk usage?

Steffen Möller — 2013-06-17T18:14:53

Hello,
 
I presume nobody wants to have the user play too much with any folders. I liked the "all on a ramdisk" idea, but BOINC occupies more than 10GB on the machine(s) in question, and it will similarly ask for that much (too much with today's common 16 or 24 GB setups when adding the memory for the apps) also for other setups.

The checkpointing interval I understood (asked around) to be ignored by the client apps of a quite a few projects. Mine is set to some 7200 seconds (two hours) and the IO does not decrease. It would not be my prime target for optimisation.

Could it be the graphics? We once observed that SETI without configuration for graphics is a few %ages faster than the same client with the same compilation options but graphics-savvy, even though no graphical client was run. If much IO was used for that, this might be an explanation. I cannot test this at the moment.

Cheers,

Steffen


Gesendet: Montag, 17. Juni 2013 um 17:20 Uhr
Von: "Eric J Korpela" <korpela< at >ssl.berkeley.edu>
An: "Steffe

Re: Can we do shared memory with no disk usage?

Eric J Korpela — 2013-06-17T15:20:10

We considered using memory mapped files for the checkpoint state
information in SETI< at >home, but decided that it was virtually impossible to
guarantee synchronization on exit.  And, of course, the problem with using
a ram disk for checkpoints that the checkpoint data disappears if power is
lost.


But if you want checkpoints to not occur, there is a preference for that...
"Tasks checkpoint to disk at most every:   60 seconds"

You can set it to 8 hours.  But you'll also want to choose "suspend to
memory".  Of course the output of the app (which depending on the app might
be more frequent than checkpoints) will still spin up the disks.

If you really want to operate from a ram disk, just copy your project
directory to /dev/shm and link it into /var/lib/boinc/projects.   You'll
still need to back it up occasionally for those times when the power goes
out.






On Mon, Jun 17, 2013 at 1:04 AM, "Steffen Möller" <steffen_moeller-Mmb7MZpHnFY< at >public.gmane.org>wrote:

opendir() failures saga continues

Steffen Möller — 2013-06-17T09:37:33

Dear all,

It took three days of rather intense compute and the opendir() error resurfaced. I (now :-) ) agree with David that my past patches would not have any effect ... that it is the older statically linked clients causing the problem. The application run was mostly rosetta, this time. Nobody else seeing this? I get it in irregular intervals on all three BOINC-long-running machines. The three days it took now are exceptionally short.

My next attempt will be to get the problem to surface with a SETI rebuilt against our libboinc.

Cheers,

Steffen

$ tail /var/lib/boinc-client/stderrdae.txt
dir_open: Could not open directory 'slots/0' from '/var/lib/boinc-client'.
dir_open: Could not open directory 'slots/7' from '/var/lib/boinc-client'.
dir_open: Could not open directory 'slots/2' from '/var/lib/boinc-client'.
dir_open: Could not open directory 'slots/1' from '/var/lib/boinc-client'.
dir_open: Could not open directory 'slots/4' from '/var/lib/boinc-client'.
dir_open: Could not open directory 'slots/8' f

Re: Can we do shared memory with no disk usage?

Steffen Möller — 2013-06-17T08:04:21




The checkpointing and the client_state.xml for my 24/7
machines I would very much like to just switch off or
have updated only every hour or have their update initiated
only upon request by the boinc client.


It should be the shm_open with mmap together, i.e. just substituting the
call to open that BOINC currently performs with shm_open.
From http://pubs.opengroup.org/onlinepubs/009695399/functions/shm_open.html

fd = shm_open("/myregion", O_CREAT | O_RDWR, S_IRUSR | S_IWUSR);
rptr = mmap(NULL, sizeof(struct region),
       PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);

Should be disk-free, with "/myregion" only having a symbolic meaning.
Here
http://stackoverflow.com/questions/4836863/shared-memory-or-mmap-linux-c-c-ipc
I also found a reference to an interesting IPv6 to localhost with multicast idea,
No idea if that is applicable for BOINC. The trend is more towards open_shm+mmap.

Many thanks for the swift reply

Steffen


_______________________________________________
boinc_dev mailing list
boinc_dev< at >

Re: Can we do shared memory with no disk usage?

David Anderson — 2013-06-17T06:46:20

Manager/client communication uses TCP - no disk I/O.
So the possible sources of large disk I/O are:

1) checkpoint or output file generation by apps
2) writing of client_state.xml (or maybe other files)
    by the BOINC client
3) client/app communication via memory-mapped files.
    According to my calculations,
    this should generate less than 1 MB/Hour per task.
    Note: we use memory-mapped files (mmap) instead of
    pure shared memory segments (shmget)
    because Mac OS X has a system-wide limit of 32 shared-mem segments,
    and some Linux systems have limits.
    Maybe there's a way to configure memory-mapped files
    to not write back to the disk file, but I can't find one.

Can someone investigate which of these is the source of the large I/O?

Re: prepared to accept patches for llvm, i.e. clang, compatibility?

Steffen Möller — 2013-06-16T21:02:29

Hi Charlie,

Have many thanks! I can now also confirm BOINC to build with clang. A couple of warnings are left, but those also appear with the gcc.

Cheers,

Steffen

_______________________________________________
boinc_dev mailing list
boinc_dev< at >ssl.berkeley.edu
http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.

Can we do shared memory with no disk usage?

Steffen Möller — 2013-06-16T18:03:31

Hello,

iostat gives rather drastic values for the amount of data that is written to disk by
BOINC and/or its applications.  Some good fellow once crafted a but report about it
   http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=636075
and my own reply was not overly helpful at the time. To reduce the IO overhead is
certainly helpful. Reduced latency is one thing, but with an outreach to the mobile
world there are many SSD / flash device users who care so much about write endurance.

It kept bugging me, and quite a while back it came to me that this may not be
because of the application's work but instead be mere overhead of a file based
communication between the app and the boinc-client / boinc-manager. I just never
got around chasing this up, also having read so often about communication done via
shared memory, which should not need much IO, and if so, then not with disk devices
but something like tmpfs. "Let's see how it is done", I just thought.

From what I found, there are two functions to create share

Re: create_work --target_host ID

David Anderson — 2013-06-09T06:06:58

This feature is called "targeted jobs"; see
http://boinc.berkeley.edu/trac/wiki/AssignedWork

The transitioner doesn't handle these jobs.
When you create a targeted job it creates a record in the "assignment" table.
The scheduler looks for these,
and creates result records as needed.

create_work --target_host ID

Radim Vančo — 2013-06-08T12:54:36

Hi,
I have another question, I noticed that there are attributes 
--target_host ID, --target_team ID and --target_user ID in CREATE_WORK. 
But if I try one of these, transitioner create workunit but no results 
for sending. Is this a bug or test feature? I didn't notice those 
attributes before.

Radim

Re: Fwd: Get OpenCL details on coprocessor line in account->computer details

David Anderson — 2013-06-05T21:51:09

GPU details are sent in scheduler request messages,
and are used to make scheduling decisions.
The details are not stored in the database.
PHP pages can see only what's in the database.

On 05-Jun-2013 2:51 PM, Jorden van der Elst wrote:

Re: Fwd: Get OpenCL details on coprocessor line in account->computer details

Jorden van der Elst — 2013-06-05T21:51:14

It's not stored in the database, but it is sent to the project.
<coprocs>
<coproc_ati>
   <count>1</count>
   <name>AMD Radeon HD 6790/6850/6870 series (Barts)</name>
   <available_ram>2112880640.000000</available_ram>
   <have_cal>1</have_cal>
   <have_opencl>1</have_opencl>
   <req_secs>0.000000</req_secs>
   <req_instances>0.000000</req_instances>
   <estimated_delay>0.000000</estimated_delay>
   <peak_flops>2976000000000.000000</peak_flops>
   <CALVersion>1.4.1741</CALVersion>
   <target>17</target>
   <localRAM>2048</localRAM>
   <uncachedRemoteRAM>2047</uncachedRemoteRAM>
   <cachedRemoteRAM>2047</cachedRemoteRAM>
   <engineClock>775</engineClock>
   <memoryClock>1000</memoryClock>
   <wavefrontSize>64</wavefrontSize>
   <numberOfSIMD>12</numberOfSIMD>
   <doublePrecision>0</doublePrecision>
   <pitch_alignment>256</pitch_alignment>
   <surface_alignment>4096</surface_alignment>
   <maxResource1DWidth>16384</maxResource1DWidth>
   <maxResource2DWidth>16384</maxResource2DWidth>
   <maxResource2DHeight>1

Re: Fwd: Get OpenCL details on coprocessor line in account->computer details

Michael Goetz — 2013-06-05T17:20:33

One of the problems you need to watch out for if you're trying to gather
performance data (or reliability data) about GPUs is that many crunchers
have systems with multiple GPUs and BOINC often will switch a task between
GPUs if the task is interrupted before it completes.  Unless the app itself
records somewhere (e.g., in stderr.txt) when it's starting and on which GPU
it's starting, you won't even know whether the app ran on a single GPU or
on multiple GPUs of potentially different speeds, models, and architectures.

Mike

On Wed, Jun 5, 2013 at 9:30 AM, Jon Sonntag <jon-pH6JACUKs6gLA5E/oR8kMQ< at >public.gmane.org> wrote: