gmane.comp.lang.ruby.rake

Re: Why do MultiTask and Task send different args to their prerequisites?

Michael Bishop — 2012-11-21T18:18:31

Done. Thanks for the explanation of the code.

_ michael


On Nov 21, 2012, at 12:58 PM, Jim Weirich <jim.weirich< at >gmail.com> wrote:

Re: Why do MultiTask and Task send different args totheir prerequisites?

Jim Weirich — 2012-11-21T17:58:40

On Nov 22, 2012, at 4:33 AM, Michael Bishop <mbishop< at >me.com> wrote:


I like the commit.  It makes the concurrent path look much more like the non-concurrent one.

The new_scope is making a task argument object that is tailored for each task.  For example: if task a with args x and y call prerequisite b with argument x, then task a gets an argument list with "x" and "y" in it and task b gets an argument list with just "x" (and initialized to the value of the the "x" from task a).

The weird thing is that trying to lookup y will still work because of the parent link in the task args.  I'm not sure I like the parent link idea, but that's another issue for another time.

So yes, the multitask path should also invoke new_scope.

If you make this a pull request, I'll merge it.

Re: Why do MultiTask and Task send different args to their prerequisites?

Michael Bishop — 2012-11-21T17:33:49

I made a patch in my branch to make these the same. The test-suite runs to completion, but I'm still wary of it since I don't know if there are good reasons for them to be different :)

 https://github.com/michaeljbishop/rake/commit/ca283160b610c95e1f28cf561af8811fd37d10c3

_ michael


On Nov 21, 2012, at 12:08 PM, Jim Weirich <jim.weirich< at >gmail.com> wrote:

Re: Why do MultiTask and Task send different args totheir prerequisites?

Jim Weirich — 2012-11-21T17:08:28

On Nov 21, 2012, at 2:36 PM, Michael Bishop <mbishop< at >me.com> wrote:


I have no memory of why task has it and multitask doesn't. I'll have to research this and refresh my memory on the whole task arguments implementation before being able to answer.

Stubbing out new_scope to just return the original arguments does cause a test to fail, so evidently its good for something.  I suspect that multitask just didn't get updated when the new_scope was introduced.

Why do MultiTask and Task send different args to theirprerequisites?

Michael Bishop — 2012-11-21T03:36:09

Hello everyone,

I've been looking at something that has been confusing me for a while. It appears that MultiTask and Task send different arguments to their prerequisites and I can't figure out the reason why. I'm hoping someone on the list will know right away.

I've looked and the code for each case and expanded it and they are basically identical except for this last remaining functional difference between the two:

TASK

< at >prerequisites.collect { |p|
  prereq = lookup_prerequisite(p) # returns a task
  prereq_args = args.new_scope(prereq.arg_names)
  prereq.invoke_with_call_chain(prereq_args, invocation_chain)
}

MULTITASK

< at >prerequisites.collect do |p|
  prereq = lookup_prerequisite(p) # returns a task
  prereq.invoke_with_call_chain(args, invocation_chain)
end


Why the extra 

  prereq_args = args.new_scope(prereq.arg_names)

in the TASK case? How does its absence affect MultiTask argument processing? Shouldn't they be the same?

Any ideas?

Thank you,

_ michael

______________________________________

Why do MultiTask and Task send different args to theirprerequisites?

Michael Bishop — 2012-11-21T03:36:09

Hello everyone,

I've been looking at something that has been confusing me for a while. It appears that MultiTask and Task send different arguments to their prerequisites and I can't figure out the reason why. I'm hoping someone on the list will know right away.

I've looked and the code for each case and expanded it and they are basically identical except for this last remaining functional difference between the two:

TASK

< at >prerequisites.collect { |p|
  prereq = lookup_prerequisite(p) # returns a task
  prereq_args = args.new_scope(prereq.arg_names)
  prereq.invoke_with_call_chain(prereq_args, invocation_chain)
}

MULTITASK

< at >prerequisites.collect do |p|
  prereq = lookup_prerequisite(p) # returns a task
  prereq.invoke_with_call_chain(args, invocation_chain)
end


Why the extra 

  prereq_args = args.new_scope(prereq.arg_names)

in the TASK case? How does its absence affect MultiTask argument processing? Shouldn't they be the same?

Any ideas?

Thank you,

_ michael

______________________________________

Re: Planning on Releasing on Monday: Rake 0.9.3 and10.0.0

Jim Weirich — 2012-11-11T04:48:37

On Nov 10, 2012, at 11:34 PM, Luis Lavena <luislavena< at >gmail.com> wrote:


Thanks for the patch!

Re: Planning on Releasing on Monday: Rake 0.9.3 and10.0.0

Jim Weirich — 2012-11-11T04:48:05

On Nov 10, 2012, at 10:49 PM, Luis Lavena <luislavena< at >gmail.com> wrote:


0.9.3 is master
10.0.0 is next-major-release

After the release I will promote master to 10.0.x

Re: Planning on Releasing on Monday: Rake 0.9.3 and10.0.0

Luis Lavena — 2012-11-11T04:34:47

The following patches fix 1 failure and one noisy output during tests:

https://gist.github.com/4053715

The noisy output should be using the "skip" instead, so I've changed it.

This happens on both master and next-major-release branches.

With patches applied:

next-major-release (10.0.0)

ruby 1.9.3p327 (2012-11-10) [i386-mingw32]
455 tests, 1298 assertions, 0 failures, 0 errors, 1 skips

ruby 2.0.0dev (2012-11-10 trunk 37612) [i386-mingw32]
455 tests, 1298 assertions, 0 failures, 0 errors, 1 skips

ruby 2.0.0dev (2012-11-10 trunk 37612) [x64-mingw32]
455 tests, 1298 assertions, 0 failures, 0 errors, 1 skips

master (0.9.3):

ruby 1.9.3p327 (2012-11-10) [i386-mingw32]
474 tests, 1389 assertions, 0 failures, 0 errors, 1 skips

ruby 2.0.0dev (2012-11-10 trunk 37612) [i386-mingw32]
474 tests, 1389 assertions, 0 failures, 0 errors, 1 skips

ruby 2.0.0dev (2012-11-10 trunk 37612) [x64-mingw32]
474 tests, 1389 assertions, 0 failures, 0 errors, 1 skips

Shouldn't 10.0.0 have more tests than 0.9.3? Either way, l

Re: Planning on Releasing on Monday: Rake 0.9.3 and10.0.0

Luis Lavena — 2012-11-11T03:49:55

Hello Jim,

Can you confirm which branch is each of the versions?

Is not clear to me which one represents version 0.9.3 (guess
next-major-release is 10.0.0)

Please confirm so I can provide you test results.

Thank you.

Re: Planning on Releasing on Monday: Rake 0.9.3 and10.0.0

Luis Lavena — 2012-11-11T03:26:51

I was just about to run tests against Ruby 1.9.3, 2.0.0 on Windows, on
both x86 and x64 configurations.

Will send the results in a bit.

Planning on Releasing on Monday: Rake 0.9.3 and 10.0.0

Jim Weirich — 2012-11-11T03:25:34

I'm planning on publishing Rake 0.9.3 and 10.0.0 on Monday, so's here's the last chance on reporting any bugs in either of those versions.

Re: Rake 0.9.3.beta.2 with -j option

2012-10-27T19:03:50

Hello to all,

Am 24.10.2012 um 01:05 schrieb Jim Weirich:


one solution could be do have an API function that must have been called from the rakefile to allow concurrent execution of Tasks. If that function wasn't called, -j defaults to 1 (is ignored). This has the drawback that a rakefile has to explicitly enable parallel execution but on the other side, thread unsafe rakefile won't executed in parallel. Example:

rakefile:

enable_parallel

task :one do
  #compile file one
end

task :two do
  #compile file two
end

task :all => [:one, :two] do
  #link file one and two
end

running rake with -j 2 could execute task :one and :two in parallel. Without the call to enable_parallel(), -j would effectively ignored.

kind regards 
Torsten

Re: Rake 0.9.3.beta.2 with -j option

Vassilis Rizopoulos — 2012-10-24T10:12:07


I have a much better use case and since my patch for allowing this 
within the tasks was rejected because of abuse potential I'm dreading 
losing the ability to use invoke within tasks.
What I do is

task build
  t=calculate_task_with_dynamic_dependencies(params)
  Task[t].invoke
end

Now for various reasons import and other tricks do not work for my use 
case (there's a bit more info on the pull request for dynamic prereqs 
https://github.com/jimweirich/rake/pull/103) but the above idiom works 
really well.
Not allowing it would be fatal for my system.
I'll also +1 the differentiation of -m and -j.
Much prefer explicitly specifying MultiTask instead of having to hunt 
down subtle race condition and resource contention bugs because of the 
implicitly multi threaded environment.
Cheers,
V.-

Re: Rake 0.9.3.beta.2 with -j option

Hugh Sasse — 2012-10-24T09:41:12

On Tue, 23 Oct 2012, Jim Weirich wrote:


I've done a little of this parallel programming in Ruby for an EM
solver, and it does get tricky to find this sort of bug.  And I
tried to simplify it with Tuplespaces.  Does any of this community
have contacts in the Fortran 90,95,2003,2008 community?  From what
I have read of modern Fortran, the compilers are pretty good (i.e.
much better than me) at figuring this stuff out), so there may be
things that could be learned.  The question then becomes: "Is it
tractable for a dynamic language like Ruby?".  Also, do the
algorithms permit one to detect certainty of success, so one can
reject parallel approaches if it comes back "uncertain"?

Actually, this is beginning to sound like a PhD project.


Although GNUmakefiles probably make more use of variables than traditional
ones do, this is essentially true.

Quite often enough at GHz speeds running for days, weeks!
        Hugh

Re: Rake 0.9.3.beta.2 with -j option

Mark Watson — 2012-10-24T00:30:49

With GNUMake it usually safe to assume that a project will *not* work
with -j by default.  Like you said there are probably a bunch of
subtle and not so subtle race conditions.  Even if the developers of a
makefile use -j, you can pretty sure it doesn't work for some build
targets.  So, yeah, I agree that would argue in favor of multitask and
the developer of the rakefile making explicit that they want to allow
a task to execute it's dependencies in parallel.

On 23 October 2012 16:05, Jim Weirich <jim.weirich< at >gmail.com> wrote:

Re: Rake 0.9.3.beta.2 with -j option

Jim Weirich — 2012-10-23T23:05:47

On Oct 23, 2012, at 4:54 PM, Mark Watson <watsonmw< at >gmail.com> wrote:


So you check out a new project from GitHub and decide to run rake on it.  How do you decide if its safe to run with -j or not?  Try it and see?  Wait for subtle unreproducible race conditions to manifest?


GNUMake mainly deals with shelling out to commands.  I suspect Rakefiles that mainly shell out to compilers and linkers will have little problem with -j.

It's the Rakefiles that execute significant Ruby code in process that I'm concerned about.  And maybe I'm overly concerned about this issue, but I've dealt with real-time systems and multiple threads in a past life and know how tricky it can be to get things right.[1]

Re: Rake 0.9.3.beta.2 with -j option

Jos Backus — 2012-10-23T22:38:42


Heh, good one.



Okay, just checking :)



Separating them sounds like it would give us more flexibility. All I was
worried about was mainly a change of semantics of -j down the road. This
approach avoids that, good to hear.



Ah, so it's a general thread-safety issue.

Thanks, Jim.

Jos

Re: Rake 0.9.3.beta.2 with -j option

Mark Watson — 2012-10-23T20:54:38

What about having the old code called by default and if you specify -j
the new parallel code is executed?  That way old rakefiles still work,
and new ones can take advantage of the -j feature (after all that was
good enough for GNUmake).  This is what I've done with my own
parallelization patch (From the number of patches it seems -j is
certainly a much wanted rake feature! :)

https://github.com/watsonmw/rakecpp/blob/master/minusj/minusj.rb

On 23 October 2012 13:34, Jos Backus <jos< at >catnook.com> wrote:

Re: Rake 0.9.3.beta.2 with -j option

Jim Weirich — 2012-10-23T21:06:02

On Oct 23, 2012, at 4:34 PM, Jos Backus <jos< at >catnook.com> wrote:

<an aside>
Saw a post that read: "I have CDO.  It's like OCD but with the letters in alphabetical order."
</an aside>



Oh, yes.  Certainly.  The fault is my own laziness.  


Michael's proposal introduces both a -j and -m flag.  The -j flag sets the thread pool size and the -m turns tasks into multi-task.  The drake behavior is to use -j to do both jobs and leave no way of setting the thread pool for multitasks.


The problem is not incomplete dependency specifications, but using shared/mutable objects in tasks (that suddenly could be executed in multiple threads).  I doubt there is any completely safe way to do this in general, but would like to hear ideas on reducing risk.

Re: Rake 0.9.3.beta.2 with -j option

Michael Bishop — 2012-10-23T20:03:45

Hi Everyone,

I've been thinking about this question of the Drake implementation vs. the ThreadPool implementation and I wanted to share my thoughts. I had no idea the resulting email would be so long. It's my hope to offer interesting points for discussion.

These are all ordered by importance so you can bail when you like :)

Please bear with me...


What Should -j mean? (Part 1.)

There are two features for which I've made pull requests:

 1 - Limit the number of concurrent tasks executing.
 2 - All tasks process their prerequisites in parallel.

Both of these features are activated with separate flags: -j and -m, respectively. Neither feature requires the other. They are complementary.

Drake uses one flag to specify both features but there is no technical reason why Rake couldn't also activate both features with a single -j.

I raise this to separate the issue of "what -j means" from the possibly larger issue of the advantages of the drake implementation.


A Perk of the ThreadPool Implementation

The r