gmane.comp.lang.ruby.rake

Overloading the divide symbol on String to facilitate path concatenation?

Michael Bishop — 2012-04-30T19:00:12

Hello fellow rake developers,

I started using this construction in my rake files and I wanted to get your opinion on it (though if there is a better forum, please direct me there).


Problem
-------

In my rake scripts, I like to assign directory paths to variables so I can use the variables later in my program. The problem is, when concatenating the variables, I end up with ugly combinations like:

  File.join(BUILD_D, "bin", "x86")
  File.join(CACHE_D, "scripts", SCRIPT_TEMP_NAME)


Possible Solution
-----------------

I find this hard to read so I tried something:

  class String
  
    def /(path)
      if !path
        self
      else
        File.join(self, path)
      end
    end
  
  end

Now I can make paths like:

  BUILD_D/"bin"/"x86"
  CACHE_D/"scripts"/SCRIPT_TEMP_NAME


Discussion
----------

My spidey sense is on overload from former C++ days where I was filled with "operator overloading is bad. Always. Never ever do it. Ever... ever"

But I have to say, the more I use this, the more I like it. I can't think of a situation where the operator overloading will get me in to trouble but maybe I haven't thought it through hard enough.

What do you think? Do you have strategies for making path concatenation less ugly in your scripts?

_ michael

ANN: (yet another) version of -j implementation (includes -m for multitask)

Michael Bishop — 2012-04-27T13:54:53

Hello Everyone,

My last implementation of -j had a fatal flaw which could have led to stack overflow for large amounts of tasks. I discovered this after prototyping a version with drake's feature of turning tasks into multitasks.

I have a new version with a rewritten MultiTask implementation which removes that flaw, is smaller, and easily fits into Rake code. In addition, I've added an optional "--multitask -m" flag to turn every task into a multitask (in direct homage to drake).

   https://github.com/michaeljbishop/rake

I've again added a pull-request for the inclusion of the change to the master branch (as always allowing for further changes to match the style and simplicity of the original)

  https://github.com/jimweirich/rake/pull/113

Questions and comments as always, are welcome.

Sincerely,

Michael Bishop


---


## PROBLEM SUMMARY (THE CASE FOR -j and -m)

Rake can be unusable for builds invoking large numbers of concurrent external processes.

## PROBLEM DESCRIPTION

Rake makes it easy to maximize concurrency in builds with its "multitask" function. When using rake to build non-ruby projects quite often rake needs to execute shell tasks to process files. Unfortunately, when executing multitasks, rake spawns a new thread for each task-prerequisite. This shouldn't cause problems when the build code is pure ruby (for green threads), but when the tasks are executing external processes, the sheer number of spawned processes can cause the machine to thrash. Additionally ruby can reach the maximum number of open files (presumably because it's reading stdout for all those processes).

## SOLUTION SUMMARY

This request includes the code to add support for a `--jobs NUMBER (-j)` command-line option to specify the number of simultaneous tasks to execute.

  * To maintain backward compatibility, not passing `-j` reverts to the old behavior of unlimited concurrent tasks.

As a nod to [drake](http://drake.rubyforge.org), a `--multitask (-m)` flag is also included which when supplied, changes tasks into multitasks.

## SOLUTION

Rather than spawning a new thread per prerequisite `MultiTask` now sends its prerequisites to a `WorkerPool` object. `WorkerPool.new(n).execute_blocks` has the same semantics as `Thread.new`...`join` but caps the thread count at `n`.

### Core Change

    threads = < at >prerequisites.collect { |p|
      Thread.new(p) { |r| application[r, < at >scope].invoke_with_call_chain(args, invocation_chain) }
    }
    threads.each { |t| t.join }

...becomes...

    < at >< at >wp ||= WorkerPool.new(application.options.thread_pool_size)
    
    blocks = < at >prerequisites.collect { |r|
      lambda { application[r, < at >scope].invoke_with_call_chain(args, invocation_chain) }
    }
    < at >< at >wp.execute_blocks blocks


To support `-m`, the `MultiTask` implementation has moved to `Task#invoke_prerequisites_concurrently` and is called from `MultiTask#invoke_prerequisites`. This enables concurrent behavior for `Task`  when `-m` is used.

### Details

`WorkerPool#execute_blocks` adds the passed-in blocks to a queue, ensures there are enough threads to execute them (under the maximum), and sleeps the current thread until the blocks are processed.

This creates a few potential problems:


Yes it would. This is solved as `#execute_blocks` removes the current thread from the thread pool just before it sleeps and creates a new one in its place. When all the blocks are processed, the current thread is added back to the pool (adjusting for the max-size). There are always enough available threads in the thread pool for processing.


`WorkerPool#execute_blocks` knows how many threads are waiting for their blocks to be processed. If, upon its awakening, it notices there are no threads waiting on blocks, it shuts down the thread pool.

### Statistics

     ---LINES--     ----LOC---
      old   new      old   new   File Name
     ----------     ----------   ----------
      598   605      477   484   lib/rake/application.rb
       16    13       11     8   lib/rake/multi_task.rb
      327   341      210   222   lib/rake/task.rb
            111             80   lib/rake/worker_pool.rb
     4264  4393     2696  2792   TOTAL
     --------------------------------------
           +129            +96   SUMMARY

## TESTS

Tests are included for all new functionality

## REQUIREMENTS

The Ruby version requirements remain the same. `lib/rake/worker_pool.rb` adds two new requirements: `thread` and `set`

Announcement: Alternate -j implementation

Michael Bishop — 2012-04-19T11:36:37

Hello Everyone,

I've recently finished an implementation of -j <max_concurrent_jobs> and would be delighted if the members of this list would take a detailed look at it

It's in this rake branch on github:

https://github.com/michaeljbishop/rake/commit/295c7a4d6d58b3e10c27b940a8259dd3e01c52f0
or
http://bit.ly/JMVARy

In addition, I've added a pull-request for the inclusion of the change to the master branch (allowing of course for further changes to match the style and simplicity of the original)

https://github.com/jimweirich/rake/pull/112

My apologies if this has been hashed out many times on this list. I'm hoping to offer a fresh look at the problem. Thank you for your consideration.

Sincerely,

Michael Bishop

---

SUMMARY
-------

USER INTERFACE:

The user-interface is simply a new -j flag that specifies the maximum number of tasks that can execute simultaneously (discussed here before as I've seen). If the -j flag is omitted, the old behavior of unlimited concurrent tasks is retained.

IMPLEMENTATION:

The implementation is inspired by Apple's Grand Central Dispatch model. The core of the problem is solved via changes to the file "multi-task.rb".

The current implementation of MultiTask#invoke_prerequisites spawns a new thread for each prerequisite and then waits to return until all the threads have completed.

In the alternate implementation, MultiTask#invoke_prerequisites creates a list of blocks inside which each prerequisite is called. Each block is then added to a thread-safe Queue for processing. A class-member ThreadPool is then expanded to include enough threads to consume and call the blocks, but only up to the limit as passed to -j.

Then, rather than MultiTask#invoke_prerequisites sleeping by joining to the newly spawned threads, it participates in the processing of the queued blocks while it waits. This prevents deadlock situations should more threads call MultiTask#invoke_prerequisites.

How then does MultiTask#invoke_prerequisites know when its prerequisites have finished? Before enqueing the prerequisite blocks, it surrounds the original prerequisite block with *another* block which maintains bookkeeping tasks as to when the prerequisite is completed and which thread is working on it. That block is what is added to the queue.

Two conditions need to be met for MultiTask#invoke_prerequisites to return:

1 - It notices that its prerequisites have all been processed
2 - It notices there are no more blocks on the queue but its prerequisites are not yet finished. In this case, it joins those threads that are still executing its prerequisites.

What is attractive to me about this implementation is that the flow retains the simplicity of the original: MultiTask#invoke_prerequisites sends all the prerequisites to be executed then waits until they are done.

THE CASE FOR -J
---------------

(Quoted from the pull-request)

PROBLEM SUMMARY:

Rake can be unusable for builds invoking large numbers of concurrent external processes.

PROBLEM DESCRIPTION:

Rake makes it easy to maximize concurrency in builds with its "multitask" function. When using rake to build non-ruby projects quite often rake needs to execute shell tasks to process files. Unfortunately, when executing multitasks, rake spawns a new thread for each task-prerequisite. This shouldn't cause problems when the build code is pure ruby (for green threads), but when the tasks are executing external processes, the sheer number of spawned processes can cause the machine to thrash. Additionally ruby can reach the maximum number of open files (presumably because it's reading stdout for all those processes).

SOLUTION SUMMARY:

This request includes the code to add support for a "--jobs NUMBER (-j)" command-line option to specify the number of simultaneous tasks to execute.

SOLUTION:

The solution creates a work queue to which blocks calling the task-prerequisites are added and a thread pool to process them. To prevent deadlock, the task that added the pre-requisites processes items on the queue (alongside the thread pool) until its prerequisites have been processed.

To maintain backward compatibility, not passing -j reverts to the old behavior of unlimited concurrent tasks.

REQUIREMENTS:

The Ruby version requirements remain the same. "multi-task.rb" adds two new requirements: 'thread' and 'set'

_______________________________________________
Rake-devel mailing list
Rake-devel< at >rubyforge.org
http://rubyforge.org/mailman/listinfo/rake-devel

Merging drake and rake?

Jos Backus — 2011-09-17T05:39:04

Hi,

Are there any plans to merge drake's features back into rake?

Jos

No longer able to run Rake tests due fork() usage

Luis Lavena — 2011-08-16T17:03:08

Hello,

I guess the subject explain it completely, but I'm not able to run
Rake tests anymore due fork usage in them.

This not only affects Windows but also JRuby.

I know this is related to session gem removal, but was wondering if
there isn't another workaround?

Thank you.

Beta 1 for Rake 0.9.3 is out

Jim Weirich — 2011-08-06T03:05:45

I've pushed a beta version of 0.9.3 out to Rubygems.org.  

Here is the change list.

* The rake test loader now removes arguments it has processed.  Issue #51
* Rake::TaskArguments now responds to #values_at
* RakeFileUtils.verbose_flag = nil silences output the same as 0.8.7
* Rake tests are now directory-independent
* Rake tests are no longer require flexmock
* Commands constant is no longer polluting top level namespace.
* Show only the interesting portion of the backtrace by default (James M. Lawrence).
* Added --reduce-compat optiont to remove backward compatible DSL hacks (James M. Lawrence).

The rake tests task pattern issue isn't explicitly mentioned here, but it seems to be resolved.  Eric, I'm assuming you did something to fix that (cf. https://github.com/jimweirich/rake/issues/51)

Anyways, grab the beta and give it a spin and report back if there are issues.

Thanks.

3 Failures at master (a774ef8)

Luis Lavena — 2011-06-23T23:37:33

Hello,

I'm getting the following failures running against 1.8.7, 1.9.2 and trunk:

  1) Failure:
test_directory_win32(TestRakeDirectoryTask)
[C:/Users/Luis/Projects/oss/jimweirich/rake/test/test_rake_directory_task.rb:32]:
Expected Rake::FileCreationTask, not Rake::FileTask.

  2) Failure:
test_egrep_with_output(TestRakeFileList)
[C:/Users/Luis/Projects/oss/jimweirich/rake/test/test_rake_file_list.rb:345]:
Expected "xyzzy.txt:2:XYZZY", not "xyzzy.txt:2:XYZZY\r\n".

  3) Failure:
test_egrep_with_block(TestRakeFileList)
[C:/Users/Luis/Projects/oss/jimweirich/rake/test/test_rake_file_list.rb:356]:
Expected ["xyzzy.txt", 2, "XYZZY\n"], not ["xyzzy.txt", 2, "XYZZY\r\n"].

Is worth mentioning that I'm using Windows :-)

Seems to me that #2 and #3 are related to newlines output in the
console, perhaps there is another way to verify the output
independently of LF/CR?

Rake 0.9.0.beta.5 is out

Jim Weirich — 2011-03-14T01:24:13

No big functionality changes in this one, just cleaning up some of the cruft leftover from the namespacing experiment. I expect this version to become 0.9.0 final unless someone discovers a big problem.

Environment changes in the beta

James M. Lawrence — 2011-03-02T03:55:18

% cat mystuff.rb
module MyStuff
  CONST_B = 44
end

% cat Rakefile
require 'rake'
require './mystuff'

module MyStuff
  CONST_A = 33
end

p MyStuff::CONST_A
p MyStuff::CONST_B
# => uninitialized constant Rake::Environment::MyStuff::CONST_B

task :default

% rake --version
rake, version 0.9.0.beta.1

% rake
33
rake aborted!
uninitialized constant Rake::Environment::MyStuff::CONST_B

Writing ::MyStuff::CONST_B instead works, and conversely
::MyStuff::CONST_A is an error.

It seems that Rake::DSL already provides the necessary partitioning.
If I wanted to keep the global scope clean, I would do

module MyStuff
  extend Rake::DSL

  task :default do
    # ...
  end
end

Could an example like that in the README replace the need for
environment.rb?

New Beta Version of Rake

Jim Weirich — 2011-02-28T12:46:28

A new beta version of Rake is out (0.9.0.beta.1) and is ready for folks to kick the tires.  It's been a while since the last release and with Eric's help we've been whipping this into shape.

Install with:   gem install rake --pre

The CHANGES file should have all the details.

About that new magic comment ...

James M. Lawrence — 2011-02-28T04:10:02

Assume the following tasks were written for Rake 0.8.x.

  # This comment describes a non-toplevel-task.
  task :complex_task do
    # ...
  end

  # For internal use only -- washes laundry, takes out trash, other
  # crap.
  task :house do
    # ...
  end

  #
  # I prefer this comment style.
  #
  task :style do
    # ...
  end

#### output of rake --tasks:

  rake complex_task  # This comment describes a non-toplevel-task.
  rake house         # crap.

Fortunately my own commenting style avoids this problem. Others will
likely encounter some annoyance (small or possibly great).

One could make the comment-grabber smarter, but the question is: How
should a programmer document a top-level task? There is already an API
method for doing so: desc. The new comment-to-desc feature does not
solve an actual problem or deficiency.

There could hundreds of these new magic-comment 'desc's cropping up,
making the --tasks output useless. Even one newly-public comment could
be an issue (think of a programmer making a veiled complaint about a
co-worker). A programmer may have good reason to use '#' instead of
'desc'. Why should a new version of Rake suddenly dismiss that reason?

Surely there is someone out there who wrote

  (0...1000).each do |i|
    # blah blah ...
    task "blah#{i}" do
      # ...
    end
  end

#### rake --tasks output:

  rake blah0         # blah blah ...
  rake blah1         # blah blah ...
  rake blah10        # blah blah ...
  rake blah100       # blah blah ...
  rake blah101       # blah blah ...
  rake blah102       # blah blah ...
  rake blah103       # blah blah ...
  rake blah104       # blah blah ...
  rake blah105       # blah blah ...
  rake blah106       # blah blah ...
  rake blah107       # blah blah ...
  rake blah108       # blah blah ...
  rake blah109       # blah blah ...
  rake blah11        # blah blah ...
  rake blah110       # blah blah ...
  rake blah111       # blah blah ...
  rake blah112       # blah blah ...
  ...

In exchange for annoying that programmer, what do we get in return?
Usually compatibility is purposefully broken in order to fix a flaw.
What is the flaw in 'desc'? Why is '#' better than 'desc'?

The worst-case scenario is an auto-generated web page of the --tasks
output. With the new Rake, a password in the comments gets exposed.

Previously '#' meant internal documentation for a non-top-level task.
Now it means public documentation for a top-level task, an effective
alias of 'desc'. For internal docs, leave a space between the comment
and the task definition. That is natural for me, but requiring all
programmers to do that in all Rakefiles seems rather unnatural.

Task for nested instance of Rake

Jesper Tholstrup Pedersen (JUP — 2011-02-22T08:39:17

Hi.

I'm a .NET developer. I recently decided to use Rake to handle things
such as deployment and database management for my .NET projects.
The projects are still build using MSBuild cause of Visual Studio (VS).

One problem I continually encounter is that there is no natural way to
order dependent projects using Rake. 
I'll give an example: In .NET it is quite natural to order projects in a
nested structure with a master project that have a number of
sub-projects in a hierarchical structure (as shown below). 

C:\Projects\Project1\master vs file
C:\Projects\Project1\Rakefile.rb
C:\Projects\Project1\... (master project files)

C:\Projects\Project1\SubProject1\sub1 vs file
C:\Projects\Project1\SubProject1\Rakefile.rb
C:\Projects\Project1\SubProject1\... (sub1 project files)

C:\Projects\Project1\SubProject2\sub 2 vs file
C:\Projects\Project1\SubProject2\Rakefile.rb
C:\Projects\Project1\SubProject2\... (sub2 project files)
...

Now if you for any reason have an interest in separating the
sub-projects from the master in VS, you would still order the files and
folders in a hierarchy since there is a dependency.

By separating the individual sub-projects from the master you no longer
have the option to say run all tests in all projects, this is where Rake
becomes interesting. 

The rakefile for each sub-project are quite easy to create and maintain
since they individually are isolated, the problem is when we get to the
master rakefile. I want to stress the point that the sub-projects
rakefiles are completely isolated meaning that they are not and should
not be aware of the master rakefile.

I have not been able to find a satisfactory solution where the master
rakefile can call each sub-projects rakefile. In other build management
solutions like Ant, NAnt and MSBuild there is a task that can run a
nested instance on a specified build file.

There appears to be some half-baked solutions for this with Rake:

1. Run a rake command for each sub-projects rakefile on the system as a
command line.
2. Import the tasks in each sub-projects rakefile and run them.

The number 1 solution has a few problems:
1. Propagating the initial Rake options (e.g. --trace) must be performed
manually. 
2. A failed sub-project build will not stop the master from calling the
next sub-project.

The number 2 solution has the problem that the sub-projects rakefile is
dependent on all other rakefiles in the project since there must not be
any task name clashes. This is unacceptable since you would no longer
have the ability to run a sub-projects rakefile with the standard
assumptions (like running "rake test" on a subproject).

Adding a "Rake task" which enables Rake to run a nested instance would
be the perfect solution for me, I just don't have that much experience
with Ruby yet to be able to develop it myself.

So the real question is: does anybody have this need?
And would somebody with the Ruby experience like to create this task?

/Jesper

Parameter not task key?

2011-02-20T12:19:22

Hello,
I'm currently trying to build an existing C++ project with rake. I want to build different "flavors" (debug, release, etc.) of the software. Every flavor uses it's own output directory, so the different flavors can live in parallel. To pass the flavor to rake, I use the task parameter. The tasks are build automatically from the existing source directory structure. What puzzled me  a little bit is, that the parameter isn't part of the key (or name) of a task. When I try to build two different flavors of the same task, the task gets executed once, not twice as I've thought it should be. Example:

rake libs[debug] libs[release]

Is this intended behavior or should I fill a bug-report / feature-request?

best regards
Torsten

btw: Thanks for the wonderful tool (rake)

rake check_manifest deletes my doc Directory

Zeno Davatz — 2011-02-11T15:52:26

Hi

I am using rake for a lot of our web projects. Those project all have
a _doc_ directory.

Now if I go by the books and run

rake check_manifest

my doc directory is wiped away. That is sad and not so userfriendly.

Any hints how I could prevent that?

Thank you for your Feedback.

Best
Zeno

Adding usage data gathering functionality to rake

Vassilis Rizopoulos — 2011-01-25T06:15:48

I've posted this in the github issue tracker 
(https://github.com/jimweirich/rake/issues/#issue/21).
Here's hoping this list s still active, since I was only able to find 
archives till June last year.

I'm experimenting with gathering usage data from rakefiles.
Primarily I would like to know which task was called, its dependency 
chain and the time execution took (ideally total time and time per task).

I thought that if rake provided this out of the box it would be extra 
convenient (using a profiler I have to aggregate method calls, time only 
works on *x platforms etc.) and would provide the right level of 
granularity.

I've done a quick proof of concept (it's linked in the comments of the 
issue).
Any thoughts?
V.-

New loading behavior

James M. Lawrence — 2010-06-07T16:26:42

task :default do
end

extra = "Rakefile.extra"
File.open(extra, "w") { |f| f.puts "p SPEC" }
SPEC = 99

# OK in rake-0.8.7; error in rake-0.8.99 (uninitialized constant SPEC)
load extra

# OK in rake-0.8.99; error in rake-0.8.7 (load_rakefile not present)
load_rakefile extra

Is this the expected behavior?

A handful of issues

James M. Lawrence — 2010-06-07T07:03:29

==== 1 - Task#needed? applies to immediate prereqs only

file "a" => "b" do
  touch "a"
end

file "b" => "c" do
  touch "b"
end

file "c" do
  touch "c"
end

task :default do
  touch ["a", "b"]
  sleep 1
  touch "c"
  p Rake::Task["a"].needed?  # => false
  # ... but "a" is needed because "c" is newer
end

==== 2 - Task#needed? can be wrong even for immediate prereqs

file "a" => "b" do
  touch "a"
end

file "b" do
  touch "b"
end

task :default do
  touch "a"
  rm_f "b"
  p Rake::Task["a"].needed?  # => false
  # ... but "a" is needed because "b" will be newly created
end

==== 3 - Filesystem timestamp granularity

memo = []

file "a" => "b" do
  memo << "a"
  touch "a"
end

file "b" do
  memo << "b"
  touch "b"
end

task :default do
  touch "a"
  rm_f "b"
  Rake::Task["a"].invoke
  p memo # => ["b"]
  # ... but should be ["b", "a"] since "b" is newer
end

==== 4 - Dependency relation should be transitive

memo = []

file "a" => "b" do
  memo << "a"
  touch "a"
end

file "b" => "c" do
  memo << "b"
  # does not create b
end

file "c" do
  memo << "c"
  touch "c"
end

task :default do
  touch ["a", "b"]
  rm_f "c"
  sleep 1 # filesystem timestamp workaround
  Rake::Task["a"].invoke
  p memo # => ["c", "b"]
  # ... but "a" is needed since it (transitively) depends on "c"
end

==== 5 - Trace output is misleading

file "a" => "b" do
  touch "a"
end

file "b" => "c" do
  touch "b"
end

file "c" do
  touch "c"
end

task :default do
  touch ["a", "b"]
  sleep 1
  touch "c"
  Rake.application.options.trace = true
  Rake::Task["a"].invoke
  # =>
  #   touch a b
  #   touch c
  #   ** Invoke a (first_time, not_needed) <--- Liar! It is needed.
  #   ** Invoke b (first_time)
  #   ** Invoke c (first_time, not_needed)
  #   ** Execute b
  #   touch b
  #   ** Execute a <--- See?
  #   touch a
end

**** Discussion

==== 1 - Task#needed? applies to immediate prereqs only

Having Task#needed? recursively check all prereqs is needlessly slow,
so the current behavior is fine.  However I think its use should be
deprecated in the published API.  We could alias it to
Task#locally_needed? and additionally write Task#globally_needed?.

==== 2 - Task#needed? can be wrong even for immediate prereqs

Should be fixed if Task#needed? (or Task#locally_needed?) is
"officially" supported.

http://github.com/quix/rake/commits/mainline-file-task-needed

==== 3 - Filesystem timestamp granularity

Unless you have ext4fs, your filesystem most likely records timestamps
only to the nearest second, making it easy to create same-time files.

The fix is easy: change timestamp comparison from > to >=.  It's a
pessimization: "When in doubt, rerun."  It sounds safer, but what
consequences lurk?

http://github.com/quix/rake/commits/mainline-timestamp-granularity

==== 4 - Dependency relation should be transitive

Let ~> denote "eventually depends on".  If a~>b and b~>c, then we want
a~>c.  Thus if c executes then a should execute, as is implied by the
dependency graph.  Parallel Rake has to assume transitivity, so its
behavior is different.

I think a file task which executes but does not update the file is
reneging.  Regarding this as an error would make regular Rake
consistent with parallel Rake, sidestepping the transitivity issue.

==== 5 - Trace output is misleading

This is related to item 1 since "not_needed" reflects Task#needed?.
Perhaps change the output to "not_immediately_needed" or some such?

Rake 0.8.7 broken on Windows

Hartin, Brian — 2010-05-26T20:42:29

Hi all,

In Rake::AltSystem#repair_command, you're now appending "call " to the
beginning of the command.  This has the effect of losing the working
directory.  For example, running 'rake test' from the root directory of
a project will fail to find its unit tests.  It also causes a lot of gem
installation failures, as 'gem install' often seems to run rake from the
installed gem directory to complete the installation, and fails to find
necessary files.

This is not backward compatible, obviously.

1 - Is there a workaround?

2 - Why is the 'call' necessary?  Is this part of preserving the linux
'sh' semantics you mentioned?  If so, how would one avoid this problem
in linux?

Thanks,

Brian Hartin

Comments about parallel building

Heath Kehoe — 2010-05-19T18:00:53

For our project one of the top goals was minimizing build times; so 
being able to invoke compilers/linkers in parallel to take advantage of 
multiple-core computers was a necessity.

(Since finishing my implementation, I have learned about drake. I intend 
to take a look at that as a possible alternative approach.)

The user specifies "j=N" on the command line, similar to make's -jN. 
When N>1, tasks are created as MultiTasks. Since FileTasks aren't 
MultiTasks, when there's a library or application, a MultiTask is 
created that has all the object files as prerequisistes; then the 
library/app includes that MultiTask in its prerequisites. This allows 
the object files to compile in parallel.

Since MultiTasks run all their prereqs at the same time, I made a custom 
version of the 'sh' method which synchronizes using a semaphore; that 
semaphore is initialized with the j=N value that the user specified. 
That way, only N commands will be invoked at once.

Recently, I tried updating to ruby 1.9.1. My MultiTask approach fell 
down because it created too many threads. 1.9 uses 'real' threads which 
consume more resources than 1.8's 'fake' (but very lightweight) threads. 
When I modified MultiTask to spawn threads in batches to limit their 
number, it worked but it was many times slower than before, probably due 
to the additional overhead of creating/joining lots of OS threads.

So we're sticking with ruby 1.8 for now. As I said, perhaps drake's 
approach will be better for 1.9, I'll try that out sometime.

-heath

______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
______________________________________________________________________

Comments from a new rake user

Heath Kehoe — 2010-05-19T17:13:55

Hello all,

I have recently implemented a build system based on Rake for a 
decently-sized project (~8000 source files) which builds code and data 
for several libraries and applications targeting four separate platforms.

I was new to both rake and ruby when I started; and it's a testament to 
the awesomeness of both that I was able to learn them quickly to be able 
to implement a fairly complicated system in less than a month.

Anyway, I wanted to pass along some comments.

Firstly, when I first got stuff building with rake (replacing a build 
environment that used gnu make) I noticed that a null build (where 
everything was up to date) took a really long time. I used the ruby 
profiler and found that approx. half the run time was spent doing file 
stats (exist? and mtime), with an average of 70 calls *per file* during 
a rake run. As it runs under Cygwin on Windows those file stat 
operations are more expensive than they are on Linux. I should note that 
I'm using ruby 1.8.7 and the rake that gem installed (0.8.7).

If you look at the FileTask code, the needed? method calls File.exist? 
then timestamp, which calls File.exist? then File.mtime. That's three 
stats in a row right there for the file itself; then each prerequisite 
is asked for its timestamp which generates two stats for each (for 
FileTasks that is). My approach was to create a simple global cache that 
uses the filename as the key and stores the file's mtime.

module Rake
     # Modify Rake's FileTask to use our cached file tests
     class FileTask < Task
         def needed?
             ! File.cached_exist?(name) || out_of_date?(timestamp)
         end

         def timestamp
             if File.cached_exist?(name)
                 File.cached_mtime(name.to_s)
             else
                 Rake::EARLY
             end
         end

         def execute(args=nil)
             ret = super
             File.invalidate_cache(name)
             ret
         end
     end
end

The invalidate_cache method simply deletes the cache entry for the given 
file; which is necessary if the file was changed by the task's action. 
The execute method does this for the FileTasks's own target; if an 
action creates or modifies other files as a side-effect, I explicitly 
call invalidate_cache in the action block for each side-effect file to 
make sure the cache doesn't contain any stale info.

This change resulted in an order of magnitude improvement in run-time.

Now, I'm not saying rake should adopt this specific optimization; 
however I think you should consider some type of caching to reduce the 
quantity of exist?/mtime calls. Perhaps the FileTask could simply cache 
its own exist?/mtime results (invalidated when execute runs).

I have more to say about dependency generation and multitasking, but 
I'll send those thoughts in separate emails.

-Heath



______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
______________________________________________________________________

Add path ( Pathname to file tasks)

Dominic Sisneros — 2010-05-18T23:15:16

_______________________________________________
Rake-devel mailing list
Rake-devel< at >rubyforge.org
http://rubyforge.org/mailman/listinfo/rake-devel