gmane.comp.lang.ruby.rake

Overloading the divide symbol on String to facilitate path concatenation?

Michael Bishop — 2012-04-30T19:00:12

Hello fellow rake developers,

I started using this construction in my rake files and I wanted to get your opinion on it (though if there is a better forum, please direct me there).


Problem
-------

In my rake scripts, I like to assign directory paths to variables so I can use the variables later in my program. The problem is, when concatenating the variables, I end up with ugly combinations like:

  File.join(BUILD_D, "bin", "x86")
  File.join(CACHE_D, "scripts", SCRIPT_TEMP_NAME)


Possible Solution
-----------------

I find this hard to read so I tried something:

  class String
  
    def /(path)
      if !path
        self
      else
        File.join(self, path)
      end
    end
  
  end

Now I can make paths like:

  BUILD_D/"bin"/"x86"
  CACHE_D/"scripts"/SCRIPT_TEMP_NAME


Discussion
----------

My spidey sense is on overload from former C++ days where I was filled with "operator overloading is bad. Always. Never ever do it. Ever... ever"

But I have to say, the more I use this, the more I like i

ANN: (yet another) version of -j implementation (includes -m for multitask)

Michael Bishop — 2012-04-27T13:54:53

Hello Everyone,

My last implementation of -j had a fatal flaw which could have led to stack overflow for large amounts of tasks. I discovered this after prototyping a version with drake's feature of turning tasks into multitasks.

I have a new version with a rewritten MultiTask implementation which removes that flaw, is smaller, and easily fits into Rake code. In addition, I've added an optional "--multitask -m" flag to turn every task into a multitask (in direct homage to drake).

   https://github.com/michaeljbishop/rake

I've again added a pull-request for the inclusion of the change to the master branch (as always allowing for further changes to match the style and simplicity of the original)

  https://github.com/jimweirich/rake/pull/113

Questions and comments as always, are welcome.

Sincerely,

Michael Bishop


---


## PROBLEM SUMMARY (THE CASE FOR -j and -m)

Rake can be unusable for builds invoking large numbers of concurrent external processes.

## PROBLEM DESCRIPTION

Rake makes it easy to maxi

Re: Announcement: Alternate -j implementation

Mark Watson — 2012-04-20T21:33:40

Yes, in the sense that all tasks can be executed in parallel with other tasks.

Upside parallelism is octagonal to the method
('file'/'directory'/'task') used to define the task.

It has the downside that some tasks may not be suitable for executing
in parallel.  E.g. the dependencies are wrong, or the task itself
incorrectly shares some resource with other tasks (say multiple tasks
modify the same file as an intermediate step during the task
execution).  IMHO, making stuff work in parallel requires thinking
about the problem anyway and often leads to better code.  If one
needed to make something single thread a mutex should do the trick.
So it's not that hard to be a good -j citizen :)


Yes, this is whatever is set in the -j option.


Yes, it's to help people debug rakefile dependencies, as you've
probably figured :)

On 20 April 2012 09:51, Michael Bishop <mbishop< at >me.com> wrote:

Re: Announcement: Alternate -j implementation

Michael Bishop — 2012-04-20T08:51:17

Quick question for the list:

After reading more about drake, I wanted to summarize the differences for my own understanding.

Differences in drake from rake:

1 - All tasks are implicitly multitask
2 - There is a cap on the number of concurrent tasks
3 - There is a control to create a repeatable single-threaded random invocation of tasks (--randomize[=SEED])

Is this correct?

_ michael

Re: Announcement: Alternate -j implementation

Steven Parkes — 2012-04-20T02:25:38

On Apr 19, 2012, at 2:40 PM, Hongli Lai wrote:


I went through this just this week, after getting tired of some C++ builds using only 1 core. I first looked at multitask but got stopped by the file dependence issue. Then I did some wider searching and stumbled across drake. Been happy ever since.

I used make a lot in the past and always made the effort to make all the dependencies explicit (rather than implicit through dependence list order) so that make -j would work. But a lot of Makefiles don't do this and break with make -j just as many rakefiles would.

I can see this being a bigger issue with rake since its expressiveness makes it a lot easier to write things that have side-effects and won't parallelize. (Just like I see people write rakefiles that don't converge: that rebuild artifacts because they haven't described the dependencies correctly.)

But in any case, for my part, with (d)rake -j, I don't go back to make anymore at all, which makes my world a little nicer.

Re: Announcement: Alternate -j implementation

Hongli Lai — 2012-04-19T21:40:18

I agree. I would really love to see a 'make -j' equivalent in Rake.

Re: Announcement: Alternate -j implementation

Mark Watson — 2012-04-19T17:03:21

Second that. The option to specify parallel builds for any task is
handy because the task may not derive from multitask, e.g. it could be
a file task to build a c library from multiple source files that you
want to build in parallel.

Personally I've used my own patch to rake to do this, that was before
drake, but I still use it due to drake having an extra dependency.

https://github.com/watsonmw/rakecpp/blob/master/minusj/minusj.rb

On 19 April 2012 17:21, Jos Backus <jos< at >catnook.com> wrote:

Re: Announcement: Alternate -j implementation

Michael Bishop — 2012-04-19T16:28:24

On Apr 19, 2012, at 11:48 AM, Jim Weirich wrote:


That is exactly right. It only affects Multitask.

Some additional interesting tidbits: There is a queue of blocks that is a class member of Multitask which all Multitask instances use to queue their prerequisites. So, your thread's call to MultiTask#invoke_prerequisites may end up processing prerequisites for another's call to MultiTask#invoke_prerequisites. No matter though, your call won't return until all your prerequisites are completed.


Thank you for that clarification. I'd only assumed that, given my rather cursory look, and it's nice to have it confirmed.

_ michael

Re: Announcement: Alternate -j implementation

Michael Bishop — 2012-04-19T16:36:32

On Apr 19, 2012, at 11:48 AM, Jim Weirich wrote:


That is exactly right. It only affects Multitask.

Some additional interesting tidbits: There is a queue of blocks that is a class member of Multitask which all Multitask instances use to queue their prerequisites. So, your thread's call to MultiTask#invoke_prerequisites may end up processing prerequisites for another's call to MultiTask#invoke_prerequisites. No matter though, your call won't return until all your prerequisites are completed.


Thank you for that clarification. I'd only assumed that, given my rather cursory look, and it's nice to have it confirmed.

_ michael

Re: Announcement: Alternate -j implementation

Jos Backus — 2012-04-19T16:21:54

To be honest, I prefer the drake semantics as they are closer to make's.

I would love to see rake and drake merged.

Jos

Re: Announcement: Alternate -j implementation

Jim Weirich — 2012-04-19T15:48:04

On Apr 19, 2012, at 10:34 AM, Hongli Lai wrote:


Michael can correct me if I'm wrong, but the -j option just limits the number of concurrent tasks used to service the "multitask" task targets.  Multitask is an existing rake facility that the -j option makes more usable.

Drake actually changes the semantics of the regular rake tasks so that they run in parallel.

Re: Announcement: Alternate -j implementation

Michael Bishop — 2012-04-19T15:48:03

Hi Hongli Lai,

This is a good question. I only knew of the Drake project as of last night so I haven't had a long period of time to investigate its implementation.

My initial analysis based on the Drake official pull-request announcement indicates a few differences but please add corrections if I am misinformed. I might be.

Drake links:

  https://github.com/quix/rake/tree/mainline-merge-candidate#readme

  http://quix.github.com/rake/files/doc/parallel_rdoc.html


For the purposes of discussion, I'll refer to my submitted rake as "mjbrake" here.


Dependency differences:
-----------------------

Drake relies on an external gem named comp_tree for parallelism. mjbrake relies on no library but instead, like the original rake, leverages the stack to make sure all the prerequisites are executed in order.


Code size differences:
----------------------

Github shows the change for mjbrake at

  "Showing 3 changed files with 74 additions and 7 deletions."

Drake shows 79 lines (+257 lines of code for comp_tree

Re: Announcement: Alternate -j implementation

Hongli Lai — 2012-04-19T14:34:08

Have you also looked at http://drake.rubyforge.org? How does your
implementation differ from his?

On Thu, Apr 19, 2012 at 1:36 PM, Michael Bishop <mbishop< at >me.com> wrote:

Announcement: Alternate -j implementation

Michael Bishop — 2012-04-19T11:36:37

Hello Everyone,

I've recently finished an implementation of -j <max_concurrent_jobs> and would be delighted if the members of this list would take a detailed look at it

It's in this rake branch on github:

   https://github.com/michaeljbishop/rake/commit/295c7a4d6d58b3e10c27b940a8259dd3e01c52f0
 or
   http://bit.ly/JMVARy

In addition, I've added a pull-request for the inclusion of the change to the master branch (allowing of course for further changes to match the style and simplicity of the original)

 https://github.com/jimweirich/rake/pull/112

My apologies if this has been hashed out many times on this list. I'm hoping to offer a fresh look at the problem. Thank you for your consideration.

Sincerely,

Michael Bishop


---


SUMMARY
-------

USER INTERFACE:

The user-interface is simply a new -j flag that specifies the maximum number of tasks that can execute simultaneously (discussed here before as I've seen). If the -j flag is omitted, the old behavior of unlimited concurrent tasks is retained.


IM

Re: Merging drake and rake?

Hongli Lai — 2011-09-17T07:28:59

I would also like to know.

Merging drake and rake?

Jos Backus — 2011-09-17T05:39:04

Hi,

Are there any plans to merge drake's features back into rake?

Jos

No longer able to run Rake tests due fork() usage

Luis Lavena — 2011-08-16T17:03:08

Hello,

I guess the subject explain it completely, but I'm not able to run
Rake tests anymore due fork usage in them.

This not only affects Windows but also JRuby.

I know this is related to session gem removal, but was wondering if
there isn't another workaround?

Thank you.

Beta 1 for Rake 0.9.3 is out

Jim Weirich — 2011-08-06T03:05:45

I've pushed a beta version of 0.9.3 out to Rubygems.org.  

Here is the change list.

* The rake test loader now removes arguments it has processed.  Issue #51
* Rake::TaskArguments now responds to #values_at
* RakeFileUtils.verbose_flag = nil silences output the same as 0.8.7
* Rake tests are now directory-independent
* Rake tests are no longer require flexmock
* Commands constant is no longer polluting top level namespace.
* Show only the interesting portion of the backtrace by default (James M. Lawrence).
* Added --reduce-compat optiont to remove backward compatible DSL hacks (James M. Lawrence).

The rake tests task pattern issue isn't explicitly mentioned here, but it seems to be resolved.  Eric, I'm assuming you did something to fix that (cf. https://github.com/jimweirich/rake/issues/51)

Anyways, grab the beta and give it a spin and report back if there are issues.

Thanks.

3 Failures at master (a774ef8)

Luis Lavena — 2011-06-23T23:37:33

Hello,

I'm getting the following failures running against 1.8.7, 1.9.2 and trunk:

  1) Failure:
test_directory_win32(TestRakeDirectoryTask)
[C:/Users/Luis/Projects/oss/jimweirich/rake/test/test_rake_directory_task.rb:32]:
Expected Rake::FileCreationTask, not Rake::FileTask.

  2) Failure:
test_egrep_with_output(TestRakeFileList)
[C:/Users/Luis/Projects/oss/jimweirich/rake/test/test_rake_file_list.rb:345]:
Expected "xyzzy.txt:2:XYZZY", not "xyzzy.txt:2:XYZZY\r\n".

  3) Failure:
test_egrep_with_block(TestRakeFileList)
[C:/Users/Luis/Projects/oss/jimweirich/rake/test/test_rake_file_list.rb:356]:
Expected ["xyzzy.txt", 2, "XYZZY\n"], not ["xyzzy.txt", 2, "XYZZY\r\n"].

Is worth mentioning that I'm using Windows :-)

Seems to me that #2 and #3 are related to newlines output in the
console, perhaps there is another way to verify the output
independently of LF/CR?

Re: Deprecating Import ???

Jim Weirich — 2011-03-14T18:05:02

On Mar 14, 2011, at 9:37 AM, Wayne E. Seguin wrote:


Ola tweeted that the JRuby 'import' is not recommend anymore and jruby_import is the recommended method now.  Given that, I think the question is moot at this point.

I'm leaning to either A (or C with a default to A).

Re: Deprecating Import ???

Wayne E. Seguin — 2011-03-14T13:37:15

I  have never used import myself directly, I vote for B.  IMHO namespacing a
loading mechanism
specifc to the library (Rake.import) is always a better option in order to
avoid general conflicts even
with other libraries.

  ~Wayne

On Mon, Mar 14, 2011 at 6:55 AM, Jim Weirich <jim.weirich< at >gmail.com> wrote: