gmane.comp.lang.ruby.cruise-control.devel

Re: Some updates,and a couple of test requests

Brian Guthrie — 2011-07-03T13:18:16

c.f. https://github.com/thoughtworks/cruisecontrol.rb/commit/17ca0f8b3f03e166fe1d6f240e1ef45c5e604a8a

On Sun, Jul 3, 2011 at 7:46 PM, Brian Guthrie <btguthrie-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: Some updates,and a couple of test requests

Brian Guthrie — 2011-07-03T09:46:53

It depends on which pid file you're referring to. The main server pid
file, tmp/pids/server.pid, will do whatever script/rails server -d
does. It's possible that means it will just do the dumb Mongrel thing.
init.rb:

      Rails::Server.new.tap { |server|
        require ENV_PATH
        Dir.chdir(Rails.application.root)
        server.start
      }

Stopping:

    def stop_server
      pid_file = Rails.root.join("tmp", "pids", "server.pid")

      if pid_file.exist?
        exec "kill -KILL #{pid_file.read.chomp}"
        pid_file.delete
      end
    end

    def stop_builders
      Rails.root.join("tmp", "pids", "builders").children.each do |pid_file|
        Platform.kill_child_process(pid_file.read.chomp)
      end
    end


The builder files don't try to be smart at all. They will create pid
files as necessary or reuse them if they already exist.
script/builder:

  Platform.project_pid_file(project.name).tap do |f|
    f.dirname.mkpath
    f.open("w") {|f| f.write Process.pid }
  end

  ProjectBlocker.block(project)

This is exactly the previous pid-creation behavior, except that the
logic has been moved to the builder, and I'm attempting a pid cleanup
as well as the old ProjectBlocker.release on error or interrupt.

Looking more closely at daemon_helper, it's clearly handling a lot of
useful stuff beyond what's in place in init, so it looks like it needs
to be fixed before I can sign off this bug. But I'm confused as to why
you'd keep your pid-management code separate when the use already
expects -d to handle this stuff. What's the benefit?

As a maintainer, I wish there was a lot more test coverage of both
places. :) This my work has not improved.

Cheers,

Brian

On Sun, Jul 3, 2011 at 5:30 PM, Chad Woolley <thewoolleyman-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: Some updates,and a couple of test requests

Chad Woolley — 2011-07-03T07:30:44

If a process dies or gets killed, and leaves a stale pid file behind,
is all the pid-handling code smart enough to detect this and recreate
a new pidfile for the new pid?  I.e. we don't act like mongrel, and
refuse to start with a stale pid?

Some updates,and a couple of test requests

Brian Guthrie — 2011-07-03T06:37:22

I did some work today to try to fix Cruise daemonization. Everything
looks ok to me on my Mac and Ruby 1.9.2 but I'm having trouble
installing and running JRuby locally. If others could take a poke at
this I'd greatly appreciate it.

Also, as far as I know, we don't need or use anything in the daemon/
subdirectory now; we're relying on Rails' server daemonization, the
killtree script, and straight-up kill to do the work.

The init.rb#stop_server method is using kill -KILL rather than kill
-INT, which is what I'd prefer, because shutting down the process
using -INT while the server is daemonized in Prod mode seems to cause
an infinite loop that I haven't locked down yet. Doesn't seem to
happen in dev mode.

Builders are now creating and cleaning up their own pid files, rather
than Platform creating the pid files at builder start time (and
getting cleaned up, well, never). If someone could explain why Cruise
is creating and using both .pid files and individual builder.lock
files that would be extremely helpful to me as a maintainer, since I
wasn't around for the creation of that code.

After this, nothing much is keeping us from cutting an RC1.

Brian

Replacing or supplementingConfiguration.disable_build_now

Brian Guthrie — 2011-06-22T05:01:05

Hi all,

Recently the number of things that the dashboard UI can do with your
builds has expanded a bit; you used to just being able to trigger
builds, but now you can add and kill them as well. So far we've been
using Configuration.can_build_now as a catch-all for these features
(when we guard them at all), but its name doesn't bear much
resemblance to its current usage.

There are a few ways we could go about improving this:

 1) deprecating Configuration.disable_build_now in favor of
Configuration.disable_admin_ui (or something similar)
 2) supplementing Configuration.disable_build_now by adding other
switches, like Configuration.disable_kill_builders and
Configuration.disable_add_project
 3) something else
 4) nothing else

My preference is for #1, because it's simple and captures the intent
of the switch better. #2 is more fine-grained, which might be helpful
in some cases. My suspicion is that you probably don't need that
granularity and anyone looking to disable Build Now probably wants to
disable those other things as well. I'm open to other suggestions but
I'd prefer to not ignore it.

Cheers,

Brian

Recent progress

Brian Guthrie — 2011-06-22T04:54:24

Hi all,

Recently we've seen a number of big improvements:

 - new UI has been merged into master
 - a couple of fixes to Git support
 - you can now build and run a cruisecontrolrb gem
 - various improvements to documentation, help files, links, contributors list
 - improved management of environment variables in cruise_config
 - fixes to Bundler support
 - added a Kill Builder button (thanks to the folks at Square for this)
 - removed old and unused files from Rails 2 days
 - tests now pass in Windows (although builds still fail)

We're getting closer to our target date for a 2.0 release, which is
targeted for the 1st of July. Once gem package support is complete, in
the next couple of days, we'll tag and package a release as 2.0.0pre1
and push it out for general usage.

Brian

Re: Next steps...

Chad Woolley — 2011-06-12T02:34:51

Makes sense.  The only hard requirement should be that all options can
be overridden/disabled - i.e. don't lock in any specific usage of
bundler or options.

Re: Next steps...

Brian Guthrie — 2011-06-12T01:39:24

I'd like to do this but it doesn't seem to square nicely with the
desire to provide a nice out of the box end user experience.

It speeds things up quite a bit to run bundle check prior to install
(which is what CC.rb does now); those commands share some subset of
arguments, and it would be nice if we could avoid the complication of
requiring them to be independently configurable; there are various
bits and bobs with the full Ruby and Gemfile paths that you have to
get right; installing to --path=vendor seems like a must if you're
planning on building multiple projects on the same box; --no-color
strips out unwanted console color output.

Brian

Re: Next steps...

Chad Woolley — 2011-06-11T19:18:16

Maybe I missed bundler discussion on another thread, but why not just
default to "bundle install" with no args, and allow a string to be
passed to override args?  Why do we need to pass any other args by
default?

Re: Next steps...

Chad Woolley — 2011-06-11T19:16:49

On Sat, Jun 11, 2011 at 9:12 AM, Ketan Padegaonkar
<ketanpadegaonkar-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Is that safe to assume?  Maybe everything isn't packaged?  Shouldn't
be normal, but I wouldn't want to force any option...

Re: Next steps...

Ketan Padegaonkar — 2011-06-11T16:12:32

We may be in a position to apply the '--local' arg to bundler by
checking existence of vendor/cache directory.

Ketan
studios.thoughtworks.com | twitter.com/ketanpkr




On Sat, Jun 11, 2011 at 6:07 AM, Brian Guthrie <btguthrie-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: Next steps...

Brian Guthrie — 2011-06-11T13:07:35

Hi Ketan - I've updated the 2.0 milestone on Github with the release
dates you've outlined. I've attempted an improvement at Bundler
support today, under the assumption that CC.rb itself needs to veer a
little bit outside of the mainstream; we're vendoring our .gem files,
and my assumption (which may be incorrect) is that most Ruby projects
aren't currently doing that. Even if they are, they'll need the
ability to turn --local off.

So my proposed solution is a configurable array of bundler_args on
Project. It's not a thrilling solution; feels a bit like a compromise
to me - a little bit more manageable and idiomatic than a plain
string, a little less complex than trying to support full-on args
rewriting. Please, everyone, let me know if this won't work for you,
and particularly if you'd like to handle it a better way.

Brian

On Sat, Jun 11, 2011 at 4:39 AM, Ketan Padegaonkar
<ketanpadegaonkar-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: Next steps...

Ketan Padegaonkar — 2011-06-10T18:39:34

Calling this a 2.0 sounds good to me given that its been a while and
its been a lot of improvements.


Theres quite a few improvements that could be made. I'm thinking
things like paths to ruby, rake bundler etc. I'm not sure if we can
push all this into master without pushing the timeframe up ahead by a
couple of weeks which I understand, and am fine with.


Rexml is a good alternative to nokogiri, it's not all that fast, but
we're not doing a lot of XML parsing. So we should be good in that
regard.

- Ketan
studios.thoughtworks.com | eclipse.org/swtbot | < at >ketanpkr

Re: Next steps...

Brian Guthrie — 2011-06-09T01:21:45

Hi Ketan,

I've been tying all of these features to a 2.0 milestone, including
the Java work you've been doing. You can see the milestones here:

https://github.com/thoughtworks/cruisecontrol.rb/issues/milestones

If that's more appropriately tagged as 1.5, I think that's a
worthwhile conversation; I picked 2.0 because I figured enough time
had passed, and the structural changes were significant enough.
There's also some "future release" stuff lumped under 2.1. If you'd
like to add a release date to those milestones, which I haven't done
yet, I think that would be great.

The release dates you've suggested sound reasonable to me. I think the
17th is about as aggressive as I'm willing to go, though, I think.

There are more improvements I'd like to see to Bundler support,
including on that I think needs to happen prior to the release
(arguments need to be properly configurable, for one). I have some
local changes here that I'll push up soon.

We still need proper Windows support. As far as I can tell, there's no
good way to get Nokogiri running there, so I'm planning on removing
that dependency.

Brian

On Thu, Jun 9, 2011 at 3:36 AM, Ketan Padegaonkar
<ketanpadegaonkar-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Next steps...

Ketan Padegaonkar — 2011-06-08T17:36:26

Hi,

Some of you may be wondering who I am. I'm Ketan from ThoughtWorks and 
one of the newest committers to ccrb. I've recently been doing some work 
to run ccrb as a jar (java -jar ccrb.jar) as a simple way to distribute 
ccrb. This is still work in progress. Later worked on fixing the build 
to make it go green 
<http://cruisecontrolrb.thoughtworks.com/builds/CruiseControl/bcc1d7267c145420606bce26006579ba3b17b0a7>.

So far here's what's been done so far(sorry, if this sounds like a 
repeat from previous conversations):

* A brand new ccrb machine - we scrapped the existing machine that was 
running an ancient OS and software which we could not figure out how it 
was setup(thanks < at >AjeyGore and Ranjib Dey from ThoughtWorks 
Infrastructure Support team)
* This has the latest version of ccrb deployed on it(rev bcc1d7).
* Get builder to work for ccrb. There was some work needed to make 
builder to work with rvm.
* Fix documentation generation - this was broken because of an api 
change between rails 2.x and 3.0.

Next steps:
===========

I feel the next steps should be to ensure that we are able to cut out a 
release(the last one was in mid 2009 - 2 years ago). To that effect I'd 
like to propose the following, and gather some consensus around the next 
steps.

This is fairly aggressive with the goal being able to cut a new release 
without having to stop the line for developing new features. Feel free 
to add anything that you feel should go into this list. These dates are 
just tentative in nature, we can be a lot aggressive if we feel we're up 
for it :)

* 1.5 beta end of next week(17th June)

Cut out a branch for 1.5 work. No new features land on this branch, 
anything new goes to the master branch to avoid regressions.

* 1.5RC week after(24th June)

Monitor any feedback around potential bugs that crop up and get them 
fixed. Cherry pick everything to master if needed.

* 1.5 week after(1st July)

Cut out a final release. Cherry pick everything to master if needed.

Just my 2 cents. Curious to hear from everyone else.

Improving cross-platform support for process creation & destruction (was: Why there was no bundle in CC.rb originally)

Brian Guthrie — 2011-05-22T09:56:59

Sorry for the delay, folks - was away from the Internet for the last few days.

I've been spiking out ChildProcess
(https://github.com/jarib/childprocess; summary: extracted from
Selenium gem, a cross-platform library to launch and kill processes)
as a platform-relatively-agnostic replacement for CCRB's own
hand-rolled process-management code, and according to that's project's
implementation JRuby on Windows is the only platform that won't yield
a PID after process creation, which might actually create more
problems than it solves if we want to be able to kill live builders.

Questions to authors: Is there a particular reason why the master
Platform.create_child_process method is responsible for creating the
pid file, rather than the child builder creating it on launch? Also,
there's a reference in the comments that appears to suggest that the
builder process tries to hold a lock on its pid file, but that's not
the case in the code - it's using a separate file called builder.lock,
and the existence of the lock is used by the UI to determine
externally whether or not a given builder is running (rather than,
say, the builder's own reported status). Why is that the case? Was it
a concession to platforms that don't provide pids?

It feels to me like both the builder itself and the UI's use of it are
crying out for some sort of first-class domain model, but their
interactions with the builder are rather different - the UI launches,
reads the status of, and (someday) kills builders, builders themselves
read and write their own status and respond to kill and build
requests. Both require some subset of the same functionality. Right
now this stuff is floating around in some combination of Platform,
BuilderStatus, BuildBlocker, FileLock, and BuilderStarter, which has
been difficult for me to understand and maintain and exacerbates the
problem of poor builder status reporting in the UI.

Brian

Re: Why there was no bundle in CC.rb originally (and probably still shouldn't be now)

Brian Guthrie — 2011-05-22T09:59:12

On Fri, May 20, 2011 at 6:22 AM, Alexey Verkhovsky
<alexey.verkhovsky-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

That's a good point, although I hasten to add that I wasn't
_seriously_ planning on building binaries on Windows; I just thought
it was interesting that it existed at all. :) I'll plan on ridding the
Gemfile of Nokogiri the next chance I get, and get Windows tests
running properly, and that should in principle eliminate the need for
a separate build box.

Cheers,

Brian

Re: Why there was no bundle in CC.rb originally (and probably still shouldn't be now)

Sidu Ponnappa — 2011-05-19T22:54:04

Would it be fair to support Windows only via JRuby? That would make
life much easier for you.

Best,
Sidu.
http://c42.in

On 20 May 2011 01:52, Alexey Verkhovsky <alexey.verkhovsky-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: Why there was no bundle in CC.rb originally (and probably still shouldn't be now)

Alexey Verkhovsky — 2011-05-19T20:22:55


Not many.



Would be a great idea for a *very seriously* supported project. For an
open-source project that sees a bit of ongoing development every once in a
while, you don't want to maintain complex infrastructure of this sort. :)

Why there was no bundle in CC.rb originally (and probably still shouldn't be now)

Brian Guthrie — 2011-05-19T19:50:47

task :release => :test
  `bundle install --deployment`
  `gem package`
  `gem push ccrb-#{RUBY-PLATFORM}.gem`
end

Something like that. Replace gem package with tar and zip for non-gem
Linux and Windows releases respectively if needed.

I've looked into binary gems. Nokogiri has crept into the gemfile via
Mechanize (which I believe exists for the performance build, which I'm
not currently maintaining - any volunteers?). It can be eliminated as
a dependency by removing mechanize and adding Builder explicitly as a
dependency. This still leaves many failing tests, which I haven't
begun to look into. Do we have some idea of how many Windows users
CC.rb has?

Apparently there is now a way to build many C-based gems in Windows
(see https://github.com/oneclick/rubyinstaller) and so one could
conceivably run a Windows release task on a Windows box that built the
necessary binaries on demand and automatically uploaded them to the
main CC.rb site.

Cheers,

Brian

Re: Why there was no bundle in CC.rb originally (and probably still shouldn't be now)

Alexey Verkhovsky — 2011-05-19T16:09:41



nokogiri

Not all CC.rb users are Ruby developers. At least, this used to be the case
when I kept awareness of such things.