gmane.comp.lang.ruby.god

Monitoring Unicorn Workers with God

Nitesh Goel — 2012-05-09T19:16:35

There seems to be no easy to way monitor unicorn workers with god. Github 
has a hack, where they use a separate thread to monitor the workers, but 
the post is two and a half years old 
(https://github.com/blog/519-unicorn-god).

Have things changed since?

Different restart commands based on the restart condition

Nitesh Goel — 2012-05-07T08:47:03

Hey Guys
I am monitoring a process where my restart command is based on the restart 
condition failed. E.g.

If memory usage too high:

w.restart = "command 1"

if cpu usage too high:

w.restart = "command 2"

Is there a way to configure a watch as such?

Thanks.

God-rb doesn't auto-daemonize unicorn well?

Guang-Nan Cheng — 2012-05-06T09:07:16

God stores a wrong pid for auto-demonizing unicorn. I can get things 
working by set the pid file on both unicorn and god. But do I have to do it 
this way? Why god doesn't auto-daemonize unicorn well?

god overwrites ".kilobytes" in Rails

Stefan Adamov — 2012-04-25T23:28:58

Setup:

Gemfile
gem 'god'

rails c
irb: warn: can't alias exit from irb_exit
=> 5120

work fine if I follow https://github.com/mojombo/god/issues/32

Gemfile
gem 'god', :require => false

rails c
=> 5242880

how to start the process

Gitted — 2012-04-17T13:42:23

How does godrb actually start, is there a ubuntu init script that
starts the process and makes sure it starts upon reboots?

[fail] never received process exit event

Toan Nguyen — 2012-04-06T03:36:28

Hi everybody,

I'm a newbie to God. I install and run God on Ubuntu system is fine.

But I get problem when run God on Debian. Run "god check", I got
message: "[fail] never received process exit event".

My system info:

No LSB modules are available.
Distributor ID:Debian
Description:Debian GNU/Linux 6.0.4 (squeeze)
Release:6.0.4
Codename:squeeze

Linux breithorn 2.6.32-5-amd64 #1 SMP Thu Mar 22 17:26:33 UTC 2012
x86_64 GNU/Linux

I have checked cn (connector) kernel module by "lsmod | grep cn":

cn                      4563  0

I don't know why it can't check process exit event.

monitor chrome with god

Sheng LUO — 2012-04-02T13:20:40

hi there,

My ubuntu runs on 1G ram, where chrome always get fronzen or crashed 
because of flash.
So I want to watch my chrome with god and limit the cpu and memory it 
occupies.
Normally, I can run a chrome by typing "google-chrome" in terminal, so I 
write following god script:

God.watch do |w|
  w.name = "chrome"
  w.start = "/opt/google/chrome/chrome"
  w.keepalive :cpu_max => 80.percent, :memory_max => 500.megabytes
end

When I start the script above by "god -c chrome.god -D", it just keep 
opening new chrome.
Doesn't it automatically figure out there is already a chrome running?
What can I do to realize monitoring chrome?

non-daemon process using Bundler

Christopher J. Bottaro — 2012-03-16T18:36:49

Hello,

I'm having a problem with god trying to watch a non-daemon process that is 
started by "bundle exec".

God starts the process just fine, as seen by ps:

6856 ?        Ss     0:00 sh -c bundle exec rake resque:scheduler >> 
/home/rails/project_a/shared/log/scheduler.log 2>&1
6860 ?        Sl     0:23 ruby 
/home/rails/project_a/shared/bundle/ruby/1.9.1/bin/rake resque:scheduler

The problem is that god is monitoring the first process (6856), so that a 
"god stop" or "god restart" nukes the first process, but the second one 
remains.

Any ideas?  Thanks for the help.

Test issues on Ruby 1.9.x and OS X Lion

Max Lansing — 2012-02-29T07:53:20

I seem to be unable to complete a rake run with 0.12.1, using Ruby 1.9.2 or 1.9.3 on OS 10.7.3.

It hangs at TestEventHandlerOperational: test_operational, in the God::EventHandler.start call. Within that method, it's hanging on ::Process.kill('KILL', pid). When kill is called, the child process is terminated, but sticks around as a zombie, and the kill call never returns.

The tests run normally for me in 1.8.7. 

'god check', which runs similar code, runs fine in 1.9.x.

Can anyone reproduce this issue?

Thanks,

Max

Killing a process doesn't restart

Matias — 2012-02-20T18:18:50

I create the follow solr.god: https://gist.github.com/1870445

The process start fine but if I kill the process, God doesn't restart
the process again. What I miss?

notification on process restart / crash when events are disabled

2012-02-22T16:06:32

Hi. I've got a couple of questions:

1) Is this project dead? Does it have an active maintainer who'll be
able to fix bugs?
2) How do I notify our ops team of process restarts or crashes when
god's running with user privileges, thus not having access to events
interface?

Thank you.

Custom Notifications

Dan Cook — 2012-02-18T19:41:58

I need to execute a some ruby code on notification.
We have a central health monitoring service on the box and process
health is just one part of the overall reports.
I would like to put the events into a queue on the disk and have our
health monitoring service slurp up any events and send them to another
centralized server.

I assume I can just do this:

w.transition(:up, :start) do |on|
  on.condition(:process_exits) do |c|
    # custom notification method here....
  end
end

Is this correct?

Cheers,
Dan

rbenv shims

ileitch — 2012-02-16T15:16:34

Hi,

What's the correct approach for my God scripts to see my rbenv shims?

God itself starts fine, as I set PATH in /etc/init.d/god:

PATH=/root/.rbenv/shims:/root/.rbenv/bin:$PATH

The individual watchers however do not have the shims in the PATH. I
could just hard code the absolute path to the shim in my watcher, but
I feel there must be a cleaner way?

I also don't see an option for setting PATH in God.

Cheers
Ian

After installing god gem, i get “command not found”

Freddy — 2012-02-02T18:09:35

I installed the god gem using bundle install --path vendor/bundle in
site5. After installing, i am unable to run god command. Running it
gives "command not found".

Running it like this gives me this error,
vendor/bundle/ruby/1.8/bin/god

/usr/lib/ruby/site_ruby/1.8/rubygems.rb:812:in report_activate_error':
Could not find RubyGem god (>= 0) (Gem::LoadError) from /usr/lib/ruby/
site_ruby/1.8/rubygems.rb:223:inactivate' from /usr/lib/ruby/site_ruby/
1.8/rubygems.rb:1146:in `gem' from vendor/bundle/ruby/1.8/bin/god:18

Please help.

"live" CPU monitoring with god

pogodan — 2011-12-06T20:46:50

We're trying to use God to monitor some processes that occasionally
crash and peg the CPU. In top we see CPU% go to 100 or 200%, but God
still reports use as only maybe 15-20%

Poking through the code and some linux forums, it seems the CPU stats
returned by `ps` are historical over the life of the process, while
calculating realtime stats requires polling, waiting an interval and
re-polling to compare "jiffies" used over the interval. It looks like
both God's SlashProcPoller and PortablePoller are basically
calculating the historical CPU use which is not very helpful for
occasionally crashing processes

Has anyone made a stab at writing a "realtime" CPU checker?

-
Paul | http://pogodan.com

Logs are empty until a process is stopped

Ivan el Magnifico — 2011-10-27T13:47:22

Hi,

I'm relatively new to god, but I've been able to set it up mostly
right.

On issue I haven't been able to fix, is for this watch:

God.watch do |w|
  w.dir      = 'some dir'
  w.name     = 'rapns'
  w.group    = 'some app'
  w.interval = 30.seconds
  w.pid_file = "some pid.pid"
  w.gid = 'user'
  w.uid = 'group'
  w.start    = "bundle exec rapns production --foreground >
somelog.log"
  w.log = "some path/god.log"
end

Why the --foreground switch and the output redirection you ask? Well,
I've tested several approaches, and none of them write to the log.
With just "bundle exec rapns production", the god.log file does get
written, but only *after* I kill the rapns process. With the method I
show, it's the same, but for the somelog.log file.

Is this an issue with the rapns process or is there something I'm
doing all wrong?

Thanks!

- Ivan

Getting Triggered when your daemon goes to unmonitored state

Pierre Baillet — 2011-10-17T15:46:52

Hi,

I wanted to get notified on my custom trigger as much as possible.
However, the current God engine will not notify you when a daemon goes
from any-state to unmonitored (for example, if you 'god stop mydaemon'
from the command line). The patch at
https://github.com/octplane/god/commit/d606152fc729c75c53c7fa2db57e8035cece6237
will send this last event to your triggers, just before actually
unmonitoring your daemon.

Cheers,
--
Pierre.

Restarting a stopped daemon

Pierre Baillet — 2011-10-14T13:20:54

Hi,

I'm trying to restart a daemon that I have previously stopped. This
doesn't work as expected, as it seems a stopped daemon will never
start again. The sample code I've ended up with is https://gist.github.com/1287060
and basically implement a very simple daemon that runs every other
minutes (of course, this is a simple proof of concept as I want to use
the lambda feature to integrate with grid-wide daemon management
later).

If anyone has an idea on how to have that behaviour, I'd be really
cool !

Thanks for reading and thanks Tom, for god :)

--
Pierre.

Error trying to restart a process

Kristof Redei — 2011-08-26T22:45:29

Hi all,

god 0.11.0 is unable to restart processes on a CentOS box under Ruby
1.9.2 (I've tried both patchlevels 0 and 180). The log shows the error
below, which I haven't seen before and was wondering if anyone else
had.

-bash-3.2$ uname -a
Linux prodcca01 2.6.18.8-xen #2 SMP Wed Jul 29 18:05:23 EDT 2009
x86_64 x86_64 x86_64 GNU/Linux

I [2011-08-26 18:38:19]  INFO: thin-5402 move 'up' to 'restart'

I [2011-08-26 18:38:19]  INFO: thin-5402 stop: default lambda killer
I [2011-08-26 18:38:19]  INFO: thin-5402 sent SIGTERM
F [2011-08-26 18:38:19] FATAL: Unhandled exception in driver loop -
(TypeError): no implicit conversion from nil to integer
/usr/lib/ruby/gems/1.9.1/gems/god-0.11.0/lib/god/process.rb:215:in
`kill'
/usr/lib/ruby/gems/1.9.1/gems/god-0.11.0/lib/god/process.rb:215:in
`block (2 levels) in call_action'
/usr/lib/ruby/gems/1.9.1/gems/god-0.11.0/lib/god/process.rb:213:in
`times'
/usr/lib/ruby/gems/1.9.1/gems/god-0.11.0/lib/god/process.rb:213:in
`block in call_action'
/usr/lib/ruby/gems/1.9.1/gems/god-0.11.0/lib/god/process.rb:277:in
`call'
/usr/lib/ruby/gems/1.9.1/gems/god-0.11.0/lib/god/process.rb:277:in
`call_action'
/usr/lib/ruby/gems/1.9.1/gems/god-0.11.0/lib/god/watch.rb:158:in
`call_action'
/usr/lib/ruby/gems/1.9.1/gems/god-0.11.0/lib/god/watch.rb:132:in
`action'
/usr/lib/ruby/gems/1.9.1/gems/god-0.11.0/lib/god/watch.rb:127:in
`action'
/usr/lib/ruby/gems/1.9.1/gems/god-0.11.0/lib/god/task.rb:171:in `move'
/usr/lib/ruby/gems/1.9.1/gems/god-0.11.0/lib/god/task.rb:356:in
`handle_poll'
/usr/lib/ruby/gems/1.9.1/gems/god-0.11.0/lib/god/driver.rb:41:in
`handle_event'
/usr/lib/ruby/gems/1.9.1/gems/god-0.11.0/lib/god/driver.rb:151:in
`block (2 levels) in initialize'
/usr/lib/ruby/gems/1.9.1/gems/god-0.11.0/lib/god/driver.rb:149:in
`loop'
/usr/lib/ruby/gems/1.9.1/gems/god-0.11.0/lib/god/driver.rb:149:in
`block in initialize'

How do I create a watch for a process that doesn't write it's own pid, and God isn't locating the correct pid automatically.

donnoman — 2011-08-03T00:21:18

I'm trying to get God to watch Riak on Ubuntu.

The riak init script doesn't write pids, and su's as the riak user and uses 
'pings' to determine if it's up or down as opposed to tracking pids.

When trying to start the init scripts via god, it detects the wrong pid. (or 
rather the pid that spawns the process goes away)

root< at >lucid32:/etc/god# god status
riak: start

root< at >lucid32:/etc/god# cat /var/run/god/riak.pid     
16382

root< at >lucid32:/etc/god# ps aux | grep 16382        
root     16906  0.0  0.1   2140   744 pts/0    S+   00:16   0:00 grep 
--color=auto 16382


but riak is running happily

root< at >lucid32:/etc/god# ps aux | grep riak
riak     16421  0.0  0.2   2192   760 ?        S    00:08   0:00 
/usr/lib/riak/erts-5.7.5/bin/run_erl -daemon /tmp/riak// /var/log/riak exec 
/usr/sbin/riak console
riak     16422  0.2  3.9  25456 14856 pts/1    Ssl+ 00:08   0:01 
/usr/lib/riak/erts-5.7.5/bin/beam -K true -A 64 -- -root /usr/lib/riak 
-progname riak -- -home /var/lib/riak -- -boot 
/usr/lib/riak/releases/0.14.2/riak -embedded -config /etc/riak/app.config 
-name riak-savSHZN5Fh8qMp+WYRx65w< at >public.gmane.org -setcookie riak -- console
riak     16447  0.0  0.1   1956   384 ?        S    00:08   0:00 
/usr/lib/riak/erts-5.7.5/bin/epmd -daemon
riak     16509  0.0  0.1   1832   532 ?        Ss   00:08   0:00 sh -s 
disksup
riak     16511  0.0  0.1   1752   496 ?        Ss   00:08   0:00 
/usr/lib/riak/lib/os_mon-2.2.5/priv/bin/memsup
riak     16512  0.0  0.0   1748   360 ?        Ss   00:08   0:00 
/usr/lib/riak/lib/os_mon-2.2.5/priv/bin/cpu_sup
root     16848  0.0  0.1   2076   632 pts/2    S+   00:11   0:00 tail -f 
/var/log/god.log /var/log/riak/god.log
root     16912  0.0  0.2   2144   808 pts/0    S+   00:17   0:00 grep 
--color=auto riak

root< at >lucid32:/etc/god# /etc/init.d/riak ping
Attempting to restart script through sudo -u riak
pong

The current god config.

God.watch do |w|
  w.name = 'riak'
  w.interval = 5.seconds

  daemon = '/etc/init.d/riak'
  
  w.start   = "#{daemon} start"
  w.stop    = "#{daemon} stop"
  w.restart = "#{daemon} restart"

  w.start_grace = 60.seconds
  w.stop_grace = 60.seconds
  w.restart_grace = 60.seconds
  
  w.log = "/var/log/riak/god.log"

  w.transition(:init, { true => :up, false => :start }) do |on|
    on.condition(:process_running){|c| c.running = true }
  end

  w.transition([:start, :restart], :up) do |on|
    on.condition(:process_running){|c| c.running = true }
  end

  # start if process is not running
  w.transition(:up, :start) do |on|
    on.condition(:process_exits) do |c|
      c.notify = %w[ localhost ]
    end
  end

  w.transition(:up, :unmonitored) do |on|
    on.condition(:flapping){|c|
      c.to_state = [:start, :restart]
      c.times = 5
      c.within = 5.minute
      c.transition = :unmonitored
      c.retry_in = 10.minutes
      c.retry_times = 5
      c.retry_within = 2.hours
      c.notify = %w[ localhost ]
    }
  end
end

How do I override the test for process_running to use the pings instead of 
checking for the pids?

God on Solaris 10 (SPARC) Events??

2011-07-11T12:17:10

All,

Am developing an in house app that uses God to manage a series of Ruby
workers (they are Resque workers that manage scheduled jobs within a
Rails app). We are moving the app from RHEL to Solaris on SPARC (our
present production hardware) and it'd be nice to keep using God to
manage these.

Looking at the archives, it seems it should work fine (cool). However,
the last discussions were quite some time ago (2008), so was checking
what works and what doesn't, especially regarding events.

So the short of it is...
# Does God work on Solaris 10 (SPARC)
# Does it support Events
# Any gotchas from people who have it working or tried

Thanks

Dan