gmane.comp.sysutils.supervision.general

Re: Suddenly sv does not start, gives a timeout

Peter Hickman — 2013-05-22T14:22:36

Aaargh found the cause and it was not sv :)

The way ruby was installed on the machine had changed but the change was
not visible if you logged on as the user and ran the commands manually.
However when sv did it's magic it didn't have the same PATH values and
failed.

I am off to view the log files and see who I should castigate >_<

Thank you for your time and sorry for wasting it

Re: Suddenly sv does not start, gives a timeout

Charlie Brady — 2013-05-22T13:40:16




You have a race condition here - process 2980 may have already died. Use 
"sv d services/scorecard_cricket_scores_importer.rb" to stop the process.

You also should not be using -9 unless you have exhausted other options. 
Use -TERM or -QUIT. Using -9 is a bad habit to have.


Why start it in the background?


Then your service is faulty. Failing silently is not satisfactory.

Use strace to see what your process is doing, and when and why it is 
exiting.

Re: Suddenly sv does not start, gives a timeout

Peter Hickman — 2013-05-22T13:32:14

Well this is what we have. Firstly we manually started it so lets kill it:

$ ps ax | grep scorecard
  731 ?        S      0:11 runsv scorecard_cricket_scores_importer
 2980 ?        Sl     0:34 services/scorecard_cricket_scores_importer.rb


16599 pts/0    S+     0:00 grep scorecard
$ kill -9 2980
$ ps ax | grep scorecard
  731 ?        S      0:11 runsv scorecard_cricket_scores_importer
16671 pts/0    S+     0:00 grep scorecard

The process has gone and will not be restarted no matter how long you wait.
So we try and start it with sv:

$ sv start ./service/scorecard_cricket_scores_importer/
timeout: down: ./service/scorecard_cricket_scores_importer/: 1s, normally
up, want up
$ ps ax | grep scorecard
  731 ?        S      0:11 runsv scorecard_cricket_scores_importer
16868 pts/0    S+     0:00 grep scorecard

Still not started. So we try it manually:

$ ./service/scorecard_cricket_scores_importer/run &
[1] 16929
$ ps ax | grep scorecard
  731 ?        S      0:12 runsv scorecard_cricket_scores_importer
16929

Re: Suddenly sv does not start, gives a timeout

Robin Bowes — 2013-05-22T10:16:44


In the logs. What do the logs say?

Or try stopping the service and running it manually from the command
line so you can see the output from the run script.

R.

Suddenly sv does not start, gives a timeout

Peter Hickman — 2013-05-22T09:30:54

One of our servers has started to have a problem with runit. Even after a
reboot we get this:

$ sv start ./service/unicorn/
timeout: down: ./service/unicorn/: 1s, normally up, want up

This has just started without (as far as we can tell) there being any
change to the server. I've even nuked the ./service/* directory so that it
will get rebuilt when the application is deployed (via capistrano - this is
a Rails app) but that does not seem to help.

The other 23 servers which are set up in the same way have no problem so I
am at a loss as to where to start looking.

Any idea of where I should look for clues?

Re: Default permissions on supervise/ok

Laurent Bercot — 2013-05-21T15:10:52

 I think it would work, yes, but I haven't thought too much about it.
Don't quote me on that ^^



 That shouldn't cause any problems.

Re: Default permissions on supervise/ok

2013-05-21T15:29:50

Hm, interesting. Though 622 permissions would correct this, right?
Writable, but not readable?

Any concerns about making the directory readable and executable?

Re: Default permissions on supervise/ok

Laurent Bercot — 2013-05-21T09:58:14

 Here's an exploit:

 If runsv is down, "sv status" will normally return a "runsv not running"
message. But if some user has supervise/ok open for reading, "sv status"
will mistakenly think runsv is running, because it assumes runsv is the
only process able to keep supervise/ok open for reading. So, a user can
deny a service recovery routine from running.

 Don't meddle with supervise/ permissions. If you want to trust a group
of users with service checking/control abilities, make the sv binary
setgid.
 Unfortunately, runit does not separate service checking and service
control abilities at the executable level, so I'm afraid you cannot
achieve what you want here.

 Shameless plug: you could switch s6, which gives you s6-svok and
s6-svstat executables for service checking, separate from the s6-svc
executable for service control. You can make s6-svok and s6-svstat
setgid at your convenience, without endangering the reliability of the
service.

Re: Default permissions on supervise/ok

2013-05-21T04:43:22

sv refuses to run if it can't open the fifo as writable. My intent is to
allow any user to inspect the process state, but not influence it.

My understanding is that runsv will never read from the fifo, and will
only use opening it as a test to verify the process exists the other end.
This doesn't seem exploitable as it's a read-only channel. At least, as
far as I can tell.

supervise/status already has 644 perms, which is fine because `sv` opens
it readonly, but the default permission on the parent supervise/ directory
are 700 which prohibits access.

Control of runsv is performed over supervise/control, which I would not
make world or group writable, of course.

Re: Default permissions on supervise/ok

Charlie Brady — 2013-05-21T00:47:37

On Mon, 20 May 2013, eam< at >frap.net wrote:


Why wouldn't you use 644 for supervise/ok? Remember that you have no 
guarantee that Joe User will use 'sv' to access the file.

Default permissions on supervise/ok

2013-05-20T23:52:58

I'd like to allow any user to access supervise/ok, in order to run
`sv stat`, but not to access supervise/control. My understanding is that
this is safe, as supervise/ok is a read-only interface. Is this accurate,
and is this a reasonable idea? Anything I should be warned about? Am I
overlooking anything important?

chmod 755 supervise
chmod 666 supervise/ok

Re: s6-portable-utils-1.0.0 compile error

Laurent Bercot — 2013-05-15T08:01:13

 This stumps me. The s6-ln code *should not* pull in taia_clockmon or
taia_clockmon_init, ever - it is not calling any time-related function !
 Thanks for the report, I will investigate.

 For non-supervision-related skarnet.org software, please use the
skaware< at >skarnet.org mailing-list. ;)

s6-portable-utils-1.0.0 compile error

Vallo Kallaste — 2013-05-14T20:17:43

Hi

This seems to be corner case as I have never before set the
flag-usert, flag-usemon, flag-noipv6, flag-forcedevr flags for
skalibs.

exec ./compile s6-ln.c
exec ./load s6-ln -lrandom -lstdcrypto -lstddjb `cat socket.lib`
/package/prog/skalibs/library/libstddjb.a(taia_clockmon.o): In function `taia_clockmon':
taia_clockmon.c:(.text+0x22): undefined reference to `clock_gettime'
/package/prog/skalibs/library/libstddjb.a(taia_clockmon.o): In function `taia_clockmon_init':
taia_clockmon.c:(.text+0xce): undefined reference to `clock_gettime'
/package/prog/skalibs/library/libstddjb.a(sysclock_get.o): In function `sysclock_get':
sysclock_get.c:(.text+0x21): undefined reference to `clock_gettime'
collect2: ld returned 1 exit status
make[2]: *** [s6-ln] Error 1
make[2]: Leaving directory `/package/admin/s6-portable-utils-1.0.0/compile/skaembutils'
make[1]: *** [.done] Error 2
make[1]: Leaving directory `/package/admin/s6-portable-utils-1.0.0/compile/skaembutils'
make: *** [.done] Error 2

Re: Installing on Centos

Joshua Timberman — 2013-05-09T15:37:53

Opscode's runit cookbook supports CentOS (and other EL family distros) first class now, in version 1.0 and higher of the cookbook. It is a complete installation solution via the default recipe, and the runit_service resource is a fully developed feature complete service resource. 

http://ckbk.it/runit

I'm biased of course since I wrote the code, but I think this is the best way to manage runit if you're using chef already. 

-Joshua

Re: Installing on Centos

Charlie Brady — 2013-05-09T15:08:33

On Thu, 9 May 2013, Peter Hickman wrote:


Sorry, my bad. I was thinking of rhel6/centos6.


You can add this in /etc/inittab:

sv:345:respawn:/etc/runit/2

Re: Installing on Centos

Peter Hickman — 2013-05-09T15:05:43

Thanks but it would seem that Centos 5.8 does not have an /etc/init
directory only an /etc/init.d

But thanks to your hint I have found the command that does the work

# cat runsvdir-start
#!/bin/sh

PATH=/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin

exec env - PATH=$PATH \
runsvdir -P /etc/service 'log:
...........................................................................................................................................................................................................................................................................................................................................................................................................'

Now all I need to do is turn this into an init.d script

Re: Installing on Centos

Charlie Brady — 2013-05-09T14:49:35

On Thu, 9 May 2013, Peter Hickman wrote:


Add /etc/init/runit.conf:

# runit - start runit supervisors
#
# This task runs the runit start script, which runs runsvdir

start on runlevel [345]

stop on runlevel [!$RUNLEVEL]

task

export RUNLEVEL
console output
exec /etc/runit/2 $RUNLEVEL

Re: Installing on Centos

Peter Hickman — 2013-05-09T13:03:53

All our chef scripts were written with ubuntu 10.04 in mind and we had not
correctly separated the os dependencies from the application deployment.
Then we had to deploy to Centos :)

So our chef scripts broke in all sorts of interesting ways. I don't ascribe
this to a problem with chef when it could so easily be our scripts being
badly written so don't worry about chef.

However we need to get runit working on Centos regardless so we have hacked
up some scripts to do this.

Runit installs fine, or at least without error but things are not working
out.

For instance there is no /etc/init.d/runsvdir or similar. The truth be told
there isn't one on Ubuntu either but some how Ubuntu is managing to start
up "runsvdir -P /etc/service" without any problem. So one question would be
how to get this to happen on Centos 5.8?

Re: Installing on Centos

Joshua Timberman — 2013-05-07T17:47:15

Hi Peter,

On May 7, 2013, at 5:17 AM, Peter Hickman <peterhickman386< at >googlemail.com>
 wrote:


The Chef cookbook for runit builds an RPM for you, from this repo:

https://github.com/imeyer/runit-rpm

No one seems interested in getting runit into RHEL family platforms in any official capacity (e.g., EPEL), unfortunately. I'd love a better solution for this, but alas, this is the "best" solution for now.

If you're having trouble with the Chef recipe[0] on CentOS, please provide more information like the version of the runit cookbook you're using, and debug log output from chef-client/chef-solo. I'm happy to help off-list, too.

Cheers,
Joshua Timberman | Opscode, Inc.
Twitter/GitHub: jtimberman


[0]: Pedantically, Chef recipes are not simply "scripts" :). A great explanation about how they work is here: 

http://erik.hollensbe.org/2013/03/16/the-chef-resource-run-queue/

Re: Installing on Centos

Peter Hickman — 2013-05-07T11:17:50

Thanks that a place to start at least. The problem for me is that all our
previous installs were from a chef script and things just magically worked.
The scripts borked on Centos and I have to try and work out how the working
system got tat way.

Re: Installing on Centos

Alex Efros — 2013-05-07T10:56:25

Hi!

On Tue, May 07, 2013 at 10:30:15AM +0100, Peter Hickman wrote:

I didn't installed runit from source for years, so I may be wrong, but:


Runit doesn't hardcoded to use some predefined directory, you can put your
services in any directory you like - /service or /etc/sv - you choose.
So, neither of these directories created automatically when installing
runit - you've to create them yourself. Some linux distributives when
creating packages for runit make this decision instead of you and
preconfigure runit in some way, which include creating such directories.


That's because you didn't started it, I suppose. :)


Check your $PATH, different distributives configure it in different ways -
some include */sbin/ directories in $PATH for non-root users, others don't.
Same for /usr/local/{bin,sbin}/. Check where your runit's binaries was
installed and is that directory included in non-root's $PATH.