gmane.network.reconnoiter.user

SNMP Implementation Change From Net-SNMP toSNMP4J

Philip Maddox — 2013-03-01T21:34:26

Hi everybody:

I have been trying to edit the SNMP check to allow people to use SNMP
Version 3. Net-SNMP does NOT support this very well, and I have run into
numerous difficulties with it (it's not thread safe, it doesn't allow
changing passwords once one for a user has been established without
manually clearing out the cache of username/password combinations, it
stores each individual engine ID/username combo in memory and won't allow
creations of numerous people with the same username/security engine ID,
etc, etc).

With this in mind, I am currently working on converting the SNMP check to
use the SNMP4J Java library instead. All parameters and current checks
should work just fine - it should be a seamless and invisible transition
once the new code is built. This will allow this check to be modified and
built upon in the future much more easily and will build a stronger
foundation to build on. It should also provide more stability, since
Net-SNMP has issues with large file descriptors that have been causing

Re: Encrypting traffic between collectd and noit

Bo Agerskov Rising — 2013-02-05T11:32:52

No one got any ideas, as to why this happens or how to fix it :)

Regards,
Bo

On 21/01/2013, at 14.11, Bo Agerskov Rising <borising< at >gmail.com> wrote:

Re: Encrypting traffic between collectd and noit

Bo Agerskov Rising — 2013-01-21T13:11:46

Hi again,

I tested a bit more, and tried upgrading to collectd 5.2.0 on my Ubuntu 12.04 LTS server, but same problem.

So to summarize it all:
- Noitd SecurityLevel 2 and collectd SecurityLevel "Encrypt" I get the "collectd: Decrypting AES256 part failed with status -1." error in noitd.
- Noitd SecurityLevel 1 and collectd SecurityLevel "Sign" I get lots of lines with "[2013-01-21 13:59:21.037597] collectd: Received signed network packet but can't verify it because no secret has been configured. Will accept it." error in noitd, and then the noitd will become unresponsive.
- Noitd SecurityLevel 1 and collectd SecurityLevel "None", I receive data from collectd, but it's not encrypted.

Also on the collectd side (in all 3 tested versions) I receive the following 2 lines:

Jan 21 13:59:06 reconmq01 collectd[6648]: Libgcrypt warning: missing initialization - please fix the application
Jan 21 13:59:06 reconmq01 collectd[6653]: Libgcrypt warning: missing initialization - please fix the application

Regards,
Bo

On

Encrypting traffic between collectd and noit

Bo Agerskov Rising — 2013-01-20T20:57:56

Hi guys,

I have been trying to setup collectd, by following the documentation. But I can't get the collectd agent to send data encrypted or signed into the noit, but if I specify None in the security level at the client side, and setting the SecurityLevel in noit to 1, data gets transported without problems.

When running noit in debug mode, and setting encrypt in the client, I get the following output:

[2013-01-19 13:26:07.854922] collectd: Decrypting AES256 part failed with status -1.
[2013-01-19 13:26:07.859262] collectd: Decrypting AES256 part failed with status -1.
[2013-01-19 13:26:07.865078] collectd: Decrypting AES256 part failed with status -1.

Any of you that have seen this problem before?

I have been trying to run collectd 4.10 on ubuntu 10.04LTS and 4.11 on 12.04LTS.

Regards,
Bo

Re: Scaling reconnoiter configurations

Theo Schlossnagle — 2012-12-10T15:55:23

So... we namespace all out "name" attributes as such:

c_<accountid>_<checkid>::normalnameofcheck

The rabbitmq driver actually knows how to parse that and will encode the
accountid in the route of the message.

both accountid and checkid are integers that are known by an external
system and have nothing to do with reconnoiter proper.

We parse that out in other places of our code.

On Thu, Dec 6, 2012 at 12:55 AM, Ask Bjørn Hansen <ask< at >develooper.com>wrote:

Re: Scaling reconnoiter configurations

Michal Taborsky — 2012-12-06T10:02:59

Hello Ask,

here is how we do it for a part of our setup. We are monitoring quite a few
(hundreds) of URLs with Reconnoiter (besides other things). We have an
external database table where we manage the URLs via simple web interface.
Each URL is assigned a UUID on creation and also a flag, set by a trigger
when the row is inserted/updated/marked for delete, that signifies the
change should be pushed to noit. We have a periodic job that uses the noit
REST API to push the checks to Reconnoiter. There is no way to query the
noit checks, so we decided to build it like this, because we manage the URL
in the database anyway.

So if you have a database of some sorts your endpoints that you'd like to
measure (and I assume you do), just add a UUID attribute somewhere in there
and use the API to push changes. If you have a large number of checks and a
lot of changes, it might make sense to use some queue for the change
requests, but in our case the cron job is enough and is just few lines of
PHP code.

BTW: I am a mem

Re: Notifications?

Gavin Sandie — 2012-12-06T09:38:46

Alerting and notifications are something you need to build yourself just now. 

There are a few posts in a "Alarms, Thresholds, Notifications" thread from this mailing list to help and explain how it works:

https://labs.omniti.com/labs/reconnoiter/lists/users/2010-October/000541.html
https://labs.omniti.com/labs/reconnoiter/lists/users/2010-October/000543.html

Also Mark Steele, who I believe is still on this list, has done some really good blog posts on running sample queries and having something that will send alerts. Lots of good information there for getting started:

http://www.control-alt-del.org/2011/10/25/monitoring-with-reconnoiter/

Cheers, Gavin

Scaling reconnoiter configurations

Ask Bjørn Hansen — 2012-12-06T05:55:21

Hi everyone,

(I mean scaling not as in "make it technically work" but rather seeking advice in making it manageable).

I haven't used reconnoiter for a few years, but am dabbling with it again now.

On use-case is that I'd like to try having reconnoiter help with the monitoring of the NTP Pool servers, but for various reasons I'd like the systems to be as loosely coupled as possible (mostly risk aversion, lack of hardware in the right place and lack of development resources/time).

What's the best way to "tag" the checks in the noitd configuration with an account name/number?  I'm trying to avoid having to use an account/server -> uuid mapping table in my external system. It'd be convenient if some of the noitd configuration data got cached in the pgsql database (at least the "path" to the check id).

Maybe I just need to suck it up and do that and the work to keep them in sync?

Otherwise an obvious approach is to use (abuse?) the hierarchy/sections in the noitd configuration to specify "owner", but it see

Notifications?

Joshua J. Kugler — 2012-12-04T01:24:22

I see in the Reconnoiter goals here:

https://labs.omniti.com/labs/reconnoiter/wiki/Goals

this goal is mentioned:

"Applying policies (such as notification and escalation) to large groups of 
service"

I have read through the docs, and see nothing about configuring notifications. 
Reconnoiter is obviously in production at OmniTI.  What does OmniTI use to 
pull data from the Reconnoiter database and do notifications and escalations?

Thanks!

j

Clarification of stratcon console metric

Gavin Sandie — 2012-09-13T09:24:59

We recently had stratcon disconnected from noit and it is now catching up processing the data.

In the stratcon console I'm able to get some information on the eventer jobq: 

stratcon# show eventer debug jobq
=== datastore_0.0.0.0_default_default ===
concurrency: 1
backq: default_back_queue
queue size: 0
=== iep_submitter ===
concurrency: 1
backq: default_back_queue
queue size: 0
=== default_back_queue ===
concurrency: 0
backq: (undefined)
queue size: 0
=== default_queue ===
concurrency: 5
backq: default_back_queue
queue size: 0
=== datastore_x.x.x.x_default_*.x.x ===
concurrency: 1
backq: default_back_queue
queue size: 2653




For the datastore entries, does the queue size reflect the amount of outstanding objects/queries that need to be run against the database? Is the concurrency something that is managed internally by stratcon? I wasn't able to see any options for it in the postgres_ingestor code or docs.


Cheers, Gavin

Re: Noitd crash

Gavin Sandie — 2012-09-03T10:08:57

I've been struggling to get the ssh module to compile when attempting to add this change.

I've read http://www.gnupg.org/documentation/manuals/gcrypt/Multi_002dThreading.html to get an understanding of what the line was, and from that I included gcrypt.h into ssh2.c to ensure that the GCRY_THREAD_OPTION_PTHREAD_IMPL macro was available.

From reading the code, I thought that ssh2 was being initialised on the following line, as the ssh2_init function doesn't seem to perform any actual libssh2 setup (I could be wrong, my C knowledge isn't the best):

https://github.com/omniti-labs/reconnoiter/blob/master/src/modules/ssh2.c#L159

However when I've added the cgry_control line here I get an error:

error: ‘gcry_threads_pthread’ undeclared (first use in this function)

Any pointers as to where I'm going wrong?

Gavin

Re: Noitd crash

Theo Schlossnagle — 2012-08-31T16:07:11

If I had to guess, this is a bug in libssh2... but can you try adding:

gcry_control (GCRYCTL_SET_THREAD_CBS, &gcry_threads_pthread);

into the libssh2 initialization function in ssh2.c in noitd.

On Fri, Aug 31, 2012 at 11:49 AM, Gavin Sandie
<g.sandie< at >digital-science.com> wrote:

Noitd crash

Gavin Sandie — 2012-08-31T15:49:45

Hello,  

I'm running the current head from github and I'm seeing occasional crashes in noit.

If I run noit in debug mode I get an error:

noitd: ath.c:193: _gcry_ath_mutex_lock: Assertion `*lock == ((ath_mutex_t) 0)' failed.

I've attached gdb and get the following backtrace when the abort happens:

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7f143d603700 (LWP 19380)]
0x00007f144c6aad05 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
in ../nptl/sysdeps/unix/sysv/linux/raise.c
(gdb) bt
#0 0x00007f144c6aad05 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00007f144c6aeab6 in abort () at abort.c:92
#2 0x00007f144c6a37c5 in __assert_fail (assertion=0x7f14439afea6 "*lock == ((ath_mutex_t) 0)", file=<value optimized out>, line=193,·
function=<value optimized out>) at assert.c:81
#3 0x00007f144396cac2 in ?? () from /lib/x86_64-linux-gnu/libgcrypt.so.11
#4 0x00007f14439723b2 in ?? () from

Review request for Front End Privileges

Nagaraj Ramakanth shet — 2012-08-20T12:52:02

Hello ,

    I have uploaded patches on Front End Privileges long back, still I
didn't get reply for that.
Please review it and give me feed back on it.

Pull Request link : https://github.com/omniti-labs/reconnoiter/pull/82

Thank you.
Nagaraj R Shet
_______________________________________________
Reconnoiter-users mailing list
Reconnoiter-users< at >lists.omniti.com
http://lists.omniti.com/mailman/listinfo/reconnoiter-users

Re: statsd plugin not receiving stats

Gavin Sandie — 2012-08-14T16:22:58

Hi,

Thanks for that. I've updated my dev server and tested again with the target set as a hostname.

Looks fixed to me. The metrics are now showing up under the check results :)

Thanks again,

Cheers, Gavin

Re: statsd plugin not receiving stats

Theo Schlossnagle — 2012-08-13T13:45:11

Looks like Phil Maddox fixed this in master. Can you update and verify
it is working for you now?

On Fri, Jul 27, 2012 at 3:26 PM, Theo Schlossnagle <jesus< at >omniti.com> wrote:

Re: Definition of "target"

Joshua J. Kugler — 2012-08-13T03:43:43

OK, that makes sense. 

Thank you!

j

On Sunday, August 12, 2012 20:50:48 Theo Schlossnagle wrote:

Re: Definition of "target"

Theo Schlossnagle — 2012-08-13T00:50:48

Any machine on the network can send collectd to noit if it is running
the collectd module.  For each host sending data, you need a check
configured with that host's IP address as the target of that check.
The docs mean you can only have one such check for each host on the
network.

On Fri, Aug 3, 2012 at 7:34 PM, Joshua J. Kugler <joshua< at >azariah.com> wrote:

Re: Definition of "target"

Joshua J. Kugler — 2012-08-12T22:10:47

Re: Monitoring on-host metrics

Gavin Sandie — 2012-08-03T23:37:13

I've used both collectd and resmon for collecting most of my host specific metrics. Currently using resmon and enjoying the simplicity of how it works, and the ease of plugin creation.

You need to graph all your metrics. It is possible to setup templates to ease the graphing, but since graphing is such a domain specific thing, you need to setup your own graphs and templates. There are none pre-loaded.

Cheers, Gavin
--
Gavin

On 4 Aug 2012, at 00:16, "Joshua J. Kugler" <joshua< at >azariah.com> wrote:




********************************************************************************   
DISCLAIMER: This e-mail is confidential and should not be used by anyone who is
not the original intended recipient. If you have received this e-mail in error
please inform the sender and delete it from your mailbox or any other storage
mechanism. Neither Macmillan Publishers Limited nor any of its agents accept
liability for any statements made which are clearly the sender's own and not
expressly made on behalf of Macmillan Pub

Definition of "target"

Joshua J. Kugler — 2012-08-03T23:34:56

In reading about the collectd plugin, I came across this line:

http://labs.omniti.com/labs/reconnoiter/docs/ch05s02.html

"Only one is supported per target."

I couldn't find a definition of "target" here: 
http://labs.omniti.com/labs/reconnoiter/docs/archecture.html

Does that mean only one system can send stats from collectd?

OR

Only one collectd listener can be configured per noit instance, but the 
noit instance can accept statistics from multiple collectd instances?

Thanks!

j