gmane.comp.configuration-management.users

Re: Puppet: a few questions and issues

Luke Kanies — 2011-04-29T12:40:36


As Justin says, Puppet's reports are the best fit for this, and if you
send your reports to the Dashboard, pretty much everything you're
asking for comes for free.

Re: Puppet: a few questions and issues

John Jasen — 2011-04-27T20:45:18

Indeed. One of my peers suggested writing a similar capability for
puppet, where the changes were committed to (syslog |& a state file)
with status, so it could be queried as needed, and to the level desired.

Re: Puppet: a few questions and issues

Narayan Desai — 2011-04-27T20:36:02

This isn't a direct solution to your problem, but we've addressed this sort of problem in bcfg2 by including a full feedback loop in the configuration process, so that you can inspect what has gone on across your clients from a single, centralized location. (or detect outright tool failure by the lack of feedback data at predictable intervals)

I'd suggest that you want something like that, preferably able to tell you about underlying failures in some detail, as opposed to a high level thumbs up/thumbs down.  
 -nld

On Apr 27, 2011, at 3:25 PM, John Jasen wrote:

Re: Puppet: a few questions and issues

Justin Clayton — 2011-04-27T20:31:03

I would recommend taking a look at the report functions of puppet as a start. There are many different supported reporting methods such as email, puppet's dashboard app, or even additional local logging.

http://docs.puppetlabs.com/guides/reporting.html
http://docs.puppetlabs.com/references/latest/report.html

Hope this helps you in your situation.

Justin Clayton
justin-gYCc/7lCvzDk1uMJSBkQmQ< at >public.gmane.org

Not sent from my iPhone

On Apr 27, 2011, at 1:25 PM, John Jasen wrote:

Puppet: a few questions and issues

John Jasen — 2011-04-27T20:25:15

As a brief summary, $work had a puppet module call the underlying OS
package manager. The package manager (apt) apparently failed, reporting
a non-zero exit status to puppet, which seems to have aborted the rest
of the puppet run. The systems were left in a half-provisioned state,
which escaped our monitoring system for a few days.

In looking at the matter, we were not quickly able to come up with a way
to detect failed puppet runs. Mining syslog may be an option, but see
the point about that further in. Is there an acceptable external way
(say through nagios) to determine if the last puppet run succeeded or
failed?

Second, it seems that puppetd -t and puppetd -t --noop both log the same
to syslog and also update the state.yaml file. Is there a way to
differentiate --noop or test runs from syslog? Is there a way to make it
not update the state file? If not, should there be? :)

Thanks in advance.

Re: Config management and asset tracking on tickets

Matt Baker — 2011-02-27T08:41:40

Following the OT thread, here's how we do this.

In Bcfg2 we can define a hierarchy of groups (or classes as I mentioned 
before):

Profile (e.g. x86-debian-lenny-php5-prod
  * Arch (x86)
   * sub arch (i386)
  * Distribution release (debian-lenny)
   * Distribution (debian)
    * OS (linux)
     * System Type (xen-vm)
  * Service (httpd-service)
   * Server (apache2-php5-server), [...]
   * Client (mysql-client), [...]
  * Cluster (web-cluster)

At each point I can inherit "Bundles" (groups of objects cfg, pkg, 
service, etc).

For the "what" we can populate a name space for templating via 
"Properties" files which, for the above example, contains:

Properties
  * Projects
   * Project (attributes: name, POSIX group/user)
    * Contact (attr: type(admin, service-manager, developer), name)
    * Instance (attr: name, status(prod, dev, pilot, etc))
     * Apache (attr: affinity)
      * VirtualHost (attr: name, port, ipaddr, ssl, etc)

I can construct definitions and properties (from attributes containing 
variable data) for any class which can be accessed at any stage of the 
"asset" instantiation and I can extract and reuse that data elsewhere 
(e.g. in nagios, reporting, CRM system, etc).

Probes are basically scripts which run on the server which the results 
either set dynamic groups (classes) or the data is stored for access later.

I can also use tools (or even build tools from Properties) to query this 
data. For example we use fabric (fabfile.org) to push changes out to 
only the hosts we are interested in. e.g.

fab hosts:apache2-php5-server push:mysql-client

the value to host on the left is the class name and the value to push is 
a bundle name (which matches a class name). This command will ask the CM 
server which hosts to apply the change and trigger an update just of 
just this bundle.

Automatic updates of monitoring and reporting services update data from 
attributes in the Properties and Probes. We define Properties for 
different services and cluster attributes, such as Admins, Contacts, 
Networks, Organisational Units, Service config, etc. We can source data 
into the name space as "Connectors" from different sources by creating 
plugins, such as LDAP or MySQL or XML.

Back to asset tracking as mentioned before, we have the requirement for 
resource discovery as people tend to buy their own kit without letting 
us know (University context). We also need to be capable of tracking 
assets which cannot be configured using a single CM tool (printers, lab 
kit). The data collected by this tool is very useful to the CM and the 
CMDB contains useful info for the asset tracker so some sort of API or 
RESTful interface is desirable. A connector could be configured for this 
to make data available. However, we recognise these are two separate 
areas of concern and that two systems are needed to maintain these areas.

Cheers,

Matt

Re: Config management and asset tracking on tickets

Atom Powers — 2011-02-27T05:42:08

This is a bit off-topic for the OP but I'll bite.

Using groups is a good option, and it doesn't require any special
schema changes. But I went a different route; since in the majority of
cases I'm trying to find out "how is this host configured" instead of
"which host has this configuration" I define the configuration
attributes in the host entry to reduce the number of queries to the
directory. I have scripts that read this information out of LDAP and
set classes or variables for cfengine, build configurations for
nagios, etc.

I admit this is a bit of a hack, but I really wanted to put as much
data in LDAP as possible and avoid creating new attributes every time
I wanted to use a new variable. I'd love to hear of a better way.

For example:
--
digipenConfigurationVariable: cfeclass:ntp_server
digipenConfigurationVariable: cfeclass:bind_server
digipenConfigurationVariable: cfeclass:dhcp_server
digipenConfigurationVariable: dhcp-service:us
digipenConfigurationVariable: ldap-role:consumer
digipenConfigurationVariable: cfeclass:openldap_server
digipenConfigurationVariable: ldap-sync-rid:81
digipenConfigurationVariable: cfeclass:samba_server
digipenConfigurationVariable: samba-role:pdc
digipenConfigurationVariable: samba-role:print
digipenConfigurationVariable: samba-name:printers
... etc.

Re: Config management and asset tracking on tickets

John Rouillard — 2011-02-27T05:22:28

Hi Matt:

On Sat, Feb 26, 2011 at 12:33:45AM +0000, Matt Baker wrote:

Yup. The CMDB component is part of DACS and is version
controlled. Checkins to the svn system that underlies DACS requires a
ticket reference. So identifying all the changes for a particular
ticket can be easily gotten from svn log.


Hmm, that's a good idea. Under DACS it would be a bit more difficult
because the files that get pushed are generated from version
controlled files. But I see where you are going.


DACS has that already. It logs updated files on ahost as part of the
distribution, but that info isn't actually looped back into an asset
tracker to allow searching. However a postprocessing step that went
over the distribution record, accessed the svn history for that file
and grabbed the ticket identification info could update the ticket's
asset list for the host scope at least.

Reverse engineering the files that are pushed to an asset subsystem
might be trickier, but usually there are only a handful of asset
subsystems that change on a given ticket, so those could be manually
updated. Also that provides a second source of info on which parts of
the infrastructure generate the most work (which can be further
refined from the svn change logs).
 

Yup very nice.
 

Ok, so this is what I was thinking of. Extending the definition of
"asset" beyond the financial asset tracking concept.


It does.

Re: Config management and asset tracking on tickets

John Rouillard — 2011-02-27T04:55:52

Well part of making a CM system palitable is making sure that it is
easy to track the cause of changes to a system as well. This has two
benefits:

  1) justification for the change
  2) ability to determine if the change affects systems I am
     responsible for. This is particularly important in a distributed
     sysadmin setup.

Re: Config management and asset tracking on tickets

John P. Rouillard — 2011-02-27T04:50:46

In message <AANLkTi=7dmf61rhTODPZssPEw0ex5VQ8q+=_e5kL5pST-JsoAwUIsXosN+BqQ9rBEUg< at >public.gmane.org>,
Atom Powers writes:


Yup RT was an intended target.


How do you define attributes for a host? E.G. say a host is running
apache, so you need cfengine to push apache configs and you need
nagios configs to enable apache monitoring. Do you assign hosts to
some type of ldap group?

the SVN commit logs for cfengine and yell at the admin sitting next to

All of the svn log messages for our CM checkins include a ticket
reference. When trying to figure out why a given line is in a config
file, you can trace it back to the original ticket to see what
functionality the implementor was attempting to supply. It has stopped
us from breaking things when the purpose of the change wasn't obvious.

However a single file change can result in changes to 1 or 100
machines and that's what we would like to capture in a ticket so we
can better determine the scope of the modifications caused by the
ticket. It allows you to isolate the tickets that are the most likely
causes for changes in behavior on a particular host. However selecting
100 hosts is error prone not to mention tedious.

But the host is only one asset in a sense. The service(s) that are
modified is also a sort of asset and provides a second way of
classifying the scope of a ticket, but most "asset" trackers use a
more financial/traditional physical hardware definition.

--
-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.

Re: Config management and asset tracking on tickets

Atom Powers — 2011-02-26T03:09:15

As mentioned, RequestTracker can do this with plugins. You may also
want to look at OCSNG.


I store all my asset information in LDAP. I have an in-house
application to manage that data. We also have helper scripts for
cfengine and nagios to read information from LDAP for assets and
update configurations as necessary (also for DNS and DHCP). I haven't
gone so far as to plug it into my ticketing system, which is used
primarily for customer support. (We are small enough that I can check
the SVN commit logs for cfengine and yell at the admin sitting next to
me if he changes something that caused a problem... but it's usually
me.)

Re: Config management and asset tracking on tickets

Matt Baker — 2011-02-26T00:33:45

Hi John,

On 25/02/11 20:42, John P. Rouillard wrote:

Keep your CMDB in a version control system. When you get a ticket keep a 
link from the commit revision (for the change, say in websvn) to the 
ticket ID (simple to do this in RT or use trac, etc) (also add a link 
for the ticket ID in the commit log) (we update RT via a commit hook). 
Then when you push the change out to the "asset" make the CM create a 
syslog entry containing either the ticket id and/or the revision. Then 
use something like a distributed/centralised log analysis tool (e.g. 
splunk) to mine the ids/revisions from the logs. You can then see what 
"assets" got what revisions and when.

As for seeing what changes occurred on what hosts for a point in time. 
We use bcfg2 here and that has a nice reporting interface (web based 
django) that keeps statistical historical data which allows you to pick 
a point in time and see the changes that were applied (for config files 
as diffs).

I definitely think one should define different "assets" as necessary. 
For example, we have services, servers (the software kind), clients (the 
sw kind), clusters, VMs, OS (types, distributions, releases). As these 
are all defined as classes they can be picked out easily. We currently 
stick all asset information (i.e. conventional stuff like serials, mac 
addresses, manufacturer, support model) in a separate database, but I'd 
like to see it more integrated in the CMDB. There is an amount which is 
probed from the hosts which forms part of this info but could be better 
integrated.

Hope that helps,

Matt

Re: Config management and asset tracking on tickets

Michael C Tiernan — 2011-02-25T21:50:01

I have not investigated deeply but there's an extension to RT that supports assent/inventories.

Re: Config management and asset tracking on tickets

Lamont Granquist — 2011-02-25T21:10:24

just make it easy for "me" to push platform config management changes out 
to every single "asset" in the company.  it needs to be easy both in terms 
of interface, and it needs to be process and politically feasible.  i 
can't possibly come begging on hand and knee for permission from every 
software group to do changes like this.  and the more onerous platform 
changes become the more the systems just rot, or else the more that 
aggressive engineers will just evalute the risk of bypassing change 
management controls completely.

On Fri, 25 Feb 2011, John P. Rouillard wrote:

Config management and asset tracking on tickets

John P. Rouillard — 2011-02-25T20:42:08

Hi all:

Here is my senario:

  ticketing system with an asset tracker

  tickets should include info on which hosts (asset) were modified
    for a given ticket so when problems occur I can search for a
    host and see all the recent changes on the host.

does anybody do anything like this? If so, does your CM system supply
you with the asset information or do you add it manually?

In a CM managed environment, do we need a different/enhanced idea of
an asset (e.g. an asset is a service rather than a host) to make
troubleshooting problems easier?

Any quips, comments, evasions, questions or answers welcome.

--
-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.