My Opera is closing 3rd of March

Cronache di Sarvegia

...because every new challenge hides an opportunity

Adding bulks of SNMP hosts to cacti

Wednesday, November 3, 2010 4:08:28 PM

, ,

Update: while the general explanation presented here is still valid, the Makefile has been updated. You can find it here.

Cacti is a well known and widely used tool, and many IT enterprises use it. We are no exception.

The challenge of the day was to configure snmpd on a few dozens of servers, and then add them to cacti for reporting and graphs. While the first part was made trivial with puppet (but it would be trivial with any configuration management tool, it must be!), adding them to cacti was a different story.

When you have to add one or a few servers to cacti, it's easy to just go through the web interface and do the clickwork: a bit less than five minutes per machine, and you are done. But when you have many servers to add at once, that clickwork becomes too tedious and prone to errors, and you'll cry for automation. So, what to do?

Luckily enough, cacti has a CLI. It's far from a full-featured product, but at least it provides all the means to add hosts and graphs (want to remove a bulk of them? bad luck, sorry!). A starting point of how to use the CLI is available at the Vim-Fu blog. The post explains well enough how to use the CLI to accomplish a number of things, but it fails at showing how to automate it all. That's where my work kicks in wink

As it often happens, I used GNU make for the automation. I'll show my final makefile chunk by chunk and explain it.

The first chunk is an header and the configuration of the Makefile itself (sort of):

########################################################################
#
# This makefile tries to help you to automate the following tasks:
#
# - add an SNMP host to cacti
# - add a set of graphs to the host
# - add the host to a tree
#
# Before using this makefile, please check that the following settings
# are correct for your installation.
#
########################################################################
PHP=/usr/bin/php
CLI_DIR=/opt/cacti/cli

# Default tree where we add hosts to
DEFTREE_ID=1

# Default host template to use
HOST_TID=9

# In the following variables, you have:
# *_GTID:	Graph Template ID
# *_QID:	Query ID
# *_QTID:	Query template ID

# Network interfaces
IFACE_GTID=2
IFACE_QID=1
IFACE_QTID=13
IFACE_FIELD=ifOperStatus
IFACE_VALUE=Up

# Individual CPUs
CPUS_GTID=27
CPUS_QID=9
CPUS_QTID=19
CPUS_FIELD=hrProcessorFrwID

# Cumulative CPU info
CPU_GTID=38

# Load average
LA_GTID=11

# Memory
MEM_GTID=39

# Processes
PROC_GTID=37

# Used space
US_GTID=26
US_QID=8
US_QTID=18
US_FIELD=hrStorageDescr


# You shouldn't need to edit anything below this line
########################################################################

We begin by setting the correct path for our php interpreter, and the cacti CLI directory. Right after that, we set a number of variables that will tell the script the ID of the tree where we'll add our hosts, which host template we are going to use, and which parameters we'll use for each graph (or group of graphs).

Using this information, we build a new small set of variables:

ADD_GRAPHS=$(PHP) $(CLI_DIR)/add_graphs.php
ADD_TREE=$(PHP) $(CLI_DIR)/add_tree.php
ADD_DEVICE=$(PHP) $(CLI_DIR)/add_device.php
LIST_HOSTS=$(ADD_GRAPHS) --list-hosts

Other variables will be set on the make command line. We'll use NAME (the name or description of the device), FQDN (the fully-qualified domain name for the host/device, or IP address), and ID (the ID of the device once it is loaded into cacti). The latest one you won't know until you add the host to cacti, of course, but as long as you provide NAME and FQDN that won't be a problem at all (if you want to know why, just keep reading).

So, we'll mangle this stuff a bit:
MATCH=$(or $(FQDN),$(NAME))
RESOLVE=$(if $(MATCH),$(shell $(LIST_HOSTS) | grep "$(MATCH)" | head -n 1 | awk '{ print $$1 }'))

# The following will set ID to the value of RESOLVE, but only if ID was not
# already defined
ID ?= $(RESOLVE)

This means that MATCH will have the value of FQDN, or NAME's in case FQDN was not set. If neither FQDN nor NAME are set, MATCH will be empty.
If MATCH is set, then RESOLVE will also be set, and it will contain the ID of the first host that matches the value of MATCH in cacti's host list.

Phew! That was a lot already, let's relax with something easy:

.SILENT: nothing

Normally, make will print each command it is going to execute for each target. If we don't want this kind of "echo" for a selected set of targets, we can use the .SILENT pseudo target. All targets listed as .SILENT dependencies won't echo their commands. In our case, there is only one silent target, this one:

nothing:
	echo "Impossible is nothing [$(RESOLVE)]"
	echo "FQDN: $(FQDN)"
	echo "NAME: $(NAME)"
	echo "ID:   $(ID)"

This does actually nothing but show the value of some variables. Being the first real target of the makefile, it's the one that will be run if you just type make. I our default target is a no-op: that's safe.

A number of targets will require the ID parameter to be set to the host ID in cacti, so we are going to check that over and over. A dedicated target is the right place to do that:

_idcheck:
	@if [ -z "$(ID)" ] ; then echo "No ID specified" ; exit 1 ; fi
	@echo "Selected ID: $(ID)"

NAME and FQDN are no different:

_namecheck:
	@if [ -z "$(NAME)" ] ; then \
		echo "Please specify a name/description for this device" ; \
		exit 2 ; \
	fi

_fqdncheck:
	@if [ -z "$(NAME)" ] ; then \
		echo "Please specify a fqdn or IP address for this device" ; \
		exit 3 ; \
	fi

That "@" before the shell commands means that the command is silent. You may notice that these three targets are good candidates for .SILENT, too.

Now let's start to put pieces together. How to add a new host to cacti? This way:

add_host_to_cacti: _namecheck _fqdncheck
	$(ADD_DEVICE) \
		--description="$(NAME)" \
		--ip="$(FQDN)" \
		--template=$(HOST_TID)

Once the host is in, we can start adding graphs to it. Network interfaces:

interfaces: _idcheck
	$(ADD_GRAPHS) \
		--host-id=$(ID) \
		--graph-type=ds \
		--graph-template-id=$(IFACE_GTID) \
		--snmp-query-id=$(IFACE_QID) \
		--snmp-query-type-id=$(IFACE_QTID) \
		--snmp-field=$(IFACE_FIELD) \
		--snmp-value=$(IFACE_VALUE)

CPU information:

cpu_cumulative: _idcheck
	$(ADD_GRAPHS) \
		--host-id=$(ID) \
		--graph-type=cg \
		--graph-template-id=$(CPU_GTID)

What if the CPU is multicore? No problem!

cpu_single: _idcheck
	$(ADD_GRAPHS) \
		--list-snmp-values \
		--host-id=$(ID) \
		--snmp-query-id=$(CPUS_QID) \
		--snmp-field="$(CPUS_FIELD)" | \
	grep "^[0-9]" | \
	while read CPUID ; \
	do $(ADD_GRAPHS) \
		--host-id=$(ID) \
		--graph-type=ds \
		--graph-template-id=$(CPUS_GTID) \
		--snmp-query-id=$(CPUS_QID) \
		--snmp-query-type-id=$(CPUS_QTID) \
		--snmp-field="$(CPUS_FIELD)" \
		--snmp-value=$$CPUID ; \
	done

Can we merge these targets together? Of course we can!!!

cpus: cpu_cumulative cpu_single

Load average, memory and processes are quite similar to each other, so I'll throw them here together:

loadaverage: _idcheck
	$(ADD_GRAPHS) \
		--host-id=$(ID) \
		--graph-type=cg \
		--graph-template-id=$(LA_GTID)

memory: _idcheck
	$(ADD_GRAPHS) \
		--host-id=$(ID) \
		--graph-type=cg \
		--graph-template-id=$(MEM_GTID)

processes: _idcheck
	$(ADD_GRAPHS) \
		--host-id=$(ID) \
		--graph-type=cg \
		--graph-template-id=$(PROC_GTID)

Information on disk partition and other memory usage is again a bit complicated, but doable:

usedspace: _idcheck
	$(ADD_GRAPHS) \
		--list-snmp-values \
		--host-id=$(ID) \
		--snmp-query-id=$(US_QID) \
		--snmp-field="$(US_FIELD)" | \
	grep -v "^Known" | \
	while read DEVID ; \
	do \
	if [ -z "$$DEVID" ] ; then continue ; fi ; \
	$(ADD_GRAPHS) \
		--host-id=$(ID) \
		--graph-type=ds \
		--graph-template-id=$(US_GTID) \
		--snmp-query-id=$(US_QID) \
		--snmp-query-type-id=$(US_QTID) \
		--snmp-field="$(US_FIELD)" \
		--snmp-value="$$DEVID" ; \
	done

And this was the last set of graphs. Again, can we merge everything together? Sure we can!!!

add_all_graphs: _idcheck interfaces cpus loadaverage memory processes usedspace

Once all the graphs are added, we needed to add the host to a tree. Again, it's not that hard:

add_host_to_tree: _idcheck
	$(ADD_TREE) \
		--type=node \
		--node-type=host \
		--tree-id=$(DEFTREE_ID) \
		--host-id=$(ID)

Now, can we automate it all? Yes, and "all" is the right word to use in this case:

# add_host_to_cacti will require FQDN and NAME to be defined, so if
# this target is running, we are sure they are OK
# These variables are then passed along to the sub-instances of make
# automatically, hence there is no need to pass them along explicitly.
# Having them set, and not having ID manually set, will assign ID the
# id value of the newly added host. So, once again, no need to set it
# explicitly on the command line.
# I like this stuff :-D
all: add_host_to_cacti
	make add_all_graphs
	make add_host_to_tree

As you can see, the "all" target does nothing but what was declared in the header section of the makefile.

We could stop here, but you'll find more. As a bonus, I added a "fulldump" target that will print all the information that is available from cacti, which is particularly useful if you are trying to understand which values you need to assign to the variables at the top of the makefile.

Now, to add a host to cacti, you'll just need to run something like

make all NAME=myserver FQDN=myserver.example.com

and that will take care of everything. And to add a bunch of them, just use the kind of loop you deem as appropriate.

The full makefile is below. Enjoy!!!

########################################################################
#
# This makefile tries to help you to automate the following tasks:
#
# - add an SNMP host to cacti
# - add a set of graphs to the host
# - add the host to a tree
#
# Before using this makefile, please check that the following settings
# are correct for your installation.
#
########################################################################
PHP=/usr/bin/php
CLI_DIR=/opt/cacti/cli

# Default tree where we add hosts to
DEFTREE_ID=1

# Default host template to use
HOST_TID=9

# In the following variables, you have:
# *_GTID:	Graph Template ID
# *_QID:	Query ID
# *_QTID:	Query template ID

# Network interfaces
IFACE_GTID=2
IFACE_QID=1
IFACE_QTID=13
IFACE_FIELD=ifOperStatus
IFACE_VALUE=Up

# Individual CPUs
CPUS_GTID=27
CPUS_QID=9
CPUS_QTID=19
CPUS_FIELD=hrProcessorFrwID

# Cumulative CPU info
CPU_GTID=38

# Load average
LA_GTID=11

# Memory
MEM_GTID=39

# Processes
PROC_GTID=37

# Used space
US_GTID=26
US_QID=8
US_QTID=18
US_FIELD=hrStorageDescr


# You shouldn't need to edit anything below this line
########################################################################

ADD_GRAPHS=$(PHP) $(CLI_DIR)/add_graphs.php
ADD_TREE=$(PHP) $(CLI_DIR)/add_tree.php
ADD_DEVICE=$(PHP) $(CLI_DIR)/add_device.php
LIST_HOSTS=$(ADD_GRAPHS) --list-hosts

MATCH=$(or $(FQDN),$(NAME))
RESOLVE=$(if $(MATCH),$(shell $(LIST_HOSTS) | grep "$(MATCH)" | head -n 1 | awk '{ print $$1 }'))

# The following will set ID to the value of RESOLVE, but only if ID was not
# already defined
ID ?= $(RESOLVE)

.SILENT: nothing

nothing:
	echo "Impossible is nothing [$(RESOLVE)]"
	echo "FQDN: $(FQDN)"
	echo "NAME: $(NAME)"
	echo "ID:   $(ID)"

# add_host_to_cacti will require FQDN and NAME to be defined, so if
# this target is running, we are sure they are OK
# These variables are then passed along to the sub-instances of make
# automatically, hence there is no need to pass them along explicitly.
# Having them set, and not having ID manually set, will assign ID the
# id value of the newly added host. So, once again, no need to set it
# explicitly on the command line.
# I like this stuff :-D
all: add_host_to_cacti
	make add_all_graphs
	make add_host_to_tree


interfaces: _idcheck
	$(ADD_GRAPHS) \
		--host-id=$(ID) \
		--graph-type=ds \
		--graph-template-id=$(IFACE_GTID) \
		--snmp-query-id=$(IFACE_QID) \
		--snmp-query-type-id=$(IFACE_QTID) \
		--snmp-field=$(IFACE_FIELD) \
		--snmp-value=$(IFACE_VALUE)

cpus: cpu_cumulative cpu_single

cpu_cumulative: _idcheck
	$(ADD_GRAPHS) \
		--host-id=$(ID) \
		--graph-type=cg \
		--graph-template-id=$(CPU_GTID)

cpu_single: _idcheck
	$(ADD_GRAPHS) \
		--list-snmp-values \
		--host-id=$(ID) \
		--snmp-query-id=$(CPUS_QID) \
		--snmp-field="$(CPUS_FIELD)" | \
	grep "^[0-9]" | \
	while read CPUID ; \
	do $(ADD_GRAPHS) \
		--host-id=$(ID) \
		--graph-type=ds \
		--graph-template-id=$(CPUS_GTID) \
		--snmp-query-id=$(CPUS_QID) \
		--snmp-query-type-id=$(CPUS_QTID) \
		--snmp-field="$(CPUS_FIELD)" \
		--snmp-value=$$CPUID ; \
	done

loadaverage: _idcheck
	$(ADD_GRAPHS) \
		--host-id=$(ID) \
		--graph-type=cg \
		--graph-template-id=$(LA_GTID)

memory: _idcheck
	$(ADD_GRAPHS) \
		--host-id=$(ID) \
		--graph-type=cg \
		--graph-template-id=$(MEM_GTID)

processes: _idcheck
	$(ADD_GRAPHS) \
		--host-id=$(ID) \
		--graph-type=cg \
		--graph-template-id=$(PROC_GTID)

usedspace: _idcheck
	$(ADD_GRAPHS) \
		--list-snmp-values \
		--host-id=$(ID) \
		--snmp-query-id=$(US_QID) \
		--snmp-field="$(US_FIELD)" | \
	grep -v "^Known" | \
	while read DEVID ; \
	do \
	if [ -z "$$DEVID" ] ; then continue ; fi ; \
	$(ADD_GRAPHS) \
		--host-id=$(ID) \
		--graph-type=ds \
		--graph-template-id=$(US_GTID) \
		--snmp-query-id=$(US_QID) \
		--snmp-query-type-id=$(US_QTID) \
		--snmp-field="$(US_FIELD)" \
		--snmp-value="$$DEVID" ; \
	done


add_host_to_cacti: _namecheck _fqdncheck
	$(ADD_DEVICE) \
		--description="$(NAME)" \
		--ip="$(FQDN)" \
		--template=$(HOST_TID)

add_host_to_tree: _idcheck
	$(ADD_TREE) \
		--type=node \
		--node-type=host \
		--tree-id=$(DEFTREE_ID) \
		--host-id=$(ID)

add_all_graphs: _idcheck interfaces cpus loadaverage memory processes usedspace

fulldump: _idcheck dump_hosts dump_graph_templates dump_queries_and_types dump_snmp_fields_per_query_type dump_fields_and_values dump_input_fields


dump_hosts:
	@echo "# HOSTS"
	@$(LIST_HOSTS)

dump_graph_templates:
	@echo "# GRAPH TEMPLATES"
	@$(ADD_GRAPHS) --list-graph-templates

dump_queries_and_types:
	@echo "# QUERIES AND TYPES"
	@$(ADD_GRAPHS) --list-snmp-queries | grep "^[0-9]" | \
	while read ID NAME ; do \
		echo "= $$ID - $$NAME =" ; \
		$(ADD_GRAPHS) --list-query-types  --snmp-query-id=$$ID ;\
		 echo ; \
	done

dump_snmp_fields_per_query_type: _idcheck
	@echo "# SNMP FIELDS PER QUERY TYPE"
	@$(ADD_GRAPHS) --list-snmp-queries | grep "^[0-9]" | \
	while read ID NAME ; do \
		echo "= $$ID - $$NAME =" ; \
		$(ADD_GRAPHS) --list-snmp-fields \
			--host-id=$(ID) \
			--snmp-query-id=$$ID ; \
		echo ; \
	done

dump_fields_and_values:
	@echo "# SNMP FIELDS AND VALUES"
	@$(ADD_GRAPHS) --list-snmp-queries | grep "^[0-9]" | \
	while read ID NAME ; do \
		echo "= $$ID - $$NAME =" ; \
		$(ADD_GRAPHS) --list-snmp-fields \
			--host-id=$(ID) \
			--snmp-query-id=$$ID | grep -v "^Known" | \
		while read FIELD ; do \
			if [ -z "$$FIELD" ] ; then continue ; fi ; \
			echo "== $$FIELD ==" ; \
			$(ADD_GRAPHS) --list-snmp-values \
				--host-id=$(ID) \
				--snmp-query-id=$$ID  \
				--snmp-field="$$FIELD" ;\
			echo ; \
		done ; \
		echo ; \
	done

dump_input_fields:
	@echo "# INPUT FIELDS"
	@$(ADD_GRAPHS) --list-graph-templates | grep "^[0-9]" | \
	while read ID NAME ; do \
		echo "= $$ID - $$NAME =" ; \
		$(ADD_GRAPHS) --list-input-fields --graph-template-id=$$ID ; \
	done

_idcheck:
	@if [ -z "$(ID)" ] ; then echo "No ID specified" ; exit 1 ; fi
	@echo "Selected ID: $(ID)"


_namecheck:
	@if [ -z "$(NAME)" ] ; then \
		echo "Please specify a name/description for this device" ; \
		exit 2 ; \
	fi

_fqdncheck:
	@if [ -z "$(NAME)" ] ; then \
		echo "Please specify a fqdn or IP address for this device" ; \
		exit 3 ; \
	fi

Suggestion for next year budgetLa valigia rossa...