Owl Monitoring System Sensor Installation Manual

1. Operational Overview of the Owl Monitoring System

	1.1.		Owl Sensor Overview
	1.2.		Owl Manager Overview
	1.3.		Data Retention on Owl Hosts

The Owl Monitoring System has a conceptually simple method of operations. There is a manager host and a set of sensor hosts. The sensors periodically make a set of DNS queries and time how long it takes to get a response. The response times are saved to data files. After a certain amount of time, the data files are transferred from the sensor to the manager host. For any particular sensor, this data transfer may be configured to be initiated by either the manager or by the sensor. The manager transfers the data into a monitoring package, which allows easy display of the data.

Below is a diagram of the flow of Owl sensor data through two Owl monitoring installations.

Take note of the following:

• There are three distinct administrative domains, two of which have Owl managers.
• Owl Manager 1 receives sensor data from all three domains.
• Owl Manager 2 receives sensor data from itself and one other domain.
• Domain 1's sensors do not provide data to other domains.
• Domain 2 contains an Owl sensor, but not an Owl manager.
• Owl sensors in the domains 2 and 3 provide data to Owl managers outside of their own domain.
• Owl sensors can query multiple name servers.

The remainder of this section provides a little more information on the components that make up the sensor and manager software.

1.1. Owl Sensor Overview

An Owl sensor makes timed DNS queries and records the time required for a response. The response time data are transferred to the Owl manager after a certain period of time. The transfer may occur at the instigation of either the manager or the sensor. The Owl sensor's three primary programs that handle these tasks are described below.

The Owl sensor's actions are controlled by a configuration file. This file defines the queries that must be performed, how often the queries must happen, how frequently the response data files are transferred to the manager, data and logging information, and other required information.

The owl-dnstimer program performs periodic DNS lookups, as specified by the configuration file. These lookups are timed so that the sensor can record how long it took to make a particular request. The configuration parameters used by owl-dnstimer include the nameserver to query, the target zone to query about, the type of DNS record, and the frequency of queries.

If the Owl system is configured so that the sensor transfers its data to the manager, then the transfer is performed by the owl-transfer program. The transfers are performed using rsync over an ssh connection. The sensor host must be able to do a password-less ssh into the manager host. The frequency of transfers is defined in the configuration file. Choosing the transfer frequency is a trade-off between lower impact on the sensor, manager, and network (less frequent) and lower latency of data display on the manager (more frequent.)

The owl-sensord program is an optional controller for owl-dnstimer and owl-transfer. If used, it will run these daemons and restart them if they stop. If one of the daemons tries to restart too often in too short a time, then owl-sensord will warn an administrator (via email) and temporarily stop trying to execute the program. owl-sensord is not required in order to run owl-dnstimer and owl-transfer; the Owl sensor can run fine without it. However, it provides a convenient way to keep them running.

As part of the installation process, you must decide whether you will run the Owl sensor using the owl-sensord daemon to control execution or if you will run owl-dnstimer and owl-transfer directly.

The Owl sensor has a number of programs for administrative support. Most of these are not required, but they assist in sensor administration. In particular, the owl-dataarch and owl-heartbeat programs are intended to run as cron jobs. owl-dataarch moves "old" sensor data into an archive directory, in order to keep the sensor-manager data transfer from bogging down (within rsync) as the data collection grows. The owl-heartbeat program periodically touches a webpage hosted on the Owl manager, in order to let the manager know that the sensor host is still running. owl-heartbeat is not critical and can be left off, but owl-dataarch is essential. There are a few other administrative commands available; see section 6 for a brief description of all the Owl sensor commands.

1.2. Owl Manager Overview

The Owl manager's purpose is to receive sensor data and make it available for display. The majority of the Owl manager is actually third-party packages, and the Owl manager software provides the "glue" that allows the Owl sensor data to work with those packages. In particular, Owl provides data to the Nagios monitoring system, which then displays the data.

The Owl sensors may report to managers that are under separate administrative control. A particular installation of the Owl Monitoring System may use multiple manager hosts, but this document assumes that there is only one.

The manager can have either an active or passive role in the process of moving Owl sensor data from the sensor to the manager. If the manager is active, it will use the owl-transfer-mgr program to pull sensor data from the Owl sensor host. If the manager is passive, then the sensor will transfer the data when it sees fit.

The are two groups of "glue" programs that Owl uses to interface with Nagios. The owl-dnswatch and owl-perfdata inject Owl sensor DNS response data into Nagios for display and graphing. The owl-sensor-heartbeat.cgi and owl-stethoscope programs insert Owl "heartbeat" data into Nagios so it can track availability of the Owl sensors.

The Owl manager also has a number of programs for administrative support. Most of these are not required, but they assist in manager administration. In particular, the owl-archdata and owl-monthly programs are intended to run as cron jobs. owl-archdata moves "old" sensor data into an archive directory, in order to keep data transfer from the sensor from bogging down as the data collection grows. The owl-monthly program archives the previous month's sensor data into a compressed tarfile. owl-newsensor and owl-initsensor assist in setting up the manager for a new sensor. There are a few others available; see section 7 of the Owl Manager Installation Manual for a brief description of all the Owl manager commands.

The following third-party software packages are used in with this distribution of the Owl Monitoring System. The packages are not included with the Owl distribution, but must be retrieved as required for your operating system. These packages must be installed on the manager:

	Nagios	Monitoring system.
	Nagios plugins	Monitoring system.
	nagiosgraph	Provides graphing for Nagios.
	rrdtool	Database tool for use by nagiosgraph.
	drraw.cgi	Draws graphs for nagiosgraph of data stored by rrdtool.

Instructions are given in this document for integrating these packages with the Owl software. It would be for the best to read the installation instructions below for each package prior to installing them.

Different management, graphing, and database tools may be used instead of those listed above, if so desired. Integrating the Owl manager software with other packages is beyond the scope of this manual.

David Josephsen's Building a Monitoring Infrastructure with Nagios is very helpful for understanding Nagios and how the various pieces work together. If you're going to be running a Nagios monitor, you would be doing yourself a favor to read this book.

1.3. Data Retention on Owl Hosts

Owl sensors generate a large amount of DNS response data. For example, a sensor running five queries, once a minute each, generated roughly 517KB of data per day. (Due to the way the data are stored, this will be affected by the amount length of target and nameserver hostnames used.) You may or may not want to retain any of this data. The data files can be useful for rebuilding the Owl manager's rrdtool databases (or building with different parameters.)

On both the Owl manager and the Owl sensors, sensor data are stored in a data directory. After a few days, the data files are moved to a data archive directory. A full month's data files will be collapsed into a compressed format.

The Owl Monitoring System provides tools that assist with managing Owl sensor data. After the sensor data have been moved from the sensor to the manager, the sensor has no further need of the data. On the manager, after the data have been moved into the data archive, the Owl Monitoring System has no further need of the old data; it may be retained or deleted. (The data will have been moved into the manager's rrdtool databases by then, so the data is not lost.) Each installation must decide their own data retention policy.