Manage notifications

Background information

This page describes advanced configuration options of notifications. For background information on notifications, see the following:

• For a general description of how notifications work in OP5 Monitor, see Checks, alerts, and notifications in Key concepts of OP5 Monitor.
• For descriptions of the notification options you can set on hosts and services, see Host options in Manage templates and time periods and Service options in Manage templates and time periods.

Notification process

OP5 Monitor determines whether it needs to send out notifications through the service check and host check logic. It sends out host and service notifications in the following instances:

• When a hard state change occurs.
• When a host or service remains in a hard non-OK state and the time specified by the configuration setting notification_interval in the host or service configuration has passed since the last notification was sent out for that specific host or service.
• If the notification filters permit it. For more information, see Notification filters.

Note: If a host is in a soft or hard DOWN or UNREACHABLE state, the service notifications for services belonging to that host will be blocked.

Each host and service definition has a contact and contact_groups option that specifies which contacts receive notifications for that particular host or service.

When OP5 Monitor sends out a host or service notification, it notifies each individual contact or member of any contact groups specified in the contact_groups option of the service definition. If a contact is a member of more than one contact group, OP5 Monitor removes duplicate contact notifications before sending notifications out.

The default behaviour is that if a host has contacts configured in the host configuration, the specified contacts receive notifications for the host and the services on the host. There are exceptions to this default behaviour:

• If a service on a host has different contacts set to the host, the contacts on the host receive all the notifications, except for the service that has contacts defined.
• If a service in a host group has contacts set, only those specific contacts receive the notification.

The following notification options inherit their values from the host they are associated with, if you do not specify a value on the service:

• contacts and contact_groups
• check_period
• notification_interval
• notification_period

Notification filters

OP5 Monitor uses a number of filters to determine whether to send a notification or not. You can find more information about time periods and service, host, and contact settings in Manage templates and time periods.

Filter	Description
Program-wide	Whether notifications are enabled on a program-wide basis. Program-wide notification settings are managed in the Operations menu of Manage > Process information. For more details, see Program-wide management.
Service and host filters	Is the host or service in scheduled downtime? Is the host or service in a flapping state? Are the host or service notification options enabled for this type of notification? Are we in the right time period for notifications at the moment? Have we already sent a notification about this alert? Has the host or service remained in the same non-OK state as it was when the last notification went out?
Contact filters	Do the contact notifications options says that this type of notification is supposed to be sent? Are the contact notification options enabled for this type of notification?

Naemon notification documentation

The notifications in OP5 Monitor follow an extensive rule set that is inherited from the core Naemon daemon. You can find more information about Naemon notifications in the Naemon notifications documentation.

Update the host name link in the notification email

The notification email includes a link back to the OP5 Monitor server that sent the notification. You can configure the email settings to link to a different host.

This is useful if OP5 Monitor is configured for load-balanced monitoring, where notifications can be sent out from different sources, but one of the peers is the preferred system for configuration and viewing data. This is the recommended way to configure load-balanced environments.

You can update the URL linking back to OP5 Monitor by updating file /etc/op5/notify.yml with a different hostname value. For example:

...
# The notification email includes a link back to the OP5 Monitor server that
# sent the notification. You can configure the email settings to link to a
# different host.
# If not specified, the system hostname is used.
hostname: master1.op5.com
...

 

In the above example, a notification sent from master2.op5.com in a load-balanced configuration appears as follows:

op5 Monitor

Service CUSTOM detected  2016-07-03  16:36:11.
'Certificate Expiration Check'  on host  'master01'  has passed the CRITICAL threshold.
https://master01.op5.com/monitor/index.php/status/service/master01

Additional info;
CRITICAL - File /opt/plugins/custom/certificate-expire is  21770079  seconds old

Host:    master01
Address:  172.27.0.12
Alias:   OP5 Monitor Server
Status:  UP
Comment: /etc/op5/notify.yml configured on master2

The link in the notification email takes you to master1 to view details of the problem.

For more information on /etc/op5/notify.yml, see Default configuration file.

Update the sender email address in the notification email

By default, OP5 Monitor sends notifications from email address op5monitor without any domain. The mail transfer agent (MTA) adds the local domain name, which by default is @localhost.localdomain.

To change the email address that OP5 Monitor sends notifications from, you can use the --from argument in the notification command, or reconfigure your MTA and host name in OP5 Monitor to send the message from the correct domain.

To change the sender email address in the notification command from op5monitor@localhost.localdomain to op5notification@mycompany.com:

1. In the OP5 Monitor user interface, click Manage > Configure.
2. In the Checks and Notifications section, click Commands.
3. In the Command to edit field, search for host-notify or select the command from the drop-down list, then click Go.
4. In the command_line field, add --from op5notification@mycompany.com, as in the following example:

$USER3$/notify/notify --from op5notification@mycompany.com -c "$CONTACTNAME$" -h "$HOSTNAME$" -f "$NOTIFICATIONTYPE$" -m "$CONTACTEMAIL$" -p "$CONTACTPAGER$" "HOSTALIAS=$HOSTALIAS$" HOSTADDRESS=$HOSTADDRESS$ " " HOSTSTATE=$HOSTSTATE$ " " HOSTSTATEID=$HOSTSTATEID$ " " HOSTSTATETYPE=$HOSTSTATETYPE$ " " HOSTATTEMPT=$HOSTATTEMPT$ " " HOSTLATENCY=$HOSTLATENCY$ " " HOSTEXECUTIONTIME=$HOSTEXECUTIONTIME$ " " HOSTDURATION=$HOSTDURATION$ " " HOSTDURATIONSEC=$HOSTDURATIONSEC$ " " HOSTDOWNTIME=$HOSTDOWNTIME$ " " HOSTPERCENTCHANGE=$HOSTPERCENTCHANGE$ " " HOSTGROUPNAME=$HOSTGROUPNAME$ " " HOSTGROUPALIAS=$HOSTGROUPALIAS$ " " LASTHOSTCHECK=$LASTHOSTCHECK$ " " LASTHOSTSTATECHANGE=$LASTHOSTSTATECHANGE$ " " LASTHOSTUP=$LASTHOSTUP$ " " LASTHOSTDOWN=$LASTHOSTDOWN$ " " LASTHOSTUNREACHABLE=$LASTHOSTUNREACHABLE$ " " HOSTOUTPUT=$HOSTOUTPUT$ " " HOSTPERFDATA=$HOSTPERFDATA$ " " HOSTACKAUTHOR=$HOSTACKAUTHOR$ " " HOSTACKCOMMENT=$HOSTACKCOMMENT$ " " NOTIFICATIONNUMBER=$NOTIFICATIONNUMBER$ " " CONTACTALIAS=$CONTACTALIAS$ " " DATETIME=$DATETIME$ " " SHORTDATETIME=$SHORTDATETIME$ "DATE=$DATE$" "TIME=$TIME$" "TIMET=$TIMET$" "HOSTACTIONURL=$HOSTACTIONURL$" "HOSTNOTESURL=$HOSTNOTESURL$" "ADMINPAGER=$ADMINPAGER$" "ADMINEMAIL=$ADMINEMAIL$" "NOTIFICATIONCOMMENT=$NOTIFICATIONCOMMENT$"

5. Click Submit.
6. Click Save at the top of the configuration page.

Note: To change this for service notifications, repeat the steps above for the command service-notify.

Options for date/time format

The following configuration settings are available for /etc/op5/notify.yml:

Option	Description
timestamp_format	Sets the time format following strftime(3). For more information, see strftime(3) — Linux manual page. The following presets are provided in the /etc/op5/notify.yml: RFC 3339 %Y-%m-%d %H:%M:%S%z Example output: 2021-05-25 14:16:26+0100 Default: Enabled System locale preferred format %c Default: Disabled US-style time stamp with time zone %d/%m/%Y %I:%M:%S %p %Z Example output: 25/05/2021 03:16:26 PM CET Default: Disabled
utc_timestamp	If set to yes, then notifications will use the UTC time stamps instead of the local system time. Default value: no

Naemon macros that use Unix time

There are a few Naemon macros that get reported in Unix time (time_t) by default. These are converted to your configured date/time format:

• LASTHOSTCHECK

• LASTHOSTSTATECHANGE

• LASTHOSTUP

• LASTHOSTDOWN

• LASTHOSTUNREACHABLE

• LASTSERVICECHECK

• LASTSERVICESTATECHANGE

• LASTSERVICEOK

• LASTSERVICEWARNING

• LASTSERVICEUNKNOWN

• LASTSERVICECRITICAL

• TIMET

• PROCESSSTARTTIME

• EVENTSTARTTIME

Caution: Macros not included in this list do not necessarily reflect Unix time. Before you convert a reported time stamp, verify the date/time format used for the macro.

The SMS service.PROBLEM skin

The SMS service.PROBLEM skin is very simple, since there is a limit to how much data you can send by SMS:

#SERVICEDESC# on #HOSTNAME# is #SERVICESTATE#. #SERVICEOUTPUT#

The mail service.PROBLEM skin

Mail notifications can contain a lot more data, so the mail skin is more complex:

From: #FROMEMAIL#
To: #CONTACTEMAIL#
Subject: [op5] #NOTIFICATIONTYPE#: '#SERVICEDESC#' on '#HOSTNAME#' is #SERVICESTATE#
Content-Type: text/plain; charset=utf-8

#extra_host_vars#

ITRS OP5 Monitor

Service #NOTIFICATIONTYPE# detected #LASTSERVICESTATECHANGE#.
'#SERVICEDESC#' on host '#HOSTNAME#' has passed the #SERVICESTATE# threshold.
#STATUS_URL#

Additional info;
#SERVICEOUTPUT#

Host:    #HOSTNAME#
Address: #HOSTADDRESS#
Alias:   #HOSTALIAS#
Status:  #HOSTSTATE#
Comment: #NOTIFICATIONCOMMENT#

Service: #SERVICEDESC#
Status : #SERVICESTATE#
Latency: Check was #SERVICELATENCY# seconds behind schedule
Misc   : Check took #SERVICEEXECUTIONTIME# seconds to complete

Additional links (requires configuration);
Host actions: #HOSTACTIONURL#
Host notes:   #HOSTNOTESURL#
Service actions: #SERVICEACTIONURL#
Service notes:   #SERVICENOTESURL#

Keywords

Skins contain keywords in the following format:

• #SERVICEDESC#
• #HOSTNAME#

Keywords are replaced with the value of a command line argument in the format X=Y. So a command line such as this generates a keyword with the name X having the value Y.

If a notification macro, or other object supplying a keyword value, is missing from the notification command, it does not stop the notification from being sent; only the replacement value will be missing.

Create a custom notification skin

If you need to change the default notification skins, you need to create them in a different directory.

To create custom notification skins:

1. From the OP5 Monitor command line, change to the /notify directory:

   cd /opt/monitor/op5/notify

2. Create a directory called custom-skins:

   mkdir custom-skins

3. Copy the skins.* directories to the custom-skins directory:

   cp -R skins.* custom-skins/

4. Make your changes to the skins. The new skins are used directly after you save the changes.

Note: If you have a peered configuration, you need to make these changes on all peers.